David Y.J. Kim, Ashley Granquist, Evan Patton, Mark Friedman, Hal Abelson, Speak Your Mind: Introducing APTLY, The Software Platform That Turns Ideas Into Working Apps, Submitted to iCERi.
Abstract: MIT Aptly is a tool that uses the technology of large language models to automatically generate mobile apps from written or spoken natural language descriptions. Similar to Github’s Copilot, it is based on OpenAI’s Codex, a specially tuned version of GPT-3. Aptly lets people create programs without requiring any use of coding or knowledge of programming. For example, one can tell Aptly by speaking or typing:
Make an app with a text box, a list of six languages and a button that says “translate.” When the button is clicked, translate the text into the selected language and show the translation.
The result is a complete functioning app for Android or iPhone. The app has a field for user input and six buttons labeled English, Spanish, French, German, Italian, Japanese. Pressing one of the buttons translates the input to the corresponding language. Aptly’s app generation is more than just a syntactic transformation of the input text. Aptly draws upon a large body of code with which it has been trained to provide a context for its app creation. In the above example, Aptly has independently chosen the six languages, something that was not specified in the input text.
As most large language models do, Aptly’s performance depends on the input given to OpenAI’s Codex. These inputs are referred to as prompts. Aptly crafts a prompt by providing a set of example pairs (a textual description of an example app and its corresponding code) along with the description of the desired app. Such prompt engineering is referred to as few-shot prompts. In order to optimize Aptly’s performance, when selecting example pairs, we choose the ones that are semantically close to the description of the desired app.
The prospect of no-code platforms is currently sparking considerable ferment in enterprises concerned with professional programming careers. Similarly, Aptly poses challenges for research in computational thinking education for K-12 students. Much of the present-day curriculum emphasizes implementing computational artifacts using text-based coding with Python or block-based coding with Scratch or App Inventor. What will be the foundations for that curriculum when tools like Aptly are common and the transition from ideas to running programs can be accomplished automatically? This presentation will demonstrate Aptly’s preliminary performance and review its implementation, which incorporates OpenAI Codex, Amazon Alexa and MIT App Inventor.
Jessica Van Brummelen, Maura Kelleher, Mingyan Claire Tian, Nghi Hoang Nguyen, What Do WEIRD and Non-WEIRD Conversational Agent Users Want and Perceive? Towards Transparent, Trustworthy, Democratized Agents, Submitted to CHI 2023.
Abstract: A majority of researchers who develop design guidelines have WEIRD, adult perspectives. This means we may not have technology developed appropriately for people from non-WEIRD countries and children.
We present five design recommendations to empower designers to consider diverse users' desires and perceptions of agents. For one, designers should consider the degree of task-orientation of agents appropriate to end-users' cultural perspectives. For another, designers should consider how competence, predictability, and integrity in agent-persona affects end-users' trust of agents.
We developed recommendations following our study, which analyzed children and parents from WEIRD and non-WEIRD countries' perspectives on agents as they create them. We found different subsets of participants' perceptions differed. For instance, non-WEIRD and child perspectives emphasized agent artificiality, whereas WEIRD and parent perspectives emphasized human-likeness. Children also consistently felt agents were warmer and more human-like than parents did. Finally, participants generally trusted technology, including agents, more than people.
Jessica Van Brummelen, Mingyan Claire Tian, Maura Kelleher, Nghi Hoang Nguyen, Learning Affects Trust: Design Recommendations and Concepts for Teaching Children -- and Nearly Anyone -- about Conversational Agents, Submitted to EAAI.
Abstract: Research has shown that human-agent relationships form in similar ways to human-human relationships. Since children do not have the same critical analysis skills as adults (and may over-trust technology, for example), this relationship-formation is concerning. Nonetheless, little research investigates children's perceptions of conversational agents in-depth, and even less investigates how education might change these perceptions. We present K-12 workshops with associated conversational AI concepts to encourage healthier understanding and relationships with agents. Through studies with the curriculum, and children and parents from various countries, we found participants' perceptions of agents -- specifically their partner models and trust -- changed. When participants discussed changes in trust of agents, we found they most often mentioned learning something. For example, they frequently mentioned learning where agents obtained information, what agents do with this information and how agents are programmed. Based on the results, we developed recommendations for teaching conversational agent concepts, including emphasizing the concepts students found most challenging, like training, turn-taking and terminology; supplementing agent development activities with related learning activities; fostering appropriate levels of trust towards agents; and fostering accurate partner models of agents. Through such pedagogy, students can learn to better understand conversational AI and what it means to have it in the world.
H. Nicole Pang, Robert Parks, Cynthia Breazeal, Hal Abelson, “How Can I Code A.I. Responsibly?”: The Effect of Computational Action on K-12 Students Learning and Creating Socially Responsible A.I., Submitted to EAAI.
Abstract: Teaching young people about artificial intelligence (A.I.) is recognized globally as an important education effort by organizations and programs such as UNICEF, OECD, Elements of A.I., and AI4K12. A common theme among K-12 A.I. education programs is teaching how A.I. can impact society in both positive and negative ways. We present an effective tool that teaches young people about the societal impact of A.I. that goes one step further: empowering K-12 students to use tools and frameworks to create socially responsible A.I. The computational action process is a curriculum and toolkit that gives students the lessons and tools to evaluate positive and negative impacts of A.I. and consider how they can create beneficial solutions that involve A.I. and computing technology. In a human-subject research study, 101 U.S. and international students between ages 9 and 18 participated in a one-day workshop to learn and practice the computational action process. Pre-post questionnaires measured on the Likert scale students’ perception of A.I. in society and students’ desire to use A.I. in their projects. Analysis of the results shows that students who identified as female agreed more strongly with having a concern about the impacts of A.I. than those who identified as male. Students also wrote open-ended responses to questions about what socially responsible technology means to them pre- and post-study. Analysis shows that post-intervention, students were more aware of ethical considerations and what tools they can use to code A.I. responsibly. In addition, students engaged actively with tools in the computational action toolkit, specifically the novel impact matrix, to describe the positive and negative impacts of A.I. technologies like facial recognition. Students demonstrated breadth and depth of discussion of various A.I. technologies’ far-reaching positive and negative impacts. These promising results indicate that the computational action process can be a helpful addition to A.I. education programs in furnishing tools for students to analyze the effects of A.I. on society and plan how they can create and use socially responsible A.I.
Jasmine Shone, Robin Liu, Evan Patton, David Y.J. Kim, Design and Optimization of an Automatic Mobile Application Generating Learning Platform, Submitted to EAAI.
Abstract: We present a large language model-based learning platform that lets students automatically generate mobile applications for smartphones and tablets from natural language descriptions. Furthermore, we show that the user-generated apps can be optimized with simple modifications to the generative model’s input (“prompts”). This paper explores three different methods of modifying the prompt: 1) changing the selection mechanism of example pairs, 2) varying the number of example pairs, and 3) altering how the pairs are ordered within the prompt. Prompts are constructed from a set of example pairs (a textual description of an example app and its corresponding code) along with the description of the desired app. We evaluated the model’s performance with 18 possible candidate mobile apps, ranging from simple to complex, and used the BLEU score to compare the models outputs to manually created apps. Our results show that appropriate example pair selection and variation of the number of example pairs make a difference in the quality of the generated apps, but alteration of example pair ordering does not. We conclude with a discussion about the potential implications for CS education in light of generative models for code.
Nicole Pang, Robert Parks, Cynthia Breazeal, Hal Abelson, The Effect of Computational Action Process on Students’ Software Prototype Ideas, 2022 ICERI proceedings.
Abstract: Teaching engineering design has shown documented benefits for middle and high school classrooms, including high engagement, problem-solving, and group sensemaking for diverse learners. But learning computer science engineering concepts can challenge students’ confidence and motivation, often posing a barrier to sustained engagement with projects over time. We report research results on a novel computational action process, a learning framework intended to emphasize purpose-driven learning in computer programming settings. Educators in two pilot workshops used online tools and materials to scaffold students in deepening their problem definition formation during their engineering design processes. Interventions included facilitating students in creating “mind maps” of problems they cared about in their communities, gathering ethnographic data on user needs, using digital tools to weigh ethical considerations of their proposals, and employing other digital tools to plan and prototype software and AI-based solutions. In the three-hour online sessions, a majority of students refocussed their proposed software solutions to highlight social impact and align more closely with user needs. Representative examples include students whose first design goals appeared fairly generic (one student proposed “a mobile touchscreen canvas for artists”) or just-for-fun (“A random red button in the middle of the screen”). In the workshop, students employed mind maps to explore particular United National Sustainable Development Goals (UNSDG) they found relevant to their communities (for example: “Conservation”; “Land where animals are protected”; “Why are animals endangered?”); formulated user research questions (“What do you like/dislike about sharks?”); and drafted an online “impact matrix” enumerating potential positive and negative effects of the design. In post-workshop materials, most students shifted their goals toward solutions rooted in real problems and many describe impacts on those affected. Preliminary research into pre-/post-intervention artifacts from students appears to show a distinct increase in students reframing solutions toward “self-transcendent” design goals and toward a greater disciplinary understanding of ethical considerations. Results also indicate the need for refinement and further testing of materials while co-designing in classroom settings. For instance, questions remain regarding how teachers can facilitate problem definition in ways that emphasize a team’s personal or community connection to a UNSDG problem. Additional questions include how best to support students’ persistence in the design processes, and to gather data on which students show persistence in self-transcendent design goals versus just-for-fun goals. From these findings and directions for future work, we discuss implications for curriculum design of an engineering process tuned to AI-powered computational action solutions.
Julius Adebayo, Towards Effective Tools for Debugging Machine Learning Models, PhD dissertation, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2022.
Abstract: This thesis addresses the challenge of detecting and fixing the errors of a machine learning (ML) model—model debugging.
Current ML models, especially overparametrized deep neural networks (DNNs) trained on crowd-sourced data, easily latch onto spurious signals, underperform for small subgroups, and can be derailed by errors in training labels. Consequently, the ability to detect and fix a model’s mistakes prior to deployment is crucial.
Explainable machine learning approaches, particularly post hoc explanations, have emerged as the defacto ML model debugging tools. A plethora of approaches currently exist, yet it is unclear whether these approaches are effective.
In the first part of this thesis, we introduce a framework to categorize model bugs that can arise as part of the standard supervised learning pipeline. Equipped with the categorization, we assess whether several post hoc model explanation approaches are effective for detecting and fixing the categories of bugs proposed in the framework. We show that current approaches struggle to detect a model’s reliance on spurious signals, are unable to identify training inputs with wrong labels, and provide no direct avenue for fixing model errors. In addition, we demonstrate that practitioners struggle to use these tools to debug ML models in practice.
With the limitations of current approaches established, in the second part of the thesis, we present new tools for model debugging. First, we introduce an approach termed model guiding, which uses an audit set—a small dataset that has been carefully annotated by a task expert—to update a pre-trained ML model’s parameters. We formulate the update as a bilevel optimization problem that requires the updated model to match the expert’s predictions and feature annotations on the audit set. Model guiding can be used to identify and correct mislabelled examples. Similarly, we show that the approach can also remove a model’s reliance on spurious training signals.
The second debugging tool we introduce uses the influence function of an estimator to help identify training points whose labels have a high effect on an ML model’s disparity metric such as group calibration.
Taken together, this thesis makes advances towards better debugging tools for machine learning models.
Jessica Van Brummelen, Empowering K-12 Students to Understand and Design Conversational Agents: Concepts, Recommendations and Development Platforms, PhD dissertation, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2022.
Abstract: Conversation influences nearly every aspect of human life, and has done so for ages. Recently, people have begun to converse with technology. This method of technology interaction has rapidly become prominent, and raises unique questions about human-computer interaction. For instance, how do such human-like, relational interactions affect people’s trust of computer systems? Researchers have started to investigate such questions with respect to adults, finding correlations between trust and anthropomorphism of agents. However, very little research investigates children’s perceptions of these devices, and even less investigates how interventions might change these perceptions. This is despite how educational interventions have previously changed how people perceive and trust other technology. It is also despite how conversational technology is uniquely positioned to appeal to children, influence them relationally and potentially spread misinformation.
This dissertation presents educational interventions for K-12 students, which aim to encourage healthier understanding and relationships with conversational agents. This includes conversational agent curricula, development platforms and conceptual frameworks. Through studies with children and parents from Western, Industrialized, Educated, Rich and Democratic (WEIRD) and non-WEIRD countries, I found different subsets of the participants’ perceptions of agents changed differently through the activities. For instance, after learning to program agents, participants from non-WEIRD countries felt agents were more competent, more dependable and more like authority figures than those from WEIRD countries did. Children consistently felt agents were warmer and more humanlike than parents did. When participants discussed their trust of agents, I found they frequently mentioned where agents obtained their information, what agents do with the information they are given and how agents are programmed. I also found participants most often mentioned learning something when discussing why their trust changed.
These studies, as well as a systematic literature review and an analysis of various agent development platforms, informed the creation of a pedagogical framework of forty foundational conversational agent concepts, seventeen conversational agent design recommendations and thirteen conversational agent K-12 pedagogy recommendations. For instance, I recommend designing agents with more task-orientation in general, while considering the end user audience. I also recommend informing end users about the trustworthiness of agents through agent design and educational interventions. This is to increase transparency and allow end users to calibrate their trust accordingly. Educators may increase agent transparency through teaching the foundational agent concepts in the framework, which fall under the categories of natural language understanding; conversation representation; dialog management; data access and conversation context; and human-agent interaction. With conversational agents becoming increasingly ubiquitous, it is increasingly important for users of this technology—including and especially children—to be able to calibrate healthy perceptions and levels of trust towards it. This research aims to empower children to do this through a conversational agent design and pedagogy framework.
Jasmine Shone, Yes, You Can Make an App Too: A Systematic Study of Prompt Engineering in the Automatic Generation of Mobile Applications from User Queries, Research Science Institute, 2022.
Abstract: Aptly aims to use large language models to allow for on-the-fly generation of full apps given only a user description of an app idea. In order to optimize this platform and to provide guidance for other platforms or companies aiming to personalize large language models (LLMs) for their needs, we embark on one of the frst systematic studies of prompt engineering for the Codex model, a LLM that produces code from a natural-language input. Specifcally, we examine the effect of varying the token length, mechanism of choosing examples (random selection, least to most token size, cosine similarity, or an adapted version of Minimum Redundancy Maximum Relevance), and how they are ordered within the prompt (highest to lowest ranked, lowest to highest ranked, randomly shuffled) on the quality of generated code. We improve the pipeline's performance from baseline for complex apps by 55% (0.10 increase in BLEU score) using example selection mechanisms and 43% (0.13) for simple apps.
Nicole Pang, Computational Action in Action: Process and Tools that Empower Students to Make a Real-world Impact Using Technology, M.Eng thesis, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2022.
Abstract: How can we help K-12 students who are learning computer science and artificial intelligence (A.I.) feel motivated, competent, and empowered? The computational action framework, proposed by Tissenbaum, Sheldon, and Abelson, suggests that the preferable way is to ensure that young people are creating technology projects that address issues in their community. I add to this framework by creating the computational action process, which is composed of curriculum, toolkit, and website that teach five key concepts: defining a real-world problem; understanding users and communities; designing responsibly with and for users and communities; teamwork, project management, and implementation; and planning and making a long-lasting impact. From a research study conducted with 101 international young people in middle school and high school, results show that after learning the computational action process, students showed significant increase in computation skill, digital empowerment, and self-efficacy. Students also demonstrated an improved understanding of the impact of technology on people and society and improved ability to work towards solutions to ambiguous problems. This thesis describes the computational action process, presents the research, and analyzes the results, concluding with key findings, recommendations, and how this work contributes to the field of K-12 computer science education and A.I. literacy.
Elizabeth Harkavy, Accessible AI That’s Out of This World: Globalizing AI Literacy through Problem-Based Learning and Deep Learning Models in a Low Code Environment, M.Eng thesis, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2022.
Abstract: From phones, to advertisement, to search engines, AI is a constant presence in daily life. As AI continues to permeate our everyday behaviors, products, and relationships with the world, an urgent need for equitable, effective AI education emerges. While existing research has shown the importance of democratizing access to AI, this effort has been predominantly geared toward WEIRD (White Educated Industrialized Rich Democratic) populations which leads to biased results and false generalizations.
In order to expand equitable AI, an interface was implemented to allow for interaction with complex Natural Language Processing models in a low code environment, curriculum was crafted for a problem-based approach to teaching AI, a series of workshops were run with students from WEIRD (United States) and Non-WEIRD (India) countries, and the results were analyzed.
The research showed students’ confidence and demonstrated ability grew significantly after the workshops; students were able to demonstrate key AI literacy skills, build complex technology projects, and leverage AI to come up with original and creative solutions to specific, real-world problems. Students’ perceptions of Conversational AI agents became more positive after the workshops and notably, their trust in AI increased. When comparing students from WEIRD and Non-WEIRD countries, Non-WEIRD students were less critical of technology than their WEIRD counterparts and believed AI would be used to complete tasks for humans rather than with them.
Overall, this work showcases the possibility for gaining AI understanding and literacy skills through a problem-based approach to AI education in conjunction with hands-on tooling, as well as highlights the importance of expanding research to include populations beyond WEIRD demographics.
Ting-Chia Hsu, Hal Abelson, Evan Patton, Shih-Chu Chen, Hsuan-Ning Chang, (2022). Self-efficacy and behavior patterns of learners using a real-time collaboration system developed for group programming, "International Journal of Computer-Supported Collaborative Learning".
Abstract: In order to promote the practice of co-creation, a real-time collaboration (RTC) version of the popular block-based programming (BBP) learning environment, MIT App Inventor (MAI), was proposed and implemented. RTC overcomes challenges related to non-collocated group work, thus lowering barriers to cross-region and multi-user collaborative software development. An empirical study probed into the differential impact on self-efficacy and collaborative behavior of learners in the environment depending upon their disciplinary background. The study serves as an example of the use of learning analytics to explore the frequent behavior patterns of adult learners, in this case specifically while performing BBP in MAI integrated with RTC. This study compares behavior patterns that are collaborative or individual that occurred on the platform, and investigates the effects of collaboration on learners working within the RTC depending on whether they were CS-majors or not. We highlight advantages of the new MAI design during multi-user programming in the online RTC based on the connections between the interface design and BBP as illustrated by two significant behavior patterns found in this instructional experiment. First, the multi-user programming in the RTC allowed multiple tasks to happen at the same time, which promoted engagement in joint behavior. For example, one user arranged components in the interface design while another dragged blocks to complete the program. Second, this study confirmed that the Computer Programming Self-Efficacy (CPSE) was similar for individual and multi-user programming overall. The CPSE of the homogeneous CS-major groups engaged in programming within the RTC was higher than that of the homogeneous non-CS-major groups and heterogeneous groups. There was no significant difference between the CPSE of the homogenous non-CS group and the CPSE of the heterogeneous groups, regardless of whether they were engaged in individual programming or collaborative programming within their groups. The results of the study support the value of engaging with MAI collaboratively, especially for CS-majors, and suggest directions for future work in RTC design.
Peer Reviewed Paper
Jessica Zhu, Jessica Van Brummelen (2021). Teaching Students About Conversational AI Using Convo, a Conversational Programming Agent, IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).
Abstract: Smart assistants, like Amazon’s Alexa or Apple’s Siri, have become commonplace in many people’s lives, appearing in their phones and homes. Despite their ubiquity, these conversational AI agents still largely remain a mystery to many, in terms of how they work and what they can do. To lower the barrier to entry to understanding and creating these conversational AI agents for young students, we expanded on Convo, a conversational programming agent that can respond to both voice and text inputs. The previous version of Convo focused on teaching only programming skills, so we created a simple and intuitive user interface for students to use those programming skills to train and create their own conversational AI agents. We also developed a curriculum to teach students about key concepts in AI and conversational AI in particular. We ran a 3-day workshop in partnership with MIT’s eSPARK program, with 15 participating middle school students. Through the data collected from the pre- and post-workshop surveys as well as a mid-workshop brainstorming session, we found that after the workshop, students tended to think that conversational AI agents were less intelligent than originally perceived, gained confidence in their abilities to build these agents, and learned some key technical concepts about conversational AI as a whole. Based on these results, we are optimistic about Convo’s ability to teach and empower students to develop conversational AI agents in an intuitive way.
Preprint Video Presentation
Ting-Chia Hsu, Hal Abelson, Jessica Van Brummelen, (2021). The Effects of Applying Experiential Learning into the Conversational AI Learning Platform on Secondary School Students. Accepted by the IRRODL Special Issue "AI e-Learning and Online Curriculum".
Abstract: The purpose of this study was to design a curriculum of artificial intelligence (AI) application for secondary schools. The learning objective of the curriculum was to allow the students to learn the application of conversational AI on a block-based programming platform. Moreover, the empirical study actually implemented the curriculum in the formal learning of a secondary school for a period of 6 weeks. The study evaluated the learning performance of the students who were taught with the cycle of experiential learning in one class, while also evaluating the learning performance of the students who were taught with the conventional instruction, which was called the “cycle of doing projects” in the current study. Two factors, learning approaches and gender, were taken into account. The results showed that the learning effectiveness of females was significantly better than that of males regardless of whether they used the learning approaches of experiential learning or conventionally doing projects. Most of the males tended to be distracted from the conversational AI curriculum because they misbehaved during the conversational AI process. In particular, in the performance of using the Voice User Interface of the students learning with the conventional learning approach, the females outperformed the males significantly. The results of two-way ANCOVA revealed a significant interaction between gender and learning approaches on the computational thinking concepts. The females with the conventional learning approach of doing projects had the best computational thinking concepts in comparison with the other groups.
Jessica Zhu, Creating Your Own Conversational Artificial
Intelligence Agents Using Convo, a Conversational
Programming System, M.Eng thesis, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2021.
Abstract: Smart assistants like Amazon’s Alexa, Apple’s Siri, and Google’s Google Home have
become commonplace in many people’s lives, appearing in their phones and homes.
Despite their ubiquity, these conversational AI agents still largely remain a mystery
to many, in terms of how they work and what they can do.
To lower the barrier to entry to understanding and creating these conversational
AI agents for young students, I expanded on Convo, a conversational programming
agent that can respond to both voice and text inputs. I created a simple and intuitive
user interface for students to input training data, create programs, and test the
conversational AI agents they create. To further assist anyone in using Convo, I
also produced a couple of video and PDF tutorials that outline how to use Convo.
Additionally, I also developed a curriculum to teach students about key concepts in
AI and conversational AI in particular, including the Big 5 AI Ideas and the difference
between constrained and unconstrained natural language models.
I ran a 3-day workshop in partnership with MIT’s eSPARK program, with a total
of 15 participating middle school students. Through the data collected from the preand
post-workshop surveys as well as a mid-workshop brainstorming session, I was
able to explore how students’ perceptions, understanding, literacy, and visions of
conversational AI agents changed. During the workshop, students were able to create
their own conversational AI agents. I also found that after the workshop, students
tended to think that conversational AI agents were less intelligent than originally
perceived, gained confidence in their abilities to build these agents, and learned some
key technical concepts about conversational AI as a whole. Based on these results,
I am optimistic about Convo’s ability to teach and empower students to develop
conversational AI agents in an intuitive way.
Joy Yu, Using Facemesh in MIT App Inventor to Empower Students to Apply Artificial Intelligence, M.Eng thesis, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2021.
Abstract: It’s possible that a teenager’s first direct contact with artificial intelligence (AI) comes
in the form of facial filters that are readily accessible on Instagram and Snapchat.
These filters allow users to seemingly transform themselves into animals, put on pieces
of clothing, or try out new “looks”. Accompanying active usage, however, is often lack
of knowledge or awareness of the technology behind the fun applications. In my
work, my goal was to allow youth to access, apply, and understand both AI and AI
ethics. I built an open-source FaceExtension tool, an MIT App Inventor extension
that uses Facemesh, through which users gain access 486 different facial landmarks on
the face. I also designed a middle-school level curriculum to build a cat or lion facial
filter camera, published a corresponding online sidebar tutorial on the official "AI
with App Inventor" page, and ran workshops with 7-8th grade students. Students
with no background in coding or AI could successfully complete the curriculum with
enthusiasm, demonstrated by consistent attendance at workshops. Survey results
show they gained not only a better understanding for and increased interested in AI,
but also new realizations on the importance of AI ethics when applying AI tools.
Students also are interested in making different AI facial filters as a means of selfexpression
and social impact. Importantly, after attending the workshops, students
became empowered to create their own apps using AI. Furthermore, professional
educators have also tested the curriculum; they not only demonstrated excitement to
use FaceExtension in the classroom, but also published their own AI projects using
Murielle Dunand, Developing Visual Accessibility Options to Empower
Grade School Students in Designing Inclusive
Mobile Applications, M.Eng thesis, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2021.
Abstract: As technology becomes more available to the general public, it is important that it be as
accessible as possible. Accessibility for mobile apps is crucial given how pervasive smartphone
use has become. For students who use MIT App Inventor -- an online platform that enables
users to make their own mobile apps -- part of being an effective app designer is to appreciate
the importance and practice of inclusive design.
I have empowered young students to make their apps more visually accessible by: (1)
offering new options for larger text, higher contrast, and alternate text in App Inventor apps, (2)
creating a curriculum for students aged 13-18 about the principles of visually accessible design,
and (3) running the workshop three times and collecting student feedback on the curriculum.
After the workshop, students reported a more accurate understanding of the nature of low vision
as well as increased comfort with making visually accessible apps. Overall, this work has shown
that it is both simple and effective to teach the principles of accessible design to students as
young as middle-school age. The code changes have been added to MIT App Inventor and the
curriculum is available on the App Inventor website.
Jessica Van Brummelen, Phoebe Lin, (2021). Engaging Teachers to Co-Design Integrated AI Curriculum for K-12 Classrooms. ACM CHI Virtual Conference on Human Factors in Computing Systems (CHI 2021).
Artificial Intelligence (AI) education is an increasingly popular topic area for K-12 teachers. However, little research has investigated how AI curriculum and tools can be designed to be more accessible to all teachers and learners. In this study, we take a Value-SensitiveDesign approach to understanding the role of teacher values in the design of AI curriculum and tools, and identifying opportunities to integrate AI into core curriculum to leverage learners’ interests. We organized co-design workshops with 15 K-12 teachers, where teachers and researchers co-created lesson plans using AI tools and embedding AI concepts into various core subjects. We found thatK-12 teachers need additional scaffolding in AI tools and curriculum to facilitate ethics and data discussions, and value supports for learner evaluation and engagement, peer-to-peer collaboration, and critical reflection. We present an exemplar lesson plan that shows entry points for teaching AI in non-computing subjects and reflect on co-designing with K-12 teachers in a remote setting. Preprint
Jessica Van Brummelen, Viktoriya Tabunshchyk, Tommy Heng, (2021). Alexa, Can I Program You?: Student Perceptions of Conversational Artificial Intelligence Before and After Programming Alexa.
Growing up in an artificial intelligence-filled world, with Siri and Amazon Alexa often within arm’s—or speech’s—reach, could have significant impact on children. Conversational agents could influence how students anthropomorphize computer systems or develop a theory of mind. Previous research has explored how conversational agents are used and perceived by children within and outside of learning contexts. This study investigates how middle and high school students’ perceptions of Alexa change through programming their own conversational agents in week-long AI education workshops. Specifically, we investigate the workshops’ influence on student perceptions of Alexa’s intelligence, friendliness, aliveness, safeness, trustworthiness, human-likeness, and feelings of closeness. We found that students felt Alexa was more intelligent and felt closer to Alexa after the workshops. We also found strong correlations between students’ perceptions of Alexa’s friendliness and trustworthiness, and safeness and trustworthiness. We recommend designers carefully consider personification, transparency, playfulness and utility when designing conversational agents for learning contexts. Preprint
Ting-Chia Hsu, Hal Abelson, Natalie Lao, Yu-Han
Tseng, (2021). Behavioral-Pattern Exploration and
Development of An Instructional Tool for Young Children to Learn
AI. Computers and Education: Artificial Intelligence,
This study aimed at developing an instructional tool for artificial
intelligence education of the young students, and using learning
analytics to find out the sequential learning behavioral patterns of
the students during the process when they lean with the
instructional tool. The instructional experiment spent 9 weeks in
total. The first stage spent 5 weeks of the course involving
individual learning of experiencing the first two chapters which
were MIT App Inventor and Personal Image Classifier. The second
stage spent 4 weeks of the course involving cooperative learning of
experiencing the last two chapters which were the maker of a robot
car and the competition of the computational thinking board game. At
the second stage, there were two students divided into a group. Each
group would make a robot car together. Finally, they played the
computational thinking board game with the personal image
classification application they developed by themselves at the first
stage and the robot car they made by themselves at the second stage. The
innovative studies found the meaningful behavioral patterns when the
young students learned the application of artificial intelligence
with the instructional tool developed and proposed by the study.
Peer reviewed paper (preprint)
Natalie Lao, Reorienting Machine Learning Education Towards
Tinkerers and ML-Engaged Citizens, PhD dissertation, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2020.
Abstract: Artificial intelligence (AI) and machine learning (ML) technologies are appearing in everyday contexts, from recommendation systems for shopping websites to self -riving cars. As researchers and engineers develop and apply these technologies to make consequential decisions in everyone's lives, it is crucial that the public understands the AI-powered world, can discuss AI policies, and become empowered to engage in shaping these technologies' futures. Historically, ML application development has been accessible only to those with strong computational backgrounds working or conducting research in highly technical fields. As ML models become more powerful and computing hardware becomes faster and cheaper, it is now technologically possible for anyone with a laptop to learn about and tinker with ML.
This includes audiences without programming or advanced mathematics knowledge, who can use public ML tools to construct both useful technical artifacts and meaningful socio-ethical artifacts around ML. My research investigates the following question: How do we reorient ML engineering education to transform "ML consumers" to be "ML contributors"? In order to illuminate this problem, I have created a Machine Learning Education Framework. In this dissertation, I present the framework and three applications of it: (1) a course based on the framework that aims to develop ML self-efficacy in general college-level audiences; (2) a curriculum rubric based on the ML Education Framework and analyses of 6 ML/AI courses using the rubric; and (3) two public ML tools for high school and above ML tinkerers that empower them to create personal ML applications through image and audio classification.
The ML Education Framework is organized into three interconnected categories: Knowledge (General ML Knowledge, Knowledge of ML Methods, Bias in ML Systems, Societal Implications of ML), Skills (ML Problem Scoping, ML Project Planning, Creating ML Artifacts, Analysis of ML Design Intentions & Results, ML Advocacy, Independent Out-of-Class Learning), and Attitudes (Interest, Identity & Community, Self-Efficacy, Persistence). The three top level categories of the framework were chosen to encompass the holistic process of teaching people ML and ensuring continued participation in the field. "Knowledge" about ML will be necessary for individuals to thrive in today's ML-driven world and builds foundations for further engagement with the technology. ML "Skills" define specific actions and activities that students need to do and practice in order to become ML contributors.
The category "Attitudes" reflects long-term societal goals for educational initiatives in ML, and addresses the perspectives that students should develop through an ML course. This framework can be used in creating new ML curricula, in analyzing the results of ML interventions, and in situating support tools within ML education for students of a variety of backgrounds and levels. Today a large portion of the world solely resides in the "ML consumer" population. Regulation of ML technologies is nascent. As policymakers think about reasonable legal structures, the public should be able to actively participate in these discussions. It is imperative that we empower these "ML consumers" to become "ML tinkerers" and "ML-engaged citizens".
Kevin Weng, Examining Conversational Programming Design Needs with Convo, a Voice-First Conversational Programming System Using Natural Language, M.Eng thesis, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2020.
Abstract: The emergence of voice-based technology and voice assistants has paved the way for new opportunities to democratize learning computational thinking and programming skills, especially for people who do not have access to the traditional programming experience, due to external circumstances or disabilities. In this thesis, I created Convo, a voice-first conversational programming system that allows users to cre- ate programs and develop programming skills simply through natural interactions and conversations with a programming agent and voice feedback. Additionally, I conducted a user study to study the effectiveness of voice-first conversational pro- gramming systems like Convo as well as receive user feedback to explore the design needs of such systems. The user study involved forty-five participants ranging from fourteen to sixty-five years old completing tasks in Convo using and comparing three different input modalities - using just voice inputs; using just text inputs; and using both voice and text inputs. The participants answered questions about their experi- ences with each input modality and general feedback on conversational programming systems. Results showed that participants have preference towards text and found voice-based programming the most difficult to use among the three input modalities. However many participants, especially those new to programming, saw the value and future potential of voice-based programming, especially when speech recognition be- comes more accurate. Additionally, I created design recommendations based on the results for conversational programming systems, including being flexible and accessi- ble, reducing cognitive load and improving speech recognition and natural language understanding.
Jessica Van Brummelen, Kevin Weng, Phoebe Lin, Catherine Yeo, (2020). CONVO: What does conversational programming need? An exploration of machine learning interface design. IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).
Vast improvements in natural language understanding and speech recognition have paved the way for conversational interaction with computers. While conversational agents have often been used for short goal-oriented dialog, we know little about agents for developing computer programs. To explore the utility of natural language for programming, we conducted a study (n=45) comparing different input methods to a conversational programming system we developed. Participants completed novice and advanced tasks using voice-based, text-based, and voice-or-text-based systems. We found that users appreciated aspects of each system (e.g., voice-input efficiency, text-input precision) and that novice users were more optimistic about programming using voice-input than advanced users. Our results show that future conversational programming tools should be tailored to users' programming experience and allow users to choose their preferred input mode. To reduce cognitive load, future interfaces can incorporate visualizations and possess custom natural language understanding and speech recognition models for programming. Peer Reviewed Paper Preprint
Phoebe Lin, Jessica Van Brummelen, Galit Lukin, Randi Williams, Cynthia Breazeal, (2020). Zhorai: Designing a Conversational Agent for Children to Explore Machine Learning Concepts. Tenth Symposium on Educational Advances in Artificial Intelligence (EAAI).
Understanding how machines learn is critical for children to develop useful mental models for exploring artificial intelligence (AI) and smart devices that they now frequently interact with. Although children are very familiar with having conversations with conversational agents like Siri and Alexa, children often have limited knowledge about AI and machine learning. We leverage their existing familiarity and present Zhorai, a conversational platform and curriculum designed to help young children understand how machines learn. Children ages eight to eleven train an agent through conversation and understand how the knowledge is represented using visualizations. This paper describes how we designed the curriculum and evaluated its effectiveness with 14 children in small groups. We found that the conversational aspect of the platform increased engagement during learning and the novel visualizations helped make machine knowledge understandable. As a result, we make recommendations for future iterations of Zhorai and approaches for teaching AI to children. Peer Review Paper
Jessica Van Brummelen, Catherine Yeo, Kevin Weng, (2020). Learning to program conversationally: A conversational agent to further democratize programming. 14th annual International Technology, Education and Development Conference (INTED2020).
Interacting graphically, as in App Inventor, is no longer the only primary mode of human-computer
interaction. With the rise of conversational agents, like Amazon Alexa and Siri, many people
interact using voice. Its convenience and efficiency make it a viable method for sending messages
and gathering information. For similar reasons, we see voice as an effective method to lower the
barrier to entry to programming and teach an even wider audience computational thinking (CT)
skills. Imagine telling your computer to make your toy race around a track while it plays your
favorite song—and it actually happening!
We developed a voice-based, conversational agent to do just this: empower nearly anyone to
program—just by having a conversation. The system is capable of program creation, debugging
and execution. Peer Reviewed Abstract
Nikhil Bhatia, Using Transfer Learning, Spectrogram Audio Classification, and MIT App Inventor to Facilitate Machine Learning Understanding, M.Eng thesis, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2020.
Abstract: Recent advancements in deep learning have brought machine learning and its many applications to the forefront of our everyday lives. As technology has become more and more integrated into our educational curriculum, researchers have focused on creating deep learning tools that allow students to interact with machine learning in a way that incites curiosity and teaches important concepts. My research contribution focuses on applying transfer learning and spectrogram audio classification methods to teach basic machine learning concepts to students. I introduce the Personal Audio Classifier (PAC), a web application that allows users to train and test custom audio classification models that can classify 1-2 second sound bites recorded by the user. Alongside this in-browser machine learning tool, I provide a set of best practices for spectrogram audio conversion in machine learning applications. I also contribute a custom App Inventor extension that allows users to use the output of the web interface to create App Inventor applications that rely on their trained custom audio classification model. Finally, I provide a high school workshop curriculum based on PAC and the App Inventor extension and detail the results of running workshops with three high school classes at the Boston Latin Academy. My results show that high school students with no prior machine learning knowledge are able to grasp important technical concepts related to machine learning and its applications, as well as build and explore custom machine learning models in the browser through the exploration of a hands-on curriculum based around PAC.
Natalie Lao, Nikhil Bhatia, (2020). Using Transfer Learning, Spectrogram Audio Classification, and MIT App Inventor to Facilitate Machine Learning Understanding, International Conference on Computational Thinking Education 2020 (CTE2020).
Abstract: Recent advancements in deep learning have brought machine learning and its many applications to the forefront of our everyday lives. As technology has become more and more integrated into our educational curriculum, researchers have focused on creating deep learning tools that allow students to interact with machine learning in a way that incites curiosity and teaches important concepts. Our research contribution focuses on applying transfer learning and spectrogram audio classification methods to teach basic machine learning concepts to students. We introduce the Personal Audio Classifier (PAC), a web interface that allows users to train and test custom audio classification models that can classify 1-2 second sound bites recorded by the user. We also contribute a custom App Inventor extension that allows users to use the output of the web interface to create App Inventor applications that rely on their trained custom audio classification model. Peer Reviewed Paper
Natalie Lao, Irene Lee, Hal Abelson, (2020). Experiences from Teaching Actionable Machine Learning at the University Level through a Small Practicum Approach, International Conference on Computational Thinking Education 2020 (CTE2020).
Abstract: Machine learning (ML) courses have traditionally been taught through a math-first approach. They generally begin by establishing mathematical theories behind ML, such as the perceptron algorithm, logistic regression, and backpropagation, and then use these building blocks to motivate more complex structures such as neural networks. Such educational resources may not be sufficient or preferable for audiences who wish to use ML to build useful artifacts but do not have a strong mathematical or programming background. In this paper, we introduce a new framework for teaching actionable ML that combines three components in a Use-Modify-Create progression: (1) technical modules taught through hands-on labs, (2) a capstone project, and (3) supplemental lectures for new areas of research. This framework was applied in two iterations of a semester-long practicum at Massachusetts Institute of Technology (MIT) as a beginner-accessible course aimed at helping a broad range of students gain the ability to ideate and implement independent ML projects. We present the curriculum, student projects, pre and post- course survey responses, assignment grades, reflective discussions, and learnings from both iterations of the course. Our results indicate that the proposed actionable pedagogical framework for ML along with the content and practices of the course were effective for increasing students’ practical self-efficacy in ML and computational identity as developers of ML applications. The findings of this study illuminate patterns of interaction with ML systems that support a practical approach to teaching ML in order to increase knowledge acquisition, future learning ability, and motivation in beginner students. Peer Reviewed Paper
Fred Martin, Irene Lee, Nicholas Lytle, Sue Sentance, Natalie Lao, (2020). Panel: Extending and Evaluating the Use-Modify-Create Progression for Engaging Youth in Computational Thinking, SIGCSE ’20 (March 11–14, 2020, Portland, OR, USA).
Abstract: The Use-Modify-Create progression (UMC) was conceptualized in 2011 after comparing the productive integration of computational thinking across National Science Foundation-funded Innovative Technology Experiences for Students and Teachers (ITEST) pro- grams. Since that time, UMC has been widely promoted as a means to scaffold student learning of computational thinking (CT) while enabling personalization and allowing for creative adaptations of pre-existing computational artifacts. In addition to UMC’s contin- ued application, it has recently been utilized to scaffold student learning in topics as diverse as machine learning, e-textiles, and computer programming. UMC has also been applied to instruc- tional goals other than “supporting students in becoming creators of computational artifacts.” This panel will re-examine the UMC progression and refine our understanding of when its use is suitable, and when not, and share findings on evaluations and extensions to UMC that are productive in new and different contexts. Organized Panel
Jessica Van Brummelen, Tommy Heng, Viktoriya Tabunshchyk, (to appear 2021). Teaching Tech to Talk: K-12 Conversational Artificial Intelligence Literacy Curriculum and Development Tools. AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-21).
With children talking to smart-speakers, smart-phones and even smart-microwaves daily, it is increasingly important to educate students on how these agents work-from underlying mechanisms to societal implications. Researchers are developing tools and curriculum to teach K-12 students broadly about artificial intelligence (AI); however, few studies have evaluated these tools with respect to AI-specific learning outcomes, and even fewer have addressed student learning about AI-based conversational agents. We evaluate our Conversational Agent Interface for MIT App Inventor and workshop curriculum with respect to eight AI competencies from the literature. Furthermore, we analyze teacher (n= 9) and student (n= 47) feedback from workshops with the interface and recommend that future work leverages design considerations from the literature to optimize engagement, collaborates with teachers, and addresses a range of student abilities through pacing and opportunities for extension. We found students struggled most with the concepts of AI ethics and learning, and recommend emphasizing these topics when teaching. Preprint
Jessica Van Brummelen, Phoebe Lin, (2020 in submission). Engaging Teachers to Co-Design Integrated AI Curriculum for K-12 Classrooms.
Artificial Intelligence (AI) education is an increasingly popular topic area for K-12 teachers. However, little research has investigated how AI education can be designed to be more accessible to all learners. We organized co-design workshops with 15 K-12 teachers to identify opportunities to integrate AI education into core curriculum to leverage learners' interests. During the co-design workshops, teachers and researchers co-created lesson plans where AI concepts were embedded into various core subjects. We found that K-12 teachers need additional scaffolding in the curriculum to facilitate ethics and data discussions, and value supports for learner engagement, collaboration, and reflection. We identify opportunities for researchers and teachers to collaborate to make AI education more accessible, and present an exemplar lesson plan that shows entry points for teaching AI in non-computing subjects. We also reflect on co-designing with K-12 teachers in a remote setting. Preprint
Xiaofei Zhou, Jessica Van Brummelen, Phoebe Lin (2020 in submission). Designing AI Learning Experiences for K-12: Emerging Works, Future Opportunities and a Design Framework.
Artificial intelligence (AI) literacy is a rapidly growing research area and a critical addition to K-12 education. However, support for designing tools and curriculum to teach K-12 AI literacy is still limited. There is a need for additional interdisciplinary human-computer interaction and education research investigating (1) how general AI literacy is currently implemented in learning experiences and (2) what additional guidelines are required to teach AI literacy in specifically K-12 learning contexts. In this paper, we analyze a collection of K-12 AI and education literature to show how core competencies of AI literacy are applied successfully and organize them into an educator-friendly chart to enable educators to efficiently find appropriate resources for their classrooms. We also identify future opportunities and K-12 specific design guidelines, which we synthesized into a conceptual framework to support researchers, designers, and educators in creating K-12 AI learning experiences. Preprint
Jessica Van Brummelen, (2019). Conversational Artificial Intelligence Development Tools for K-12 Education. AAAI Fall Symposium.
Abstract: Artificial intelligence (AI) education is becoming increasingly
important as the adoption of AI technology, including
conversational agents such as Amazon Alexa, Siri, and
the Google Assistant, increases. Current educational and programming
tools enable non-programmers to develop simple
conversational agents, or advanced programmers to develop
complex agents. However, to the author’s knowledge, there
are no tools for non- or novice programmers to develop conversational
agents for the purpose of learning AI and programming
skills. This paper describes AI curriculum that includes
content about conversational agents, machine learning
(ML), and AI ethics, as well as a blocks-based conversational
AI interface developed within MIT App Inventor. During a
series of six workshops, students used this interface to develop
conversational agents for social good, including a memory
aide, math tutor, speech visualizer, and recycling assistant.
In this paper, I present (1) the blocks-based interface,
(2) the conversational AI curriculum, (3) how conversational
AI directly relates to computational thinking skills,
and (4) results from an initial small-scale study. The results
show that through the curriculum and using the blocksbased
conversational AI interface, students learned AI and
ML concepts, programming skills, and to develop conversational
agents for social good. Peer Reviewed Paper
Jessica Van Brummelen, Judy Hanwen Shen, Evan W. Patton, (2019). The Popstar, the Poet, and the Grinch: Relating Artificial Intelligence to the Computational Thinking Framework with Block-based Coding. International Conference on Computational Thinking Education 2019 (CTE2019).
Abstract: As the world becomes increasingly saturated with artificial
intelligence (AI) technology, computational thinking
(CT) frameworks must be updated to incorporate AI concepts.
In this paper, we propose five AI-related computation
concepts, practices, and perspectives: classification;
prediction; generation; training, validating, and testing;
and evaluating. We propose adding them to a widely-used
CT framework. We also present an MIT App Inventor extension
that explores this framework through projectbased
learning. Through the framework and practical
block-based coding artifacts, students and educators will
be empowered to learn about the rapidly-growing field of
AI, and gain skills necessary for the future of work. Peer Reviewed Paper
Evan W. Patton, Audrey Seo, and Franklyn Turbak,
(2019). Enhancing Abstraction in App Inventor with Generic
Event Handlers. Blocks & Beyond 2019.
Abstract: Work on code
smells in blocks languages has found that programmers often duplicate
blocks code rather than abstracting over common patterns of
computation using procedure-like features. For example, previous
analyses of over a million MIT App Inventor projects have revealed
that procedures are used surprisingly rarely in the wild and that many
users miss opportunities for using procedural abstraction to avoid
code duplication in their projects.
In this work, we use data analysis to explain how particular features
of App Inventor create barriers to abstracting over event handlers. In
many cases, duplicated code in event handlers cannot be extracted into
a procedure without using so-called generic blocks that abstract over
a particular component (e.g., a label). Generic blocks are rarely used
in practice, possibly because programmers do not know about them or
find them difficult to use. But even proceduralization with generic
blocks does not remove the need for duplicating the event handlers
We address these issues with two enhancements to App Inventor. First,
we add generic event handlers, a new form of abstraction that allows
specifying a single handler for all components of a particular
type. Second, we add a way to easily convert between specific and
generic blocks to facilitate genericization, that is,
abstracting actions over a particular component to apply to a group of
components of that type.
We also discuss related design choices and ways to encourage
programmers to use the new features to avoid code duplication. Our
work is an example of data-informed programming language design, in
which the creation or modification of features is informed by the
analysis of large datasets of programs from the language's target
audience. Peer Reviewed Paper
Evan W. Patton, (2019). Exploring Blockly
Performance: Case Studies in MIT App Inventor. Blockly
User Summit 2019 (Oct 10-11, 2019, Mountain View, CA).
Abstract: As the first project to use Blockly, the MIT App Inventor
users have pushed the boundaries of Blockly's capabilities. In our
datasets we have seen projects with upwards of 1400 blocks. I
present two ways in which we measure performance and four different
techniques used in App Inventor's production system to improve on
Blockly performance for large workspaces. These improvements include
reducing the amount of time spent in rendering and three different
improvements to worst case runtime for bulk operations on workspaces
with many blocks. Invited Talk
Evan W. Patton, (2019). Realtime Collaboration
with Blocky: Experiences with MIT App Inventor. Blockly
User Summit 2019 (Oct 10-11, 2019, Mountain View, CA).
Abstract: In her Masters of Engineering thesis, Deng explored three
different approaches to multi-user collaboration on MIT App Inventor
projects. I briefly introduce the different computational models and
network infrastructures for collaboration, and explain how we chose
the implementation we did for realtime collaboration in App
Inventor. A brief overview of Deng's user study and preliminary
results are also presented. Invited Talk
Danny Tang, Yuria Utsumi, and Natalie Lao, (2019). PIC: A Personal Image Classification Webtool for High School Students. IJCAI 2019 EduAI Workshop.
Abstract: As machine learning (ML) permeates our society
and manifests itself through commonplace tech-
nologies such as autonomous vehicles, facial recog-
nition, and online store recommendations, it is nec-
essary that the increasing number of people who
rely on these tools understand how they work. As
such, we need to develop effective tools and cur-
ricula for introducing ML to novices. Our work
focuses on teaching core ML concepts with image classification, one of the most basic and widespread
examples of machine learning. We built the PIC, a
web interface that allows users to train, test, and
analyze personalized image classification models
on pictures taken with their webcams, as well as a companion extension for MIT App Inventor that al-
lows users to use their models to classify objects in
a mobile application. We also created high school
level curricula for workshops based on the afore- mentioned interface and MIT App Inventor,
and ran pilot workshops with two classses of high school students from Boston Latin Academy.
Our findings indicate that high school students with no ML background are able to learn and understand gen- eral concepts and applications of ML through us- ing our tools, as well as successfully utilize models they build for personal use. Peer Reviewed Paper
Natalie Lao, Irene Lee, and Hal Abelson, (2019). A Deep Learning Practicum: Concepts and Practices for Teaching Actionable Machine Learning at the Tertiary Education Level.
ICERI 2019 Conference.
Abstract: Despite a rapid expansion of machine learning (ML) across fields and industries, little is known about
how to best prepare students for work in ML. Most ML courses today are taught at the college or
professional level using a theoretical programming approach. Existing educational resources may not
be sufficient or preferable for many audiences, particularly those who do not have a strong computer
science background, who wish to attain basic understanding of and ability to use ML. We present
learnings from a beginner level, semester-long actionable machine learning course taught at
Massachusetts Institute of Technology meant to be accessible for students with minimal computer
science knowledge. Based on analysis of survey responses and student projects, we find 5 core
concepts (Multilayer Networks, Convolutional Neural Networks, Transfer Learning, Recurrent Neural
Networks, and Embeddings & Generative Models) and 8 core skills (scoping a problem, choosing
datasets, creating datasets, choosing models, modifying models, creating models, modifying learning
rates, and training & testing) that helped lead to student self-efficacy as independent ML developers
when mastered. We conclude by discussing implications of this research on effective course design
and educational efforts for beginner level university courses in ML. Peer Reviewed Paper
Jessica Van Brummelen, (2019). Conversational Agents to Democratize Artificial Intelligence. 2019 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).
Artificial Intelligence (AI) technology can be found nearly
everywhere. However, developing and controlling this tech-
nology is generally limited to large companies or those with
extensive training in computer science. It is difficult for many
who have application ideas for AI to even know where to start
when developing AI technology.
In , I empower students with little to no programming
experience to develop conversational AI agents (programs that
use AI and natural language to communicate with humans).
Through high school workshops and block-based coding tools,
students created agents to help people remember forgotten
words, determine how to recycle items, and learn math
concepts. Students learned computational thinking (CT) and
machine learning (ML) concepts through developing these
Nevertheless, this system has limitations. It uses the Alexa
Skills platform, which requires users to have access to Alexa
Developer and Amazon Web Services accounts. As well, to
speak to the agents, one must begin all conversations with
the word, ”Alexa”, which can be awkward and frustrating.
Furthermore, the visual coding tool limits its user-base to
those with regular vision and those who know how to use
a traditional computer. Conference Paper
Tissenbaum, M., Sheldon, J., & Abelson, H. (2019). From Computational Thinking to Computational Action. Communications of the ACM, March 2019, Vol. 62 No. 3, Pages 34-36.
Abstract: Computational action, a new framing for computing education, proposes that while learning about computing, young people should also have opportunities to create with computing that have direct impact on their lives and their communities. In this Viewpoint, we outline two key dimensions of computational actioncomputational identity and digital empowermentand further argue that by focusing on computational action in addition to computational thinking, we can make computing education more inclusive, motivating, and empowering for young learners. Learners have the capacity to develop computational products that can have authentic impact in their lives from the moment they begin learning to code, all they need is to be situated in contexts that allow them to do so.
Too often, K-12 computing education has been driven by an emphasis on kids learning the "fundamentals" of programming. Even more progressive CS education that centers around the development of learners' computational thinking has largely focused on learners understanding the nuanced elements of computation, such as variables, loops, conditionals, parallelism, operators, and data handling.10 This initial focus on the concepts and processes of computing, leaving real-world applications for "later" runs the risk of making learners feel that computing is not important for them to learn. It begs the question far too many math or physics students have asked, "When will we use this in our lives?"
See the article at CACM
Kevin Zhu, An Educational Approach to Machine Learning with Mobile Applications, M.Eng thesis, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2019.
Abstract: Machine learning has increasingly become a major topic in computer science for stu- dents to learn. However, it can be quite technical and thus difficult for students to grasp, especially those in high school and under. To make machine learning and its applications more accessible to younger students, we developed a series of machine learning extensions for MIT App Inventor. MIT App Inventor is a web application for users with minimal programming experience to easily and quickly build mobile applications, and these extensions allow users to build applications that incorporate powerful machine learning functionality. These extensions were tested over a 6-week class with about 10 students and can be used as an educational tool.
Danny Tang, Empowering Novices to Understand and Use Machine Learning With Personalized Image Classification Models, Intuitive Analysis Tools, and MIT App Inventor, M.Eng thesis, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2019.
Abstract: As machine learning permeates our society and manifests itself through commonplace technologies such as autonomous vehicles, facial recognition, and online store recom- mendations, it is necessary that the increasing number of people who rely on these tools understand how they work. As such, we need to develop effective tools and curricula for introducing machine learning to novices. My work focuses on teaching core machine learning concepts with image classification, one of the most basic and widespread examples of machine learning. I built a web interface that allows users to train and test personalized image classification models on pictures taken with their computers webcams. Furthermore, I built an extension for MIT App Inventor, a platform for building mobile applications using a blocks-based programming lan- guage, that allows users to use the models they built in the web interface to classify objects in their mobile applications. Finally, I created high school level curricula for workshops based on using the aforementioned interface and App Inventor extension, and ran the workshops with two classes of high school students from Boston Latin Academy. My findings indicate that high school students with no machine learning background are able to learn and understand general concepts and applications of machine learning through hands-on, non-technical activities, as well as successfully utilize models they built for personal use.
Kelsey K. Chan, Building an Online Community of Creators Through MIT App Inventor, M.Eng thesis, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2019.
Abstract: An online community whose primary activity involves sharing personal creations is called an Online Community of Creators (OCOC). These communities provide a space where one users work can serve as inspiration for others and where users can offer feedback to each other, thus fostering creativity and helping users learn and grow. I designed and implemented a stand-alone gallery where App Inventor programmers can showcase their projects and receive feedback on their work. The goals of this new gallery were (1) to encourage more users to share their projects and (2) to increase the discoverability of shared projects among the App Inventor community. Based on the user study results, the new gallery addresses many usability pain points that users faced in the old gallery. Users expressed that they are more likely to share projects and view projects using the new gallery. By continuing to improve the usability of the gallery user interface, we hope to encourage even more users to publish projects to the gallery and to explore projects created by other members of the App Inventor community.
Jessica Van Brummelen Tools to Create and Democratize Conversational Artificial Intelligence, M.S. thesis, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2019.
Abstract: The world is becoming increasingly saturated with voice-first technology, such as Ama- zon Alexa and Google Home devices. As this technology becomes more complex, the skill set needed to develop conversational AI applications increases as well. This work bridges the gap, democratizes AI technology, and empowers technology consumers to become technology developers. In this thesis, I develop block-based Alexa Skill programming tools, enabling anyone even elementary school students to create complex conver- sational AI applications. During high school workshops, students created Alexa Skills to help others remember forgotten words, learn math concepts, ease recycling, and display Alexas speech on screen for those hard of hearing. Additionally, I developed a conversational AI curriculum and partnered with MITs High School Studies Program to provide workshops to the Boston community. We taught students about the capabilities, limitations, and implications of conversational AI, and explored research questions, such as "What do students believe, understand, and think about conversational agents?"; "Can students develop their own conversational AI applications?"; and "What do students envision for the future of conversational AI?" The results from a pre-workshop assessment suggested that despite not understand- ing how conversational agents worked, students could think of ways for conversational agents to solve problems. The post-workshop assessment suggested that through the workshops, students learned conversational AI and machine learning concepts; could identify capabilities and limitations of conversational agents; felt proud of their project development; were interested in developing their projects further; and were generally hopeful and excited about the future of conversational AI. Through this research, students learned about the power and limitations of AI, were empowered to solve real-world problems using AI, and developed socially useful conversational AI agent applications.
Nichole I. Clarke, Through the Screen and into the World: Augmented Reality Components with MIT App Inventor, M.Eng thesis, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2019.
Abstract: Augmented reality (AR) is an immersive technology which augments the real-world with virtual content. The added, virtual content is grounded in the real-world, provides in-context information, and creates experiences that could not be perceived by ones senses alone.
As an educational technology, augmented reality has the power to motivate students and greatly shape their learning [1,2]. However, theres a lack of educational AR applications, and many educators and learners do not have the technical training to develop such applications.
I hypothesize that one solution is to empower students by providing them with direct access to AR development tools and grounding their learning through computational action. My work focuses on creating such augmented reality development tools, which are comprehensible and employable by anyone.
I built upon MIT App Inventor, a blocks-based mobile development tool, to democratize augmented reality application development. I developed a set of 20 augmented reality components, which allows users to build iOS augmented reality applications and enables the development of curriculum that promotes computational action with AR. I created a set of tutorials for the AR components and designed sample applications using those components with increasing levels of complexity. Finally, I conducted a pilot study with MIT students where they built applications using the augmented reality components.
My findings indicate that study participants felt empowered by working with the AR components. Participants, who originally doubted their ability to create AR applications, believed that they could build augmented reality applications after using the AR components.
Evan W. Patton, Michael Tissenbaum, and Farzeen Harunani (2019). MIT App Inventor: Objects, Design, and Development. In: Computational Thinking Education, S. C. Kong & H. Abelson (Eds.), Springer Nature, New York.
Abstract: MIT App Inventor is an online platform designed to teach computational
thinking concepts through development of mobile applications. Students create applications by dragging and dropping components into a design view and using a visual
blocks language to program application behavior. In this chapter, we discuss (1) the
history of the development of MIT App Inventor, (2) the project objectives of the
project and how they shape the design of the system, and (3) the processes MIT uses
to develop the platform and how they are informed by computational thinking literature. Key takeaways include use of components as abstractions, alignment of blocks
with student mental models, and the benefits of fast, iterative design on learning.
Mike Tissenbaum, Josh Sheldon, and Hal Abelson, (2018). Reducing the Barriers for Computational Action: How Blocks-Based Programming Can Support Digital Empowerment. In: Proceedings of the BLOCKS+ Workshop, co-located with SPLASH 2018 (Nov. 5, 2018, Boston, MA).
Abstract: Computational action, by which learners are given opportunities to build projects with real-world applicability, is, we argue, a preferable way to engage young people in learning how to program computers. Mobile computing, as the dominant form-factor of computing currently, provides an especially meaningful opportunity for youth to create these projects. Meanwhile, platforms with powerful abstractions, that make operations that might otherwise be difficult simple, build student capability to create projects that enhance their computational action.
Natalie Lao, (2018). CloudDB in MIT App Inventor. In: Proceedings of the BLOCKS+ Workshop, co-located with SPLASH 2018 (Nov. 5, 2018, Boston, MA).
Abstract: Data sharing through Cloud technology is one of the most powerful new computer science concepts of the past few decades. As such, developing powerful and easy-to-use tools for incremental learning and application of shared data concepts is an important endeavor. My work focuses on using MIT App Inventor, a popular blocks-based mobile application development tool for teaching computational thinking to young students, to make shared data technology understandable and usable by anyone without the need for extensive computer science training. I present the deployment of CloudDB, a set of coding blocks for MIT App Inventor that allows users to store, retrieve, and share various types of data in tag-value pairs on a Redis server for their mobile applications.
Evan W. Patton, Stefania Druga, Jessica Van Brummelen, Joe Mazzone, and Xavier Puig Fernandez, (2018). Panel: Block Abstractions for Artificial Intelligence. In: Proceedings of the BLOCKS+ Workshop, co-located with SPLASH 2018 (Nov. 5, 2018, Boston, MA).
Abstract: Block-based programming languages, such as Scratch and MIT App Inventor, make computational concepts accessible to those as young as primary school students. As technology advances, so do the abilities of such block-based programming languages. This panel will discuss the state-of-the-art in artificial intelligence (AI) abstractions in block-based languages. Specifically, it will address block-based AI challenges, pedagogical value, and applications.
Evan W. Patton, Mark Sherman, and Michael Tissenbaum, (2018). Research Tools for MIT App Inventor. In: Proceedings of the BLOCKS+ Workshop, co-located with SPLASH 2018 (Nov. 5, 2018, Boston, MA).
Abstract: Understanding how ideas move through a classroom as students explore, learn, and share their findings with other students is important for computational thinking education. We present a set of enhancements to MIT App Inventor to enable fine-grained analysis of both qualitative and quantitative data. Our framework combines real-time event streams, project snapshots, and screen captures with audio recording to allow for targeted queries around when components or concepts not previously seen appear in a project. This automated approach reduces the amount of time spent sifting through data by providing researchers new tools for performing analysis of App Inventor projects.
Evan W. Patton and Danny Tang, (2018). JSON Interoperability in MIT App Inventor. In: Proceedings of the BLOCKS+ Workshop, co-located with SPLASH 2018 (Nov. 5, 2018, Boston, MA).
Wen Xi and Evan W. Patton, (2018). A Blocks-Based Approach to Internet of Things in MIT App Inventor. In: Proceedings of the BLOCKS+ Workshop, co-located with SPLASH 2018 (Nov. 5, 2018, Boston, MA).
Abstract: Internet of Things (IoT) integrates physical devices and creates opportunities for people to interact with the surrounding
environment. While a number of blocks-based approaches
exist for programming some hardware, such as Snap! for
Arduino, Microblocks, and Scratch extensions, this is still an
underexplored area. In this paper, we propose a block-based
programming approach using MIT App Inventor to enable
novices be able to build mobile apps integrated with IoT
technology. We also review other block languages applied
for IoT. We conclude with some thoughts on how blocks
languages might inspire people to create with IoT.
J. Van Brummelen and H. Abelson. "'What's conversational AI?' with MIT App Inventor and Amazon Alexa", Amazon Research Days 2018, Boston, Oct. 2018.
Abstract: This work aims to democratize conversational AI technology. Currently, MIT App Inventors block-based coding tools empower anyone to develop their own mobile apps. Soon, users will also be able to program their own Alexa skills, conversational AI, and neural networks using blocks. Through middle school workshops, research questions, such as "Can students learn about the capabilities, limitations and implications of AI through developing conversational AI mobile apps?", will be explored.
S.C. Kong, D. Andone, G. Biswas, G., T. Crick, T., H.U. Hoppe, T. C. Hsu, R.H. Huang, K.Y. Li, C.K. Looi, M. Milrad, J. Sheldon, J.L. Shih, K.F. Sin, M. Tissenbaum, & J. Vahrenhold (Eds.),
On Tools that Support the Development of Computational Thinking Skills: Some Thoughts and Future Vision, Proceedings of the International Conference on Computational Thinking Education 2018. Hong Kong: The Education University of Hong Kong.
Abstract: Development of Computational Thinking (CT) is an area of many initiatives in the last years, due to the importance of having CT skills. There are many environments that allow learners to develop such skills, for instance Scratch and MIT App Inventor, in a visual and intuitive way. As in professional software development, assisting tools that help and guide learners are starting to appear. In this paper, we discuss the current status of these tools, based on an analysis of what state-of-the-art CT assessment tools, such as Dr. Scratch for Scratch and CodeMaster for App Inventor, offer. We report their limitations and envision and discuss future enhancements.
Peer Reviewed Paper
C.H. Tseng, M. Tissenbaum, W.H. Kuan, F.C. Hsu, C.C. Wong. A design-based approach to implementing a computational thinking curriculum with App Inventor and the Internet of Things, CTE 2018, July 2018
Abstract: The growing ubiquity of everyday devices connected over the internet, know generally as the Internet of Things (IoT), has opened up new avenues for students to explore their worlds and think and create computationally.
Combining IoT with mobile technologies (such as smartphones), enables students to move their designs and computational thinking out of traditional classroom settings and into the real world. This article outlines a design-based IoT curriculum that connects Taiwanese students with the personally-relevant issue of air pollution. The curriculum employs student-driven smartphone application design, using MIT's App Inventor, with Wi-Fi enabled IoT devices (LinkIt 7697
Wi-Fi/BLE MCU board). This paper reports on changes to the curriculum based on a preliminary pilot and observations of student engagement during the most recent enactment.
Peer Reviewed Paper
L. Tang. Constrained Sets: The Effects of Multi-Layered Environments in Learning App Inventor term paper, Massachusetts Institute of Technology, June 2018
Abstract: MIT App Inventor is a mobile application development platform that seeks to democratize the construction of mobile apps by making app development accessible to people with little to no experience with script-based programming. It uses block-based programming to introduce and teach programming concepts to its users. Users drag and drop functional and visual components onto their planned app in the screen editor, and construct the logic behind those components by using blocks in the block editor. In this thesis, we design and implement Constrained Sets, a system that allows instructors and developers to allow access to only a subset of App Inventor functionality by hiding component and block access. This system allows for the construction of multi-layered interfaces, which we then use to conduct an experiment that explores how novice App Inventor users learn App Inventor in different interface environments. Furthermore, we discuss and test the possibility of using a React based implementation of the App Inventor designer, and what implications that may have on creating more flexible user interfaces.
K. Prutz, Democratizing Cross-Platform Mobile Development with MIT App Inventor, term paper, Massachusetts Institute of Technology, Spring 2018.
Abstract: App Inventor is an intuitive, blocked-based programming environment that allows everyoneeven childrento build fully functional mobile apps for smartphones and tablets. App Inventor has been around for 8 years now, has users in almost every country in the world, and has 6.8 million users across the globe.
App Inventor democratizes mobile development through block-based coding, but because App Inventor targets only the Android operating system, this excludes the 20% of people worldwide who use Apple devices, including schools that are deploying iPads for use in the classroom. In this project, weve extended MIT App Inventor by natively developing components for iOS in Swift, allowing for the development of iOS applications through block-based programming. This provides users with the ability to develop Android and iOS versions of the same application, without any additional coding and without the need for platform-specific development hardware. Upon randomly sampling 1,831,065 existing App Inventor projects, 68.805% of the projects could be developed for both iOS and Android. This solution takes great steps toward democratizing cross-platform mobile development for Android and iOS applications.
Student Term Paper
K. Prutz, H. Abelson, Expanding Device Functionality For the MIT App Inventor IoT Embedded Companion, term paper, Massachusetts Institute of Technology, May 17, 2018.
Abstract: AppInventor allows anyone, even - and especially - children, to create fully functional apps in a quick and trouble-free way. AppInventor+IoT expands this to include communication between the device running the app and remote devices, most notably Arduinos. The next step with AppInventor+IoT is to allow for fully autonomous projects to run on the Arduino. This is what the ?App Inventor IoT Embedded Companion (hereafter Embedded Companion) makes possible. While this was in development, I ?created project tutorials to complement the addition of the EmbeddedCompanion into AppInventor+IoT. I made tutorials for two projects, the Autonomous Healthy Plant and the Puzzle Box, to showcase the power of autonomously-run projects. I also made slight adjustments to the existing how-to documents for setting up the Arduino and setting up a BLE connection between the app and the Arduino. Since the EmbeddedCompanion was in development while I was building the tutorials, I extended device functionality to meet the needs of these two projects: light sensor, humidity and temperature
sensor, moisture sensor, touch sensor, buzzer, and LCD.
Student Term Paper
K. Wang, Creating a Database Log History Page for Cloud Components in App Inventor to Support Software Development Education term paper, Massachusetts Institute of Technology, 2018.
Abstract: CloudDB, an experimental feature of the MIT App Inventor Platform, allows users to send data across several different machines using tag (descriptor) and value (data) pairs. However, currently with CloudDB, users, by deploying several blocks, can only view specific values corresponding to known tags. Since App Inventor, a blocks-based app development interface, emphasizes the usability of its program, in this work I implement a web interface where users can view and search the entirety of their CloudDB data history. The user, for each separate project, can see their tags, the corresponding stored data, and a
history of that data. Detailed user studies were conducted to discover efficacy in teaching shared data concepts using this tool. Coupled with educational workshops designed for increasing computational thinking, this extension of CloudDB helps bolster student comprehension of advanced computing concepts and strengthen their independence in design, development, and debugging processes. This work demonstrates that a comprehensive visual tool appended
to a cloud database system can increase understanding and thinking about a technology so prevalent in many of todays technologies. Student Term Paper
M. Sherman, assisted by O. M. Alkhatib, Technovation Computational Thinking 2018, Massachusetts Institute of Technology, 2018.
Abstract: The Technovation Challenge is the largest technological entrepreneurship program for young
women in the world. The program is run by Iridescent, who has already collected evidence
that the challenge increases the rate of majoring in computer science by the girls in their
program. This is especially important, and women are consistently underrepresented in
Technovation requires a mobile app prototype, and around 80% of apps submitted this year
were built with MIT App Inventor. In this analysis, a team from MIT App Inventor took a closer
look at half of those projects. The MIT team broke down the projects into a number of
features and metrics, provided interpretation for the data patterns that were found, and
provided abstracted data representing these features so that Iridescent may conduct further
This analysis was conducted under an agreement between Iridescent and MIT App Inventor.
The mechanism of data collection, extraction, and analysis follows in the methodology.
Link to Analysis
M. Tissenbaum, J. Sheldon, L. Seop, C.H. Lee, N. Lao, Critical computational empowerment: Engaging youth as shapers of the digital future
Global Engineering Education Conference (EDUCON), 2017 IEEE, 1705-1708, April 2017
Abstract: While there is growing recognition of the need to support young learners as they develop computational thinking (CT) skills, this paper advocates for an increased focus on computational identity (CI) and digital empowerment (DE) in particular, as we posit that these CT skills will help young learners become self-motivated, innovative creators. When nurturing identity and empowerment, it is a persistent challenge to provide youth with real-world experiences and suitable development tools. This paper advances two key approaches to address these issues: 1) A critical computational literacy approach to engage youth in developing personally meaningful applications that have impact in the real world; 2) the use of a tool, like MIT App Inventor, that lowers barriers to creating useful, impactful technology. Using two case-based studies, we show how these two approaches have supported youth in developing applications that respond to meaningful challenges in their communities and helped them to establish their computational identities and digital empowerment.
Peer Reviewed Paper
S. Kong, H. Abelson, J. Sheldon, A. Lao, M. Tissenbaum, M. Lai, Curriculum Activities to Foster Primary School Students Computational Practices in Block-Based Programming Environments The Education University of Hong Kong, Hong Kong, 84 2017, CTE 2017
Abstract: As digital technology is increasingly a part of all sectors of society, educational approaches must be developed in order to nurture students ability to see the world through a computational lens. One way to achieve this goal is to promote Computational Thinking (CT) for young learners. The CoolThink@JC project is a four-year curriculum pilot designed to integrate CT into Hong Kong upper-primary level schools. The CoolThink framework for curriculum development is structured around computational concepts, practices and perspectives adapted from the framework of Brennan and Resnick (2012). This adapted framework motivated the choice of learning activities for CoolThink. This paper focuses on one aspect of that framework, namely computational practices. Here, we describe how activities in the CoolThink curriculum can promote the computational practices highlighted by the framework.
Peer Reviewed Paper
L. L. Ruan, E. W. Patton, and M. Tissenbaum, Evaluations of programming complexity in App Inventor, presented at CTE 2017, Hong Kong, Hong Kong, July 2017.
Abstract: To understand computational thinking in App Inventor, it is important to be able to effectively evaluate computational complexity in block-based programming languages. In the past, there have been a handful of complexity measures proposed for text-based languages (Weyuker, 1988). In this paper, we will attempt to implement 2 such measures, Halsteads Programming Effort and statement count, in App Inventor on a dataset of projects from 50 random users. The goal is to determine whether or not text programming standards for complexity can be generalized to block programming languages. This paper shows that the 2 complexity measures we implemented are not adequate measures for complexity in App Inventor. This result indicates a need for different measures of complexity that more accurately portray block programming proficiency. We hope this study will be a gateway into a better understanding of the intricacies of App Inventors block programming language and its unique contributions to the development of computational thinking.
Peer Reviewed Paper
D. Bau, J. Gray, C. Kelleher, J. Sheldon, and F. Turbak, Learnable programming: Blocks and beyond, Comm. of the ACM, vol. 60, no. 6, pp. 72-80, Jun, 2017.
Abstract: A global push to broaden participation in computer science has led to an explosion of interest in blocks-based programming. Visual blocks are used by numerous programming tools. Millions of students receive their first exposure to programming via these tools in courses and activities like Code.orgs Hour of Code. Blocks allow beginners to compose programs without struggling with the frustrations of syntax (Figure 1).
There is increasing interest in developing and studying blocks languages. At VL/HCC 2015, a small workshop session called Blocks and Beyond ballooned to a large event, with 51 submissions and 36 presenters. Researchers shared work in new blocks languages, interface innovations, domain-specific applications of blocks, and ways to make blocks languages more effective and accessible for diverse coders.
This article explores how blocks impact the learnability of programming. We begin by reviewing studies on the effectiveness of blocks languages. Then we discuss the key features of blocks languages and how they relate to learning. Finally, we look at applications of blocks in new domains and discuss tools for creating your own blocks language.
Peer Reviewed Paper
M. Tissenbaum, J. Sheldon, E. W. Patton, A. Gupta, E. Zhang, and D. Gopinath, Off the screen, and into the world of everyday objects: Computational thinking for youth with the Internet of Things, presented at CTE 2017, Hong Kong, Hong Kong, Jul, 2017.
Abstract: This paper discusses the opportunities presented by the growth of the Internet of Things (IoT) to provide youth opportunities to develop their computational thinking and digital empowerment. This paper argues that to support youth in developing these literacies, we need to develop platforms that reduce the barriers of entry while still allowing them to explore and develop their computational identities. To this end, this paper introduces an extension to App Inventor by MIT that enables students to quickly design, develop, and implement IoT applications. We outline one IoT activity for youth and future directions for both curricular and technical development.
Peer Reviewed Paper
N. Lao, "CloudDB: Components for Exploring Shared Data with MIT App Inventor" ,
Data sharing through Cloud technology is one of the most powerful new computer science concepts of the past few decades. As such, developing powerful and easy-to-use tools for incremental learning and application of shared data concepts is an important endeavor. My work focuses on using MIT App Inventor, a popular blocks-based mobile application development tool for teaching computational thinking to young students, to make shared data technology understandable and usable by anyone without the need for extensive computer science training. I present the ongoing development of CloudDB, a set of coding blocks for MIT App Inventor that allows users to store, retrieve, and share various types of data in tag-value pairs on a Redis server for their mobile applications.
Peer Reviewed Paper
N. Lao, Developing cloud and shared data capabilities to support primary school students in creating mobile applications that affect their communities, M.Eng thesis, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2017.
Abstract: As technology and society become further intertwined, it is imperative that we democratize the creation of technology and educate people to be capable of harnessing the full power of computational thinking. As such, developing meaningful tools and curricula for incremental learning of computational thinking concepts starting in primary education is an important endeavor . My work focuses on making Cloud technology, one of the most powerful new computer science concepts, understandable and usable by anyone without the need for extensive computer science training.
I used MIT App Inventor, a blocks-based mobile application development tool for teaching computational thinking to young students, as the platform for my research. I developed CloudDB, a set of coding blocks for MIT App Inventor that allows users to store, retrieve, and share various types of data in tag-value pairs on a Redis server for their mobile applications. I created middle and high school level curricula based on CloudDB along with assessment tools to evaluate my materials and the extent to which young students can understand and utilize the concepts around shared data. Finally, I ran one of those workshops with middle school students in the MIT area.
My findings indicate that teaching shared data as a core computational thinking concept is entirely feasible to students as young as middle school level. Students are capable of inferring and extrapolating other use cases and potential problems with the Cloud, such as storage limits and security concerns. When given the context of solving a problem in their lives, they are very driven and able to design and create complex independent mobile application projects using MIT App Inventor and CloudDB.
X. Deng, Group collaboration with App Inventor, term paper, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2017.
Abstract: Collaboration becomes increasingly important in programming as projects become more complex. With traditional text-based programming languages, programmers typically use a source code management system to manage the code, merge code from multiple authors, and optionally lock files for conflict-free editing. There is a limited corpus of work around collaborative editing of code in visual programming languages such as block-based programming. I present a collaborative programming environment to MIT App Inventor, a web-based visual platform for building Android applications with blocks, which enables many programmers to collaborate in real time on the same MIT App Inventor project. I design and implement three collaboration models to evaluate the efficacy of these different collaborative models for users of App Inventor so as to understand which approach best enables collaboration. Our results demonstrate that real-time programming decreases the completion time of a task, improves the interaction between users, and increases users likeability towards collaborative programming. I anticipate that this new collaborative programming environment will change the way users use MIT App Inventor, and more curriculum based on the collaborative tools will be designed.
L. Tang, Creating customized user interfaces for App Inventor, term paper, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2017.
Abstract: Many of App Inventors users pick up App Inventor outside of a formal learning environment by going through online walkthroughs and tutorials. By using a block-based programming language, App Inventor democratizes technology by lowering the bar of entry for people around the world. For my thesis, I plan to create and test a feature that allows for the formation of customized component and block sets in App Inventor.
Student Term Paper
X. Deng, E.W. Patton, (2017) Enabling Multi-User Computational Thinking with Collaborative Blocks Programming in MIT App Inventor. In: Proceedings of the 1st International Conference on Computational Thinking in Education (July 13-15, 2017, Hong Kong, China).
Collaboration becomes increasingly important in programming as projects become more complex. With traditional text-based programming languages, programmers typically use a source code management system to manage the code, merge code from multiple editors, and optionally lock files for conflict-free editing. There is a limited corpus of work around collaborative editing of code in visual programming languages such as block-based programming. We propose an extension to MIT App Inventor, a web-based visual platform for
building Android applications with blocks, which will enable many programmers to collaborate in real-time on MIT App Inventor projects. We take the position that real-time collaboration within MIT App Inventor will encourage students in a group environment to interact with one another in ways that help them improve each others understanding and practice of computational thinking practices that may not be achieved in the traditional one user-one project paradigm that is currently provided.
Peer-reviewed Conference Paper
E. W. Patton, X. Deng, (2017). Collaborative Mobile App Development with MIT App Inventor. In: Proceedings of Blocks & Beyond 2017 (Oct. 10, 2017, Raleigh, NC).
Abstract: MIT App Inventor has been used around the world to introduce students to explore computational thinking in the context of designing and building mobile applications. Currently, App Inventor has a limitation of one user per project. This makes group work in an educational setting difficult because it creates an artificial barrier to collaborative work. The rise of cloud-based, realtime systems to collaborate on other types of creative works, such as Google Docs, suggests that there is a gap between the production capabilities of App Inventor and what users might expect from online tools.
In this presentation, we will demonstrate an extension to App Inventor that allows multiple users to simultaneously edit shared projects in both the designer and blocks editors. Actions are synchronized between workspaces in near realtime, allowing for a "Google Docs"-like experience for group work on App Inventor projects. User awareness is enhanced through colored highlights to indicate the presence and focus of other users in the shared workspace. The demonstration will include hands-on use of the realtime system.
G. Campbell, Extending cloud data store component for MIT App Inventor, term paper, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2017.
Abstract: MIT App Inventor required a new cloud storage component to provide cloud sharing capabilities for Android applications built with App Inventor. I helped design and implement such a component, named CloudDB, which allows App Inventor users to store application data and media files on an MIT hosted Redis server. The component initially provided key-value storage and basic list operations. I improved it by adding advanced list operations, image, sound and other media file storage, as well as ensuring that data stored on the server would be transmitted securely to and from user devices via SSL.
Student Term Paper
E. Giurleo, Keyboard interactions in block-based coding environments, term paper, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2017.
Abstract: App Inventor is a website that allows users to make their own Android apps using a block-based coding language. While App Inventor has almost 3 million users, it contains few accessibility features. In particular, there is no way for a user to navigate the block workspace on the App Inventor site without using their mouse; this poses a significant problem for users who use screen readers or who cannot operate a mouse. I have designed and implemented a set of keyboard shortcuts that allow users to operate the App Inventor website in an intuitive way. I have also conducted a usability study to determine the effectiveness of this keyboard-accessible interface. Based on the results of the study, I conclude that much of the interface is intuitive to users with experience using the keyboard in other domains, but that there are changes that can be made to make the design even more intuitive.
Student Term Paper
J. Im, User-defined blocks for MIT App Inventor, term paper, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2017.
Abstract: MIT App Inventor is an open source web platform that lets users build their own Android Apps quickly by using blocks-based programming. MIT App Inventor currently has diverse types of components, from general blocks to specific task oriented components like sensors. However, it does not provide every type of block that an user needs. For example, currently there are no components for intricate graph visualization in App Inventor. However, there is a limitation to keep adding new categories of blocks, not only because the complexity of maintaining the system will increase, but also because an individual users need is necessarily not a general need for others. In order to keep lowering the barriers of creating special goal oriented apps and to foster the learning of computational thinking, while also preventing the situation of adding every possible block to the system, a new feature called user-defined blocks (customizable blocks) has been implemented during 2017 Spring semester.
Student Term Paper
W. Li, Helping the helpers: A Toolkit for mobile humanitarian assistance apps, M.Eng. thesis, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2016.
Abstract: My research investigates the following question:? ?How can relief workers be empowered to create useful mobile apps to support the work of first responder teams? Mobile devices are reshaping the disaster management domain and they make fast and targeted support possible. However there is the major obstacle of relief workers and volunteers often lacking the technical abilities to build and deploy the right mobile app in response to a particular disaster. In addition, data interoperability is often missing in many of the applications since they were developed without any prior agreement on the data schema. Linked Data technology solves the data interoperability problem by defining a method of publishing structured data and these structured data can be interlinked and become more useful. In order to explore and provide a solution to the gap between the knowledge of the relief workers and volunteers and the technical abilities needed to
create an app, I conducted a participatory design workshop with the people at the International Committee of Red Cross and developed two mobile applications with one of the project managers there. In addition, I have created a DoItYourself (DIY) toolkit that includes: (1) a Mobile Linked Data App Kit: a streamlined process for creating mobile apps and (2) an appbuilding methodology: a set of principles for relief workers to follow while creating apps using the toolkit. The goal of this toolkit is to enable aid workers to create vital and necessary mobile apps and to empower the humanitarian community. User testing of the framework with first responders has shown that the idea of using, modifying, and creating an app is greatly favored by our
participants, that mobile Linked Data applications can aid humanitarian organizations by increasing their impact and effectiveness, and that humanitarian workers can successfully develop
and deploy mobile applications by using this toolkit.
A. Colter, Evaluating and improving the usability of MIT App Inventor,
M.Eng. thesis, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2016.
Abstract: MIT App Inventor is a blocks-based programming language for Android apps designed to teach programming skills to middle school and high school students. We aim to make app development accessible for all. Anyone learning to program must learn computational thinking methods; App Inventor users must also learn how to use the service. Our target users, teenagers and people without programming experience, often conflate the two learning processes: they think App Inventor is hard because learning to program is hard. As such, App Inventor needs a
user interface that matches the conventions of commercially- available software our users already know how to use. Such an interface will allow them to focus on learning how to program and to transfer their knowledge and skills to other programming languages and environments.
I designed several tasks and conducted a usability study on the existing, publicly- available App Inventor service. Users encountered 75 unique issues and a total of 157 issues. This is an average of 5 unique issues and 10.5 total issues per user. I made changes to the App Inventor source code that addressed 34 of the most common issues encountered. My intent was to make App Inventor both more usable to novice programmers and more similar to the programming environments that experienced programmers use. Finally, I conducted a usability
study with the same tasks on the modified version of App Inventor. Users encountered 65 unique issues, including 19 issues encountered in the first study, and 107 total issues. Based on user
comments and behavior, I conclude that my solutions resolved 21 of the original issues, partially mitigated 9 issues, and did not improve the usability of 4 issues.
B. Xie, Progression of computational thinking skills demonstrated by App Inventor users, M.Eng thesis, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2016.
Abstract: I analyze skill progression in MIT App Inventor, an open, online learning environment
with over 4.7 million users and 14.9 million projects/apps created. My objective is to
understand how people learn computational thinking concepts while creating mobile
applications with App Inventor. In particular, I am interested in the relationship
between the development of sophistication in using App Inventor functionality and the
development of sophistication in using computational thinking concepts as learners
create more apps. I take steps towards this objective by modeling the demonstrated
sophistication of a user along two dimensions: breadth and depth of capability. Given
a sample of 10,571 random users who have each created at least 20 projects, I analyze
the relationship between demonstrating domain-specific skills by using App Inventor
functionality and generalizable skills by using computational thinking concepts.
I cluster similar users and compare differences in using computational concepts.
My findings indicate a common pattern of expanding breadth of capability by
using new skills over the first 10 projects, then developing depth of capability by
using previously introduced skills to build more sophisticated apps. From analyzing
the clustered users, I order computational concepts by perceived complexity. This
concept complexity measure is relative to how users interact with components. I
also identify differences in learning computational concepts using App Inventor when
compared to learning with a text-based programming language such as Java. In
particular, statements (produce action) and expressions (produce value) are separate
blocks because they have different connections with other blocks in App Inventors
visual programming language. This may result in different perceptions of computational concepts when compared to perceptions from using a text-based programming language, as statements are used more frequently in App Inventor than expressions.
This work has implications to enable future computer science curriculum to better
leverage App Inventors blocks-based programming language and events-based model
to offer more personalized guidance and learning resources to those who learn App
Inventor without an instructor.
S. Tsai, Understanding App Inventor forums, M.Eng. thesis, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2016.
Abstract: MIT App Inventor is a visual blocks language that allows users to create mobile applications for Android. App Inventor users have the option of posting in a public support forum to discuss anything from specific programming issues to education is- sues. In order to leverage the information on the forums to improve App Inventor, we must first understand what is being discussed. In this thesis, we used unsupervised machine learning methods to automate discovery of discussion topics. First, we transformed posts into feature vectors using a bag-of-words model. Next, we clustered posts using k-means clustering and evaluated our results both quantitatively, by calculating the average silhouette of the posts, and qualitatively, by simply looking at the clusters of posts. Finally, we used LDA topic modeling to determine the topics being discussed and compared the extracted topic words to a manual evaluation of each cluster. Using this technique, we were able to uncover common problems with App Inventor that users encountered. We hope to use this information to improve users experience with App Inventor.
K. Asmus, A system of applications for the integration of BLE beacons in museums, M.Eng. thesis, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2016.
Abstract: There is great potential for BLE beacon technology to be incorporated into museum exhibits to provide more enriching content to visitors and gather feedback to improve the museum overall. I developed a system of three applications to accomplish these goals: one for museum staff to easily deploy and update beacons, another for visitors to engage with the supplemental material transmitted by beacons at various exhibits, and a third for museum staff to understand visitors interactions and feedback provided by the beacons.
I implemented two versions of the system according to two different beacon protocols: Eddystone- UID, which broadcasts to a dedicated mobile app; and Eddystone-URL, which broadcasts to mobile browsers via the Physical Web.
Each system was tested in a real-world context through experiments hosted by the MIT Museum. I analyzed the success and potential of such systems based on observations, participant responses, and data gathered via the system during the experiments. The reliability of BLE connections remains an issue, and beacon content only appealed to a small subset of visitors; however, those who interacted with the system valued the experience highly and the MIT Museum expressed interest in providing a richer experience for these visitors.
B. Xie, I. Shabir, and H. Abelson, Measuring the usability and capability of App Inventor to create mobile applications, in PROMOTO 2015, Pittsburgh, PA, 2015, p. 108.
Abstract: MIT App Inventor is a web service that enables users with little to no previous programming experience to create mobile applications using a visual blocks language. We analyze a sample of 5,228 random projects from the corpus of 9.7 million and group projects by functionality. We then use the number of unique blocks in projects as a metric to better understand the usability and realized capability of using App Inventor to implement specific functionalities. We introduce the notion of a usability score and our results indicate that introductory tutorials heavily influence the usability of App Inventor to implement particular functionalities. Our findings suggest that the sequential nature of App Inventor's learning resources results in users realizing only a portion of App Inventor's capabilities and propose improvements to these learning resources that are transferable to other programming environments and tools.
Peer Reviewed Paper
R. Morelli, C. Uche, P. Lake, & L. Baldwin, Analyzing year one of a CS principles project, in 46th ACM Technical Symposium on Computer Science Education, Kansas City, MO, 2015, pp. 368-373.
Abstract: The Mobile Computer Science Principles (Mobile CSP) project is an NSF-funded CS 10K project. Its goal is to train a new cohort of high school computer science teachers to teach an Advanced Placement (AP) level course based on the emerging Computer Science Principles (CSP) framework. Mobile CSP uses App Inventor, a graphical programming language for Android devices, to engage students in app building as
a means to get them interested in computer science. This paper reports on the first year of this effort. In addition to describing the projects Professional Development (PD) course, a 6-week, full-time summer course for teachers, and the Mobile CSP curriculum, the paper provides a preliminary analysis of demographic and performance data obtained from high school students who took the course during the 2013-2014 academic year.
Peer Reviewed Paper
F. Turbak, M. Sherman, F. Martin, D. Wolber, and S. Crawford Pokress, Events-first programming in App Inventor,Journal of Computing Sciences in Colleges, vol. 29, no. 6, Jun, 2014, pp 81-89.
Abstract: Events are a central concept in computer science. They are becoming more important with the prevalence of mobile and web platforms that use event-based programming. Yet, events are typically taught late in the CS curriculum -- e.g., in a web programming or operating systems course. We have introduced events to CS0 students from day one using MIT App Inventor, a blocks-based programming environment that enables students to create apps for Android devices. This paper presents the system's event-based model, along with typical coding problems and best-practice approaches for solving them. We advocate for increasing early emphasis on events in the CS curriculum.
Peer Reviewed Paper
D. Y. Miao, PrivacyInformer: An automated privacy description generator for the MIT App Inventor, M.Eng. thesis, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2014.
Abstract: With the Advent of smart mobile phones and ubiquitous mobile applications, the pace at which people generate, access, and acquire data has accelerated significantly. In this thesis, we first examine how privacy concerns in the mobile apps market compromise the well-being of both app consumers and developers. Then, we study the problem with the lack of usable privacy documents in mobile apps. Subsequently, we propose a technical solution named PrivacyInformer, as an extension to the MIT App Inventor, a do-it-yourself mobile app building platform. This platform has a vast international user base, as well as a growing impact on the democratizing of mobile app building. We show that by analyzing the source code of mobile apps, PrivacyInformer can produce simple and useful privacy descriptions in both human-readable and machine-readable format. Finally, we conduct an exploratory user study to evaluate the effectiveness of PrivacyInformer from the app developer's perspective, as well as discuss the policy impact of such a tool in the mobile app development community.
D. Wolber, H. Abelson, and M. Friedman, Democratizing computing with App Inventor, Mobile Computing and Communications Review, vol. 18, no. 4, pp 53-58, Oct. 2014.
Abstract: MIT App Inventor is a visual blocks language that enables beginners and non-programmers to create apps for their phones and tablets. It has empowered thousands to create software with real-world usefulness, and see themselves as creators rather than only consumers in the mobile computing environment. Educationally, it offers a gateway drug that can help broaden and diversify participation in computing education.
Peer Reviewed Paper
J. Schiller, F. Turbak, M. Friedman, H. Abelson, J. Okerlund, J. Dominguez, and A. McKinney, Live programming of mobile apps in App Inventor, in Programming for Mobile and Touch (PRoMoTo), Portland, OR, 2014, pp. 1-8.
Abstract: MIT App Inventor is a programming environment that lowers the barriers to creating mobile apps for Android devices, especially for people with little or no programming experience. App Inventor apps for a mobile device are constructed by arranging components with a WYSIWYG editor in a computer web browser, where the development computer is connected to the device by WiFi or USB. The behavior of the components is specified using a blocks-based graphical programming language. A key feature in making App Inventor accessible to beginning programmers is live programming: Developers interact directly with the state of the evolving program as it is being constructed, and changes made in the web browser are realized instantaneously in the running app on the device. This paper describes the live programming features of App Inventor and explains how they are implemented.
Peer Reviewed Paper
F. Shih and J. Boortz, Understanding peoples preferences for disclosing contextual information to smartphone apps, in International Conference on Human Aspects of Information Security, Privacy, and Trust (HAS 2013), Las Vegas, NV, 2014, pp 186-196.
Abstract: Smartphones have become the primary and most intimate computing devices that people rely on for their daily tasks. Sensor-based and network technologies have turned smartphones into a context-aware information hub and a vehicle for information exchange. These information provide apps and third party with a wealth of sensitive information to mine and profile user behavior. However, the Orwellian implications created by context-awareness technology have caused uneasiness to people when using smartphone applications and reluctance of using them . To mitigate peoples privacy concerns, previous research suggests giving controls to people on how their information should be collected, accessed and shared. However, deciding who (people or the application) gets to access to what (types of information) could be an unattainable task. In order to develop appropriate applications and privacy policies it is important to understand under what circumstances people are willing to disclose information.
Peer Reviewed Paper
F. Zhang, F. Shih and D. Weitzner, No surprises: Measuring intrusiveness of smartphone applications by detecting objective context deviations,in ACM CCS Workshop on Privacy in the Electronic Society (WPES 2013), Berlin, Germany, 2013, pp 291-296.
Abstract: We address the challenge of improving transparency for smartphone applications by creating tools that assesses privacy risk. Specifically, we invented a framework for qualitatively assessing and quantitatively measuring the intrusiveness of smartphone applications based on their data access behaviors. Our framework has two essential components. The first component is the Privacy Fingerprint, a novel visualization that is concise yet holistic. It captures each app's unique access patterns to sensitive personal data, including which types of behaviors and under which privacy-relevant usage contexts the data are collected. The second component is a new Intrusiveness Score that numerically measures out-of-context data collection, based on real data accesses gathered from empirical testing on 33 popular Android apps across 4 app categories. Specific attention is paid to the proportion of data accesses that occurs while the user is idle, raising the perceived level of intrusiveness and exposing the profiling potential of an app. Together, these components will help smartphone users decide whether to install an app because they will be able to easily and accurately assess the relative intrusiveness of apps. Our study also demonstrates that the Intrusiveness Score is helpful to compare apps that exhibit similar types of data accesses.
Peer Reviewed Paper
A.K. Saigal, Information accountability for mobile financial applications, M.Eng. thesis, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2010.
Abstract: In this thesis, I designed and built three sets of applications for three different demographics - young people, elderly people, and people in the developing world - to enable them to be involved in their personal banking. Members of these demographics are not actively involved in their personal banking when compared to others. We believe that part of the discrepancy lies in the lack of convenience and accountability. Thus, we have developed applications, whereby the issues of convenience and accountability are addressed. The applications are built around mobile devices, which will likely make them accessible since members of these demographics are often on mobile devices. The applications are also built around rules that can be set by a responsible party, so that the users know exactly what can and cannot be done with their money (such as a father restricting the amount of money his son can withdraw, or what the money can be used for). Finally, we keep a history of every transaction, who initiated it, and an explanation given by the initiator so we can understand why it occurred. Using our applications, built around convenience and accountability, will allow banks to reach youth, elderly people, and people in the developing world in ways that they have not been able to previously.
F. Shih, O. Seneviratne, D. Miao, I. Liccardi, L. Kagal, E. W. Patton, P. Meier, and C. Castillo, Democratizing mobile app development for disaster management, in AIIP Workshop on Artificial Intelligence Problems and Approaches for Intelligent Environments and Workshop on Semantic Cities, Beijing, China, 2013, pp 39-42.
Abstract: Smartphones are being used for a wide range of activities including messaging, social networking, calendar and contact management as well as loca- tion and context-aware applications. The ubiquity of handheld computing technology has been found to be especially useful in disaster management and relief operations. Our focus is to enable developers to quickly deploy applications that take advantage of key sources that are fundamental for todays net- worked citizens, including Twitter feeds, Facebook posts, current news releases, and government data. These applications will also have the capability of empowering citizens involved in crisis situations to contribute via crowdsourcing, and to communicate up-to-date information to others. We will leverage several technologies to develop this application framework, namely (i) Linked Data principles for structured data, (ii) existing data sources and ontologies for disaster management, and (iii) App Inventor, which is a mobile application development framework for non-programmers. In this paper, we describe our motivating use cases, our architecture, and our prototype implementation.
Peer Reviewed Paper
R. Morelli, D. Wolber, S. Pokress, F. Turbak, and F. Martin, Teaching the CS principles curriculum with App Inventor (abstract only), in Special Interest Group on Computer Science Education 44th Association for Computing Machinery Technical Symposium on Computer science education, Denver, CO, 2013, pp. 762-762.
Abstract: The CS Principles Project is an NSF-funded initiative to develop a breadth-first advanced placement (AP) course in computer science. App Inventor is a visual, blocks-based programming language that makes sophisticated computing concepts accessible to a broad range of students. This hands-on workshop, aimed at high school and undergraduate teachers, will introduce participants to lessons, homework exercises, project assignments, and assessment materials (quizzes, grading rubrics) that can be used in an App Inventor-based CS0 course. Participants will develop simple Android apps, using devices provided by the workshop, and will use them in the context of lessons and assignments that fit within the CS Principles framework.
Peer Reviewed Paper
MS. Crawford Pokress and J. J. Dominguez Veiga, MIT App Inventor: Enabling personal mobile computing, presented at Programming for Mobile and Touch (PRoMoTo), Indianapolis, IN, 2013.
Abstract: MIT App Inventor is a drag-and-drop visual programming tool for designing and building fully functional mobile apps for Android. App Inventor promotes a new era of personal mobile computing in which people are empowered to design, create, and use personally meaningful mobile technology solutions for their daily lives, in endlessly unique situations. App Inventor's intuitive programming metaphor and incremental development capabilities allow the developer to focus on the logic for programming an app rather than the syntax of the coding language, fostering digital literacy for all. Since it was moved from Google to MIT, a number of improvements have been added, and research projects are underway.
Peer Reviewed Paper
F. Shih and W. Li, Unlocking smartphone data for educational use in teaching and learning environment, presented at the Smart University Workshop in Eighth International and Interdisciplinary Conf. on Modeling and Using Context, Annecy, France, 2013.
Abstract: We address the challenges of applying smartphone data to improve the experiences of teaching and learning in the university environment. Specifically, we created a framework to collect, monitor and analyze student behavioral data and increase
social interactions for educational uses. The framework has three essential components. The first component is App Inventor, a web-based platform with drag-and-drop tools for novices to create mobile apps that collect personal and environmental data. The second component is Reactive Data Store, an intuitive mechanism that connects multiple data sources and automates the tasks for analyzing personal data. The third component is a push notification mechanism that delivers personalized or group messages as
the results of the real-time analysis and triggers more actions accordingly. Together, these components will transform people's campus experiences by unleashing the values of smartphone data. Lastly, we introduced the framework to a group of teachers in
higher education with a focus group discussion to identify the plausible use cases in terms of performance assessments, pedagogy design and collaborative learning. Peer Reviewed Paper
Hsu, Y. -C., & Ching, Y. -H. (2013). Mobile app design for teaching and learning: Educators experiences in an online graduate course. The International Review of Research in Open and Distance Learning, 14(4), 117-139.
This research explored how educators with limited programming experiences learned to design mobile apps through peer support and instructor guidance. Educators were positive about the sense of community in this online course. They also considered App Inventor a great web-based visual programming tool for developing useful and fully functioning mobile apps. They had great sense of empowerment through developing unique apps by using App Inventor. They felt their own design work and creative problem solving were inspired by the customized mobile apps shared by peers. The learning activities, including sharing customized apps, providing peer feedback, composing design proposals, and keeping design journals (blogging), complemented each other to support a positive sense of community and form a strong virtual community of learning mobile app design. This study helped reveal the educational value of mobile app design activities and the web-based visual programming tool, and the possibility of teaching/learning mobile app design online. The findings can also encourage educators to explore and experiment on the potential of incorporating these design learning activities in their respective settings, and to develop mobile apps for their diverse needs in teaching and learning. Peer Reviewed Paper
A. Bhagi, Android game development with App Inventor, M.Eng. thesis, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2012.
Abstract: Ever since App Inventor found a new home at MIT's Center for Mobile Learning in early 2012 (it was previously housed at Google), it has been receiving significant attention from educational communities that hope to use the tool to facilitate technology learning in the classroom and beyond.
Much of App Inventor's current user base is teachers in high-schools and colleges, and as the tool continues to attract more users, it is likely that many of these users will be teachers of students in grades 3 thru 12 (i.e. students who are old enough to know how to use computers and design/develop apps, yet not experienced enough to get their hands dirty with the Android SDK). Students of this age group are quite fond of gaming, and so we anticipate that there will be a strong demand for developing graphically appealing, interactive single-player and multiplayer games using App Inventor in the coming years. It is therefore important that the App Inventor team at MIT prepare for the imminent growth of the App Inventor game development community. Accordingly, this thesis looks at where App Inventor currently stands with respect to game development and how its game development capabilities can be improved and extended. Accepted Thesis
F. Turbak, S. Sandu, O. Kotsopoulos, E. Erdman, E. Davis and K. Chadha, Blocks languages for creating tangible artifacts, in Institute of Electrical and Electronics Engineers Symposium on Visual Languages and Human-Centric Computing, Innsbruck, Tyrol, Austria, 2012, pp 137-144.
Abstract: Logo turtles and Hendersons picture language have long been used to teach computational thinking by inspiring learners to construct programs that
create complex geometric designs. We have developed visual blocks-based versions of these languages, TurtleBlocks and PictureBlocks, that allow users to transform
their designs into tangible artifacts produced by laser cutters and vinyl cutters. Our languages embody two novel features. First, they use constructive area geometry to
convert the geometric designs generated by our programs into formats suitable for laser and vinyl cutters. Second, they leverage static typing and polymorphism to provide
a new way to reference the names of procedure parameters and local variables in a blocks language. Peer Reviewed Paper
Lauren E. Margulieux, Mark Guzdial, and Richard Catrambone, Subgoal-labeled instructional material improves performance and transfer in learning to develop mobile applications, Proceedings of ICER 2012, p 71-78. Mental models are mental representations of how an action changes a problem state. Creating a mental model early in the learning process is a strong predictor of success in computer science classes. One major problem in computer science education, however, is that novices have difficulty creating mental models perhaps because of the cognitive overload caused by traditional teaching methods. The present study employed subgoal-labeled instructional materials to promote the creation of mental models when teaching novices to program in Android App Inventor. Utilizing this and other well-established educational tools, such as scaffolding, to reduce cognitive load in computer science education improved the performance of participants on novel tasks when learning to develop mobile applications. Peer Reviewed Paper
A.K. Saigal and A. Saigal, Saathimobile and the rapid deployment of prototypes to build applications for social enterprise in the developing world, in 1st International Conference on Wireless Technologies for Humanitarian Relief, Amritapuri, Kollam, Keralla, India, 2011, pp. 351-356.
Abstract:This paper describes a sales, marketing, and inventory management application, SaathiMobile, for the distribution of low-cost sanitary napkins, made out of waste banana fiber,
to women in rural areas. It is being pilot tested in Hubli, Karnataka. This product has the potential to have great impact on the health and economic welfare of women in these areas.
The product was developed using App Inventor, a rapid prototype deployment environment for Android. Peer Reviewed Paper
B. Magnuson, "Building blocks for mobile games: A multiplayer framework for App Inventor for Android," M.Eng. thesis, Elect. Eng. Comput. Sci., Massachusetts Inst. of Technol., Cambridge, 2010.
Abstract: Building Blocks for Mobile Games is a client-server multiplayer game building framework for the App Inventor for Android platform. The framework includes an App Inventor component and a game server running on Google App Engine. The client side component packages the complexity of web service calls, data transfer and game state management into a set of graphical code blocks that allow users without programming experience to create Android applications that can access the game server API. The default game server provides basic functionality that can be used to create simple multiplayer games and message passing applications, such as a multi user bulletin board. By using custom server modules, the game server can be enhanced to provide server commands that implement a wide range of operations. Custom modules were used with the system to develop a multi- player card game, a version of Bulls and Cows with a shared scoreboard, an application that accesses Amazons book search API and a pair of applications for creating, managing and voting in polls. Using App Inventor and the game framework, each of these applications can be created with only a basic understanding of mobile phones and client-server principles. Accepted Thesis