Learning computational concepts in Augmented Reality with Anna Fusté, MIT Medialab at the Art-AI Festival

Teaching Computational Thinking and Machine Learning using Augmented Reality lets children understand concepts drawn from their physical surroundings, from their most immediate and tangible reality. This is what I am exploring at the MIT Medialab where I work as a Research Assistant at the Living Mobile Group. At the Leicester Art-AI Festival, I have lead three workshops for 10 year-old kids, on my recently developed applications Paper Cubes (an AR Experiment developed along with Judith Amores and the Google Creative Lab) and HyperCubes, my current research project.

We are still far from exploiting the full potential of Augmented Reality. The majority of applications being released don’t make use of the spatial awareness and the connectivity to devices and objects in our physical surroundings. As Baron Webster explained, a lot of these ’superimposing demos’ get limited to a kind of ‘Additive AR’ placing very cinematic content in front of the user. But the richness of our physical environment has so much more to offer for developing useful and very promising Augmented Reality applications.

Paper Cubes is one of the prototypes we developed for the Google AR Experiments platform this past summer. The main idea is to have a DIY Augmented Reality kit. With just some paper, scissors and glue the user can make their own paper cubes that will serve as basic commands for the AR content. Each cube has a behavior attached to it and the user can place them in space to control a bunch of stick figures walking, in a very ‘Lemmings style’. There are basic commands such as ‘jump’, ’turn’ or ’stop’ and then there is an ‘AI Cube’ that makes these stick figures smarter over time. The kid playing with the application can visualize the evolution of these stick figures and start understanding complex computational concepts such as the effects of a Neural Network over time.

In HyperCubes the user has more freedom of creation with much more commands and digital content to play with. HyperCubes incorporates tracking of surfaces along with the basic tracking of the cubes. The kid can place the cubes everywhere and start building flowing systems in space, in a similar way as traditional visual programming interfaces. Combining a meaningful use of space and tangible objects in teaching resources are key strategies that help children learn complex concepts, especially in STEM education. The Logo Turtle is the cornerstone example for this. Now we have the possibility of expanding it in 3D space with digital animated content. Children become creators and learn while tinkering with commands such as transformations in space or Recurrent Neural Networks that control the little characters. Infinite opportunities open up where the children can be an artist, a coder and just a kid having fun and learning at the same time.

During the Arts-AI Festival in Leicester, students from Sandfield Close had the chance to mount their own cubes and play with both applications. The workshops served as a kickoff to a bunch of other activities developed during the festival. Both students and teachers had a great time and the feedback gathered was very positive.

WORKSHOP FEEDBACK: 

‘The workshop was amazing. I loved the black cube [AI Cube] the most’ – Ashish, 10 years old

‘The workshop was really good, we learnt new things about Paper Cubes and to know they get very, very clever’ – Amaar Arshad, 10 years old

Interviewer: Whats different about using computers this way?

‘This one is much easier for children cause computers they normally have loads of buttons and apps. With this all you need to do is tap it and then you get all this animations which is quite cool’. – Dylan, 10 years old

Interviewer: Is it more like a game than programming?

“It’s sort of both really. You get sort of addicted to it, like my teacher did” – Dylan, 10 years old

Mrs Eammes [teacher]: ‘The program that they are using builds upon the knowledge of what it is learnt in the first cube and you can put different cubes together and come up with different outcomes so children is communicating with each other: “have you tried this? Have you found a sheep? How did you do this?” so it’s a lot more interactive than I first thought. I thought it might just be the children sitting there with an iPad or a phone and it would be a very insular thing. Whereas actually them talking and communicating with each other it’s been really good to see.’

 

Anna’s workshop was part of the Art-AI Festival and involved children from Leicester’s Sandfield Close Primary School.  All images included in this post are copyright of Art-AI Festival, Anna Fuste and Sandfield Close Primary School.

You can follow Anna’s work and projects on Twitter @AFuste or her website at http://annafuste.com/

 

Paper Cubes

Augmented reality (AR) is an explanatory medium that takes advantage of the physical space, objects and surfaces. It gives us the opportunity to create interfaces to better understand what AI does in a visual and spatial manner.

Paper Cubes [1] and HyperCubes were created to teach basic computational skills as well as more advanced programming concepts such as Artificial Intelligence (AI) and Machine Learning (ML) using AR in a more visual and engaging way. This workshop is intended to teach computational thinking skills and machine learning concepts to children using these DIY Augmented Reality tools. Paper Cubes and HyperCubes are two AR applications that help children in understanding complex behavioral concepts by building, creating and having fun. With just some paper, scissors, glue, and a phone; they can explore how to make the physical world augment the digital one. Basic behaviors involve simple commands to control a character with the physical cubes such as turn, split, jump, etc. More advanced behaviors teach ML concepts such as having a Neural Network to make the characters smarter over time using an evolution strategy with the AI paper cube. Children between 7-12 years old will be able to build their own cube patterns. Once they have the paper patterns, they can play with the AR applications using the cubes they built, to control the digital content in space and learn about the different behaviors and algorithms. The workshop objectives are:

  • Engage in a DIY origami activity making physical cubes with paper.
  • Learn basic computational concepts used for coding such as basic control commands (turn, jump, split, etc) making use of space in Augmented Reality.
  • Learn more advanced concepts such as Machine Learning and Artificial Intelligence Algorithms visualizing RNNs making use of space in Augmented Reality.
  • Build, create and have fun!

[1] Fusté, Anna, Amores, Judith & Google Creative Lab. (2017). Paper Cubes Google AR Experiments.

More about the artist here