Creating Impact in Education through Mixed Reality

Collaboration with Microsoft Design Expo 2017


Hannah Rosenfeld, Leah Jiang, Manjari Sahu, Manya Krishnaswamy

16 weeks – Microsoft Design Expo and Carnegie Mellon University, 2017

Instructor: Bruce Hannington and Peter Scupelli


Design Research and Analyses, Concept Ideation, Augmented Reality Prototyping, Service Flow, Wireframes, Motion Graphics, Videography & Editing, Workshop Facilitation.

Tools & Processes

Adobe Creative Suite, Sketch, Invision, Hololens, Cinema 4D, paper.io,

Interview, Surveys, Territory mapping, Collaborative Workshops, Design Jam, Storyboards, A/B testing,


How might Mixed Reality transform automotive training to empower the next generation of technicians?

Dash - Driving Automotive Technical Education

Dash, is an educational platform that integrates contextually relevant information into Mixed Reality environments to promote critical problem solving and data-driven learning in automotive repair education.

The Dash system is composed of three main parts; the Augmented Reality safety glasses, a tablet backend to customize courses and AR settings and an internet-enabled scan tool that senses real car data and makes it available via Dash’s AR system.

Mapping the Territory

Being interested in the education sector, we thought Mixed Reality was particularly suited for embodied learning. Our initial research led us to automotive technical training, a field that exists in the physical boundaries of learning as well as technological advancement and know-how.

Using territory mapping as an exercise, we explored the various stakeholders, education platforms, challenges and scope for mixed reality within this field.

Exploratory Research

Being a 5-month long project, we aimed to work with as many stakeholders working in the automotive field, conducting interviews, workshops, surveys, and research. We spoke to car hobbyists, experts in the IoT and AR technology, automotive and aviation repair experts, educators and technical students to understand the scope of the work and identify key challenges.

Exploring with Co-creation Workshops

Working with students and instructors at Rosedale Technical College (a vocational school in Pittsburgh), we ran multiple research and co-creation workshops. These workshops were a vital part of our design process that led us to immerse ourselves in the field and gain a deeper understanding of the complexities of automotive training.

In one generative workshop, we asked students to envision and build tools of the future that would make them better technicians. Each student was asked to think of a specific scenario and challenge they faced and imagine a tool they wish they had to help them work through it.

The insights we learned through their models, storyboards, and scenarios helped us create our design principles.

Identifying Pain Points


Navigating Complexity

Automotive repair is complex. Not only do technicians have to juggle information from multiple sources but they also had to be immersed in the multi-sensory experience of car repair, involving all the smells, sounds and tactile feedback from a car.


Staying Relevant

In the face of rapid advances in technology and the automation of many tasks, students were worried about staying relevant and developing skills that couldn’t be computerized.


Technological Advancement

New tech in cars and in tools used in the workshop meant that students and instructors were constantly pushed outside their comfort zone.

Design Principles

Ideation and storyboarding

From our initial brainstorming, we narrowed down onto the following categories for concept development.

1. Dynamic learning environments that supported embodied learning
2. Remote support and mentorship to enable collaborative real-world practice and
complement in-classroom learning
3. Smart tools that provided real-time feedback to students
4. Administrative support to automate certain tasks

We created multiple storyboards under each category and held a speed-dating workshop with instructors and students at Rosedale, to understand which concept features were the most valuable in the workshop.

Tablet Prototyping and Testing

Dash Tablet Platform Prototyping

For the Dash tablet platform flow, we rapidly prototyped using paper and created an interactive paper wireframe on paper.io.  We aimed to gain insight into the UX flow, content, navigation and information architecture from the instructors at Rosedale.

“I love that I can capture the small learnable moments and share them with my students”

-Rosedale Technical Instructor

Tablet Dashboard UI

The dashboard is designed to give instructors greater visibility of their students and foster better collaboration with them. View-sharing allows instructors and students to share what they see and make annotations in real time.

A real-time work log uses sensor data from the car to track students’ work flow. Instructors can review students work and provide personalized and granular, feedback. Such features are especially useful when dealing with large class sizes.

Student's AR Prototyping and Testing

Video Prototyping for AR Interactions

For most of our target audience, AR and MR were abstract concepts that were unfamiliar. Through a series of short videos to simulate what interactions with a system in AR might feel like, we learnt about students’ and instructors’ needs and interaction preferences.

Student testing with Paper and HoloLens Prototype

We divided our user groups into two and tested a HoloLens UI Prototype with one, and a paper prototype with another. Our paper prototype included a clear acrylic sheet with printed UI elements and a pair of safety glasses with a laser pointer attached to it (to mimic gaze). The low-fidelity version of “augmented reality” allowed us to co-design the dashboard with students by enabling the flexibility needed to move elements around quickly as students walked through the features and offered suggestions for improvement.

AR Dashboard Design

Dash brings all the information they need for car repair in one location by creating a hybrid physical-digital workspace.  This includes car schematics, real-time sensor data, vehicle specifications and course material.

The goal is to help students focus on problem-solving and making sense of information from the car. The AR menu is, therefore, designed for easy navigation to resources and information students need in the workshop. Students can customize their digital dashboard to ensure frequently accessed information is always kept in view.