Quo Vadis Mixed Reality Evaluation?


On 19th July 2021, about 25 participants from different lines of Mixed Reality (MR) research got together to present and discuss their work and to get inspired by the work of others. The occasion was the one-day workshop “Beyond Questionnaires: Innovative Approaches to Evaluating Mixed Reality”, which was part of the 33rd British Human-Computer Interaction (HCI) conference. The goal of this workshop was to explore which evaluation methods are available for MR applications, in particular, to determine the usability and user experience, and how those methods could be extended and expanded beyond the omnipresent questionnaires. Shortcomings of questionnaires, which motivated this goal, were discussed; for example, the limitations in expressiveness, flexibility, and modality of feedback that questionnaires support, and the indirect influence of the end user's comments with an interpreting researcher in the middle.

In the first keynote, Prof. Mark Billinghurst (University of South Australia) presented his experience and outlook on how we can move beyond questionnaires to evaluate MR experiences. This ranged from his first published experiment in 1995 that explored how sketching maps can be used in the evaluation of virtual environments to future directions of research that looks into data analytics, sensors, etc.

In the second keynote, Prof. Eleni Mangina (University College Dublin) presented the ARETE project, with a focus on its evaluation activities. The aim of the ARETE project is to bring innovative AR applications to classrooms all over Europe. The main evaluations will be performed in the form of three pilot studies, for AR applications that foster English literacy skills, teach STEM topics, and to support school-wide positive behaviour intervention and support, respectively.

The keynotes were followed by six paper presentations, covering a wide variety of MR applications and their evaluation, including a learning tool that brings digital skeletons into the homes of veterinary students, a design tool for improving collaborating with robots, and an evaluating tool for trust in holographic Artificial Intelligence. A tool to support end-users in designing MR interfaces and content was also presented, as well as a mapping space to identify questionnaires or other evaluation instruments for MR applications, and Trace Plots that allow teachers to analyse student activities in virtual environments and improve their teaching accordingly.

After the lunch break, two interactive discussion sessions were conducted. For the first one the workshop participants were divided into four groups to discuss the aspects of the future of MR based on the following questions:

  • Are MR Glasses next-generation smartphones? Why?
  • How should MR technology be enhanced to make it become the future mainstream digital communication device?
  • What are possible use scenarios?
  • What innovative methods and tools need to be developed to evaluate such MR technology?

After the lively group discussion, a delegate of each group summarised their discussion points to all workshop participants. While each group thought differently in terms of MR enhancements, the consensus seemed to be that the MR Glasses are the devices of the future; nevertheless, they are unlikely to replace smartphones any time soon since MR technology requires multiple improvements, ranging from technological advancements (e.g., miniaturization, improved interaction) to social solutions (e.g., privacy, trust, acceptance) as well as cheaper solutions to be suitable for the mass market. In terms of evaluations, existing tools and sensors (e.g., EEG, GSR, EMG, heart rate) require further improvement in terms of accuracy and precision, and innovative solutions are needed to remove noise and interpret the data correctly.

In the second interactive discussion section, the participants were asked to identify grand challenges of MR evaluation methodology, based on their experience and research agendas. The discussion was then compiled into the following diagram:

To conclude the workshop, future steps (like publishing a collaboration paper, run another workshop, etc.) were discussed.

Overall, the workshop was considered a success with exchanges of ideas, stimulating future work on this burgeoning topic. Last but not least, we would like to thank the keynote speakers, authors of the papers, attendees, and student volunteers; the workshop would not have succeeded without them.