Beyond Questionnaires: Innovative Approaches to Evaluating Mixed Reality

 

Workshop in the 33rd British Human Computer Interaction (HCI) Conference

Date: 19th July 2021 (Monday)

Venue: Online

 

Important dates:

  • 4 June 2021: Submission deadline
  • 22 June 2021:  Author notification
  • 29 June 2021: Camera-ready version

 

Motivation

Mixed Reality (MR) technologies, including augmented reality (AR) and virtual reality (VR), are increasingly used in a number of sectors, thanks to their capabilities to immerse users in multisensory interaction environments. However, as indicated by some systematic reviews, questionnaire remains the main method for evaluating the interaction quality of MR. While it is imperative to recognize the usefulness of questionnaires for evaluating the subjective perception of interaction quality, the lack of innovative methods for addressing the unique features of MR technology calls for more research effort. Indeed, one distinct characteristic of MR is the use of head-mounted display (HMD), given its abilities to recognise location and motivation, support collaboration, investigate gaze-based attention, detect emotions, etc. Innovative methods are needed to evaluate such emerging interaction devices. Otherwise, the advances of MR would be dampened as evaluation insights can inform its future development.  In this workshop we aim to explore this issue by inviting participants to share their practical experiences or conceptual ideas of evaluating MR in various contexts, using different methods and tools. The ultimate aim is to produce a research agenda on this topic for the community to examine it further in the future.

 

Aim and Objectives

The main aim of the workshop is to explore the key questions: whether and what MR-specific usability and UX evaluation methods are available, how these methods are applied, which factors facilitate or hinder their wider adoption

The aim informs three objectives:

  • To gather practical cases as well as conceptual ideas of applying different evaluation methods and tools for MR technologies in various contexts.
  • To identify strengths and weaknesses of such methods and tools and challenges of applying them.
  • To develop a research agenda for developing innovative evaluation methods for MR technologies.

Note that while we encourage submissions on HMD-based applications, other MR technologies (e.g. AR markers) being evaluated with new or traditional approaches are much welcome. We aim to analyse and discuss a range of research and practice.

Submission

Two types of papers on applying methods and tools for evaluating MR technologies will be solicited.  Authors are required to use the eWiC templateFull papers (4 to 10 pages excluding references) can present theories, literature reviews or full-fledged empirical studies. Position papers (2 to 4 pages excluding references) are work-in-progress.

Authors are expected to address the following questions in their paper:

  • What MR technology is used in which application domain?
  • What is the evaluation method used (or conceptualized)?
  • How is the evaluation method applied? What challenges are encountered and how are they overcome?
  • What are the strengths and weaknesses of the evaluation method?
  • How can the evaluation method be enhanced?

Each submission will be peer-reviewed by at least two workshop organisers and/or invited external reviewers. Accepted Full Papers will be published in the conference proceedings.  Authors of the accepted papers, full and position, will be invited to present in the workshop.

Authors should submit their paper via email to arete.ule@gmail.com by 4 June 2021.  Enquiries about the workshop can also be sent to this email address.

Workflow (tentative)

It is planned to be one-day online workshop with the following timetable.

Time (UK)

  Activity

9:00 – 9:15

  Welcome & Introduction

9:15 – 10:00

  Keynote: Prof. Eleni Mangina, University College Dublin, Ireland

10:00 – 11:00

  Paper presentations Session 1

11:00 – 11:15

  Break

11:15 – 12:15

  Paper presentations Session 2

12:15 – 13:00

  Lunch

13:00 – 14:15

  Group work Session 1

14:15 – 14:30

  Break

14:30 – 15:45

  Group work Session 2

15:45 – 16:00

  Wrap up and Closing

 

Organisers

  • Effie Lai-Chong Law, University of Leicester, UK
  • Santawat Thanyadit, Hong Kong University of Science and Technology, China
  • Matthias Heintz, University of Leicester, UK
  • Abey Campbell, University College Dublin, Ireland
  • Fridolin Wild, Open University, UK

Registration

To be defined