Beyond Questionnaires: Innovative Approaches to Evaluating Mixed Reality
Workshop in the 33rd British Human Computer Interaction (HCI) Conference
Date: 19th July 2021 (Monday)
Venue: Online
Important dates:
- 11 June 2021: Submission deadline
- 22 June 2021: Author notification
- 29 June 2021: Camera-ready version
Motivation
Mixed Reality (MR) technologies, including augmented reality (AR) and virtual reality (VR), are increasingly used in a number of sectors, thanks to their capabilities to immerse users in multisensory interaction environments. However, as indicated by some systematic reviews, questionnaire remains the main method for evaluating the interaction quality of MR. While it is imperative to recognize the usefulness of questionnaires for evaluating the subjective perception of interaction quality, the lack of innovative methods for addressing the unique features of MR technology calls for more research effort. Indeed, one distinct characteristic of MR is the use of head-mounted display (HMD), given its abilities to recognise location and motivation, support collaboration, investigate gaze-based attention, detect emotions, etc. Innovative methods are needed to evaluate such emerging interaction devices. Otherwise, the advances of MR would be dampened as evaluation insights can inform its future development. In this workshop we aim to explore this issue by inviting participants to share their practical experiences or conceptual ideas of evaluating MR in various contexts, using different methods and tools. The ultimate aim is to produce a research agenda on this topic for the community to examine it further in the future.
Aim and Objectives
The main aim of the workshop is to explore the key questions: whether and what MR-specific usability and UX evaluation methods are available, how these methods are applied, which factors facilitate or hinder their wider adoption.
The aim informs three objectives:
- To gather practical cases as well as conceptual ideas of applying different evaluation methods and tools for MR technologies in various contexts.
- To identify strengths and weaknesses of such methods and tools and challenges of applying them.
- To develop a research agenda for developing innovative evaluation methods for MR technologies.
Note that while we encourage submissions on HMD-based applications, other MR technologies (e.g. AR markers) being evaluated with new or traditional approaches are much welcome. We aim to analyse and discuss a range of research and practice.
Submission
Two types of papers on applying methods and tools for evaluating MR technologies will be solicited. Authors are required to use the eWiC template. Full papers (4 to 10 pages excluding references) can present theories, literature reviews or full-fledged empirical studies. Position papers (2 to 4 pages excluding references) are work-in-progress.
Authors are expected to address the following questions in their paper:
- What MR technology is used in which application domain?
- What is the evaluation method used (or conceptualized)?
- How is the evaluation method applied? What challenges are encountered and how are they overcome?
- What are the strengths and weaknesses of the evaluation method?
- How can the evaluation method be enhanced?
Each submission will be peer-reviewed by at least two workshop organisers and/or invited external reviewers. Accepted Full Papers will be published in the conference proceedings. Authors of the accepted papers, full and position, will be invited to present in the workshop.
Authors should submit their paper via email to arete.ule@gmail.com by 11 June 2021. Enquiries about the workshop can also be sent to this email address.
Accepted papers:
- Augmented Reality for Veterinary self-learning during the pandemic: a holistic study protocol for a remote, randomised, cross-over study; Xu, X., Pan, X., Mangina, E., Kilroy, D., Kumar, A. & Campbell, A.G.
- How To Identify Questionnaires For Mixed Reality Applications; Saeghe, P.
- Virtual Reality, a tool for safe testing of user experience in collaborative robotics; Branco, D. Silva, P.A., Almeida, J., Menezes,P., Bermúdez I Badia, S., Pilacinski, A.
- A new metric scale for measuring trust towards holographic intelligent agent; Huang, X., & Wild, F.
- Beyond Paper: PDart - Participatory Design Augmented Reality tool; Heintz, M., &
Law, E. L-C. - Data Visualization for Asynchronous VR Classroom; Thanyadit, S., Heintz, M., & Law, E. L-C.
Workflow (tentative)
It is planned to be one-day online workshop with the following timetable.
|
Time (UK) |
Activity |
|
9:00 – 9:15 |
Welcome & Introduction |
|
9:15 – 10:00 |
Keynote: Prof. Eleni Mangina, University College Dublin, Ireland |
|
10:00 – 11:00 |
Paper presentations Session 1 |
|
11:00 – 11:15 |
Break |
|
11:15 – 12:15 |
Paper presentations Session 2 |
|
12:15 – 13:00 |
Lunch |
|
13:00 – 14:15 |
Group work Session 1 |
|
14:15 – 14:30 |
Break |
|
14:30 – 15:45 |
Group work Session 2 |
|
15:45 – 16:00 |
Wrap up and Closing |
Organisers
- Effie Lai-Chong Law, University of Leicester, UK
- Santawat Thanyadit, Hong Kong University of Science and Technology, China
- Matthias Heintz, University of Leicester, UK
- Abey Campbell, University College Dublin, Ireland
- Fridolin Wild, Open University, UK
Registration
To be defined
