Collaborative AR learning experiences using Orkestra library
Although there are some experiences that demonstrate the validity of the use of Augmented Reality in schools to help students understand and retain complex concepts, Augmented Reality has not been widely adopted in the education sector yet. This is partially because it is hard to use Augmented Reality applications in collaborative learning scenarios and to integrate them in the existing school curricula.
Vicomtech, as a technological centre specialised in Artificial Intelligence, Visual Computing and Interaction, has provided a set of tools that allow to address the previous gap, ensuring the communication and interaction between remote or in-person students and teachers.
Therefore, in the context of ARETE, the real time communication between multiple users sharing an augmented space has been addressed through the Orkestra library. Orkestra is a technology agnostic library, composed of a client side called OrkestraLib, where most of the logic that allows multi-user communication and synchronisation is implemented, and a server side called OrkestraServer, which manages all the connections, messages and sessions of the users.
More in detail, the features that Orkestra provides could be summarised as follows:
- Session generation: it allows to create different isolated sessions though unique session and user IDs so that students can work in pairs without interfering with the other groups.
- Real time communication: it is possible to send messages between users in real time.
- Synchronisation: it allows the synchronisation of each activity between users and devices.
- Simple interactions: it provides the possibility to implement interactions that require a single message sharing to show the same behavior on both screens.
- Advanced interactions: it provides the possibility to implement interactions that require a lot of messages per second to track the movement of 3D objects and obtain a smooth and coherent interaction in both screens.
- Connection control: it allows to check the connection of users in the same session during the game so that in case one user is disconnected the other can not continue playing alone.
Although the library was initially developed just for web applications, the nature of the applications generated in ARETE required porting the client side to CSharp. An initial attempt of integration of the library was carried out in Pilot 2 - Geography and Geometry, but the library was still in a very experimental state, so it was decided to postpone its use and improve it to be able to integrate it in Pilot 3 - Positive Behaviour Intervention & Support.
Pilot 3 started in November 2022 and the Practice Section of the PBIS AR app includes some collaborative learning activities that work on top of Orkestra:
- Greeting others: This is a role-play game in which students learn how to greet the other through quizzes and animations (See Figure 1 B).
- Stand up for others: This is a similar game in which students learn how to behave when they witness a bullying scene involving one of their colleagues (See Figure 1 C).
- Keep the workspace organised (Untidy version): In this activity, students learn how to organise the workspace interacting with virtual objects in an untidy drawer that they see after scanning a real surface such as a table (See Figure 1 D).
- Keep the workspace organised (Tidy version): This is a similar activity to the previous one but the lesson is addressed from the opposite point of view, i.e. using a tidy drawer.
Figure 1: Screenshots of the multi-user activities in the PBIS AR app. A) Main menu; B) Greet others; C) Stand up for others; D) Keep your workspace organised
All of this will allow the students to do the exercises together, interacting in the augmented space which will improve their learning experience as seen in Figure 2.
Figure 2: Multi-user PBIS AR experience
The pilot is still going on and the first results are expected by March 2023.
