Extended Reality Educational System with Virtual Teacher Interaction for Enhanced Learning

Abstract

Advancements in technology that can reshape educational paradigms, with Extended Reality (XR) have a pivotal role. This paper introduces an interactive XR intelligent assistant featuring a virtual teacher that interacts dynamically with PowerPoint presentations using OpenAI’s ChatGPT API. The system incorporates Azure Cognitive Services for multilingual speech-to-text and text-to-speech capabilities, custom lip-syncing solutions, eye gaze, head rotation and gestures. Additionally, panoramic images can be used as a sky box giving the illusion that the AI assistant is located at another location. Findings from three pilots indicate that the proposed technology has a lot of potential to be used as an additional tool for enhancing the learning process. However, special care must be taken into privacy and ethical issues.

1. Introduction

Extended reality (XR) [1] refers to a category of immersive and projected technologies that merge the physical and virtual worlds to create new environments where users can interact with digital content in more natural and intuitive ways. XR encompasses a range of technologies, including virtual reality (VR), augmented reality (AR), and mixed reality (MR). VR involves creating a fully immersive, computer-generated environment that users can experience through headsets or other specialized hardware. AR involves overlaying digital information onto the user’s view of the physical world, typically using a smartphone or tablet camera. This technology can be used to enhance the user’s understanding of a real-world environment or to create new, interactive experiences. MR combines elements of both VR and AR to create a hybrid experience that blends digital content with the physical world.

XR technologies have the potential to transform many aspects of our lives, from the way we learn and work to the way we entertain ourselves and explore the world around us. They are designed to create more engaging, immersive, and interactive experiences for users, whether they are playing games, exploring new environments, or learning new skills. The challenges facing XR are complex and multifaceted and a recent survey of drawbacks and opportunities has been recently published [2]. Recent developments in hardware for AR include lightguides [3] and holograms [4] that have great potential for future applications.

The XR market was valued at USD 25.84 billion in 2020. It is expected to reach USD 397.81 billion by 2026, registering a Compound Annual Growth Rate (CAGR) of 57.91% over the forecast period. One of the main driving factors for the market is the increasing demand for reducing the distance between people and richer visual content. On the other hand, the Global Digital Education Market size is expected to grow from USD 11.5 billion in 2021 to USD 46.7 billion by 2026, at a CAGR of 32.3% during the forecast period. Increased internet penetration; reduced infrastructure cost and increased scalability using online learning; increased efficiency, greater convenience, and more flexibility and rising demand for micro-learning are expected to be the major factors driving the growth of the Digital Education Market.

Owing to the outbreak of COVID-19, various countries follow strict lockdowns, shutdowns, and mobility restrictions to avoid the spread of the virus. Vendors have experienced high demand for Digital Education solutions during 2020. The pandemic has resulted in schools shutting all across the world. Globally, over 1.6 billion children in 188 countries were affected by school closures due to the pandemic [5]. With this sudden shift away from the classroom in many parts of the globe, some wonder whether the adoption of online learning will continue to persist post-pandemic and how such a shift would impact the worldwide digital education market. Before COVID-19, there was already high growth and adoption in digital education technology, with global edtech investments reaching USD 18.66 billion in 2019. However, whether it is language apps, virtual tutoring, video conferencing tools, or online learning software, there has been a significant surge in usage since COVID-19.

This paper tries to address the above issues and offer a solution that can be used for enhancing the teaching process through XR based on the XR4ED project [6]. XR4ED is an Innovation Action EU project aiming at the design, deployment, piloting, and market entry for a novel product along with its value-added services. In this direction, the overall methodology of the XR4ED project implementation is targeted toward achieving the commercialization of the XR4ED platform.

The focus of the paper is not on replacing the teacher but on providing extra support with an intelligent interface that can be easily used by a non-technical educator. The content that is required is only the teaching material of the teacher and optional videos and panoramic images. Artificial Intelligence (AI) takes care of the interpretation of the teaching material whereas the presentation is conducted through XR. Teachers have the option to select between two generic avatars, one male and one female. Both of them include body animations, eye gaze, head motion and lip-synchronization to increase realism.

The rest of the paper is structured as follows: Section 2 presents background information regarding VR, AR and XR in relation to education. Section 3 illustrates the potential of XR in education. Section 4 provides some insights into ethical human–AI interactions. Section 5 and Section 6 present the architecture and the main functions of the intelligent XR teacher. Section 7 illustrates the initial evaluation that was performed in two different countries, focused on teachers and professionals. Finally, Section 8 presents conclusions and future work.

2. Background

Until recently, the majority of the studies for education were focused mainly on VR [7,8] and AR [9,10] but much less on XR technologies [2]. In terms of authoring tools, a recent survey reports that there are only a few solutions for the easy design of educational AR experiences addressing the needs of teachers [11].

Moreover, social VR systems have become very popular in the last years such as VRChat (https://hello.vrchat.com/, accessed on 15 August 2024), RecRoom (https://recroom.com/, accessed on 15 August 2024), Spatial (https://www.spatial.io/, accessed on 15 August 2024), and MozillaHubs (https://hubs.mozilla.com/, accessed on 15 August 2024). However, such scenarios are usually limited to only visual and auditory stimuli, without considering the broad range of sensors that we can solicit. Ubiq is an immersive VR collaborative environment [12], provided with an internal mechanism that takes advantage of its modular architecture to facilitate easy integration of services and can be used in education.

VR affects perceived learning effectiveness only when specific tasks that fit will improve users’ learning outcomes [13]. The acceptance of VR in the classroom by educators was also assessed [14]. Another study investigated teachers’ willingness to integrate AR and VR technologies into their teaching and learning practices [15]. The assessment of engineering professors of different nationalities and universities regarding the use of VR technologies in the classroom was explored through the use of a validated questionnaire [16].

An XR roadmap shows how educational institutions can choose the right path to focus their attention on both the educational and social aspects of immersive learning experiences [17]. Existing barriers to the successful transition from individual use-cases of XR tools to broader adoption across university institutions were investigated in an Australian university [18]. Furthermore, immersive learning frameworks have been categorized according to their purpose and their elements [19]. Moreover, the different perceptions and challenges of implementing XR technologies in education were surveyed based on 48 experts [20].

The combination of XR with AI has also been examined and results found that the primary motivation is focused on training AI, conferring intelligence on XR, and interpreting XR-generated data [21]. In terms of AI in education, institutions provided conflicting feedback regarding the use of large language models such as ChatGPT [22]. The opportunities and challenges, including potential risks and benefits, of using ChatGPT in education were also recently explored [23,24].

3. The Potential of XR in Education

With the use of XR technology, learning can be enhanced in ways that are not achievable with conventional teaching techniques. While VR simulations can be used to create realistic environments that students can explore and learn from, AR applications can be used to overlay digital content onto real-world items, allowing students to manipulate 3D models [25]. However, there are still challenges to creating effective solutions [26].

Furthermore, XR has the potential to facilitate richer, effective, and more collaborative learning, than is possible with 2D alternatives and provide a high rate of knowledge transfer. When co-located in virtual learning settings, XR environments hold significant promise for empowering learners to use movement, gesture, and gaze to facilitate learning [27,28]. XR technology empowers learners with the ability to:

• Manipulate visual 3D objects, processes and procedures and elicit additional information about them
• Access inaccessible views of objects, processes and procedures
• Combine and compare multiple views
• Augment their senses

XR technologies when properly used can foster effective learning by leading to increased motivation and active engagement and participation of the students rather than passivity (learning by doing) [29]. They also have the potential to allow learners to proceed through the learning process during a broad time period at their own pace and encourage social interaction among users in collaborative settings. Furthermore, XR technologies can bridge formal and informal learning and improve the transfer of knowledge and skills to real situations through the contextualization of learning and the mastering of authentic tasks by learners. These affordances offer the following:

• Help learners to experience phenomena that are not possible in the real world.
• Visualize complex spatial relations and abstract concepts.
• Compare/contrast and interpret multiple/alternative perspectives.
• Enhance the learning experience using interactive 3D synthetic objects.
• Improve the attention of learners.
• Enhance learners’ immediacy and immersion
• Foster effective collaborative and situated learning.
• Improve practices and literacies that cannot be enacted in other traditional ICT environments.

4. Insights on Ethical Human–AI Interaction

Modeling, replicating and predicting human behavior [30] in everyday situations is useful in different applications including education. However, most current approaches to modeling human behaviors are unrealistic. To replicate realistic behaviors in credible simulations, more recent approaches have concentrated on agent-based modeling with a focus on psychological, social, and cognitive factors [31].

An interesting approach established an evaluation criterion to assess the suitability of the different ontologies for human behavior recognition [32]. Human behavior, including personality traits and emotions, can show itself in a multitude of ways, contingent upon the situation and daily occurrences. Engaging in interactions with other agents—human or not—that mimic or possess emotional experiences is also crucial.

When carrying out a task, the interaction between a person and an AI system must be viewed as a collaborative process between two agents [33]. In human–AI interactions, the sense of agency [34] plays a crucial role because it modifies our perceptions of control over our actions, raising questions about who is in charge and how to accept that fact. Therefore, it is essential to consider the possible effects of human–AI contact as well as strategies for boosting acceptability and trust.

The ethical guidelines that direct the creation of AI systems have received the majority of attention in the field of AI ethics [35,36,37]. The question of the obligations of people and the AI systems they interact with, as well as the results of their combined actions, is equally crucial, though. An interesting proposal was by cognitive scientists in 1998, defining the term “Extended Cognition” [38]. This was used to describe the way automation extends human cognition by giving it access to cognitive functions that it would not otherwise have without automated devices and has applicability in XR.

5. XR Interface

5.1. Implementation

The overall functionality of the interactive XR Intelligent Assistant is illustrated in Figure 1. It is built in the Unity game engine (version 2022.3.1f1). For the XR functionality, Vuforia (Vuforia Engine 10.25) is used to ensure compatibility with Android phones and MagicLeap2 (https://www.magicleap.com/magic-leap-2, accessed on 15 August 2024), AR head-mounted display (HMD). The AR HMD offers an immersive presentation of the intelligent teacher and the material presented but it comes with a high cost (approximately 4500 euros). On the other hand, the mobile phone version is less immersive but highly accessible to all students and teachers since the cost is much cheaper (approximately 500 euros).

ChatGPT in this research is used in the following ways. Tutoring acts as a virtual tutor, helping students understand concepts they find challenging. Information retrieval is used to assist students to ask questions and seek explanations on specific topics. Interactive learning experiences are also supported where students can have dynamic conversations, enhancing their engagement and understanding of the material. Personalized learning allows students to adapt to their own pace, providing more individual learning experiences.

The XR teacher’s avatars were designed in Character Creator 4, and are complemented by support for Ready Player Me avatars. Character Creator 4 provides advanced tools for creating highly realistic characters that can be optimized for different platforms, including low-poly models for mobile apps and high-performance models for XR applications. Communication with users is facilitated through the OpenAI ChatGPT API, utilizing JavaScript Object Notation (JSON) format for language preferences and slide navigation. The application supports different levels as shown in the next section.

5.2. Graphical User Interface

All the available functionality is accessible through a Graphical User Interface (GUI) which is operational in both the HMD and the mobile (phone/tablet) versions. The GUI has two components, the main GUI and the ChatGPT one. An overview of the the GUI is illustrated in Figure 2.

The main GUI is designed to control the multilingual settings, the gender of the predefined avatar (male or female), basic transformations (buttons ‘Rotate L’ and ‘Rotate R’) and the panoramic mode (button ‘Switch panorama’). An extra sub-menu is also provided allowing learners to start verbally communicating with the intelligent assistants (button ‘Tap to speak’). Additionally, learners can control the pace of the PowerPoint presentation (button ‘Previous’, ‘Play’ and ‘Next’) as illustrated in Figure 3.

5.3. Modes of Operation

The application supports two different modes of operation. In both cases, students can interact with the application and select the language of their preference. The first one refers to the virtual teacher delivering a presentation (Figure 4a). In this scenario, the learners can experience the virtual teacher interpreting the teaching material in real-time performance. In the second scenario (Figure 4b), the learner can participate in an intelligent conversation between two AI agents (which can be either colleague students or teachers) discussing the teaching material. In essence, ChatGPT is given a specific topic (the topic of the presentation) and the agents discuss a particular topic. The duration of the conversation is currently pre-specified to a few minutes, but this can be altered according to the needs of the learning scenarios.

An interesting feature of our application is that it is also possible to interrupt the AI conversation between the two agents and get engaged in the conversation itself [39]. Based on this setting, a pilot study with six participants explored a knowledge acquisition scenario within an AR environment featuring two ChatGPT-driven agents. The avatars, aware of users’ gaze and mimicking realistic body language, gradually displayed social exclusion behaviors. While the results were not statistically significant, the data indicated an emotional shift post-interaction, with both positive and negative emotions decreasing, consistent with previous research on social exclusion.

6. System Functionality

The functionality of the application supports four main categories including: (a) Speech Interaction and Multilingual Support, (b) Avatar Gestures and Animations, (c) PowerPoint and Video Integration and (d) Panoramic Environment.

6.1. Speech Interaction and Multilingual Support

The integration of multilingual support for lectures into the classroom has several benefits [40] and greatly enhances the educational process. It improves understanding, encourages diversity, and gives students useful skills. Azure Cognitive Services (Azure AI Services (https://azure.microsoft.com/en-us/products/ai-services, accessed on 15 August 2024)) are integrated for seamless speech interaction, such as speech-to-text (STT), text-to-speech (TTS), speech translation, and speech recognition. This enables the XR teacher to respond in multiple languages. In order to increase realism, TTS also produced visemes, which were used for synchronized avatar lipsyncing [41]. This synchronization enhances the naturalness of avatar gestures, creating a more immersive learning experience. Language detection is also performed through Azure Cognitive Services, enhancing the system’s adaptability to user preferences. Therefore, by offering lectures in a variety of languages, teachers recognize the range of linguistic proficiency among their students, which improves understanding and participation.

6.2. Avatar Gestures and Animations

Behavior modeling for virtual avatars involves creating computer-generated character models that mimic human behavior. To achieve this goal, a variety of technologies are required, integrating multiple modalities such as speech, gesture, facial expression, and body language. Although accurate solutions based on motion capture systems exist, the prohibitive cost of professional motion capture poses a barrier to its mainstream accessibility. In this research, Mixamo was used for generating avatars but later on, Character Creator 4 was employed (male and female versions). Ready Player Me’s animation library (readyplayer.me (https://readyplayer.me/, accessed on 15 August 2024)) provides animations that are categorized into locomotion, dance, expression, and idle, and are re-targeted to both feminine and masculine armatures. The avatar (i.e., the virtual teacher) supports head and eye movements during gaze shifts of virtual characters, including eyelid and eyebrow motion [42] as well as lip-syncing [41] capabilities.

Animations that mimic body language provide interactions that feel more natural and responsive and it is critical in maintaining user interest and engagement [43]. Moreover, body language animations can create a sense of presence and social connection and can enhance collaboration and learning in educational settings [44]. As a result, to make avatar interactions feel more real, several gesturing animations were incorporated into the XR application (Figure 5). These were used in order to enrich the XR teacher’s expressiveness, providing a diverse range of gestures and animations to engage users during presentations.

6.3. PowerPoint and Video Integration

The system includes a custom solution for parsing PowerPoint presentations (PPTX), extracting text from each slide. Users can manually add slide images, and the XR teacher can navigate to specific slides based on user queries, creating an interactive learning environment. To create a hybrid and immersive experience, video content can be also combined with the real world. Video integration in XR is designed for user flexibility, requiring manual addition to tailor content delivery. Teachers can enhance presentations with videos, adding a dynamic element to the learning experience essentially improving the storytelling.

6.4. Panoramic Environment

Although the next wave of clever tools for perceiving the surroundings is panoramic imaging [45], it has not been used for creating educational XR environments. They provide a 360-degree view, which enhances the sense of immersion in virtual and augmented reality environments. Panoramic images are easy to capture by teachers and students and they can be used for creating immersive and realistic experiences. When incorporated into XR, they have the advantage of transporting the viewer to different places and times, such as historical sites, natural wonders, or fictional worlds. The XR system supports the activation of various panoramas, allowing users to immerse themselves fully in the educational content while obscuring physical surroundings. Figure 6 illustrates the use of panoramic imaging combined with XR by showcasing two XR intelligent assistants. This process is performed through the GUI of the XR application depicted in Figure 2.

6.5. Collaboration

The collaborative part operates on a classic host–client model, wherein the host initiates a server on their device allowing client devices to connect. The host’s device manages the presentation of the avatar, including interfacing with OpenAI’s ChatGPT for slide explanations. When the avatar is scheduled to speak, the host device sends a message to all connected clients containing the predetermined sentences for the avatar. Upon receipt, each client device independently engages a speech synthesis service to generate the spoken audio. This audio is simultaneously played back along with synchronized gestures and lip-sync animations; the specific gesture animations are designated by numerical identifiers contained within the host’s message. Each client generates its lip-sync animation independently.

The system is designed to support multilingual functionality. Each client can translate the avatar’s speech into their preferred language and generate corresponding audio and lip-sync animations in that language, enhancing the user experience across different linguistic demographics. Both the host and clients have the capability to ask questions to the avatar. Questions, once spoken, are transcribed using speech-to-text technology and sent to the host. Using ChatGPT, the host processes these inquiries to generate relevant responses, which are then dispatched to all client devices. Each client device then autonomously generates and plays back the response in audio format accompanied by appropriate avatar animations.

7. Qualitative Evaluation

The proposed solution is planned to be quantitatively evaluated in the classroom next year. At this stage, it has been qualitatively evaluated on three different occasions, with different audiences as described below. The sample of the participants covers an age range between 20 and 60 with a total of 65 participants (50 male and 15 female). On all occasions, the procedure followed was the same. All participants were prompted to ask questions and seek information about the presentations that were delivered to them. The title of the first presentation was “Introduction to Computer Graphics” and the second one was “Extended Reality”. The third presentation was about the XR4ED project entitled “An Overview of the XR4ED EU Project”. Qualitative data were then collected from participants’ responses. It is worth mentioning that this was just an informal evaluation with no data collection, anonymous feedback only, with no real risk. Verbal consent was obtained from all participants involved in the study.

7.1. Teachers

A group of 20 high-school teachers experienced the XR intelligent teacher in October 2023 (location is removed for reviewing purposes). The majority of them did not have any previous experience with XR, which was expected. However, they also did not know what ChatGPT is about nor the latest developments on chatbots and large language models (LLMs).

On the positive side, they were all impressed with the fact that a potential student can ask interactive questions and get answers in real-time performance. Moreover, they appreciated the fact that multi-lingual support was provided, so the delivery of the material could be conducted in their language (removed for reviewing purposes), even if the presentation was in the English language. They also liked the fact that a potential student could control the pace of the presentation. Finally, they really appreciated the fact that the application was operational and fully functional on mobile devices and tablets since the schools already have the necessary hardware.

On the negative side, their main concern was the capabilities of the AI. Even if they found it interesting that ChatGPT could reply in real-time to several questions, they felt that their jobs could be in danger in the future. Several raised their concerns about ethical and privacy issues that come with AI as well as fake/wrong information. Some teachers also raised some concerns about the digital representation of the avatars, and they asked for better customization and more body language animations.

7.2. Professionals

During the Immersive Tech Week expo that took place in Rotterdam in November 2023, 35 professionals were exposed to the technology. Their background was very different from the teachers’ group, and they mainly originated from industry (apart from three University Professors). They were all experts in XR and related domains (i.e., games) since the expo was focused on XR. A screenshot of the intelligent assistant that was demonstrated in Immersive Tech Week expo is shown in Figure 7.

The professionals indicated that the application has great potential at all levels of education. They really appreciated three aspects including (a) interactive responses, (b) multilingual support and (c) customizable in terms of content (i.e., can easily change panoramic images, PowerPoint presentations and avatars). They valued very highly the fact that the same application was ported on HMDs and mobile devices. Several of them asked if they could use the application in different application domains such as entertainment, business and marketing.

The main concerns of the professionals were on the technical side of XR. Several asked for much more realistic virtual human representations. Unreal Engine supports the fast and high-fidelity creation of virtual humans, called MetaHuman Creator (https://www.unrealengine.com/en-US/metahuman, accessed on 15 August 2024). Several others wanted to have more variety in computer animations as well as computer graphics elements that can increase realism (i.e., real-time shadows). One person mentioned that it would be very interesting if there were collaborative aspects to the application so that multiple learners could access the same information at the same time. Finally, two participants were interested in having haptics so that they could physically interact with the avatars.

7.3. Researchers

During the Eurographics 2024 international conference that took place in April 2024, 10 researchers (ranging from post-docs to Professors) were exposed to the technology. All of them were experts in the domain of computer graphics including rendering, animation, computer vision and image processing. As expected the main feedback that was received concerned the rendering quality of the intelligent assistant as well as the overall scene.

Two participants complained that the body language animations are limited and they wanted more variety in the movements. One participant mentioned that there are no soft and hard shadows cast into the floor. While all participants liked the idea of using LLMs in conjunction with XR, they wanted to make use of AI to either generate animations or other relevant content (i.e., teaching material). Additionally, some mentioned customizing existing LLMs in order to be focused on education.

On the positive side, the majority of researchers liked the features of the GUI which allows them to display the features of the application (see Section 5). Several participants rated positively the eye gaze, head movement, and lip synchronization. Additionally, they commented that realistic representations of avatars are not necessarily crucial in order to avoid the uncanny valley effect.

8. Discussion

The interactive XR intelligent assistant has the potential to transform education by providing immersive and interactive learning experiences. Results from the qualitative evaluation illustrate this. A summary of the results from 65 participants in three diverse fields is illustrated in Figure 8.

However, there are several limitations that can hinder the learning experience. One of the biggest barriers is the availability of high-quality software and hardware equipment. They are often expensive and depreciate within a period of three to five years, making it difficult for some educational institutions to afford. Not all students have access to XR devices, which can create a digital divide. Additionally, some students may have physical or cognitive disabilities that make using XR challenging. Our solutions offer the choice between HMDs and smartphones which reduces the cost (since almost everyone owns a smartphone nowadays).

Moreover, there is a limited amount of high-quality educational content available for XR educational experiences. Educators need proper training to effectively integrate XR into their teaching. Without adequate support, the technology may not be used to its full potential. Developing custom content can be time-consuming and costly. However, our solution is primarily based on content that can be easily generated by educators such as PowerPoint presentations and 360 panoramic images. In terms of avatars, the system is going to be connected to the XR4ED marketplace where teachers will be able to download and use related content. Finally, another potential limitation of the interactive XR intelligent assistant is highly immersive. This might lead to students becoming distracted from the learning objectives.

9. Conclusions

This paper presents a novel XR educational system, where a virtual avatar serves as an interactive teacher, seamlessly integrating PowerPoint presentations and real-time user interactions based on AI. It represents a significant step towards dynamic and engaging learning experiences. Through the integration of such mechanisms, we can unlock novel collaborative experiences that enable teachers and learners to engage in interactive authoring and storytelling leveraging their creativity and improving their involvement in unprecedented ways. A total of 65 users have evaluated the AR intelligent assistant through three different pilots. Qualitative results indicate that the proposed technology has a lot of potential to be used as an additional tool for enhancing the learning process.

Nevertheless, intelligent XR assistants pose significant challenges and risks, such as privacy, security, ethics, and regulation. They require access to large amounts of personal and sensitive data, which may be vulnerable to hacking, misuse, or abuse. They may also influence or manipulate the user’s behavior, preferences, or opinions, which may affect their autonomy, identity, or values. These issues need to be addressed before applying intelligent XR teachers into the classroom. Summing up, XR is a rapidly evolving field, and there are many research challenges that researchers are currently grappling with such as designing software algorithms and hardware, user experience and social interaction, and ethical considerations.