A Perspective Review on Integrating VR/AR with Haptics into STEM Education for Multi-Sensory Learning ^†

Abstract

As a result of several governments closing educational facilities in reaction to the COVID-19 pandemic in 2020, almost 80% of the world’s students were not in school for several weeks. Schools and universities are thus increasing their efforts to leverage educational resources and provide possibilities for remote learning. A variety of educational programs, platforms, and technologies are now accessible to support student learning; while these tools are important for society, they are primarily concerned with the dissemination of theoretical material. There is a lack of support for hands-on laboratory work and practical experience. This is particularly important for all disciplines related to science, technology, engineering, and mathematics (STEM), where labs and pedagogical assets must be continuously enhanced in order to provide effective study programs. In this study, we describe a unique perspective to achieving multi-sensory learning through the integration of virtual and augmented reality (VR/AR) with haptic wearables in STEM education. We address the implications of a novel viewpoint on established pedagogical notions. We want to encourage worldwide efforts to make fully immersive, open, and remote laboratory learning a reality.

1. Introduction

In response to the COVID-19 outbreak, educational institutions have implemented restrictions for on-site meeting and learning. They are stepping up their efforts to use a variety of educational tools to give students remote learning possibilities while schools are closed. To assist parents, teachers, schools and school administrators in facilitating student learning and providing social care and interaction during periods of school closure, the United Nations Educational, Scientific and Cultural Organisation (UNESCO) elaborated a list of educational applications, platforms and resources [1]. Resources for psycho-social support, digital learning management systems, systems built for use on basic mobile phones, systems with strong offline functionality, Massive Open Online Course (MOOC) Platforms, self-directed learning content, mobile reading apps, collaboration platforms that support live-video communication, tools for teachers to create digital learning content, and external repositories of distance learning solutions are among these solutions [2]. Although these solutions provide critical support to society in these exceptional times, they are primarily focused on facilitating the transfer of theoretical knowledge. One of the most significant shortcomings of the multitude of existing solutions is the lack of support for hands-on laboratory work and practical experiences [3].

At all levels of scientific education and throughout all disciplines, hands-on activities are critical for significantly advancing learning. This is particularly necessary for departments of science, technology, engineering, and mathematics (STEM), which must constantly improve their laboratories and pedagogical tools to secure successful study programs to their students. The development of online material and the ability to provide immersive experiences for lab activities may be valuable in the near term to address COVID-19-related difficulties, but it may also pave the way for the introduction of new e-Learning resources in the long run.

A promising teaching strategy of STEM modules involves the ideas of Learning by Doing (LBD) [4], the approaches of Problem Based Learning (PBL) [5] and the concepts of Active Learning (AL) [6]. In reality, actively integrating students and allowing them to complete the activity in the lab is one of the most successful ways of teaching them how to conduct a practical engineering task. The LBD method is not a new instructional theory, it is exactly what it sounds like. Aristotle stated: “One must learn by doing the thing, for though you think you know it, you have no certainty until you try”. Similarly, Confucius declared: “I hear and I forget. I see and I remember. I do and I understand” [7]. On top of the necessary methodology for teaching efficiently and effectively, it is also necessary to contextualise and orient teaching activities from a socio-economic perspective. In the last several decades many of the world?s most developed countries have shifted from an industrial economy to a knowledge economy, which is based on the creation of knowledge, information, and innovation [8,9].

More recently, John Dewey became one of the strongest proponents of the LBD approach. Dewey argued: “Education is not preparation for life, it is life itself”. This educational methodology is perfectly in line with the concept of multi-sensory learning [10], which assumes that individuals learn better if they are taught using more than one sense (modality). The senses usually employed in multi-sensory learning are visual, auditory, kinesthetic, and tactile—VAKT (i.e., seeing, hearing, doing, and touching), as shown in Figure 1. The recent advances in virtual reality (VR), augmented reality (AR) and haptic technology may potentially enable the possibility of extending the multi-sensory learning approach to e-Learning and for re-designing study modules and engineering remote laboratories. Through this strategy, the community of educators and students may be enabled to explore the new frontiers of education.

In this work, we describe a unique perspective for incorporating virtual and augmented reality (VR/AR) with haptic wearables into engineering education to accomplish multi-sensory learning [11]. These immersive technologies are collectively known as extended reality (XR), a broad term which incorporates the three main technologies in use today (i.e., virtual reality—VR, augmented reality—AR, and mixed reality—MR), and others that will follow in the future, while virtual reality completely immerses the user in a computer-generated world, augmented reality allows the user to continue to see the real world, on which this technology overlays computer-generated objects. Mixed reality is a form of augmented reality in which the computer-generated objects blend more realistically with the real objects and the user interacts more profoundly with the former ones [12]. For this reason, in the current paper we will not dedicate a separate chapter to mixed reality, but will mention it in the virtual/augmented reality chapters.

The purpose of this study is to provide a new viewpoint on existing educational ideas, as well as to investigate the implications of this approach. We want to support initiatives throughout the world to make fully immersive, open, and remote laboratory learning a reality. First, we do a thorough review of the literature. Our study, however, is not a systematic literature review, but rather a viewpoint piece (with a solid foundation on the literature). This decision is deliberate because we want to encourage research—besides, the fragmented field and the combination of different aspects likely indicates that it is too early for a literature review that could synthesise the existing knowledge.

This article is organised as it follows. A review of the theoretical background from a learning perspective is given in Section 2. The strategy adopted in our perspective study is provided in Section 3. An overview of related VR, AR and haptic technology are presented in Section 4, Section 5 and Section 6, respectively. In Section 7, we review technologies that can potentially be used for evaluation and assessment of learning processes. Finally, we discuss our findings in Section 8 before concluding in Section 9.

2. Learning Theories

In this section, we provide an overview of immersive learning and present the motivation for the research described in the following sections. Some concepts may be too difficult too understand in STEM fields using the traditional pedagogies, namely addressing the relevant topics in lectures and tutorials. To increase the attraction and retention of students, many universities have adopted active learning (AL) pedagogy for over 20 years [13]. AL can be generally defined as “any instructional method that engages students in the learning process. AL requires students to do meaningful learning activities and think about what they are doing” [14]. In this student-centered approach, the learning responsibility is shared between the learner, the group, and the instructor. The instructor is responsible for organising the conditions on which effective learning depends [15]. The instructor might oversee changing learners’ aptitude towards learning to increase the learning value. As depicted in Figure 2, the aim is to progress from instructor-led activities (progressive learning) to self-directed learning in which the student defines the learning goals and uses available processes and resources to achieve them.

There are several strategies and techniques to implement AL into STEM courses. Some of the more common are project-based learning, problem-based learning, cooperative-based learning, and competency-based learning. Project-based and problem-based learning are often perceived as complementary and used together, since they have similar objectives and methodologies [15]. Students are required to develop a solution to a problem presented by the instructor, individually or in groups, during multiple learning activities [16]. Cooperative-based learning is centered on the human capability to learn socially; students are divided into groups and work together to achieve a goal, such as conducting a research project or multiple-step exercises [13]. This pedagogy promotes peer teaching and collaborative skills. Competency-based learning uses systems of instruction, assessment, grading, and academic reporting to ensure that students learn the knowledge and skills deemed to be essential to succeed in the profession.

When in-person teaching is not possible or laboratories are not physically accessible, immersive learning may be included in AL pedagogy. The opportunities offered by immersive learning are summarised in Figure 3. This considers options ranging from interactive physical environments to avatars in virtual worlds [17]. Interactive physical environments enable multiple learner collaborative experiences; engaging in virtual worlds as an avatar provides a cost-effective, safe, and expansive learning experience [18]. In immersive environments, students can securely participate in controlled experiments that would be high risky in the real world, allowing them to safely make mistakes, and learn from them. Moreover, some phenomena, that would otherwise not be observable can be visualised and presented in a clear way resulting in a deep understanding of core concepts. To achieve high fidelity immersive environments, instructors can integrate VR and AR technology into their classes [19]. The use of such technology during immersive learning experiences can optimise students’ knowledge acquisition and motivate them to be active participants in their own learning [18]. Furthermore, multi-sensory technologies enable learning with multisensorial feedback. This makes it possible to achieve higher levels of affinity between the student and the simulated learning environment. Tactile feedback, e.g., force and texture, can be rendered with the use of haptic technologies, while other multi-sensory technologies can enable olfactory and audio feedback. These technologies help develop more accurate mental models and representations of different concepts, thus enhancing learning [17].

A recent study conducted a review on the usage of immersive VR technology in higher education, learning theories for VR application design as well as the evaluation methods and the learning outcomes [20]. The study shows that VR applications engage scholars, as the article found 18 domain applications in higher education. The article points out the lack of learning theories that help with developing learning-oriented VR applications. However, this study neither examines the adoption of haptic technologies, nor how they can be relevant for STEM in higher education. A market study of available apps [21] brings complementary results but does not include haptic technologies, either.

As shown in Figure 4, to ensure successful learning based on immersive technologies, it is necessary to, firstly, plan the learning activities that the students will participate in. For this, the instructor should identify the students? learning needs, this is the knowledge and skills the students lack, and are necessary for the students? professional success, based on real-life work environment context. Next, based on the learning needs, the instructor should define the learning goals, i.e., the knowledge and competencies that the students are expected to acquire through successful participation in the learning process. The instructor should also define the pedagogical concepts and technologies that are going to be used during the learning activities. Pedagogy is a very important factor in the success of the learning process [22], and describes how the learning activities will occur. Once the learning process is defined, the instructor will choose the immersive tools that best fit the content to be learned. Then, the instructor can create the immersive learning environments and experiences or co-create them with the students to allow them to constantly interact with the virtual world and adjust the learning to their own characteristics and specific needs [23].

The effectiveness of the implemented learning process must be assessed to provide feedback to the learners and adjust their learning journey if needed. This can be done through two different perspectives: the outcome and the student experience. The first variable can be measured by a theory test or a practical conversation, where knowledge and skill acquisition and retention can be determined. The second perspective can be measured through a questionnaire, where the students answer various questions in order to determine student motivation, engagement, and immersion.

Although there is an increasing number of studies on the effectiveness of immersive learning, there is still a lot to be done to understand how to design effective learning experiences. The industry has been using this approach to train operators and some parameters of efficiency and effectiveness of the immersive learning process have been systematized, namely the speed in acquiring skills, error reduction rate in the execution of tasks after training, and accuracy in the anticipation and identification of events [24]. Moreover, factors like interest, motivation, perceived self-efficacy, embodiment, cognitive load, and self-regulation have been pointed out as crucial to achieve the defined learning outcomes [25]. The literature seems to point to a negative effect of groups on learning performance, suggesting that immersive learning is more suitable for small groups or even individuals [26]. In sum, it is necessary to continue to study the most appropriate way to use immersive technologies to ensure the effectiveness of the learning process.

The use of immersive technologies to design a learning process to achieve the goals defined by the instructor, possibly in collaboration with the learners, requires mastering of the technology and knowledge of which contents can benefit most from applying the tools. As shown in Figure 5, at the beginning of a new learning journey, the instructor starts by experimenting with the technology tools available to improve the understanding of opportunities and limitations (initiate level). At a more advanced stage, the instructor acquires a greater sensitivity to the available tools, becoming able to use them to promote the learning of a wider range of topics and define a greater variety of learning experiences. The focus is now on facilitating the communication of content and encouraging student engagement in activities that benefit from the use of immersive technologies (convert level). Finally, when the instructor masters the use of immersive technologies in learning, an optimisation process can be applied to define customised learning experiences. The focus is on learners’ expectations, learning styles and preferences. The instructor co-builds the learning journeys with the learners. In a self-directed way, the learner uses the immersive technologies to explore concepts and practices; the instructor becomes a mentor for this exploration (augment level).

3. Perspective Strategy

In this section, we highlight our strategy in surveying literature for technology that may enable multi-sensory learning with VAKT feedback. We assert that the way how these technologies have been used in education, especially in STEM for higher education, is understudied. Most literature focuses on technicalities, design, and models of interaction; it rarely examines technologies in the higher education context for STEM. Moreover, most of the existing rendering devices are still relatively costly and therefore still not available to the vast majority of students [27]. This represents a consistent gap in the systematic review of VR/AR and haptics for e-Learning. Among the existing technological solutions, the focus of this paper is on the possibility of adopting low-cost commercial off-the-shelf (COTS) components for STEM learning purposes to enable multi-sensory learning with VAKT feedback.

A possible setup for the novel e-learning approach is pictorially represented in Figure 6.

A student is at home attending an on-line course and he is wearing on their right hand three wearable haptic interfaces embedded in rings. He their exploring an object perceiving its stiffness, surface roughness and temperature through the wearable interfaces. The immersiveness of the experience is completed by the head-mounted display and the headphones that serves for video and audio streaming, respectively. This novel paradigm of e-learning needs a technological update both on the schools and students’ homes facilities.

3.1. Required School Tools

Beside the formation of the teachers and the study of specifically designed contents, the production of e-Learning courses will require an investment in the technology available in the classrooms. The creation of VR and AR contents will pass through the use of novel programming environments like Unity [28] and the adoption of 3D cameras and 3D scanners to acquire in the right format the needed content. Effort on designing mixed reality classes are on-going leading to the possibility of accessing the same contents on-site or on-line [29]. Finally, the addition of haptic contents would require both the use of databases where data-driven haptic texture and friction models are available [30] and the possibility to register tactile information using specific sensors, such as force sensors, accelerometer, temperature sensors, etc. [31]. Novel haptic contents could also be shared in an open-source way to foster the diffusion of multi-sensory e-Learning platforms.

3.2. Potential Home Solutions

To achieve a broad distribution of the devices needed for the VAKT model of e-Learning there are two main possibilities. The first is that commercial products reach a price that is affordable for most of the students. This is what happened for instance for smartphones and is what is very likely to happen also for VR/AR displays. For haptic interfaces this road appears to be longer since the technology is relatively younger, and only recently big tech companies have started investigating the tactile communication channel [32]. A possible way to speed up this process is the diffusion of open-sources repositories where it is possible to find detailed instructions to easily build wearable haptic interfaces by using off-the-shelves components. Similarly to what happened for instance with soft robotic hands [33], source files for 3D printable designs could be made available to users, together with all the instructions to assemble motors, controllers and to connect the devices with PCs. Several models of wearable haptic interfaces, that will be introduced in Section 6, would perfectly fit these initiatives. In this perspective, our research group is currently pursuing this possibilities with a research project that was funded by the European Union through the Erasmus+ Program under Grant 2020-1-NO01-KA203-076540, project title Integrating virtual and AUGMENTED reality with WEARable technology into engineering EDUcation (AugmentedWearEdu), https://augmentedwearedu.uia.no/ (accessed on 27 March 2022) [34].

3.3. Requirements to Facilities

When it comes to physical space, the idea is to make use of the existing facilities, but this needs to be altered/updated to comply with different requirements and experience of the VR/AR users. Literature has highlighted both physiological discomfort [35,36] and psychological discomfort [19] when using VR. Physiological discomfort can appear as motion sickness such as nausea and oculomotor, dizziness, blurred vision and vertigo, postural instability, drowsiness, eye fatigue, and degraded eye-hand coordination (especially in children). Other physiological signals, such as blood pressure, heart rate and sugar level, can also be affected especially when having unpleasant experiences in the synthetic environments (horrific venues, realistic unpleasant virtual venues). Moreover, psychological factors can stem from the uncomfortable feeling when wearing a VR/AR headset in public. Thus, in certain learning settings, one needs a private space when using VR for learning [19].

To prevent such potential discomfort issues, which could especially have direct impacts to the health, VR/AR haptic multisensory learning should pay attention for the following issues:

• Spacious: i.e., no obstruction, giving enough space to act without worries of hitting solid objects, such as walls or tables when using VR/AR and disconnect from “reality”.
• Flexible use: for private and public room purposes. This is true for those who want to learn in a small group as VR/AR apps require one to do certain movements or even repeat from the auditory feedback to be able to conduct the VR tasks. This solution will give an opportunity to minimise the psychological discomfort for those who need it. The same space could also be used in a larger group as a means to learn together and to get peer feedback.
• Health and safety issue: as some physiological discomforts have been reported in many studies [35,36], in the facility requirement guideline, we recommend having more than one person in a room to anticipate if an unwanted event occurs to a learner such as motion sickness symptoms, thus others can help.

These facility requirements are especially important for a strategy that includes not only procuring technology enablers for VR/AR development, but also conducting learning sessions with VR/AR devices.

4. VR Technology

There are many definitions for Virtual reality (VR). VR is investigated from different perspectives, such as technology, interaction, immersion, semantics and philosophy [37]. All of these perspectives are important when talking about applications of VR in learning. For examples, Mütterlein [38] discusses the three pillars of VR, i.e., immersion, presence and interactivity, and investigates how they are interrelated. When considering the application of VR solutions in learning, VR technology should support these three pillars as well as tackling sensory perception channels for multi-sensory learning [39].

Immersive VR devices seek to place a user in a virtual environment. The best-studied solutions are VR Cave Automatic Virtual Environment (CAVE) systems and head mounted displays (HMDs). Early HMD systems were seeking accurate and fast tracking of the head rotation. It was necessary to solve the Motion-to-Photon (End-to-End) latency problem [40], because high latency does not allow full immersion. People can sense the delay and the artificial nature of the environment. Nowadays this problem is largely solved, but some low-cost solutions, such as Google cardboard, still can not provide full immersion; even worse, response lags and jittery movements may lead to motion sickness [41]. The latter can also affect users of high-end HMDs if these do not cater for individual needs [42].

Another step into enhancing the immersiveness is allowing six degrees of movement freedom that requires tracking not only rotation, but also translation. The two approaches in use include outside-in and inside-out tracking. Outside-in tracking uses external devices (e.g., Oculus Rift external cameras [42]) to ensure head motion within tracked area. The inside-out tracking adopts cameras inside HMDs to track movements.

The immersiveness is further enhanced by tracking body parts and even external devices (e.g., pens, guns, sports equipment). This is relevant for learning applications that require the learning by doing approach. Tracking is done by using additional devices. The systems that could use such tracking are the outside-in systems. The inside-out systems can track palms and fingers, using infrared cameras, but the use-cases are limited in comparison to outside-in tracking systems. The additional trackers that are used in outside-in systems come in various dimensions and forms. For example, the VR ink device [43] from Logitech uses a similar form as a normal pen. This device or similar ones could be used for activities that require precision. Usage may span a wide range; a pen-like instrument could be used for such virtual activities as cutting tissue for a surgical procedure or for learning to solder.

VR technology enables educators to provide comprehensive assistance, because the tracked students’ activities can be used to give feedback in real-time or for briefing/debriefing purposes. Some recently introduced VR systems (e.g., HTC VIVE Pro Eye [44], HoloLens [45], …) also include eye tracking capabilities. Eye tracking makes it possible to automatically adjust the interpupillary distance (IPD) and track the gaze of the user [46]. Gaze tracking is used in many areas and it might be important for learning applications as well [47].

Providing haptic feedback in VR to make the experience more realistic has become a strong focus of research in recent years [48]. Haptic feedback has been shown to have an added value to extend immersiveness and add additional dimension related to senses [49]. Haptic experiences with VR, however, remain a challenge [50]. Currently available VR systems are mostly commercial applications and games, employing mostly hardware input. For example, the HaptX Gloves provides true-contact haptics [51], with 133 points of tactile feedback per hand. Dexmo, a hand haptic device for VR medical education by Dexta Robotics [52] is one of the few, existing, haptic VR learning systems. It uses force feedback to enable user feelings of size and shape, and captures 11 degrees of freedom (DoF) of users’ hand motion. However, most of the existing high fidelity haptic rendering devices are still relatively costly. For this reason, a few frameworks exist to facilitate the integration of haptics with different applications. For instance, Interhaptics provides hand interactions and haptic feedback integration for VR, mobile devices, and console applications [53].

5. AR Technology

AR superimposes virtual information over a user’s view of the surrounding environment, in such a way that this information seems naturally part of the real environment [54]. The main advantage over VR is that “AR connects users to the people, locations and objects around them, rather than cutting them off from the surrounding environment” [55]. This effect has a big potential in education, as it is demonstrated in an increasing number of research papers [56,57,58].

While the performance of AR technology has increased steadily over time, the main components of the hardware have stayed the same: sensors, processors, and displays. The role of the sensors is to provide information for tracking and registration. This is mainly achieved through an optical camera, with or without the help of sensors such as Global Positioning System (GPS), accelerometers, and gyroscopes. Optical tracking is categorised in the literature as marker-based, in which a static image is recognised (such as a quick response (QR) code), or marker-less, in which natural features of the surrounding environment are recognised. Other forms of tracking exist, but are much less common, such as acoustic, electromagnetic or mechanic [59].

Displays are the most prominent part of the hardware and the most impactful for the end user. They are usually visual ones, in the form of head mounted displays (HMD), handheld displays (HHD) or spatial AR (SAR) [60]. Less common forms of display are: tactile, audio and olfactory. Notably, audio displays are still much more common than tactile and olfactory displays, as there are already consumer devices specifically aimed at audio AR [61]. Input devices are often considered as a separate category, ranging from keyboards to voice inputs.

Nowadays, due to their ubiquity and versatility, handheld devices, in the form of smartphones, have become the main vehicle for AR experiences in many fields, including education [62], while hardware in smartphones has not changed fundamentally in recent years (yet continued to advance gradually), software evolution has been much more prominent (e.g., see the work presented in [63]).

Regarding software in AR, besides the low-level software that powers sensors, processors and displays to perform their tasks, higher-level software is used to enable creators to design different AR experiences. Depending on the technical skills of the creator, and the business or educational needs, one can use AR software development kits (SDKs) such as Vuforia, Wikitude, ARKit or ARCore, or all-in-one platforms, such as Cospaces Edu or EON Reality (which do not require programming skills).

A search in the scientific literature revealed no consistent existing research on exploiting AR in STEM education for fully-immersive remote laboratory learning. Most literature surveys cover the whole spectrum of target groups, from early childhood education to doctoral education. In general, AR is known to increase the understanding of the learning content, especially in spatial structure and function, compared to other forms of media, such as books or video; to aid with long-term memory retention, compared to non-AR experiences; to improve physical task performance, but also collaboration; to increase student motivation, through providing satisfaction and fun to the activities [64]. Use cases for education might be transferable from non-educational, professional AR usage such as from application that counter information overload [65].

A recent review of the literature on how AR is supporting STEM education [62] showed that the majority of the developed applications were exploration apps and simulation tools. At the same time, most were self-developed native applications, while the others used AR development tools. Furthermore, the vast majority were marker-based and only a few were location-based. These existing applications almost exclusively stimulate sight, leaving other senses unexplored. The study also surveyed what learning outcomes were measured and how, concluding that we are missing a deeper understanding of how AR learning experiences take place in STEM environments [62].

Another recent review on AR in STEM recognises intensive research in this area in recent years [66], although this work still mainly addresses early childhood education. The authors categorised the advantages of applying AR in STEM: contribution to learner, educational outcomes, student interaction and others. However, they also identified challenges. Most of these are owed to technical problems (i.e., weak detection of markers or GPS position). Other challenges include teachers’ resistance to adopting the AR technology, in which the time required prolonged periods to develop high-quality content plays an important role [66].

In contrast to VR, where we have somewhat established devices, both low-cost, low performance ones (Google Cardboard and similar) and high-cost, high-performance ones (i.e., Oculus Quest 2 or HTC Vive Pro 2), in AR we can rely only on the first type. These are the smartphones, which can actually be considered zero-cost, since the vast majority of the users already own one, and medium performance, since the a lot of effort has been put into developing high-performing AR on these devices, by big players (Google, Apple). Their disadvantage, however, in virtual labs in STEM education for example, is that (at least) one of the hands of the student is busy holding the smartphone, so haptic interaction is limited.

More appropriate and powerful AR devices, as in AR glasses, are still to become mainstream. Google’s AR glasses, launched for the general public in 2013, have been retired quickly, in 2015, and are now produced only for the enterprise domain. Apple’s AR glasses have been rumoured to be released for several years now, so they are still to come. Other brands of glasses, such as Moverio or Magic Leap One, have failed to become mainstream (at least in the sense that Oculus or HTC Vive are in the VR world).

In the area of mixed reality (MR), a form of AR in which the interaction of the user with the virtual objects is more profound [12], by far the best-known devices are Microsoft’s Hololens. These high-cost, high-performance devices have set a standard for the mixed reality technology up until now.

6. Haptic Technology

Touch is one of the most reliable and robust senses, and is fundamental to the human memory and in discerning the surrounding environment. In fact, touch provides more certainty than other senses, especially vision. To provide the user with tactile information, haptic technology can be employed. Haptic feedback, also known as haptics, is the use of the sense of touch in a human–computer interface. A variety of possible applications are made possible by the use of haptics, including the possibility of expanding the abilities of humans [67]: increasing physical strength, improving manual dexterity, augmenting the senses, and most fascinating, projecting human users into remote or virtual environments. Haptic technology is the key for achieving the tactile feedback experience of the VAKT model.

Early examples of haptic technology applied for gaining “touch” experience of the users through the sensation of forces, vibration or motion can be found in [68,69].

Most of the haptic devices available on the market like the sigma.x, omega.x and delta.x series (Force Dimension, Swiss) or the Phantom Premium (3D Systems Inc., USA) [70] are usually very accurate from a rendering perspective, and able to provide a wide range of forces. However, such devices present a limited workspace with a high cost of production. The pursuit for bigger workspaces and the possibility to achieve multi-contact interaction [71] lead researchers to the development and design of exoskeletons, a type of haptic interface grounded to the body [72]. Exoskeletons can be seen as wearable haptic systems, however they are rather cumbersome and usually heavy to carry, reducing their applicability and effectiveness.

To deal with these limitations, a new generation of wearable haptic interfaces have been investigated [31]. Haptic thimbles [73,74,75], haptic rings [76,77], and haptic armbands [78], have been designed for several applications, ranging from tele-operation to VR or AR interaction. Most of the available wearable haptic interfaces are only capable of providing cutaneous cues that indent and stretch the skin [79], and not kinaesthetic cues, i.e., stimuli that act on skeleton, muscles, and joints [80]. Wearable haptic interfaces, providing only cutaneous stimuli, do not exhibit any unstable behaviour due, for instance, to the presence of communication delay in the closed haptic loop [81]. To close-loop control the haptic feedback, the platform requires a cohesively integrated system. As a consequence, the haptic loop with wearable tactile interfaces results to be intrinsically stable. Wearable haptic devices are light, portable and can be used in combination to achieve multi-contact interaction [71]. Moreover, recent results demonstrated that wearable haptics can also be used in virtual and mixed reality to alter the perception of physical proprieties of tangible objects including stiffness, friction and shape perception [82].

Most of the proposed devices are built combining rapid prototyping techniques with off-the-shelves components including servomotors, vibromotors, programmable board, etc. This aspect can dramatically foster the diffusion of these devices in “at home” scenarios. We can imagine a future scenario where students could easily download, print and build their own devices and access haptic contents available for the novel VAKT model of e-learning.

7. Evaluation, Assessment and Eye-Tracking Technology

Haptic feedback can be used to increase the degree of presence in a virtual environment, allowing one to touch and feel virtual objects [83], which is very important for learning. Most scholars in educational research have acknowledged and concur that there is a strong connection between assessment and student learning. Traditionally, evaluation and assessment can be done using various methods such as: knowledge tests (i.e., written multiple choice, open-ended questions, oral examinations); practical knowledge evaluation and reports with narrative feedback or peer feedback and portfolio containing reflections [84]. However, we concur with Kreimeier et. al. [83] that how to assess and evaluate haptic feedback on its task-based presence and performance in virtual reality for STEM education is still rarely discussed in the literature.

Many evaluation efforts often focus on the usability of VR [85]. In particular areas, such as medical education, haptic-based VR for learning has increasingly been adopted, e.g., for simulating surgeries. However, the evaluation part often emphasises the overall impression on realism of the VR simulator, realism of tactile sensation, and other simulator elements, e.g., [86], while usability, acceptability and user experience assessment can be considered very relevant, there is also a need for exploring what evaluation and assessment are possible for multi-sensory learning and the use of VR. In Section 2, we argue on the need for integrating immersive learning into the learning process. In this perspective, we show a model proposing two evaluation measurements, i.e., measuring outcome (knowledge and skills, acquisition, and retention) and measuring experience (learner motivation, engagement, and immersion) to support multisensory learning.

Moreover, the shift in education theories from behavioralist toward a constructivist perspective [87] assume that students should be regarded as active learners of their own knowledge, skills, and competencies. The authors suggest assessment variations such as using social interaction, reflection and feedback involving peers and teacher both narrative/ oral feedback and multi-source feedback, in addition to other assessment method such as portfolio or collection of student products that reveal the achievements and efforts in specific areas. In fact, “simple” pass/fail decisions for learning assessment are gradually changing the assessment environment encouraging students to be more responsible to enhance their own learning [88]. This shift further supports the idea of incorporating immersive learning into the learning process.

Recently there are new developments and possibilities in combining eye-tracking with VR for usability and evaluation studies. Previously, eye-tracking products have appeared as screen-based devices, eye-tracking webcam, or wearable glasses. These devices have been applied in research and business settings to understand how humans interact with systems, machines, and processes. Eye tracking has also been used for understanding media habits including preferences and visual perception on various digital media devices [89], including in educational settings. By tracking gaze behaviour, researchers can measure visual attention to specific elements [90].

Thus, not only VR and haptic technology are advancing, but the eye tracking capability has been combined with VR technology. There are two variants in the market so far, i.e., a VR device that has built-in eye-tracking capabilities, for example HTC-Vive Pro Eye, and an independent eye-tracking device that can be mounted to existing VR device. Through such eye-tracking enhanced VR, it is possible to obtain user tracking data that allow researchers to learn about various behaviours with respect to the distribution of the visual attention within the virtual environment.

When taking the eye-tracking enhanced VR into learning context, different evaluation possibilities can be considered. Referring to Section 2, especially considering Figure 5, the model suggests two evaluation measurements, i.e., measuring outcome (knowledge and skills, acquisition, and retention) and measuring experience (learner motivation, engagement, and immersion). Instructors can verify whether the expected behaviours are achieved, the right objects are seen touched or moved, whether unnecessary distractions occur. There are many more criteria that can be used to evaluate the success of expected behaviour in the virtual environment. Advanced visualisation techniques presenting visual attention data can be presented in many ways such as heat-maps, as shown in Figure 7. Different eye-tracking metrics can be calculated from the raw data and have been proposed, such as pupil diameter (mean of left and right), gaze entropy, fixation duration, and percentage of eyelid closure to decide the importance objects seen by the eye-tracking users [91,92]. The power of incorporating eye-tracking into the multisensory learning case, lies especially on the capability to link kinesthetics/tactile, visual, and auditory feedback of the learners with virtual environment, and generate various data that can be evaluated after the learning session.

Instructors can predetermine some metrics to show successful sensory-based learning outcomes, e.g., by linking haptic feedback to a specific object in the virtual environment, such as objects that relate to eye-hand coordination [93]. Some scholars have used metrics such as task time, economy of motion, drops, instrument collisions, excessive instrument force, instrument out of view, and master workspace range to assess perceived workload. Higher intensity in particular object can be interpreted as perceived higher workload [91]. One can even determine distracting elements that push away user attention from their actual mission/learning points. Similar principles can be reused for integrating VR/AR with haptics into STEM education.

The use of VR in combination with eyetracker can actually enrich the variations of learning assessment method and can engage peers and teachers. For example, it can be done by projecting the virtual environment so that the peers and teachers can observe together and provide feedback on how to improve the learning acquisition and documented for the student’s own learning.

In [94,95], eye-tracking was combined with different information, such as audio, video, bio-metric data, and annotations to improve planning, execution and assessment of demanding training operations, by adopting newly designed risk-evaluation tools. This integration is the base for research on novel situational awareness (SA) assessment methodologies. This can serve the industry for the purpose of improving operational effectiveness and safety through the use of simulators. Such capability has the potential to be adopted for evaluation and assessment of the multi-sensory learning process using VR and haptic technologies, both for measuring the outcome (knowledge and skills, acquisition, and retention) and experience (learner motivation, engagement, and immersion).

VR-eye-tracking assessment can also be employed when seeking more understanding about visual attentions [96], decision making and judgement capability [97], and visual-driven emotional attention [98]. The students can reflect their own learning, perspective and other contextual factors that influence the learning stage. However, to be beneficial and maximising the learning impacts, it is the task of the teacher and facilitator to formulate the assignments, imminence of assessment, the design of the assessment system and the cues. In other words, no single solution what the best assessment systems would be, as the assessment itself really needs innovation and creativity and be anchored to the overall course goal. Our role here is to show the opportunities that the VR supported eyetracking can encourage reflective learning in STEM education, and expand the possibilities for learning assessment and evaluation.

8. Discussion

In the following, we first discuss implications of our work. We then explain its limitations before analysing the potential for future research.

8.1. Implications

It is possible to analyse the implications of virtual/augmented reality (VR/AR) technology and haptics when they are studied in the context of STEM education. The whole field of our study can be characterised as heterogeneous and highly dynamic when it comes to its maturity to be brought into STEM education supported by multi-sensory learning.

In alignment with other works (e.g., [20]), we conclude that approaches often are experimental and exploratory. Even if learning and teaching are explicit goals, the theoretical foundations in pedagogy tend to be shallow. Educators who want to use VR, AR and haptics typically need to start from scratch and try out what might work. Arguable, the state-of-the-art of AR and haptics in education is even less mature than that of VR, where educators can find more and more practical advice [19].

At the same time, the technological progress is rapid. Hardware and arguably to an even higher degree software support for VR, AR and haptics is continuously becoming more powerful, and the application in other areas—such as entertainment—will likely also pave the way for easier usage in education. Existing tools are still relatively expensive and not commonly available to all students. Even for educational institutions, buying hardware for large courses seems unfeasible. However, it can be expected that this problem will become less important with a wider spread of VR, AR and haptics applications.

The implications of our findings vary by perspective and role:

• Practitioners will find much need to advance the current technology as well as to find ways how it can be applied effectively and efficiently.
• For theory, much additional work is needed. Our work has shown that the gaps in understanding VR, AR and haptics in education are still large, and they are rather widening than being closed. We expect that more targeted research will be needed, and that research will need to be multi-disciplinary to follow pace with the technological development yet grasp the consequences for teaching.
• Educators will be able to use rich tools in the near future, but they will need much help in doing so. Selecting appropriate courses, fitting didactic, setting up equipment, and evaluating its use will need support. Moreover, close collaboration with those who develop VR/AR environment wit haptics will be required.
• Developers and designers of virtual worlds and augmented reality applications will need better tools to create multi-sensory learning environments that have high educational value. Work is also needed to improve their interface to educators, making exchange easy and fruitful.

8.2. Limitations

Our work has limitations that need to be mentioned. First, we worked rigorously with the literature. However, our paper is not a systematic literature review but rather a perspective paper (with a solid foundation on the literature). This choice was deliberate, as we want to stimulate research—besides, the fragmented field and the combination of different aspects likely means that it is too early for a literature review that could synthesise the existing knowledge.

Second, there are few works yet that combine VR or AR (let alone VR and AR) with haptics in an educational context, and in STEM subjects specifically. As a consequence of this immaturity, lessons need to be taken from the fields of VR, AR and STEM education individually. As an extension to the first limitation, it is impossible to get a full overview of any combination of VR, AR, haptics, and education. Thus, we might have missed works that do not squarely fall into our topics but would still help to get a better understanding of innovative pedagogic models for learning in immersive environments.

Third, the development in the field is rapid, which makes it hard to project future progress. Additionally, advances from non-scientific research could offer new possibilities or render current approaches obsolete.

The limitations of our work do not diminish its value—in fact, they align with the implications we identified.

8.3. Future Research

The need for maturing the multi-sensory learning concept in combination with the technological opportunities allows us for setting a list of research questions. In particular, we propose which steps research should take to leverage the current possibilities, aiming at improving STEM education. We setup the following research and implementation agenda:

• to operationalise further the multisensory learning concept for STEM education by taking advantages of current development of the VR, AR and haptic technologies, by focusing on the more affordable technologies;
• to focus on hands-on laboratory work and pedagogical tools that provide practical experiences for the students;
• to provide assessment tools for educators by having competencies evaluation on using VR, AR and wearable haptics in STEM education;
• to explore open source libraries of VR, AR and wearable haptics to be used for re-designed study modules and engineering laboratories;
• to develop teaching modules and to test our concept with engineering students in an experimental setting, to evaluate the applicability of the concept.

Based on this agenda, our research group recently presented a novel haptic-enabled framework for hands-on e-Learning [99]. The framework enables a fully-immersive tactile, auditory, and visual experience. This is achieved by combining virtual reality (VR) tools with a novel wearable haptic device, which is designed by augmenting a low-cost commercial off-the-shelf (COTS) controller with vibrotactile actuators. Results suggest that the proposed haptic-enabled framework improves the student engagement and illusion of presence. As a future work, our research group will continue the development of this framework.

9. Concluding Remarks

In this paper, we presented a perspective review on VR and AR with haptics integrated into STEM education. For this purpose, we first reviewed learning theories before discussing VR and AR technology as well as haptics. We then reviewed technologies that can potentially be adopted for evaluation and assessment of learning processes. Finally, we discussed our findings.

Our work has implications for research and practice, as well as for educators and for developers. Neither technological development nor scientific assessment of the field are close to being finished. This leaves plenty of room for future activities. Our work on this topic will continue.