Countering the Novelty Effect: A Tutorial for Immersive Virtual Reality Learning Environments

Abstract

Immersive Virtual Reality (iVR) is a new technology, the novelty effect of which can reduce the enjoyment of iVR experiences and, especially, learning achievements when presented in the classroom; an effect that the interactive tutorial proposed in this research can help overcome. Its increasingly complex levels are designed on the basis of Mayer’s Cognitive Theory of Multimedia Learning, so that users can quickly gain familiarity with the iVR environment. The tutorial was included in an iVR learning experience for its validation with 65 users. It was a success, according to the user satisfaction and tutorial usability survey. First, it gained very high ratings for satisfaction, engagement, and immersion. Second, high skill rates suggested that it helped users to gain familiarity with controllers. Finally, a medium-high value for flow pointed to major concerns related to skill and challenges with this sort of iVR experience. A few cases of cybersickness also arose. The survey showed that only intense cybersickness levels significantly limited performance and enjoyment; low levels had no influence on flow and immersion and little influence on skill, presence, and engagement, greatly reducing the benefits of the tutorial, despite which it remained useful.

1. Introduction

The first Virtual Reality (VR) technologies emerged in the 1950s and have only become widespread in society over time, due to their high cost. However, in the last decade, the cost of immersive Virtual Reality (iVR) has fallen as its popularity has risen.

The popularization of iVR has led to the development of many more iVR applications, as well as its expansion in a wide variety of fields, such as education and simulators, where they are known as iVR serious games. According to some research, iVR environments are useful in education, in order to improve both interest in learning [1] and the understanding of complex concepts [2], as well as to decrease misconceptions [3]. Besides, iVR is now understood as a core component of Extended Reality (XR), an integration platform for iVR, mixed reality, and augmented reality, a new paradigm of the possibilities of ICTs technologies to extend the sensory experiences of virtual reality that can potentially boost learning experiences [4,5].

Although more iVR applications are being developed, iVR is itself still a new technology, so it is likely that many people will not have heard of it before. Their inexperience using Head Mounted Displays (HMD) and the novelty of using unfamiliar iVR interfaces might be a source of extraneous cognitive load [6], one consequence of which could be lower satisfaction with the iVR experience. The extra cognitive load of iVR learning experiences can also limit the learning outcomes of users [7], a limitation that is caused by the novelty effect, which Huang [8] defined as new, unfamiliar, and unexpected experiences for a user. In iVR environments, the focus for novel users is on learning how to use an iVR hand-controller, exploring the iVR environment, and trying out all possible interactions [9], rather than on the proposed tasks. The curiosity of a user is due to high levels of motivation and the perceived usability of a new technology [7,10]. However, the novelty effect is short-term, as it fades with exposure to the new technology and the experience that it offers [10].

Video games usually include tutorials as an introduction that also serve to counter the novelty effect. These tutorials are designed to help the user become familiar with the controllers, giving instructions on their use, the mechanics of the video game, interaction with the environment, and the objects, etc. Tutorials are also designed so that players can acquire some skill when preparing for a real video game, in so far as players must be free to practice without the constraints of time, space, and interactions and at the same time in as complete an environment as possible, so that when the player can enjoy the video game, the skill levels match the challenges [11]. In addition, the tutorials are commonly the first point of contact for players of the video game. Another of their objectives is therefore to attract and to entertain players, so that they remain engaged and continue playing [12]. Nevertheless, the balance has to be struck between each skill and how well it has to be learnt to surmount the implicit challenges of a video game. If a player has not developed sufficient skill to negotiate a task, its effect on the gaming experience will be negative. The same will occur if the skill is overdeveloped for the tasks [13]. In addition, tutorials can have different approaches, such as context-sensitivity and context-insensitive tutorials [12].

Therefore, the design of the iVR tutorials presents different challenges, some of which are common to video-game tutorials, while others are quite different. The effects of immersion in the iVR environment, the new interfaces, and controllers, and cybersickness fall into the latter category [1]. The first time that users enter the iVR environment, they neither know how to use the iVR hand-controllers, nor how to interact with the objects. The iVR adds to immersion, providing a feeling of presence when the iVR environment is considered as their real world. However, it also provokes disorientation, a possible symptom of cybersickness, and can imply physical discomfort with the iVR experience [14]. Besides, the way the information is introduced to the user in iVR environments can also increase the user’s disorientation. Avoiding a major role for text-based content is general advice, by trying to shift this role to a 3D interactive character: whether a simple one, such as an automated robot or a complex solution, such a digital human-like character gifted with artificial intelligence and emotions [15].

Considering the research gap into tutorial design, an iVR tutorial is proposed in this research to improve user skill and engagement with iVR environments. The iVR tutorial has a general-purpose design for any iVR learning experience. It is structured into levels (or modules) of increasing difficulty, so that the users can learn how to use the iVR hand-controllers, acquire sufficient skill, and become familiar with the iVR environment at their own pace. Furthermore, the iVR was validated with 65 final users. The conclusions are intended to guide the design of iVR applications and to optimize iVR teaching.

The first key point of this research relates to the interconnection of design features and learning theories for the development of the tutorial. The second is an evaluation of its usefulness for novelty effect reduction, considering five satisfaction components (engagement, skill, flow, presence, and immersion) and their connections with tutorial design. Finally, it relates to the measurement of the effect of cybersickness and the performance of these iVR experiences, a factor that cannot be completely avoided, due to the vigorous movements and interactions that are both required and needed in an iVR tutorial.

The structure of this paper is organized as follows. In Section 2, the state of the art of tutorial design is presented and, in Section 3, the materials and methods used to design and to develop the iVR tutorial are explained. The design followed the Multimedia Learning Cognitive Theory of Mayer and included some features from other research lines: carefully designed experiences and a context-sensitive style. The development was performed in Unreal Engine and the iVR tutorial structure was divided into modules of increasing difficulty. Then, the results of the usability and satisfaction surveys administered during the validation of the tutorial are analyzed in Section 4. Finally, the conclusions of this study are presented in Section 5 and some future lines of research are outlined.

2. State of the Art

Tutorials have emerged to complement video games, so that players can learn to play and to engage themselves in the video game. However, the importance of the tutorials in non-VR video games depends on the complexity of the video game. This idea is supported by a study in which 45,000 players tried three video games of varying complexity. The duration of each video game level and the return rate were analyzed. According to the results, the tutorials were only useful when the video games were complex [12]. The tutorials positively influenced the time spent playing the video game and the number of completed levels. Tutorials for simpler video games hardly appeared necessary, because players can learn the mechanics faster through experimentation. The same can be said of some iVR applications [16], although tutorials are not usually included, as Checa et al. [1] noted. In their review, 10% of the papers reported games using tutorials for players to learn what to do in their iVR experiences, which were usually included as an introduction or first level [17,18,19,20,21]. However, some tutorials were included to accustom players to the iVR environment, and so that they could learn how to interact with the objects, as proposed in the work of Bhargava et al. [22]. SteamVR was used in the research of Shewaga et al. [23] to teach the users how to use HTC Vive hand-controllers. These tutorials both serve as an introduction for users to the iVR world and as a means of gaining familiarity with the interaction devices and their functions, thereby reducing the novelty effect [9]. One option to use the tutorial before playing was described in [24], also reviewed in Checa et al. [1].

Furthermore, tutorials can be presented in different forms in video games. One of the first approaches is the context-sensitive tutorial. The presentation style of this sort of tutorial consists of showing the video game mechanics using text when the user needs the information. The text appears in relation to the player, in a way quite unlike the context-insensitive tutorial. The context-sensitive tutorial presents the video game mechanics without a filter: all the information is shown simultaneously at the outset [12]. Tutorials can also incorporate hinting systems, which are intended to improve performance and with which most players will be familiar and will find helpful [16]. However, there is little investigation in the area of iVR applications. One example is the study of Kao et al. [16], in which different forms of iVR tutorials were compared: text instructions that interrupt the gameplay, context-sensitive, and context-insensitive. In their study, the conclusion was that the context-sensitive tutorial had a higher positive effect than presenting all the instructions with text.

In research works about iVR learning experiences, tutorials are not usually included. The acquisition and improvement of knowledge and skill are generally compared between an iVR group and a reference group following a traditional learning methodology (e.g., a desktop computer-based methodology), such as the example given in Dengel et al. [25]. However, the levels of acceptance of iVR and desktop computer technologies were not considered in their research, as those variables could not cause the expected effects in learning enhancement, even when user satisfaction was very high [17]. The conclusions of other studies were likewise that iVR had no effect on learning [26,27,28,29].

Beyond the introductory tutorials, the real-world tutorials reviewed in [30] let players practice in real and virtual environments prior to playing an iVR application. One consequence of using this type of tutorial was increased familiarity and confidence with the environment and the positive effects of practicing in both real and virtual environments were clearly demonstrated.

Despite the limited research on tutorials for iVR learning applications, it is worth considering the information to be transmitted before starting to design the tutorial. This information comprises the context of the game, the objectives, and the different functional operations and utilities of the game. Taking all this information into account, the most successful commercial iVR games with tutorials were included in the analysis. Most of the games had some form of text help (88%), graphics or images (56%) in their tutorials and labels were also used on those controllers to instruct the player (22%), as Kao et al. mentioned [16]. According to the developers of the games, the tutorial design was based on intuition, personal experience, existing examples, and user tests [12]. In addition, tutorials can be of different types, as Green et al. explained [31]. In their study, they proposed three tutorial types based on the teaching mode: by instructions, by examples, and with carefully designed experiences. Tutorials with instructions guide the user step by step while explaining the rules of the game. Tutorials with examples present the consequences of concrete actions as instances. Finally, the tutorials that feature carefully designed experiences consist of environments that can be freely explored where the user can try out actions with neither time constraints nor limited attempts.

3. Materials and Methods

In this section, both the design and the development of the iVR tutorial are described, as well as the design of a validation stage with final users in terms of usability and satisfaction. The iVR tutorial developed can be downloaded from the following URL http://3dubu.es/virtual-reality-tutorial-learning-experience/ (accessed on 23 November 2022).

3.1. Design of the iVR Tutorial

In this research, the model of a carefully designed experience was chosen, as explained in Section 2. With this approach, the user can practice the mechanics in a quiet environment in preparation for the main experience. Moreover, this model was chosen, because of the novelty effect. A user in an iVR environment for the first time wishes to explore it and try out actions, so if there are limits on interactions and time constraints, the novelty effect cannot be lost.

As a first step, the research of Frommel et al. [32] was consulted. According to their work, one of the features of an iVR tutorial is to show information that is synchronized with the pace and even with the requirements of the user. Therefore, the style of the tutorial has to be context-sensitive. Following the research of Kao et al. [16], the text modality with controller diagrams was implemented, so as to teach the user how to use the hand controllers. As explained in their study, less could be learnt from the text alone than the option of text with diagrams. The third modality that they proposed was text with controller-tool tips, although it was not included in this study, because the iVR tutorial design was for a general purpose. A text presentation may have readability issues within an iVR environment, hence the selection of HTC Vive HMD with eye-tracking. This HMD limits user freedom to movements within ten square meters, which is sufficient for the tasks that are proposed in the tutorial, although it can limit the comfort of the iVR experience. To be sure that this limitation does not play a major role in the experience´s quality, it will be evaluated in the satisfaction survey during the evaluation of the iVR tutorial.

As a second step, the iVR tutorial design was based on the Multimedia Learning Cognitive Theory of Mayer [33]. Mayer et al. [34] proposed 12 principles, in order to reduce cognitive load and to enhance learning, a theory that has in most cases been applied to desktop 2D games. It is therefore necessary to adapt certain principles for its use in iVR. In addition, the iVR tutorial includes 7 of the 12 principles that are summarized below in Figure 1.

The 7 chosen principles followed or adapted from the above model are explained as follows:

• Multimedia principle: a principle that consists of learning from a combination of graphics and text together rather than using only words. The images help the understanding of the material, so its use is recommended. As one of the main problems with iVR is learning how to operate the virtual controllers, iVR tutorials often include graphics and diagrams where controller buttons are presented alongside an explanation of their functions. This principle of teaching users how to use the iVR controllers through a combination of graphics and text is shown in Figure 2.

2.

Signaling principle. This principle states that learning can be improved when essential words are highlighted in various ways, i.e., arrows, large text, bold text, and color. In this research, the principle has been adapted so that the information appears as visual cues to conduct the user’s attention (see Figure 3). In this figure, the assistant robot, which leads the user through the iVR tutorial and the iVR learning experience, appears, offering help when the user looks directly at it (see Figure 3A where a signaling cue is highlighted by an orange rectangle) and when the robot assistant is waiting for the user help call (see the orange rectangle in Figure 3B). The signaling principle is represented by not overloading the environment with information and only showing the information when the user requires it. Furthermore, the use of visual cues has been shown in various studies to speed up learning information and to enhance learning efficacy, as Lin and Anderson explained [33], to decrease cognitive load as demonstrated earlier [35,36], and to improve the speed and the accuracy of completing tasks, as Kelleher and Pausch demonstrated [37]. In addition, the use of iVR environments faces other challenges. For example, visualization problems can occur if objects are placed in the periphery (the user would have to turn his head to see them). To overcome this problem, the main visual elements have been placed in the user’s initial field of view (FOV). Users pay much more attention to the initial FOV than to the rest of the regions, which means that if information is placed outside the initial FOV, users may lose track of events [38].

3.

Coherence principle. According to this principle, learning improves when irrelevant material is avoided. This extra material is unnecessary for the purpose of the instruction and it should be excluded from it [39]. Likewise, the use of extraneous words and diagrams can cause distraction (directing the learner’s attention to irrelevant material), disruption (the superfluous material interrupting the thought patterns of the user so that no mental model is created), and seduction (focusing on an irrelevant field of knowledge). Following this principle, the iVR tutorial of this research design is composed of neutral colors with no elements that can distract, and superfluous extra material has been avoided. Also, the scenario is the same, so it cannot hinder the perception of the iVR environment, as Wojciechowski et al. [40] showed in their research. Figure 4 shows an example of the iVR environment according to the coherence principle.

4.

Spatial contiguity principle. In keeping with this principle, the words and graphics are depicted nearby. As can be seen in Figure 2, the text and the image to which it refers are close together. Minimal distances between the text and related images facilitate learning, rather than longer distances, and information shown on separate screens.

5.

Temporal contiguity principle. This principle is related to the spatial contiguity principle, because the images appear near the text and are shown at the same time. There is no narration in the iVR tutorial, but both principles have been adapted not only with the images and text, but also with the assistant robot and its text bubble.

6.

Redundancy principle. According to this principle, learning is facilitated to a greater extent with a combination of narration and animation, rather than with a combination of animation, narration, and text. Exceptionally, learning is enhanced when the on-screen text that also highlights relevant information is shortened and placed near the image to which it refers. In this research, the principle is applied whenever the assistant robot explains each task to a user and when the information is shown on the board complementing the information from the robot and the most relevant text is highlighted. Figure 5 is an example of the redundancy principle that has been adapted to this research.

7.

Personalization principle. This principle shows that learning improves when an informal rather than a formal narrative style is used. The iVR tutorial of this research has an informal style of conversation, referring to the assistant robot and the graphics. In Figure 6, an example is highlighted with an orange rectangle.

Based on the previous design in which the Multimedia Learning Cognitive Theory was considered, the development of the iVR tutorial was executed. A previously tested and validated framework was used, in order to complete this phase of the iVR tutorial creation [41]. Its main function in this research was to reduce the time spent developing the iVR tutorial, as it simplifies the development of iVR applications, solving the main technical issues related to the iVR environment and its development, so that the researchers can concentrate more on the design.

Furthermore, some features were borrowed from the research of Andersen et al. [12]. These features were context-sensitivity, as previously explained, and freedom and availability of help. Freedom in tutorials can be defined as a modality of playing in which the player is not forced to complete any task or stay in a certain place. As presented in that research, users respond more positively to experimentation within an environment rather than without one. Access to additional information is also available through a help feature. The design of this feature was based on the visual cues proposed through the signaling principle taken from the Cognitive Theory of Multimedia Learning.

3.2. Development of the iVR Tutorial

The framework described in Section 3.1. was developed in Unreal Engine™ (4.27 version). This game engine is characterized by its potential to create photorealistic environments, its ease of use, and its compatibility with most iVR HMDs on the market. Furthermore, the most common actions in iVR applications are included in the framework, such as player movement, interactions with the scenario and objects, the creation of scene targets, and data collection. The designed framework is device agnostic and adaptable to a wide range of interaction forms. Thus, in this initial phase, the tutorial uses only iVR hand-controllers (representation of virtual hands), but it could be adapted to different types of interaction such as hand tracking or the specific VR controllers of each HMD vendor. To achieve this wide adaptability to different input modalities, the framework was programmed in order to ensure that the scene object hosts the component that allows it to be grabbed and, depending on the pawn used, it will be grabbed according to its available tools (e.g., hand tracking, VR controllers…).

One of the main objectives behind the creation of this iVR tutorial is to have a general tutorial that can be applied to a wide variety of iVR applications, an objective that can only be achieved if the iVR tutorial is a comfortable experience through which to gain familiarity with the environment. In addition, users have to learn from the iVR tutorial how to interact in the environment and the objects using the VR controllers, but it can be challenging, due to the variety of iVR applications whose interfaces and forms of interaction all differ. Considering these goals, different modules (levels with different types of interaction) have been developed. These modules can be combined, in order to adapt the iVR tutorial as much as possible to the post-user experience and, at the end of the iVR tutorial, the skill of the player is likely to match the challenges they will face in the iVR experience. The chosen modules are summarized in Figure 7.

The modules that were included are explained as follows:

• Introduction. The first space the user encounters when entering the iVR environment. Distraction is avoided, in order for the user to become accustomed to the iVR world and to become familiar with it. In addition, the design of this module follows the coherence principle, which limits the use of distracting and strange words and graphics.
• Basic interactions. The basic interactions can be started once the user is familiar with the iVR environment. The user will be able to press a button as a first basic action to initiate the tutorial. Pressing a button is a basic and accessible interaction. Moreover, the user manages the pace during the tutorial. After pressing the first button, more basic button-pressing interactions appear, so as to progress through the tutorial.
• Grab. In iVR, grabbing is one of the most common interactions. It is also usual that the grabbing interaction pursues a certain purpose, in which it is necessary to attach the grabbed item. In the iVR tutorial, users learn how to grab and to attach different objects with different purposes.
• Complex interactions. In addition to basic interactions such as pressing buttons and grabbing and attaching objects, the iVR tutorial includes complex interactions such as interacting with levers. These complex interactions can be required to accomplish certain tasks in some iVR applications.
• Interact with user interfaces. The user interfaces are commonly included as menus and information screens, so it is important for the users to know how these interfaces work.
• Explore and play. This is the final module. It has been conceived as a set of all the previous modules. The user can explore and interact with all the surrounding objects. All the mechanics already known by the user are included. When the user feels prepared, the learning experience can begin, and the users can do so with the feeling of being ready to face the tasks proposed in the iVR experience.

The duration of the tutorial was between 3-to-7 min depending on the pace of the user; sufficiently long to master all the proposed interactions. Apart from reducing the novelty effect, the other purpose is to limit cybersickness, as a recent study [42] showed that the longer the duration of exposure to iVR environments, the more severe the symptoms of cybersickness. Cybersickness affects health and safety, so it is an important factor that can condition satisfaction itself, as participants experiencing cybersickness can hardly feel satisfied. It can also affect their performance in the iVR tasks. Cybersickness levels were measured in the satisfaction and usability survey after users had completed the iVR learning experience.

3.3. Design of the Usability and Satisfaction Survey

The satisfaction and the usability of this iVR tutorial were tested as an introduction to an iVR learning experience, which is called the “PC Virtual Lab” [43]. Students who participated in this iVR experience were expected to improve their learning and their skill at specific tasks related to computing as part of their lessons. These tasks were divided into levels: mounting a desktop computer with an assistant, identifying and naming the components of a motherboard, cooling a desktop computer in a logical way, and assembling a desktop computer with specific characteristics.

The aim of using the tutorial as an introduction to “PC Virtual Lab” was to reduce the novelty effect. Therefore, the users become familiar with the VR environment, learn how to use the iVR controllers, and how to interact with the objects. After the tutorial, the users were prepared to focus on the previous levels to optimize their learning.

For the further improvement of the tutorial, and to know whether the tutorial results were easy and comfortable to use, a usability and satisfaction survey was completed by each participant after finishing the iVR learning experience. Furthermore, this survey was used to measure user satisfaction levels, in order to solve problems and include their suggestions.

The survey was based on the proposed model of Tcha-Tokey et al. [44]. The survey consisted of a Multiple-Choice Questionnaire that was composed of 21 questions. Participants were asked to rate each question on a Likert scale from 1 to 5 where 1 means totally disagree and 5 totally agree. These questions measure 5 components that Tcha-Tokey et al. [44,45] proposed for the evaluation of user experience: engagement, presence, immersion, flow, and skill. They are explained as follows:

• Engagement is the involvement that exists between the user and that user’s actions within the VR environment. Motivation is directly related to engagement, so if the user is motivated, the user will also be engaged. Furthermore, engagement is connected with presence and immersion [46]. In the proposed survey, engagement is evaluated with 3 questions.
• Presence is the illusion of ‘being there’, so that the user believes that the iVR environment is real (the dominant reality for the user) [47]. The user therefore behaves and feels as if living in a real situation [48]. Presence is evaluated with 5 questions in the survey.
• Immersion levels depend on the hardware and depend on the extent of the stimuli from the virtual world that the user perceives. The hardware replaces the user stimuli with the virtual sensory stimuli [44]. For its assessment, 4 questions were used in the survey.
• Flow is the psychological state when a user feels control, fun, and enjoyment [44]. Three questions were asked in the survey for the evaluation of flow.
• Skill is the gain in user knowledge when practicing certain activities during the VR experience. In the survey, skill was evaluated with 6 questions.

Table 1. Usability and satisfaction survey questions to evaluate the 5 components: engagement, presence, flow, immersion, and skill.

Engagement	Do you think this iVR tutorial could be useful for learning?
	Was the information provided by the application clear?
	Was the VR environment realistic?
Presence	Were the interactions with the iVR environment natural?
	Could you examine the objects from different points of view?
	Was the interaction with the VR controllers natural?
	Did the VR controllers that were monitoring the interactions distract you from assigned tasks?
	Was the interaction with the VR environment natural?
Flow	Did the VR environment respond to the actions you had initiated (i.e., picking up an object)?
	Did you perceive the actions as perfectly controllable?
	Did you know what to do in each proposed task?
Immersion	Did you enjoy the experience?
	Did you lose a notion of time because the involvement was so high?
	Did the VR experience make you feel good physically?
	Was involvement in the VR environment so high that you did not notice what was going on around you?
Skill	Did you find the VR controllers easy to use?
	Did you quickly become accustomed to the VR controllers?
	At the beginning of the experience, did the interaction with the iVR environment leave you with a good feeling?
	At the end of the experience, did the interaction with the iVR environment leave you with a good feeling?
	Was it easy to interact with the VR environment, grabbing and attaching objects the first time the computer components were placed in position (first level)?
	Was it easy to interact with the VR environment, grabbing and attaching objects the last time the computer components were placed in position (last level)?

collects the questions with which these 5 components were evaluated in the iVR experience usability and satisfaction survey.

In addition, the cybersickness level was rated on a 1-to-4 Likert-type scale where 1 indicates that the experience had to be abandoned, due to cybersickness, and 4 indicates that no cybersickness was experienced.

3.4. iVR Tutorial’s Evaluation with Final Users

Three groups of students were organized to evaluate tutorial usability and satisfaction. These three groups were chosen due to their availability as volunteers to participate in the iVR experience and their similar backgrounds with the topic included in the learning experience performed after the tutorial (an exercise on computer components). The first group was composed of Computing and Communications students following a Vocational Education and Training (VET) course; the second group of students were studying Telecommunications and Computer Systems to gain a Certificate of Higher Education (HNC); and the third group of students were following a bachelor’s degree in Videogames Design (BA). Specifically, the sample consisted of 10 students from the VET group (10% female) with an average age of 18.9 years old; 11 students from the HNC group (27% female) with an average age of 21.5 years old; and 44 students from the BA group (30% female) with an average age of 18.9 years old. All of them volunteered to participate in this experience, although it already formed part of the undergraduate classes, as they could choose the proposed iVR experience or the traditional method to learn the topic, and an extra-activity outside the learning schedule of the vocational students.

Participant pre-experience in iVR was limited. Only 25% of the 65 participants had previous experience. Specifically, 11% of the 65 participants had used HMDs and 14% had used cardboards (or any low immersive and interactive iVR devices). Therefore, the previous experience with medium-high quality iVR experiences of the test group was very limited. Therefore, that group could be considered suitable for testing the iVR tutorial and assessing the impact of the novelty effect.

Participants were organized three by three and all of them performed the whole iVR tutorial and “PC Virtual Lab” iVR learning experience. The experimental setup of the experience consisted of three workstations that were equipped with Intel Core i7-10710U, 32GB RAM and NVIDIA GTX 2080 graphics cards (California, USA). The HMDs were the HTC Vive Pro Eye with its VR hand-controllers.

Before beginning the experience, it was briefly explained to the participants what they were going to do in the iVR environment. Afterwards, the helmet was adjusted, and a calibration was performed with the eye-tracking calibration tool provided by the SDK of the HTC Vive pro eye. This calibration first detects whether the user has the helmet correctly positioned at the correct height, then detects the user’s IPD (the technician must adjust it by hand by turning the dial on the side of the helmet following the software’s instructions) and finally the user must follow the points that appear on the screen with their eyes. Eye tracking should be recalibrated for each user. This procedure ensures that all participants have their helmets properly fitted to avoid loss of immersion in the experience.

Then, participants were told to start the tutorial, followed by the iVR learning experience “PC Virtual Lab”, as tested in the research of Checa et al. [43].

The tutorial duration lasted between 3 and 7 min, and the time spent in the iVR learning experience was between 7 and 15 min, in both cases depending on the pace of the participant. Immediately after finishing the iVR learning experience, participants were invited to complete the usability and satisfaction survey. The iVR experience was undertaken at all times in accordance with COVID-19 health safety measures and the data of all participants were processed in compliance with the norms of the University of Burgos data-protection committee and current Spanish regulations. The Bioethics Committee had also approved and validated the protocol, tools, and survey before the tutorial experience and before the survey was administered (Reference Number: IR-14/2022).

4. Results

All the data analyzed in this section are collated in Appendix A included in Supplementary Materials.

The results of the usability and satisfaction survey are shown below in

Table 2. Mean (M) and Standard Deviation (SD) of satisfaction and usability survey of the three groups and for all the students treated as one single group.

Component	VET (n = 10)		HNC (n = 11)		BA (n = 44)		Average (n = 65)
	M	SD	M	SD	M	SD	M	SD
Engagement	7.97	1.87	8.82	1.45	8.27	1.86	8.32 *	1.89
Presence	7.66	2.08	8.04	1.65	7.29	2.30	7.47	2.24
Flow	7.10	1.83	7.00	1.53	6.24	1.53	6.50	1.66
Immersion	8.23	1.98	8.48	1.60	7.91	2.44	8.06	2.25
Skill	8.61	1.80	9.24	1.46	8.35	2.15	8.54 *	2.12

*: statistical difference (p-value < 0.05) between this component and the lowest component (flow).

, for which a 1-to-5 Likert-type scale was converted into a proportional scale from 1 to 10. ‘No answer’ was recorded as 0. Table 2 includes the mean values and their standard deviations for each component (engagement, presence, flow, immersion, and skill) for the three student groups and all the students were treated as one single group for statistical analysis. The highest values are highlighted in bold and statistical differences (p-value < 0.05) between components in the single group related to the low component (flow) are marked with an asterisk. Figure 8 plots the results included in Table 2 in a radar plot.

Table 2 and Figure 8 show the highest results that correspond to the HNC. Also, the component whose average has the highest rate is skill (8.54), followed by engagement (8.32), immersion (8.06), and presence (7.47). The lowest rating component is flow (6.50). In addition, the standard deviation shows a high disproportion of the three groups.

Furthermore, cybersickness was evaluated by participants. Cybersickness was rated on a 1-to-4 Likert scale. As some participants noted some of the symptoms of cybersickness (i.e., visual fatigue, headache, sweating, disorientation, nausea, and full stomach [49]), each symptom was classified according to its selected cybersickness level (in the Likert scale of the survey: 1 low; 2 medium; 3 high; 4 no cybersickness). The usability and satisfaction survey results for the four groups are shown in

Table 3. Mean (M) and average and its standard deviation (SD) results of the usability and satisfaction survey for each level of cybersickness and their Percentage Difference (PD) with the no-cybersickness group.

Component	No Cybersickness (n = 53)		Low Cybersickness (n = 10)		Moderate Cybersickness (n = 1)		Strong Cybersickness (n = 1)
	M	SD	M	PD	M	PD	M	PD
Engagement	8.46	1.78	7.30	−13.70%	9.00	+6.39%	7.33	−13.31%
Presence	7.58	2.12	6.82	−10.04%	7.60	+0.25%	5.40	−28.77%
Flow	6.58	1.52	6.43	−2.30%	6.33	−3.82%	3.00	−54.44%
Immersion	8.18	2.12	7.78	−5.11%	8.00	−2.36%	3.50	−57.28%
Skill	8.79	1.79	7.18	−18.24%	9.00	+2.43%	5.83	−33.61%
Average	7.92	1.86	7.10	−9.88%	7.99	+0.58%	5.01	−37.48%

through the mean score of each component and the percentile difference of each cybersickness group with respect to the non-cybersickness group. The average of the five components was also included for each group. These data are depicted in Figure 9 in a radar graph to illustrate these values and visualize their differences. In this figure, the values for low and strong cybersickness are normalized to the no-cybersickness group (blue line: 100% in Figure 9) and plot in terms of percentages.

As Table 3 and Figure 9 show, the strong cybersickness participant has evaluated the experience with the worst results. Flow is the most affected component (3.00) and the least affected one is engagement (7.33).

5. Discussion

As shown in Table 2 and Figure 8, satisfaction was very high in the three groups and the single group. All components, except flow, showed scores over 7. The BA group had the worst results in all components, except for engagement. A possible reason was the fact that both the VET and the HNC students had volunteered to participate in the iVR experience while the BA group was participating in the experience as part of their learning program. As Csikszentmihalyi et al. posited [50], participants are more motivated to learn in a different and new context than in traditional lessons. In addition, age may explain some of the results. The HNC group had a higher average age (21.5) than the other two groups (18.9 in VET and 18.9 in BA). This age difference might explain why the HNC group, composed of older participants, achieved the best results, a hypothesis that coincides with the study of Chen et al. [51]. In their study, they showed that the novelty effect was stronger among younger users who interacted more playfully, although they were more easily bored compared to the most mature users. Nevertheless, background noise in the data was always present, due to the small number of participants within each group.

The component with the highest score in all three groups and the single group was skill, with statistical significance related to lower flow values. Users felt that the tutorial was useful to learn and to master their actions in the iVR environment. This may be explained due to the modular structure of the tutorial at increasing levels of difficulty. Using the assistant robot in the tutorial explains the results of skill. The robot transmits encouragement and gives immediate feedback in an informal style of conversation after the user completes each task in the tutorial. This signal is for the users to let them know that the task challenges match their skill level, as Tichon explained in her research [52]. The robot also guides the user through the tutorial modules, which coincides with the research of Moody et al. [53]. In their study, they explained that the increasing difficulty and user guidance through the modules improved the capacity of users when acquiring the right skill. Furthermore, in this research and in the study of Grassini et al. [54], a high presence was shown to have a positive effect on skill development.

The component with the second-highest score was engagement with statistical significance related to the lower flow values for the single group, something which the research of Gao et al. [55] might explain. As they explain, engagement is influenced by a correct balance between challenges and skill. Related to skill, as previously explained, the feedback from the assistant robot contributes to heightening the engagement level, as explained in the study of Li et al. [56]. In addition, the novelty effect in the study of Bodzin et al. [57] was understood to have a positive effect on engagement, an effect that was also upheld in the answers to the open questions in the survey where most participants reported that the iVR learning experience was an enjoyable way of learning, something they had never experienced before. The novelty effect also increased the cognitive load, which was another factor that contributed to higher engagement levels. As Berka et al. [58] found, both workload and engagement increased proportionally with the complexity of the demands of each task. In addition, Makransky et al. [17] showed that if immersion was high, then cognitive load was also high. Nevertheless, the engagement component was not the highest in the three groups. One explanation might be participant frustration levels, which can be intensified as a consequence of difficulty when grabbing objects. Frustration might negatively affect engagement during the iVR learning experience, as Dubovi showed in her research [59] and Kahu et al. explored in their studies [60]. They found that frustration comes from an inappropriate design of certain things and from high cognitive load. Although it can lead participants to abandon the experience, if it happens within a short space of time, there is no impact on users. Therefore, the cognitive load in participants of this iVR tutorial was short-term, because it was minimized by applying the principles of Multimedia Learning Cognitive Theory. The purpose was to reduce user effort when constantly seeking to follow the next action. A combination of images and text improved the comprehension of the tasks without repeating the information. It was achieved thanks to the redundancy principle through which the information is shown in different complementary parts.

Immersion and presence were intermediate in terms of rating, and their difference was not statistically significant related to the lower flow values. Both components were closely related, so it is logical to expect them to be equally rated. Immersion contributes to high presence and vice versa. This was supported in the research of Servotte et al. [61] and Jicol et al. [62], respectively. Emotion also had a strong influence on the relationship between both components. Also, Freeman et al. [63] showed that when there is no emotion, immersion in this case has a greater influence on presence. The improvement in presence was due to arousal and immersion as both increased presence. The reason behind the contribution of arousal was because it improved attention. At the same time, the novelty effect added to arousal. In fact, the novelty effect explained why presence was rated as intermediate. At the beginning of the iVR experience, users felt the novelty effect, so their level of presence was high, but as the novelty effect slipped away over time, they lost part of their high level of presence. In this iVR tutorial, presence was influenced by the iVR environment that was a simple one, with neutral controls, in accordance with the coherence principle of the Cognitive Theory of Multimedia Learning. Users also have recognizable objects with which to interact freely, augmenting the engagement component, which is necessary to create a high level of presence, an aspect explained in the study of Huang [7]. Finally, as immersion, presence, and engagement achieve very high scores, it can be concluded that the cable connection between the workstation and the HMD does not limit significantly the comfort of the iVR experience.

The flow component had the lowest score of all three groups, which may be explained in reference to the study of Csikszentmihalyi [64]. In his study, the flow level was high when the challenge matched the skill. The results of the survey showed that skill had the highest score, so users had no difficulty with their actions. Flow was also influenced by engagement and concentration during a certain task. However, the proposed tutorial engagement level was high, which explains that the overall experience was enjoyable, but participants could not always control their actions. The answers to the open questions of the satisfaction and usability survey reinforced that idea. Most participants reported some difficulties with grabbing objects as negative aspects. This aspect must be corrected, in order to improve the flow score. In addition, prior knowledge is another factor that affects flow. Both the VET and the HNC groups were composed of participants with higher levels of computer knowledge than the BA group, as both vocational educations were fundamentally based on that field. According to the study of Huang [7], the greater the prior knowledge, the lower the cognitive load the users will have, so the flow levels will increase.

With regard to cybersickness, as shown in Table 3, it strongly reduced the satisfaction and usability of the experience, except in the case of the moderate cybersickness level, with a neutral result (+0.58% of average compared to no- cybersickness). The result of moderate cybersickness could be due to the small sample of users in this group (just one), and a mistaken self-evaluation of the user’s perceived level of cybersickness. Although the user reported moderate cybersickness, he could perform the whole experience at the same time as users with no cybersickness, raising doubts over the validity of his self-perceived level of cybersickness. Besides, the result was similar in the study of Wei et al. [65], which lends support to the idea that presence indicates success in iVR experience satisfaction, as in this case presence scored as highly as the no-cybersickness group.

In the case of low cybersickness (Figure 9), all components achieved slightly worse results than the no-cybersickness group. The average results were reduced by around 10%. Skill was the most affected component, a finding also echoed in the work of Sepich et al. [66]. Their study showed that skill and workload were positively correlated with cybersickness. Therefore, if participants had difficulty with some tasks (as the negative aspects of the survey showed), both the workload and the symptoms of cybersickness also increased. In contrast, flow was the least affected. The difficulties with interaction may explain the difference in the percentage of flow. Both the no-cybersickness and low-cybersickness groups had difficulties when interacting with the objects, so both rated flow lower than the other components.

In the case of strong cybersickness, the results were reduced by around 38% compared to the no-cybersickness group. A stronger effect was expected than low cybersickness in all the components, due to the discomfort that cybersickness generated, which meant that the user was unable to enjoy the iVR learning experience. Nevertheless, although true for four components, engagement remained unaffected (−13% from the no-cybersickness group, the same reduction as for the low cybersickness group). The worst-affected component was immersion, which coincided with the study of Salgado et al. [14]. It showed that cybersickness negatively affected immersion and task performance, which may also explain why flow was very negatively affected. The same levels of satisfaction in low and strong cybersickness groups can also be explained, because the iVR learning experience was enjoyable despite the cybersickness in both cases. Finally, although this group only included one user, his self-perception of cybersickness must be considered true, because he had to stop the experience after a while. Low skill values meant low flow values as well, due to the user having to stop the experience.

If cybersickness events are to be reduced, then the iVR tutorial needs to be improved. The design must include more visual cues to reduce text lengths. It would be helpful for some users with cybersickness who reported motion sickness when reading the text. Applying the coherence principle in more situations could also be helpful to simplify the environments and introduce the iVR world to these users at a slower pace.

6. Conclusions

An iVR tutorial was presented to avoid the novelty effect in immersive Virtual Reality. The tutorial was designed as a general introduction to an iVR learning experience. For the development of the tutorial, a design by modules was followed. These modules proposed different tasks in which the use of a certain interaction was needed, in order to cover all the basic iVR interactions. The modules were divided into introduction, basic interactions, grabbing, and complex interactions, User Interface interaction, and the explore and play level. Task complexity increased as the user advanced through the tutorial. Besides, the tutorial design was based on the following principles of Multimedia Learning Cognitive Theory (adapted to iVR): (1) the multimedia principle for graphs to facilitate understanding of the text; (2) the signaling principle to attract attention with visual cues; (3) the coherence principle to avoid distraction; (4) the spatial and temporal contiguity principles to improve understanding; and (5) the redundancy principle to complement information through different sources, rather than repeating what is shown. Finally, the personalization principle was used to adapt the information to an informal register.

The tutorial was then validated with 65 students, by means of a usability and satisfaction survey. Its results revealed high satisfaction levels. The group with the oldest participants enjoyed it most of all. The opposite occurred in the case of the group with the lowest satisfaction levels, whose iVR experience was part of their lessons, so their motivation was not so high. The highest scores were for the skill component, because participants found the tutorial useful to overcome all the challenges proposed in the iVR experience. In contrast, the flow component had the lowest ratings because the participants encountered some difficulties with certain complex actions during the experience. These results assure that the tutorial is useful to lose the novelty effect, acquire skill, and enjoy the experience, as well as making them familiar with the iVR environment.

The data on cybersickness showed low levels of affection. However, the more acute the symptoms of cybersickness, the worse the effect on user performance. If the user had strong cybersickness, the tutorial was of little or no use, which drastically reduced the scores of all the components, especially immersion and flow, although the utility of the tutorial was not significantly affected by low levels of cybersickness, having no effect on flow and immersion and only a low effect on engagement, skill, and presence. Besides, those limited cases might be unavoidable due to the nature of the tutorial that involves movement, exploration, and interaction. Finally, despite those effects, engagement was not particularly affected in both cases and participants enjoyed the experience, underlining the successful design of the tutorial.

As future lines of research, the inclusion of this tutorial in an iVR learning experience and the extension of the performance indicators to new indicators based on objective measurements (e.g., EEG, EMG, eye-tracking…) could be studied to outline its benefits for student learning performance. In addition, its use in other sectors, such as industrial training and XR formats, such as web- or mobile-based AR, might be also desirable, as well as an enlargement of the sample to balance the number of participants in all the groups. In addition, the modular design of the framework used for tutorial development will ensure future compatibility with new HMDs and mixed reality devices. It also allows the integration of more natural forms of interaction, such as hand tracking and even the possibility of using full-body avatars. Likewise, the use of artificial intelligence technologies for the data processing of user performance will mean the full adaptability of the tutorial to the pace of user learning.