Asymmetric Interface: User Interface of Asymmetric Virtual Reality for New Presence and Experience

Abstract

This study proposes an asymmetric interface that can provide head-mounted display (HMD) and non-HMD users with improved presence and an experience consistent with the user’s environment in an asymmetric virtual reality. For the proposed asymmetric interface, a controller-based hand interface is designed for portability, easy and convenient use, and high immersion. Subsequently, a three-step decision-making structure that supports accurate and efficient decision-making is defined based on the asymmetric experience structure of each user (HMD and non-HMD). Based on this process, an optimal interface that distinguishes between HMD (direct interaction) and non-HMD users (multi-viewpoint interaction) is implemented. With the objective of surveying and analyzing each user’s experience along with the presence provided by the proposed interface, an asymmetric virtual reality application is developed directly, and an experiment is conducted with the participants. Finally, it is statistically analyzed and verified that the use of the proposed asymmetric interface can provide optimal presence and user-optimized experience to both HMD and non-HMD users.

1. Introduction

Virtual reality (VR) technology provides an experience similar to reality to a user existing in a virtual environment created by a computer [1,2]. Through the advancement of head-mounted displays (HMDs) such as Oculus Rift CV1/Go and HTC Vive, the VR technology provides a more deeply immersive experience environment by combining an HMD with a hardware system such as a treadmill and VR globe. Furthermore, application studies have been conducted from various perspectives for user interfaces and haptic feedbacks in an immersive VR to interact with the virtual environment directly and control objects realistically [3,4,5,6].

An objective of immersive VR is to provide a user with a realistic experience based on the five senses regarding which actions are performed, with whom and where. Accordingly, studies were conducted on the following aspects: Displays that send stereoscopic information; auditory sense using volumetric audio sources; and tactile sense felt through a haptic system, which directly feeds a physical reaction back to a human body (hand, foot, etc.) [7,8,9]. To increase the presence in a VR by inducing a high user immersion, a realistic interaction between reality and the VR is required. To design this, the following processes should be performed: Detecting the movements of joints in a human body existing in the real space; analyzing the intention of movement and reflecting it in the virtual environment [5,10]. Consequently, studies [11,12,13] have been conducted to accurately map actual hand movements to the movements of a virtual hand model through markers (surface or optical). Accordingly, studies were conducted to develop a method of calculating a hand model precisely through a spherical-mesh tracking model [14], and a method of realistically expressing the actions of a person including hand gestures, facial expressions, and whole body movements in VR by using motion capture data [5]. Moreover, the following studies were conducted to facilitate infinite walking in limited space concerning movements of the user: A study on smooth assembled mapping that calculates walking that feels real with an isometric distortion [15], and a study that proposed a representation method of natural walking in the process of multiple VR users sharing spatial information [6]. Lee et al. [16] represented movements in VR with a straightforward method by designing a portable walking simulator using a user’s walking-in-place with a more universal approach than the conventional studies.

For the most immersive VRs, algorithms, systems, user interfaces, and interactions are designed and studied by focusing on the presence felt by VR users. Typical examples include: Precisely controlling the strength applied on the tip of a finger by using wearable haptic devices such as 3-RSR (revolute–spherical–revolute) [17] or 3-DoF [18]; and a user-friendly haptic system [8] that can be easily carried around while accurately mapping the user’s walking or action in a virtual environment through a motion capture device. However, the roles desired by the users existing in the VR or the level of their participation may vary (some may want to only observe while others may want a limited participation or close participation as a part of the experience). Therefore, rather than dwelling on the immersion in the hardware aspect, studies on immersive VR should consider various aspects such as roles and communication through social interaction.

With respect to a virtual collaboration environment, interaction methods and technology are proposed, whereby many users existing in a virtual environment can collaborate through communication in a given condition and environment [19,20,21,22]. For the virtual collaboration environments of immersive VRs, systems are designed to create immersive environments where multiple users wearing HMDs can experience immersion together. Related studies proposed interactions that can be used by multiple VR users to work on a group task together or communicate in an immersive environment; furthermore, they proposed, as a more advanced type, an asymmetric VR to allow non-immersive users (usually PC users) to participate together. ShareVR and Role VR proposed by Gugenheimer et al. [23] and Lee et al. [24], respectively, are representative studies on interactions in an asymmetric virtual environment. They proposed asymmetric interactions to provide experiences that satisfy all the HMD and non-HMD users located in the same space, in addition to an improved presence in an immersive environment. However, conventional studies have focused on VR users wearing immersion devices such as HMDs and have limitations in that only limited participation is possible for non-HMD users as assistants or spectators. Moreover, users cannot act as independent entities and they share an avatar. Consequently, there is a limitation that non-HMD users become dependent on HMD users for receiving improved presence and various experiences in a virtual environment.

This study proposes an asymmetric interface that provides non-HMD users with convenient, efficient, and immersive interactions with HMD users in an asymmetric VR. The key aspect is to provide high immersion while allowing any person to interact with others conveniently and to provide non-HMD users with user-optimized experience and improved presence as independent participants rather than as assistants to HMD users. Accordingly, a decision-making structure is designed to remind users continuously that they are sharing and experiencing together their experiences and thoughts. The main contributions of the proposed asymmetric interface are:

• A controller-based hand interface is designed to maximize the immersion of every user in VR while minimizing the user burdens. A real-hand-to-controller mapping method is developed in such a way that a user can experience interactions with users or a virtual environment by directly using a hand.
• A three-step decision-making structure (object, status, and message) is designed to allow users to share the thoughts, behavior, and emotions experienced by them in an asymmetric environment through an efficient and intuitive structure.
• It is systematically verified that providing a user-optimized interface in an asymmetric VR can provide even non-HMD users with improved presence similar to that provided to HMD users, and also with various experiences in different environments (immersive and non-immersive).

2. Related Works

2.1. Presence in Immersive Virtual Reality

Studies on immersive VRs are conducted through various approaches such as interaction, user interface, haptic system, and motion platform so that users can be provided with experiences that cross the boundary of virtual world and reality based on the five senses. Here, a user feels the presence by perceiving the VR as reality, and it is an important factor in implementing an immersive VR. To experience a high presence at a level that it will make it difficult to distinguish reality from the VR, a highly immersive environment should be provided for the VR. To this end, it is necessary to develop interaction and hardware technology to increase the immersion of users, and various approaches are required from social and cultural perspectives.

Various immersive VR studies focusing on presence have been conducted, ranging from display technology (HMD, 3D monitor, etc.) providing realistic experiences in a space by delivering 3D visual information to methods that can increase immersion in terms of hearing by using a spatial audio or interacting directly and realistically with a virtual environment [7,13]. Nowadays, mobile platform HMDs, such as GearVR, and PC-based high-resolution HMDs such as HTC Vive have become popular. Thus, it is no longer unusual to experience 3D visual information in an immersive virtual environment. To provide realistic interactions between virtual and real words based on such visual satisfaction, studies (based on marker, Kinect, leap motion device) are ongoing for representing realistic movements and actions in a virtual environment by detecting and tracking the movements of joints in a body quickly and precisely, and through this, recognizing and analyzing actions or gestures [3,11,12,25]. In addition, the immersion of users is enhanced by processing tactile senses delicately through a haptic system, which feeds back physical reactions occurring in a virtual environment or interaction between users, such as electrical actuator and wearable haptic devices [8,17,18,26,27]. Such studies developed into multi-modality studies that satisfy tactile and hearing sensations or tactile and visual sensations simultaneously [9,28]. Furthermore, with the objective of overcoming the spatial constraints of experience, studies were conducted to facilitate infinite walking in a limited space such as a flexible space [29].

Pivoting around Slater et al. [30], studies related to presence have been conducted widely, focusing on cases from various perspectives. The presence for walking of a user was classified into the cases of gaze, navigation, optical motion tracking device, etc., and compared [31]. Moreover, the effect of gaze on the presence in communications between users was analyzed [32]. Such studies confirmed that differences in perceptions and actions may occur depending on the condition and environment where VR users exist [33], and they developed into studies that analyzed the relationship between VR and users through the comprehensive approaches of psychology, neural science, social phenomenon, etc. [34]. Recently, studies have been conducted to analyze the effects of interactive methods and passive methods comparatively, with respect to learning activities in VR based on specific case studies [35]. Furthermore, studies were conducted to analyze the factors that enhance presence when interacting directly with a virtual environment or object by using gaze, hand, etc., based on the user interfaces [3,4,36]. However, a majority of studies on immersive VR focused on techniques that increase the immersion of a VR user wearing an HMD or the analysis of presence with respect to these techniques. In other words, there are very few studies on the methods for the coexistence and interaction of immersive and non-immersive VR users based on presence in VR. Therefore, this study aims to design an immersive VR optimized for the experience environment of users in the asymmetric VR based on the existing interaction studies that focus on the presence of users in existing immersive VR. To accomplish this, the experience environments of the asymmetric VR classified into HMD and non-HMD users are specified with respect to the mechanical environments of immersive (HMD) and non-immersive (monitor) devices and the role environments of manager, assistant, and participant. An asymmetric interface focusing on the presence and experience of users is then proposed.

2.2. Interaction of Collaborative Virtual Environments

Studies on presence in immersive VR were extended to virtual collaboration environments where multiple users exist. Through these studies, it is confirmed that immersion can be induced in the process of communicating or sharing thoughts and emotions between multiple users, and the presence can be enhanced by providing various experiences. Virtual collaboration environments originated from a study on distributed interactive virtual environments that defined the interaction between a user and environment or between users [37]. Subsequently, studies have been continuously conducted with the objective of providing an immersive environment for distributed collaboration [38]. Furthermore, various application studies have been conducted: A study on the comparative analysis of remote tasks, which can be effectively collaborated by multiple users while displaying creativity when designing a building in a virtual environment [39]; a study on the analysis of awareness process in a group task of multiple users existing at the same location [21]; a study proposing a technology and method for collaborative learning in VR [40]; and a study on the evaluation of user experiences for human representation in a virtual collaboration environment [41].

Such a virtual collaboration environment was further developed into an asymmetric VR where non-HMD users as well as HMD users can participate together and perform actions. Considering the example of a study that proposed an effective interaction between VR users wearing HMDs and PC-based regular users [42], asymmetric (immersive vs. non-immersive) experience environments were classified and appropriate methods were provided. From a similar perspective, Duval et al. [43] proposed an asymmetric interaction method between HMD users and PC users and an interaction method between common users and users wearing augmented reality HMDs, but not of VR. Studies on such asymmetric VRs defined the roles and actions of HMD and non-HMD users by classifying them more clearly. In Dollhouse VR [44], while an HMD user performs an action of exploring a house by walking around in a virtual space, a non-HMD user, as a designer, creates or edits the virtual space in real time, thereby providing a changed environment to the HMD user. The studies that use non-HMD users in supporting roles to maximize the presence of HMD users include Haptic Turk [45] and TurckDeck [46]. Gugenheimer et al. [23] facilitated more active actions for non-HMD users, compared with the conventional studies, through ShareVR and provided an experience environment where all users (HMD and non-HMD) existing in the same space can have satisfying experiences based on the asymmetric interactions. RoleVR proposed by Lee et al. [24] was developed to provide an enhanced presence to every user through optimized immersive interactions based on the roles divided between HMD and non-HMD users. Applications related to asymmetric VR, such as Ruckus Ridge VR Party [47] and Playroom VR [48] were released in the market and used by many people. Moreover, like Maze Commander [49] which uses Sifteo Cube, studies have been conducted to develop asymmetric VR games. However, because non-HMD users use keyboard, mouse, or gamepad for these applications, the immersion is extremely limited in the VR perspective. Asymmetric VRs can induce immersion and experience, which are different from those of immersive VRs focusing on HMD users. Notably, an asymmetric VR requires an environment where a non-HMD user does not depend on an HMD user and directly participates and immerses in the VR. Therefore, this study attempts to investigate generalized interaction in an asymmetric VR so that, without favoring HMD or non-HMD users, every user can be provided with an enhanced presence and his/her own experience in an independent environment by immersing in the VR according to his/her own system and experiential environment.

3. Asymmetric Interface for Immersive Interaction

This study proposes an asymmetric interaction to provide an experience environment where HMD users and non-HMD users can have diverse experiences with high presence in asymmetric VRs. The proposed asymmetric VR is based on an experiential environment comprising asymmetric interfaces for both HMD and non-HMD users. However, interaction and interfaces are designed in view of extensibility to a large number of users. The experience space between users considers both co-located and remote users. However, we aim to design interfaces that can effectively share statuses, thoughts, and behavior, focusing on remote users who are difficult to communicate with directly.

The proposed asymmetric interface includes the following three parts: A controller-based hand interface, which can be used conveniently at a low cost for a highly immersive interaction, a three-step decision-making structure where users can share experience by efficiently and accurately exchanging information and thoughts between themselves, and an interface consistent with the experience environment of the user. In this study, Oculus Rift CV1 HMD and a dedicated controller are used for the basic experience environment, and an integrated development environment is built on the Unity 3D engine. The Oculus integration package that can be used in the Unity 3D engine is imported, and the HMD camera is controlled using the prefabs (OVRCameraRig, OVRPlayerController, etc.) provided in the package. Furthermore, the touch controller used by HMD and non-HMD users for interaction is controlled through the OVRInput. In addition, the UI functions of Unity 3D engine are used for texts, icons, etc., in the communication process.

3.1. Controller-Based Hand Interface

Users can directly use their hands to interact with a virtual environment or objects in an immersive VR. The proposed asymmetric interface also provides an interaction environment where all users (HMD and non-HMD) can use their hands. Previous studies by Han et al. [3] and Jeong et al. [50] also confirmed that interactions using hands in immersive VRs provide higher presence compared with a gaze interface as well as a keyboard and game pad. Kim et al. [8] added a portable hand haptic system to maximize the presence in interactions using hands. Notably, existing studies emphasize that hands should be expressed more accurately and directly in a virtual environment and an easier and more convenient interface is required. However, most studies recommend the use of a lip motion device in addition to an HMD and some even require a haptic device. This study proposes a VR controller-based hand interface to provide a more popular and easier-to-use interaction environment while maintaining the highly immersive input method of using hands.

Three basic motions are defined to map the VR controller naturally with hand gestures according to the input settings of the controller. The key aspect is that the hand motions required to press a button or a trigger while holding the controller must correspond closely with the gestures of the virtual hand to maintain a high degree of immersion. Figure 1 shows the process of mapping Oculus touch, which is a dedicated controller, with a three-dimensional virtual hand model that contains joint information in the Unity 3D development environment. The key inputs are set, and the left and right hands are operated separately to map the defined basic motions (grab, point, and open) naturally with the controller. Algorithm 1 summarizes the proposed controller-based hand interface.

Algorithm 1 Design of controller-based hand interface.

1: Btn_Grab ← click state of controller’s index trigger button.

2: Btn_Point ← click state of controller’s middle trigger button.

3: procedure CONTROLLER To HAND INPUT PROCESS Btn_Grab, Btn_Point

4: grabbing ← check the state of grabbing hand.

5: pointing ← check the state of pointing gesture.

6: opening ← check the state of open hand.

7: if Btn_Grab == true then

8:     if grabbing == false then

9:         search an object to grab near the hand

10:       after collision test, grabbing by the hand

11:       store in the grab-object list

12:       grabbing = true

13:     end if

14: else if Btn_Point == true then

15:     set as a pointing gesture.

16:     pointing = true

17: else

18:     if grabbing == true then

19:       angular velocity calculation of drop object.

20:       drop the object grabbed by the hand.

21:     end if

22: end if

23: end procedure

3.2. Three-Step Communication Process for User-to-User Interaction

For effective communication between users, we consider the following six factors: Listening, verbal communication, non-verbal communication, emotional awareness, written communication, and communicating in difficult situations. This study designs a communication process to present an environment where users can share their statuses, emotions, etc., through faster and more intuitive communication (both verbal and non-verbal) in a remote experience space excluding listening. To optimize for communication, a structure is designed through a three-step communication method that considers both a non-verbal method using objects and icons, and a verbal method based on text as ways to express the current situation and status as directly as possible. By extending the current three-step process, we aim to design a process with an open structure that converts voices into icons and messages.

In an asymmetric VR, both HMD and non-HMD users interact with the virtual environment or objects using their hands. We propose a communication process that can increase immersion in VR by allowing users to share differences in experience caused by the system, role, and behavior. A three-step process-structure is designed for intuitive and accurate communication between users. This communication process can quickly and accurately deliver objects, statuses, and messages to other users in an intuitive structure under the assumption that the sense of space in VR is used to the maximum. Kim et al. [4] demonstrated that setting different dimensions (2D or 3D) in the process of selecting objects or exchanging information affected the understanding of information or immersion. Therefore, this study also defines an interface for communication by setting a dimension optimized to the type of information to be delivered and the method of communication.

Figure 2 shows the proposed flow of interaction between users. Presence is improved through collaboration between asymmetric users sharing experiences (thoughts, behavior, states, etc.) based on the differences in their independent VR environments.

The communication process is composed of three steps: Object, status, and message. The first step involves gripping various objects with the hand and delivering them to another user in a VR space. The second step involves communicating one’s status, emotion, etc. Similar to social networking services (SNS), this step uses emoticons which are often used to communicate messages directly on the Web. Finally, the message function involves accurately communicating requirements or core contents. This is not to type texts directly, but to auto-complete messages based on the two functions of object and status. Figure 3 outlines the communication process of the three-step structure. HMD and non-HMD user interfaces transfer information (environment and space, object, etc.) through objects and express their current status and emotions. In addition, they have a communication structure to send and receive simple messages by combining them.

3.3. Development of User-Optimized Interface

In an asymmetric VR, there are differences in the methods and process by which HMD and non-HMD users can participate. In other words, a user-optimized interface should be provided by analyzing the system and environment factors under the assumption of basic interactions using the hand. Figure 4 outlines the optimized roles and interaction methods based on an analysis of the differences in the experience environments between the users in an asymmetric VR. For HMD users to experience presence through high immersion as participants in VR, an environment for direction interaction must be provided in the virtual environment through an intuitive structure. non-HMD users have limitations as they must start from low presence owing to a non-immersive environment. However, they can overcome this limitation if various roles (manager, assistant, and participant) and various viewpoints (first-person and third-person) are provided as they can judge the situation as a whole by viewing the scene more broadly than HMD users do (limited view).

3.3.1. Direct Interaction for HMD User Interface

HMD users are provided with a high sense of space through stereoscopic visual information. Therefore, we design an immersive interaction that can communicate more directly with a virtual environment, objects, and other users in it. Thus, a graphical environment is implemented so that HMD users can interact directly with objects or environments using two hands and draw statuses and messages based on the local coordinates in the 3D space.

In an immersive VR, HMD user interface is provided with a high sense of space through first-person stereoscopic video transmitted to an HMD that systematically consists of a binocular camera. Therefore, an experience environment that allows users to perceive the virtual environment directly and control various objects in it intuitively is required. Figure 5 shows the proposed experience environment for HMD users, which enables users to perform various behavior and object control directly through hand motions in a virtual environment with a controller-based hand interface.

In the case of a communication process for HMD users, the user-to-user interactions are also designed around the hand because it is possible to interact with the environment and objects directly using the hand in VR. Figure 6 shows the result. First, users can deliver objects to other users by directly holding and throwing them with their hand. When users click a three-dimensional icon, which is activated around the hand as they hit a defined key with their index finger, an icon is generated above their head, which communicates the status. Finally, when they select the object and status, the message is auto-completed and delivered. HMD users enter the values in a 3D space in all the three steps. However, it is important for the non-HMD user interface that receive the information to recognize the delivered information quickly and accurately with a high degree of immersion. Therefore, it is implemented in such a manner that objects are delivered in 3D with their 3D shapes, status is delivered via 2D emoticons, which can be understood quickly and accurately [4], and messages are converted to 1D texts, which are highly legible.

The auto-completion of messages is to predefine sentence combination templates and complete a sentence by finding and substituting a sentence that matches the object and status selected by the user. As shown in Figure 7, the dialogue sentences often used in conversations or required in the application to be produced are listed abstractly in advance. The sentences are expressed by a combination of object (O) and status (S), and template sentences are predefined by considering various combinations such as object and object, object and status, and status and object. Subsequently, when the user selects (input) an object and status, a sentence is completed based on the selected information, and candidates are listed, among which the user can select the most ideal sentence (auto-completion). The message auto-completion feature is provided to non-HMD users as well.

Equation (1) represents the process of finding the appropriate message according to the input based on the predefined sentence templates. First, an integer value is mapped according to the information selected by the user (object (O): 1, status (S): –1). Then, the messages are only found to match the input parameter (I) with all the added values and the templates (T). Then, the function finds only the sentences that match the templates (T) with the sum of all the input parameters (I). That is, only the messages with the exact combination of the input values and the template are found, and the selected information is substituted and displayed to the user. Where n is the number of user-selected combinations. In this case, the order in which the messages are arranged is determined by raising the weights on the sentences which are frequently selected by the user.

$$\arg \min (|T-\sum _{i=1}^{n}I(i)|)$$

(1)

3.3.2. Multi-Viewpoint Interaction for non-HMD User Interface

Unlike HMD user interface, non-HMD user interface is present in a non-immersive experiential environment that receives visual information through a flat display such as a monitor. Therefore, to utilize this limitation as a potential advantage, we employ non-HMD users in various roles such as participant, manager, and creator, and design the supporting viewpoint, interaction, and communication. The aim of this study is to extend beyond the general presence of “being there” to the presence of “being there together” by presenting an experiential environment where non-HMD users can experience the thoughts and experiences of HMD users. This enhanced presence of non-HMD users overcomes the limitation of a non-immersive experiential environment for non-HMD users by allowing them to participate in an experiential environment as subjects just like HMD users, rather than simply acting as assistants to HMD users. Thus, we design an interface structure for non-HMD users that provides multiple viewpoints freely switching between the first- and third-person and enables interaction and communication by directly utilizing the hands just like that for HMD users.

For users to experience the presence of VR in a non-immersive VR, 3D visual information is provided through displays such as a 3D monitor or a cave automatic virtual environment [51]. However, these displays are different types of devices that replace the HMD and require considerable burden (economic and spatial) to be used as displays for non-HMD users in an asymmetric VR. Therefore, the key objective of this study is to propose a presence for non-HMD users as much as or better than the presence for HMD users and an interaction that can provide new experiences for HMD users under the assumption that non-HMD users participate in a general non-immersive environment using a PC.

From the viewpoint of non-HMD user interface, this has the advantage of observing and exploring scenes from more diverse viewpoints than an HMD user interface do. By reflecting this advantage in the interface, not only diverse roles (manager, assistant, and participant), but also diverse viewpoints can be provided. When the same first-person viewpoint as that of an HMD user interface is provided to a non-HMD user, they can participate and behave together in the virtual space as participants. Sometimes, they can play the role of an assistant to HMD users by operating a camera and judging the situation quickly. If a function for selecting a scene from a third-person viewpoint is provided, they can additionally design their roles as assistants. Considering these aspects, Figure 8 shows a structure where non-HMD users can experience a non-immersive environment through a controller-based hand interface.

For first-person non-HMD user interface, an interface where they can interact as participants or as assistants to HMD users in VR is designed. However, unlike for an HMD user interface, there is no sensor to trace the 3D orientation of the VR controller; hence, the camera is fixed at an appropriate position where the hand can be present by assuming the location using the eye. Considering the limited environment where the hands of the non-HMD user are not free, a function for selecting and controlling objects through the index finger is additionally provided. Figure 9 shows the process of expanding the controller-based hand interface and selecting objects from the first-person viewpoint of a non-HMD user interface.

In addition, a multi-viewpoint function that allows free conversion between first- and third-person observer viewpoints for non-HMD users is designed, and a manager role or different types of assistant roles are provided together. Thus, it is possible to judge every situation by perceiving the VR scenes, including the behavior and motions of HMD users, as a whole and to communicate the situation to HMD users through direct behavior; it is also possible to affect the flow of the VR scenes. Kim et al. [52] proposed third-person immersive interaction based on the god-like interaction suggested by Stafford et al. [53]. Furthermore, they verified that an interface optimized for the third-person viewpoint can provide a high degree of presence, as much as the first-person viewpoint and experiences optimized to the third-person viewpoint. This study designs multi-viewpoint interaction under the assumption that a third-person interface provides non-HMD users with new experiences in asymmetric VR. Furthermore, the effects of the multi-viewpoint interface on presence and experience are systematically analyzed through a survey.

When a controller-based hand interface is applied to the third-person viewpoint, it can include more functions than the three motions defined in Algorithm 1. Therefore, functions related to camera control (camera movement, view volume setting, viewpoint conversion, etc.) need to be applied in addition to the basic functions. Algorithm 2 outlines the functions added for third-person non-HMD user interface based on Algorithm 1. Figure 10 shows the results of free camera control from the third-person viewpoint through the algorithm.

The communication process is designed separately for non-HMD users because first- and third-person multi-viewpoints are provided. For the first-person viewpoint, the basic process is the same because it is the same as the viewpoint of an HMD user interface. Figure 11 shows a communication process for the first-person non-HMD user interface. If the function for delivering object, status, and message is the same as that for an HMD user interface, a method of selecting through a ray, which is calculated from the index finger, is added.

Algorithm 2 Input processing for third-person viewpoint camera control of non-HMD user interface.

1: Procedure 3rd-PERSON VIEWPOINT CAMERA CONTROL PROCESS

2: if grabbing == false then

3:     if pointing == true then

4:       if the current hand is the left hand then

5:         calculate the moving direction vector by subtracting the current position from the index fingertip position.

6:         move the camera to the calculated vector direction.

7:       else

8:         create the ray in the forward direction of fingertip.

9:         calculate the collision of the ray and predefined layers

10:       if click the B button on the controller then

11:           activate the non-HMD of 1st-person viewpoint with the collision coordinate.

12:         end if

13:       end if

14:     else if opening == true then

15:       if the current hand is the left hand then

16:         if click the Y button on the controller then

17:           zoom in camera

18:         else if click the X button on the controller then

19:           zoom out camera

20:         end if

21:       end if

22:     end if

23: end if

24: end procedure

In the case of third-person non-HMD users, the observation is undertaken from the top view, and a flat image is provided compared with the first-person viewpoint; thus, every communication process is expressed in two dimensions. However, the interface for using the hand is the same in this case.Figure 12 shows a communication process from the third-person viewpoint of a non-HMD user interface. The menu (object, status, and message) is enabled and selected based on the key input through a controller, and it also shows the step-by-step communication process. In the first menu, Objects, the selectable objects are output as 2D icons via coordinates on a circumference around the right hand. Furthermore, information is delivered via selection through the tip of the right index finger. The HMD user receiving the information converts the 2D icons to matching 3D objects so that 3D object information can be perceived realistically. The status is also processed in the same manner as objects, but the status is checked with 2D emoticons for HMD users. Finally, all the messages are transmitted as 1D texts.

4. Application

4.1. Design

The proposed asymmetric interface provides a controller-based hand interface and a communication process to provide a new experience and improved presence by considering the system and environment differences for each user (HMD and non-HMD). In this study, an experimental application is developed for experiencing and analyzing the presence provided to each user by the proposed interface in the asymmetric VR. In conventional VR applications, such as Rucks Ridge VR Party [47] and Playroom VR [48], there are methods where an HMD user and several non-HMD users experience an environment together or an HMD user and a non-HMD user participate together. Moreover, various participation types or proceeding methods are provided, e.g., users cooperate for a common goal or a user hinders actions of an opponent for his/her own goal. However, while an effort was made to increase the presence in the VR for HMD users, only simple interactions are provided to non-HMD users, i.e., using a keyboard or gamepad in a regular PC- or console-based experience environment. In other words, relatively speaking, there is a lack of effort for providing unique experiences and an enhanced presence comparable to that of HMD users to non-HMD users. Therefore, the asymmetric VR application of this study is developed as follows: A goal is provided and in the process of accomplishing this, HMD and non-HMD users perform actions based on their respective roles and share the experiences by exchanging information. The developed application has a limitation in generalizing the impact of the proposed asymmetric interface on the presence and experience of users. However, the design process is proposed in such a manner that the functions of the proposed asymmetric interface can be applied practically for various objectives (education, industry, entertainment, etc.) considering the system environment of the use (HMD or non-HMD), and role (participant, manager, assistant, spectator, etc.).

4.2. Flow and Role

The progression flow of the application is designed in a structure to accomplish the goal by exchanging information between users based on different roles. Four places, i.e., living room, bathroom, bedroom, and kitchen, are configured, and the users are asked to match the given words in a similar to that of the Scrabble board game. Here, an HMD user performs the role by searching for an object corresponding to the correct answer in the VR space and grabs it with a hand. The roles of a non-HMD user are divided according to viewpoints. In the case of third-person viewpoint, with the roles of observer and assistant, the non-HMD user views the hint (the number of letters, the first letter of the word) for the word and delivers the information required for finding a correct answer to the HMD user while observing the movements of the HMD user. Furthermore, in the case of first-person viewpoint, by going into the VR space directly, the non-HMD user can directly perform actions to find the correct answer as a participant alongside the HMD user. However, limits are imposed on the opportunity and time provided for participating in the VR space with a first-person viewpoint to induce fast decision-making. Figure 13 classifies the roles and actions of users and the progression flow of the proposed asymmetric VR application.

Figure 14 shows the results of the application developed for the experiments, and the images were captured by classifying the users during the execution of the application. Here, for accurate communication between the users connected via a network, a fast and accurate synchronization process is required. This study uses UNet (Unity Network) provided in the Unity 3D engine development environment. Using this, a developer can access the commands that can satisfy the communication requirements between users even if low-level specifics are not considered. In this study, the conversion information of objects is accurately synchronized by using such functions as command and client RPC as well as the attributes provided by UNet. As shown in the right-hand side images in Figure 14, it can be confirmed that, regardless of the viewpoint of HMD and non-HMD user interfaces, the conversion information for the same object is accurately synchronized. The number of frames per second (fps) is recommended to be a minimum of 80∼90 fps, and the number of polygons in a rendered scene a maximum of 2 M to use a virtual scene of immersive VR application as contents without VR sickness [2,16]. The HMD user of asymmetric VR may otherwise feel dizzy by just looking at a virtual scene due to technical problems such as screen delay and cognitive dissonance. For recording various scenes for the proposed application, the number of polygons is 197.3 K minimum and 2.1 M maximum, maintaining 315.8 K on average. Furthermore, the number of frames per second is 198.2 fps maximum and 84.9 fps minimum with the average of 149.8 fps, confirming that the hardware-based problem is not triggered.

5. Experimental Results and Analysis

The application developed for the experience environment and experiments of HMD and non-HMD users (both co-located and remote) based on the asymmetric interface in the VR was implemented by using Unity 2017.3.1f1 (64-bit) and Oculus SDK (ovr_unity_utilities 1.22.0). Furthermore, every user interacted with the VR environment or objects by mapping the Oculus Touch controller to his/her hands, whereas the HMD user received the video information through the Oculus Rift CV1 HMD. The PC used for the integrated development environment construction and experiment had the following specifications: Intel Core i7-6700, 16 GB RAM, and Geforce 1080 GPU.

Figure 14 shows images, from the developed application, of users performing the given roles and actions to accomplish the objective of the application based on the interactions classified according to the characteristics of users. In the images, the HMD user, as a participant, performed active actions based on the 3D visual information, and the non-HMD user performed various roles and actions by alternating between the third-person viewpoint (observer or assistant) and the first-person viewpoint (participant or assistant). Subsequently, in the configured experience environments, the users performed independent actions in a space (approximately 1.5 m × 1.5 m) where they could perform actions freely by using their hands while staying at the same spot. It is possible to perform actions while standing up or sitting down. Users can participate in the VR remotely, and co-located participation is also possible. In the case of co-located HMD and non-HMD users, the non-HMD user can watch the display screen of the HMD user together while a sufficiently large participation space is provided (Figure 15). In this case, as the experience of the HMD user is indirectly experienced by the non-HMD user, the presence of the non-HMD user is improved [24].

The essence of the proposed asymmetric interface is that all the users experiencing the asymmetric VR application can have new or satisfying experiences classified according to the experience environment while feeling similar presences. For a systematic analysis regarding this objective, a survey was conducted with the participants. First, the survey participants consisted of a total of 20 persons (males: 15, females: 5) between the ages of 22 and 37. Furthermore, the asymmetric interfaces classified into HMD and non-HMD users and their subsequent experience environments are set up as independent variables, while the presence, experience, and social interaction as dependent variables. First, as it is an important issue in this study to analyze the presence and experience between the HMD and non-HMD user interfaces comparatively, two persons (an HMD user and a non-HMD user) were paired up and the experiences were undertaken with ten teams with the total of 20 persons. Here, notably, the participants experienced both the situations of wearing and not wearing the HMD and answered the survey questions. To design the survey experiment process specifically, first, this study analyzed the experience differences of HMD and non-HMD user interfaces in the system and environment aspects. Based on this, the interactions were defined, suitable roles and actions were provided, and each user had the experiences by classifying them.

For non-HMD users, an interface to expand the breadth of roles and behavior through multi-viewpoint was designed as a way to overcome differences in the non-immersive experience environment. Therefore, the experience environment was designed such that the interface of a non-HMD user could be compared between a single viewpoint (third- or first-person) and the proposed multi-viewpoint. In RoleVR proposed by Lee et al. [24], when the user participating in the asymmetric VR was a co-located user, the first-person viewpoint was replaced by sharing of the monitor screen of the HMD user with the non-HMD user; through the experiment, such an experience environment improved the presence of the non-HMD user. Considering this, the first-person viewpoint was additionally provided in this study as the experience environment of remote users was considered. Accordingly, experiments were performed to analyze the relationships of presence and role comparatively in various viewpoints.

The first experiment involved the analysis of the presence comparison questionnaire. The interactions were designed with the objective that all the users (HMD and non-HMD) existing in the VR can feel high presences. Therefore, a survey was conducted to investigate this aspect. This study analyzed the user reactions for the presence in various angles by using the presence questionnaire proposed by Witmer et al. [54].

Table 1. Comparative analysis result of presence between the HMD users and non-HMD users in the asymmetric VR. * mark indicates statistical significance.

Factors	Mean	SD
HMD	6.224	0.184
non-HMD (3rd and 1st)	6.191	0.229
non-HMD (Only 3rd)	5.084	0.416
non-HMD (Only 1st)	5.844	0.413
Comparison between users
non-HMD (3rd and 1st) vs. HMD	F(1,38) = 0.113, p = 0.741
non-HMD (Only 3rd) vs. HMD	F(1,38) = 48.962, p < 0.001 *
non-HMD (Only 1st) vs. HMD	F(1,38) = 4.871, p < 0.05 *
Viewpoint type of non-HMD	F(2,57) = 24.101, p < 0.001 *

shows the results of questionnaire values recorded by the users. The participants who experienced the application developed in-house experienced both the role of an HMD user and of a non-HMD user; the mean values were 6.224 (SD: 0.184) and 6.191 (SD: 0.229), respectively, showing that similarly high values were recorded in the survey. Furthermore, for calculating the normality for the experience environments, respectively, through D’Agostino’s K-squared test [55], the significance probability (p-value) for HMD and non-HMD (third and first) users was 0.071 and 0.408, respectively, thereby confirming that the null hypothesis could not be rejected and normal distributions were followed. Through a comparison with the case where the non-HMD user’s viewpoint was fixed, it was noted that the inability to utilize the viewpoint had a direct impact on the presence. It is an evident that when a non-HMD user interface has a first-person viewpoint, the presence is high. It is observed that the visual information must become a factor of the highest priority for increasing the presence of a user. Moreover, by providing various roles to a non-HMD user through a new viewpoint, the range of participation can be expanded, which is an important factor for increasing the presence. A study by Denisova and Cairn [56] confirmed that the first-person viewpoint where the world is viewed through the eyes of a character increased immersion compared with the third-person viewpoint, regardless of the preferred viewpoint of the users. However, when interaction and an experiential environment optimized for the third-person viewpoint were presented in the immersive VR, comparative experiments with the first-person viewpoint confirmed that this also showed relatively satisfactory presence [52]. Therefore, this study attempted to present an experiential environment that could utilize both the first- and third-person viewpoints to provide satisfactory presence by inducing immersion in a non-immersive experiential environment of non-HMD user interface. In addition, for the RoleVR of Lee et al. [24], sharing the screen of an HMD user with a non-HMD user was as important a factor as providing distinguishable roles to every user. Unlike RoleVR, when the asymmetric VR of remote users was considered, it was confirmed that the multi-viewpoint interaction becomes an important factor for the presence of a non-HMD user. Based on the pairwise comparison analysis by Kim and Kim [57], which presented a comparative experimental environment between presence and immersion depending on different interaction environments of VR users and demonstrated a statistically significant difference through the one-way analysis of variance (ANOVA), this study also performed a statistical analysis on the presence and experience of asymmetric interfaces for HMD and non-HMD users. Through calculations of the statistical significance of presence through one-way ANOVA analysis, it was observed that similar values were recorded and there was no significant difference between the HMD user interface and the multi-viewpoint interaction-applied non-HMD user interface. However, a statistically significant difference was observed between the non-HMD user interface of fixed viewpoint and the HMD user interface. Finally, there was a statistically significant difference between the viewpoints provided to the non-HMD user interface as well. This proves that providing an interface that considers third-person viewpoint significantly improves the presence for non-HMD users.

In the subsequent experiment, the user experience provided by the interaction of the asymmetric interface was systematically analyzed. To this end, this study utilized the core module, positive experience module, and social interaction module of the game experience questionnaire (GEQ) [58]. The survey participants should record a value between 0 (not at all) and 4 (extremely) in the questions presented in the GEQ. Here, the core module consists of 33 items and it is possible to infer various factors such as competence and tension through a combination of items. In addition to the core module, there are questionnaire items that deal with social interaction. Therefore, this study analyzed the experiences provided by the proposed asymmetric interface to users through GEQ. First, the comprehensive impacts of the difference of roles felt by users in the asymmetric VR in terms of the immersion and interest were analyzed through the core module, and it was confirmed that a difference existed between the experiences felt by the non-HMD users and the HMD users (

Table 2. Detailed comparative analysis results for the experiences between the HMD users and non-HMD users in the asymmetric VR; * indicates statistical significance.

Components	HMD (SD)				non-HMD (SD)
Comprehensive Experience	3.408 (0.219)				3.600 (0.260)
Competence	3.094 (0.093)				3.382 (0.144)
Sensory & Immersion	3.542 (0.158)				3.482 (0.247)
Flow	3.476 (0.223)				3.674 (0.204)
Tension/Annoyance	0.250 (0.201)				0.133 (0.145)
Challenge	2.026 (0.285)				2.148 (0.153)
Negative effect	0.255 (0.176)				0.100 (0.156)
Positive effect	3.494 (0.174)				3.612 (0.282)
Factors	Overall Survey
Comprehensive Experience	F(1,38) = 2.8596, p = 0.108
Comparison between users
Factors	Comprehensive Experience	Competence	Sensory &Immersion	Flow	Tension/Annoyance	Challenge	Negative effect	Positive effect
non-HMD vs HMD	p = 0.108	p < 0.001 *	p = 0.547	p < 0.05 *	p = 0.175	p = 0.273	p < 0.05 *	p = 0.299

). Based on the maximum satisfaction level of the experience suggested by GEQ, i.e., 4.0, overall satisfaction levels of 3.408 and 3.600 were recorded for the HMD and non-HMD users, respectively. Furthermore, for calculating the normality for the experience environments the significance probability (p-value) for HMD and non-HMD users was 0.914 and 0.515, respectively, thereby confirming that the null hypothesis could not be rejected, and normal distributions were followed. This study attempted to supplement interfaces, focusing on the experience of the non-HMD users in an asymmetric environment. These points are considered to have increased the overall satisfaction. The detailed reason for this is analyzed as follows. It was confirmed that the roles provided to the users and their interactions yielded satisfying experiences in the asymmetric VR. In the results of dividing and comparing the specific components, as the non-HMD users performed various roles as an observer, assistant, and sometimes participant, and understood and managed the overall application, the flow and positive effects were high. However, although the concentration and immersion of HMD users were high as they performed actions directly in the immersive environment, annoyance and negative factors were also high owing to system factors such as VR motion sickness or relatively limited roles. Through a comparison of the questionnaire values via one-way ANOVA, it was confirmed that regarding the factors for satisfying experiences, everyone was satisfied with no significant difference between the HMD and non-HMD user interfaces. For a detailed experience, however, different trends were exhibited depending on the user. First, with respect to the immersion, challenge, and positive effect factors, the proposed asymmetric interface was determined to provide identical experience of participation as well as immersion. This was done by providing a role along with interaction optimized for the experience environment of the user. Moreover, because the proposed application contained entertainment elements, it did not trigger any special unpleasantness in the user. However, the users were satisfied with the non-HMD user interface because it provided diverse roles based on the multi-viewpoint, and the users had the competence or understood the flow of application more intuitively. By contrast, in the survey results of the HMD user interface, the negative effect was high with a significant difference because of relatively limited roles and the inconvenience arising from the blocked view. Therefore, the survey results confirmed that the proposed interface that can overcome the experience limitation of non-HMD users and induce immersion could be designed.

Finally, through conducting the comparison survey for the social interaction, it was confirmed that, based on the three-step communication process, a high mean was recorded, showing that all the users were satisfied. For calculating the normality for the experience environments, the significance probability (p-value) for the HMD and the non-HMD (third and first) was 0.563 and 0.776, respectively, thereby confirming that the null hypothesis could not be rejected, and the normal distribution were followed. In the comparison with the case where the roles were limited through a fixed viewpoint, a large difference was observed with respect to the social interaction. However, when fixed at the first-person or third-person viewpoint, as there is a difference of interaction depending on the roles, it cannot be said with certainty that one has more influence than the other. Nevertheless, it is confirmed that, by providing various roles to the non-HMD users and accordingly expanding their interactions, the social relationship with the HMD users increased, and this had a large effect on sharing experiences with HMD users, showing a positive effect on the presence.

Table 3. Comparative analysis results of social interactions between the HMD and non-HMD users in the asymmetric VR; * indicates statistical significance.

Factors	Mean	SD
HMD	3.406	0.115
non-HMD (3rd and 1st)	3.504	0.148
non-HMD (Only 3rd)	2.636	0.263
non-HMD (Only 1st)	3.093	0.156
Comparison between users
non-HMD (3rd and 1st) vs. HMD	F(1,38) = 2.474, p = 0.133
non-HMD (Only 3rd) vs. HMD	F(1,38) = 64.766, p < 0.001 *
non-HMD (Only 1st) vs. HMD	F(1,38) = 23.611, p < 0.001 *
Viewpoint type of non-HMD	F(2,57) = 44.079, p < 0.001 *

shows the results of comprehensive analysis, and a statistical significance was confirmed through the one-way ANOVA as well. An experience environment was provided, in which a non-HMD user could freely switch between the viewpoints and interact with a HMD user while performing various roles in the application; based on this, it was proven that there was no significant difference in the social relationships. However, it is confirmed that if the role in a single viewpoint is limited, similar to that of HMD users, the limitation of social interaction in the non-immersive environment possessed by the conventional non-HMD user interface cannot be overcome.

6. Limitation and Discussion

This study designed an optimized user interface whereby enhanced presence can also be provided to non-HMD users who are participating in a non-immersive experience environment in an asymmetric VR through high participation and immersion, although they are different from HMD users. However, this process was based on the premise that the experience environment minimizes the economic burden and the additional burden of the equipment so that every user can easily access and use it. Regardless of this aspect, using a motion recognition device such as Leap Motion or an additional haptic device can help increase the presence of a non-HMD user—this is another approach that can be considered. Therefore, in the future, it will be necessary to conduct a comparative experiment for overcoming the non-immersive environment by providing a non-HMD user with another inexpensive device in addition to the dedicated HMD controller. Furthermore, it will be necessary to provide various asymmetric VR applications, and through this, confirm that the interactions of the proposed interface provide an enhanced presence and new positive experience to all users in the asymmetric VR through the process of analyzing the roles of a user from multiple angles. In particular, an analysis is required for the negative factors, such as VR sickness, as much as the analysis on the presence and positive experience felt by the users in the immersive VR. This study satisfies the technical requirements (number of frames per second and polygons of rendered scene) considered in a VR application. However, because the survey experiment centering on the users was not conducted, specialized questionnaire such as simulator sickness questionnaire (SSQ) should be used in the future to conduct the HMD-user-oriented analysis for the VR sickness that might occur in the process of interacting with a non-HMD user while considering the technical requirements of asymmetric interface. Furthermore, the effect of proposed interface should be examined in various age groups by expanding the present age groups of participants (20s and 30s) in the experiment.

This study focuses on the interaction in an asymmetric VR composed of HMD and non-HMD user interfaces and aims to confirm whether non-HMD users can actively participate in interaction, rather than simply being an assistant, through a new role and a differentiated interface. Therefore, the current study does not deal with comparative analysis experiments by designing various interfaces and various systematic and environmental factors. However, we confirmed that the interface that interacted with a virtual environment or objects by directly using the hands based on the existing study [3,52] provided enhanced presence through high immersion compared with the input methods of typical interactive systems utilizing gamepads and keyboards. Existing studies on the asymmetric VR including this study mainly focused on the objective performance analysis of the proposed interfaces by using verified questionnaires (e.g., PQ, GEQ) rather than comparing existing studies as the experience environment and conditions presented were clear. Therefore, in the future, we plan to conduct experiments by designing asymmetric interfaces for HMD and non-HMD users from various perspectives (hand, gaze, immersive device, etc.) and supplementing comparative analysis studies.

7. Conclusions

The proposed asymmetric interface was designed for providing optimized interactions to the HMD and non-HMD users participating in an asymmetric VR. The key objective is to provide optimal interfaces (HMD: Direction interaction and non-HMD: Multi-viewpoint interaction) by classifying the differences between the user systems (hardware and device) and the role-based (participant, assistant, manager, etc.) experience environments based on the controller-based hand interface and three-step communication process. Through this process, satisfying presence and rich experience can be provided to the non-HMD users as well as the HMD users in the asymmetric VR.

To analyze the impacts of the proposed asymmetric interface on the presence and experience systematically, an asymmetric VR experiment application was developed in-house, targeting general participants. After experiencing the application in both HMD and non-HMD environments, regular participants (n = 20) recorded the results. The results confirmed that providing an interface specialized for the experience environment of users can allow non-HMD users to enjoy improved presence as much as HMD users and facilitate the unique experience of non-HMD users in an asymmetric VR. In conclusion, this study proved that, to induce immersion for non-HMD users in a non-immersive environment as much as for HMD users in an asymmetric VR, an interface based on the viewpoint and role that can maximize the experience environment of non-HMD users is required.