Exploring the Effect of Virtual Environments on Passive Haptic Perception

Abstract

Recent advances in virtual reality (VR) technologies such as immersive head-mounted display (HMD), sensing devices, and 3D printing-based props have become much more feasible for providing improved experiences for users in virtual environments. In particular, research on haptic feedback is being actively conducted to enhance the effect of controlling virtual objects. Studies have begun to use real objects that resemble virtual objects, i.e., passive haptic, instead of using haptic equipment with motor control, as an effective method that allows natural interaction. However, technical difficulties must be resolved to match transformations (e.g., position, orientation, and scale) between virtual and real objects to maximize the user’s immersion. In this paper, we compare and explore the effect of passive haptic parameters on the user’s perception by using different transformation conditions in immersive virtual environments. Our experimental study shows that the participants felt the same within a certain range, which seems to support the “minimum cue” theory in giving sufficient sensory stimulation. Thus, considering the benefits of the model using our approach, haptic interaction in VR content can be developed in a more economical way.

1. Introduction

Recently, virtual reality (VR) has received significant attention as a key technology for education, training, and entertainment, in that it may prove to be a more effective method of interaction, especially as the experience of users in immersive environments improves [1,2]. This technology can create various situations with useful information, allow for participant interaction, and provide the sense of being in the immersive virtual environment [3]. To increase immersion in the participant’s virtual environment, recent studies have been conducted on haptic interaction based on virtual objects [4]. For example, it is necessary to apply interaction methods to naturally manipulate an object using haptic feedback from virtual objects. Thus, although the user’s virtual environment is different compared to the real one, the virtual environment should be configured to provide the user with a sense of consistency between the two environments [5]. However, technical difficulties must be resolved to match physical configurations with different virtual spaces to enhance the user’s presence. Furthermore, virtual reality systems need to be built in an effective way while maintaining a level of presence, called the “minimum cue” [6]. In this line of thinking, this paper presents the effect of haptic perception with respect to the transformation (e.g., position, orientation, and scale) of a virtual object according to different physical conditions. Passive haptic is an interaction technique in VR that uses physical objects to provide feedback to the participant through shape, thus substantiating their virtual counterparts [7]. For instance, when a user interacts with a virtual object in a passive haptic scenario, the participant recognizes when the size of the virtual object differs from the size of the corresponding real object. In this situation, the feeling of being in the virtual environment is quickly broken; as a result, the participant’s sense of presence is lowered [3].

In general, it is difficult to use several physical objects that correspond to virtual objects to realize passive haptic scenarios in order to experience a complex virtual environment. Thus, in our study, we focused on the effect of passive haptic on users’ perception through different transformation conditions in immersive virtual environments. Then, we tried to measure the level of feeling through an “available cue” for physical objects. In other words, we found the degree to which different virtual and real objects can be implemented in an immersive virtual environment. Our method is expected to be able to create and present guidelines for the creation of immersive virtual reality content.

Figure 1 shows an example of a typical interaction situation using passive haptic with an immersive virtual environment. The user wearing the HMD sits in the chair, manipulates the physical object for passive haptic cues, and interacts with it. In terms of the user’s perception, given the availability of immersive environments, it has become possible for the user to experience virtual objects as if they were real [8]. Accordingly, VR content can create the feeling of being in a virtual environment where the virtual object is appropriately transformed to increase the level of presence. Recently, there have been a few attempts to overcome the limitation of conflicting information coming from the physical object and from the virtual one, such as situations of different positions, size, and shape [9]. Therefore, it is necessary to resolve differences between real and virtual objects, and the participant should be able to recognize the same object with a perceived coherence. The results indicated that the method helped to improve the sense of presence while dynamically aligning the physical and virtual objects, and it allowed the participant to experience correct and natural interactions in the virtual environment. In our paper, we compare the effect of passive haptic perception using different transformation conditions in immersive virtual environments (e.g., position, orientation, and scale). We observed that the participants in the virtual environment felt the same within a certain range in passive haptic situations, which seems to support the “minimum cue” theory in giving sufficient sensory stimulation. Our experimental study showed that the results felt by the participants were the same within a certain range. Additionally, our results showed that test conditions for passive haptic perception had certain rules in terms of control on the transformation of the virtual object. Through our approach, the VR environment can provide sufficient sensory stimulation related to presence, which can serve as a guideline for passive haptic cues with interaction developed in a more economical way.

The remainder of the paper is organized as follows. Section 2 presents previous work related to our research; Section 3 provides an overview of our system and approach; Section 4 details the implementation; Section 5 presents the experimental setup and procedure to evaluate passive haptic perception in the system configuration; Section 6 reports the main results, as well as providing a discussion and limitations; lastly, Section 7 summarizes the paper and concludes with directions for future work.

2. Related Works

We will outline research from three areas (i.e., consistency between real and virtual environments, haptics, and haptic retargeting) that are related to the current work.

2.1. Consistency between Real and Virtual Environments

With the recent and ongoing advances in sensing technologies, developments that enhance users’ perceptions in the interactive VR environment have become more feasible [10]. Therefore, many researchers have considered system configurations that evaluate the participant’s sense of presence when interacting in virtual environments [11]. In one notable piece of representative research, an interactive locomotion technique called redirected walking was proposed for virtual environments to allow users to walk in a virtual space larger than the real walking space [12]. In this study, they attempted to manipulate participants in the virtual environment into walking and interacting along an arc while giving the perception of walking straight. Another useful study showed a method for evaluating users’ perception of virtual objects using heterogeneous augmented reality (AR) devices to develop interactive AR content [3]. They divided several parameters related to the user’s perception such as color and scale. In the result, participants in the scale perception experiment thought the large virtual object was equal in size to the small real object. People tended generally to perceive the virtual object as small and to underestimate its dimensions. Lee et al. proposed the effects of an “out of body” tactile illusion to create feedback using vibration devices in the case of dynamic virtual objects detached from the body [13]. Their approach can be applied to generate phantom tactile sensations that seem to arise from an external virtual object. More recently, many researchers have attempted to align different physical spaces and to adapt the teleported avatar to remote sites for collaborative environments [14]. Jo et al. introduced an adjustment of the position of the teleported avatar into the local space with the corresponding information in teleconference environments [1]. Comprehensive research in terms of developing passive haptic feedback in a more economical way has yet been conducted. Our work was designed to explore the effects of passive haptic parameters on the user’s perception, comparing actual and virtual objects in relation to transformation. Specifically, in our paper, we investigate the correlation in visual perception between real and virtual objects, aiming to determine whether the participants felt the same way within a certain range to give sufficient sensory stimulation.

2.2. Haptics

In the field of robotics, many studies have been introduced on sensory design for grasp control and manipulation of real objects with human-hand-like motions [15]. In a few more recent results, researchers have introduced haptic devices and rendering. Haptic technologies have a long history of use in many pieces of research such as a dental training simulator [16]. Haptics is a field of study mainly about the sense of touch, and enables the feeling of simulated objects via an interactive method depending on the physics through a haptic interface [17]. Immersive VR systems typically focus on visual and auditory feedback; to provide more interactive content, haptics provide additional feedback to participants. There are two types of haptics methodologies: passive and active haptic. Passive haptic for natural interaction is a method where real objects that resemble virtual objects provide feedback naturally through the sense of touch of their tactile properties such as contact location, pressure, vibration, and temperature [4]. In contrast, active haptic creates forces through haptic equipment with motor-controlled actuators to provide kinesthetic perception [18]. More recently, many researchers have been concerned about haptic rendering to compute the correct interaction forces for virtual objects [17]. For example, there have been a few attempts to make collision detection algorithms to find penetrations and the contact area, as well as force response algorithms to create return values with respect to applicable forces in the collision state [17]. However, in haptic research results, it remains to be determined how passive haptics should be configured to have a high level of presence in the virtual environment to enhance the effect.

2.3. Haptic Retargeting

Haptic interaction is an output form of VR systems to engage with participants through natural interfaces [19]. To improve the quality of the experience in VR environments, haptic retargeting is being used as a major approach, increasing the participant’s sense of presence with a single physical object to provide a passive haptic representation [5]. There have been a few previous attempts to enable effective hand reaching and visual feedback when interacting with a virtual object. Typically, in this method, the virtual representation of the body is dynamically aligned with the passive haptic prop, called redirected touching [5]. Han et al. introduced remapped physical reaching for hand interactions using the passive haptic method to enable a more realistic interaction [4]. They suggested remapping techniques for reaching in situations of physical and virtual object mismatch. More recently, Clarence et al. proposed reach prediction for haptic retargeting in VR systems [20]. They presented the method of the user’s intended interaction target and reach trajectories. However, there are still research topics for passive haptic interaction such as the participant’s sense of feeling with respect to the design or effective conditions of physical props. Most existing studies have focused on the motion of the virtual hand such as to minimize reaching errors and achieve matching qualities. In our paper, we focus on an effective configuration of passive haptic interaction, investigating issues such as participants using the passive haptic prop in the virtual environment to feel the same perception in different transformation conditions. We summarized previous works in

Table 1. Classification of issues and problems regarding passive haptic perception.

Issues	Problems	References
Consistency in virtual environments	Matching between a real object and the related virtual object	Shin et al. [3], Lee et al. [9]
	Consistency between shared virtual environments	Jo et al. [1]
Haptic devices and rendering	Computing correct interaction forces	Insko [7], Hattori et al. [16], Salisbury et al. [17], Jose et al. [18]
Haptic retargeting	Remapping techniques for reaching of the virtual hand	Han et al. [4], Azmandian et al. [5], Clarence et al. [20]

.

3. System Overview and Approach

Figure 2 shows our system configuration and an example of passive haptic perception in an immersive virtual environment. A participant was seated in a chair in front of a desk, wearing a head-mounted display (HMD) to view the fully immersive space. The participant could interact with a virtual object such as a computer-generated cube. In our case, a tracking device consisting of optical marker sets was used for recognizing the participant’s hand position and orientation. Moreover, a virtual hand in our virtual environment represented dynamic physical reaching from precise transformation (e.g., position and orientation) data from rigid-body marker sets attached on the participant’s real hand. A precise tracking system with an error of less than 1 mm for the participant’s hand interactions was used. Figure 2 also represents the real and virtual scenes used to evaluate passive haptic perception in our paper. The participants could not perceive the slight difference in virtual conditions compared to the real environment (e.g., the case with different reaching degrees of the virtual object).

Table 2. Test conditions in our system regarding passive haptic perception.

Transformation Factors	Description	Test Conditions
Position	The distance between a real object and the related virtual object	Distance factors −30 cm to 30 cm (Step 5)
Rotation	The angle (or orientation) difference between a real object and the counterpart virtual object	Orientation factors −60° to 60° (Step 5)
Scale	The difference in size between a real object and the matched virtual one	Scale factors 0.5 to 1.5 (Step 5)

shows our test conditions for the experiment aimed at determining the effects of passive haptic perception. We set test conditions to measure variables for transformation factors such as position, rotation, and scale. For example, when interacting with the virtual object in immersive environments, participants touched a real object via passive haptics under different test conditions, and they were asked to rate how they felt in the given situation. Through a preliminary experiment involving eight people, we set the limits of the value of our test conditions, and the value determined by all participants in the virtual environment to be different from the real environment was set as the boundary of the test conditions. In addition, we included situations where several-times errors occur, for example, the cases where participants cannot touch the physical prop due to a long distance during a given time. In our system, several steps were used to establish appropriate test conditions; in particular, the virtual hand was used to automatically operate visual feedback to calculate the distance depending on the tactile situation of the real object according to haptic retargeting [4].

Figure 3 shows our interaction process to evaluate the effect of passive haptic perception in immersive virtual environments. For example, to explore the participant’s perception, the specific transformation condition was selected through three types of transformation conditions (adjustment of position, rotation, and scale parameters). The participant interacted with a real object for passive haptic feedback and a related virtual object as the corresponding counterpart. Then, the participant described the difference between the real and virtual objects.

In our system, the purpose was to determine the similarity between real and virtual environments constructed using passive haptics. Thus, we used a questionnaire to determine the results of a hand-interaction operation. Accordingly, we evaluated broken passive haptic situations and measured the degree of difference between both environments.

4. Implementation

In this section, we describe our implementation method concerning transformation parameters and the experimental setup for passive haptic perception. Figure 4 shows our system configuration used to evaluate passive haptic perception. We compared the transformation difference in terms of participants’ sense of passive haptic interaction. In our implementation, the participant in a virtual environment wore the HMD, and an optical-based tracking device called Optitrack Trio was installed for hand interaction [21]. Marker sets for hand tracking only visible in the optical camera were attached to the back of the hand. To set up our experiment, we adopted the method used by Shin et al., who investigated the effects of the participant’s perception on an augmented reality (AR) experience [3]. We used a real cubic puzzle (or cube) to provide passive haptic cues and applied a transformation comparison in our experiment. The participant was seated on a chair in front of a desk. Then, the subject was allowed to experience the same virtual space in the HMD as the real one. For instance, the participant could interact with the virtual cube on the desk in the virtual environment, and they compared the different types of transformation conditions in relation to passive haptic perception between the real cube and the virtual one. For the initial setup to match the virtual and real space, we applied scale calibration by measuring the size of the real environment (e.g., the desk’s horizontal and vertical dimensions) using optical reference marker sets. Because we focused on the effect on the setting parameters, manipulation of the virtual object was not applied. Figure 4 shows an example of different position factors in our system configuration of passive haptic perception. In the case of the position factor for passive haptic perception, we investigated how participants perceived the position of the virtual cube in slightly different conditions.

As outlined in Table 2, the test conditions in our system for passive haptic perception consisted of five steps for each transformation factor (e.g., the distance for the position factor was scaled in five steps from −30 cm to 30 cm). Figure 5 shows an example of different position factors by adjusting five steps along the z-axis, including the exact position of the real and virtual objects. According to prior simple experiments as already mentioned, there were many cases in which participants could not find the passive haptic prop to touch; thus, we only set the position to change along the error-free z-axis. Then, the participant was able to compare the position of the virtual cube with the actual one.

Figure 6 shows an example of passive haptic interaction with different scale parameters. In our implementation for scale perception, other parameters such as position and orientation were maintained in their same conditions. For test conditions, we used Unity3D to visualize a 3D virtual cube to represent the same shape and material as the real one, and the virtual hand was allowed to move as a rigid body without finger movement [22]. To visualize a virtual hand with resemblance to an actual person, we dynamically interpolated the position of the virtual hand during a reaching motion as a function of the distance of tracked physical reaching to calculate hand motion [4]. Moreover, when a collision situation between a virtual hand and a virtual cube was recognized in a specific area with a threshold value, the motion animation of the virtual hand holding the cube was automatically generated. In test conditions for immersive 3D displays, the subject wore a head-mounted display (Oculus Rift [23]) to view and interact with the virtual cube. We used a real cubic puzzle with a size of 5.5 cm. To provide processing, the virtual environment was operated on a personal computer running the 64-bit version of Windows 10 Pro.

5. Experiment for Effects on Passive Haptic Perception

In our paper, we proposed the effects of passive haptic perception in the immersive VR environment and focused on transformation conditions of a virtual object related to its real counterpart (or physical prop). We ran three comparative experiments (i.e., position, rotation, and scale) to evaluate the participant’s perception, as judged by the same range between the real and the virtual environments with respect to passive haptics.

We installed the HMD, the actual cube, and the tracking system. In the initial setup, the virtual cube was visualized to appear with the same size, position, and orientation as the real one. Then, we investigated how participants responded to three conditions with five steps in each condition (see Table 2). We introduced the purpose of our research study and gave instructions to the participants. Then, the within-subject experiment was conducted in a predetermined random order with a total of 15 steps, whereby an operator behind the curtain configured the experimental conditions. In each step of the experiment, the participant was asked to touch the virtual cube placed on a virtual desk. Twenty-four paid participants with an average age of 26 years participated in our experiment. The age range was between 20 and 34 years old. Considering the gender of participants, 33% were females (8 women) and 67% were males (16 men). All participants had at least one experience with an immersive VR system prior to our experiment; most participants (20 people) were playing mobile games regularly. Fortunately, all participants were right-handed, and we did not have to change the virtual hand representation.

At each step, the participant was asked to describe the difference in perception between the virtual object and a real prop with respect to transformation. Here, participants who denoted the presence or absence of a difference were also asked about the degree after carrying out the task. The degree was measured using a questionnaire with a 100-point scale, where a score of 0 denoted perfect matching between two objects of different environments (e.g., real or virtual), while a score of 100 denoted complete inconsistency. Here, to more accurately analyze the precise difference, in the experiment, we used a 100-point scale.

Figure 7 shows the number of participants who evaluated perception differences according to transformation factors. In the position factor test condition, many participants thought that there was a difference between the real and virtual environments by distance error. In the case of the rotation factor, participants distinguished a difference between the two environments more clearly than by other factors. In terms of the scale factor, there were similar results to position factors, in that people highlighted differences according to the size of the virtual object.

Figure 8 shows the survey results regarding the degrees of perceived differences between the virtual object and the real one in terms of transformation. The overall trend was similar to the participant’s perception of the difference. In the case of scale, our experimental study showed that participants had certain tendencies. A pairwise comparison using two-sample t-tests (small-size group or large-size group) showed a significant effect in the statistical analysis (t(46) = 6.90, p < 0.05). Here, we found beneficial results in the guidelines to install virtual reality systems. Details are provided in Section 6.

6. Discussion and Limitations

Our results showed that test conditions for passive haptic perception had a major effect on the participant’s transformation perception of the virtual object compared to the real one. First, participants tended to perceive a greater difference when the virtual object felt physically close to the body. It can be assumed that this result is related to proprioceptive cues depending on the distance. Note that proprioception is the sense of self-movement and body position used to detect kinematic parameters such as joint position [24]. Second, we found that people typically noticed a difference more clearly in the rotation case. According to the results in Figure 7 and Figure 8, the number of people and the degree of difference in the various orientation steps were significantly large. Furthermore, depending on the angle, when rotational gain was used, we found that we could properly adjust the angle of the virtual object to the user [25,26]. In the scale factor, participants in the experiment tended to underestimate the size of the virtual object when it was smaller than the actual object. This was also found in previous similar studies [3]. Our experiments also showed the effect of “minimum cue” in terms of position and scale factors to create VR content. Thus, in certain situations, we found that people did not notice if the virtual object was slightly different from the real object. This outcome may facilitate development of immersive VR environments with haptic interaction in a more economical way.

Figure 9 shows an example of the limitations in our experiments for passive haptic perception. Our current results need to be updated to provide more useful haptic experiences including complex situations such as interactions with both hands and manipulation of the virtual object in the air. Also, with the sensing capability of the physical prop, we will improve the system to more complex interactions; for example, participants with respect to a feeling for different-shaped virtual objects will need to analyze how they feel differently. Additionally, passive haptic should be provided anytime and anywhere in a virtual environment, and it is necessary to handle a navigation situation as well as a fixed location. Furthermore, we plan to extend the evaluation to support cases where the virtual object is a deformable soft body in passive haptic situations.

7. Conclusions and Future Work

In this paper, we explored the effect of passive haptic perception in virtual environments and compared passive haptic parameters using different transformation conditions. Many participants thought differently depending on the transformation factors (e.g., position, rotation, and scale). Specifically, in the case of rotation factor, participants discriminated better than other factors. In position and scale factor, a score greater than 60 points indicated an inconsistency in the case of close and smaller cases. On the other hand, we found that participants tended to not perceive slight differences between virtual and real objects within a certain range, thus supporting the “minimum cue” theory. VR content designers can create beneficial models using our approach consisting of haptic interaction developed in a more economical way.

In terms of the future, there are still many aspects of our system that need improvement for the practical applicability of perceptual haptic factors. In particular, it is necessary to study the shape of tangible props for passive haptic interaction, as well as to retarget destinations of the passive prop to enhance tactile feedback. Additionally, we will continue to explore various VR environments such as educational scenarios for use in the real world. We also plan to further extend quantitative evaluation by physiological means with various parameters that affect the participant’s haptic perception.