Arm Posture Changes and Influences on Hand Controller Interaction Evaluation in Virtual Reality

Featured Application

Motion-sensing interaction and virtual reality technology.

Abstract

In virtual reality (VR) applications, hand-controller interaction is largely limited by the biomechanical structure of the arm and its kinematical features. Earlier research revealed that different arm postures generate distinct arm fatigue levels in mid-air operational tasks; however, how they impact interaction performance, e.g., accuracy of target grasp and manipulation, has been less investigated. To fill this gap in knowledge, we conducted an empirical experiment in which thirty participants were recruited to complete a series of target acquisition tasks in a specifically designed VR application. Results show that (1) a bent arm posture resulted in a higher interaction accuracy than a stretched arm posture; (2) a downward arm posture interacted more accurately than an upraised arm posture; since two arms are bilaterally symmetric, (3) either selected arm interacted more accurately on the corresponding side than on the opposite side; and (4) the user-preferred or dominant arm interacted more persistently than the non-dominant one, though two arms generated little difference in interaction accuracy. Implications and suggestions are discussed for designing more efficient and user-satisfying interactive spaces in VR.

1. Introduction

Benefiting from a rapid development in motion-sensing and computer graphic (CG) technologies in recent years, VR has become an increasingly prevalent technology in sports and rehabilitation training [1], immersive entertainment [2] and education [3], and more other spatial simulation and exploration domains. For example, in geological research and educational practices, researchers used VR technology to recover and simulate geological heritages, which has proven to be an effective means for immersive visiting and learning by academics and the public [4,5]. Especially when other cutting-edge technologies, such as natural language processing (NLP) and brain-computer interface (BCI), are integrated into VR, people have more opportunities to experience novel products and services, including, but not limited to, immersive storytelling, neurogaming and psychotherapy [6,7]. In various VR applications, users are immersed in a computer-simulated stereoscopic environment through head mounted displays (HMDs) such as Oculus Rift, HTC Vive and Meta, and acquire and manipulate virtual objects through a handheld controller or bare hand, as if in the real world. There are a variety of common interaction tasks in a VR environment, including spatial navigation, information retrieval, target moving and dragging. Among these, target acquisition, which consists of two steps, i.e., target pointing and target selection, is the most fundamental, but representative, interaction task [8,9].

Hand-motion sensing-based interaction, including bare hand interaction and a hand controller-based interaction, is largely limited by the biomechanical structure of the user’s arm and its kinematic features [10,11,12]. The user performs different arm postures to acquire targets at diverse positions in mid-air, as illustrated in Figure 1. Earlier kinematic studies revealed that hand operation in some postures was perceived to be more comfortable than in other postures. In particular, a downward arm posture was generally more relaxed than an upraised arm posture, and a bent arm posture was generally more labor-saving than a stretched arm posture [13,14]. However, how arm posture and hand-operational position affect hand interaction performance, e.g., accuracy of acquiring virtual objects and hand-operational persistence, were less investigated. Because of this, user interface layout design and interactive space organization in current VR applications are often arbitrary.

To fill the knowledge gap and, more importantly, to develop guidelines or strategies for designing highly efficient and user-satisfying three-dimensional user interfaces (3DUIs) in VR applications, we recruited thirty participants to complete a specifically designed target-acquisition experiment. In this experiment, targets in the VR scene floated at different positions, and the participants performed different arm postures to acquire these targets. Except for the arm posture, the user’s handedness and hand choice were also identified and defined as independent variables in the experiment. We measured target acquisition accuracy and hand-operational persistence in different conditions of hand choice and arm posture and, based on these, concluded kinematic influences on the interaction results.

The remainder of this paper is organized as follows: Section 2 presents an overview of related work. Section 3 states the research objectives and hypotheses development. Section 4 gives an introduction to the experimental method, including information about the participants, apparatus, experimental design, procedures and analytical methods. This is then followed by Section 5, which presents the experimental results and hypotheses verification. Based on the experimental results, Section 6 provides a discussion about the experimental findings and related implications. Finally, Section 7 concludes this research work.

2. Related Work

2.1. Hand Motion Sensing-Based Interaction and Ergonomical Concerns in VR

Hand-motion sensing-based interaction, which is often simplified as hand interaction, including the direct interaction by bare hands or a handheld controller-mediated interaction, is a most widely applied form of natural user interaction (NUI) [15]. Compared with conventional mouse- and touching- based interactions, hand interaction possesses advantages of higher intuitiveness and compatibility with application scenarios [9,15,16,17]. According to surveys by Mine et al. [17] and Lemmerman and LaViola [18], hand interaction was preferred by users and perceived to be more immersive than the conventional interactions.

Despite the advantages, hand-motion sensing-based interaction has been criticized to have notable usability deficiencies, such as ambiguity of gestures, inaccuracy in manipulation and an arm fatigue issue [19,20]. Researchers continually proposed technical methods to eliminate or alleviate these deficiencies. For example, Vogel and Balakrishnan [9] made use of a Vicon^TM high-precision motion-tracking system to realize accurate hand manipulation in mid-air. Lou et al. [21] found that spatial distance between the user and the display not only reflected the user’s intention in interaction tasks, but also had a significant effect on the interaction speed and accuracy on the display. They also proposed a spatial distance-adaptive technique to make a large display-based motion-sensing interaction more efficient and accurate [22]. Techniques such as Go-Go interaction [23] and BoostHand [24], which worked by nonlinear scaling of hand positions, were proposed to assist the user in approaching any object in a virtual environment. Ens et al. [25] proposed a personal cockpit with menu items placed on a sphere centered around the user’s point of view. Gerber and Bechmann [26] also developed technique called ‘Spin Menu’ to place the menus around the user’s wrist responsively, thus making menu operations more efficient. Others, such as Haque et al. [16], used surface electromyogram (sEMG) and inertial measurement unit (IMU) detection sensors worn on the user’s arm to implement an accurate identification of hand movements and gestures. Mäkelä et al. [27] proposed the technique of ‘Magnetic Cursor’ to make acquisition of small targets easier. Additionally, Lou et al. [28] had ever put forwarded a hand-adaptive technique to aid target aiming and selection in a virtual environment.

Regarding the arm fatigue problem, Hincapié-Ramos et al. [14] proposed a metric, Consumed Endurance (CE), to quantitatively measure arm fatigue level in free-hand operational activities. They found that hand operation at waist height through a bent arm posture was more labor-saving than operation at shoulder height through a stretched arm posture. Nonetheless, hand-operational position or the arm’s bending degree are more varied than what they had investigated. Arm postures are not only relevant to the hand-operational height and distance, but also relevant to the arm’s angle relative to the body in the horizontal direction. As illustrated in Figure 2, the hand-operational height includes shoulder, waist, and above-head heights. In regard to the hand-operational distance, not only does it include the two alternatives of far and close distances, but also a mid-far distance corresponding to stretched, wholly-bent and half-bent arm postures, respectively. In regard to the hand-operational angle in the horizontal direction, there are also three basic variations: negative angle on the left side, zero angle at the central front of the body, and positive angle on the right side, as shown in Figure 2a. Earlier research, including Hincapié-Ramos et al. [14], seldom investigated the effects of all these parameters on hand interaction performance completely.

User-perceived arm fatigue is a prevalent problem in arm movement and operations in mid-air because it causes less persistent hand interaction than conventional mouse- or touching- based interactions and, thus, alternations between the two hands are inevitable and even frequent in completing a continuous interactive task [13]. Alternating hand from one to another generates a break in the task, as well as a sudden change in the operational position, which implies that hand alternations make the user’s dominant hand and the non-dominant hand more evenly used; hand alternations also make the interactive task discontinuous which ultimately brings potentially negative influences to the overall efficiency of the task. However, to the best of our knowledge, the hand alternation phenomenon and its potential influences on hand interaction performance in VR have seldom been of concern or investigated. Additionally, the cooperation and alternating mechanism of two hands are crucial but less considered issues in the hand interaction research field. Operational performance comparison between the dominant hand and the non-dominant hand has often been neglected in earlier studies. Such negligence was a ubiquitous weakness in relevant hand interaction applications, acknowledged by Lou et al. [22].

2.2. Target Acquisition in VR and Aiding Techniques

Target acquisition, namely pointing and selecting objects, remains by far the dominant interaction paradigm in most software user interfaces, including 3DUIs in a VR environment [8]. It generally contains two phases: ballistic and correction. At the former phase, the user moves the hand to touch the target, then at the following phase adjusts the hand position and triggers a selection event [29]. In the physical world, by leveraging visual perception and proprioception, users can accurately acquire objects located around their body [17,30]. However, when interacting in a VR environment, real visual information is replaced by computer-simulated scenes and tactile feedback is often lacking. It becomes more difficult to acquire targets in simulated stereoscopic scenes.

Recent research by Yan et al. [31] and Cockburn et al. [32] measured target-acquisition efficiency and accuracy in a virtual environment and revealed that visually guided acquisition within the user’s visual field was more accurate than eyes-free acquisition. Wang and MacKenzie [33] found that user interaction was more efficient when visual and motor spaces were superimposed or coupled closely within the arm’s reachable space. The potential influences of the user’s arm posture and the hand-operational position on target-acquisition performance, however, was not considered in their work. Other research by Lubos et al. [34,35] introduced an arm joint-centred interaction technique to make target acquisition more efficient in VR. In this technique, the user’s wrist, elbow and shoulder joints were treated as reference points and targets were located around these points, thus they could be acquired more promptly. It is actually a responsive user–interface technique in which interactive objects were located flexibly and dynamically. However, in situations where dynamic user interfaces are inapplicable, joint-centred interaction becomes invalid. Therefore, having an investigation into the intrinsic features of the arm’s movement and hand interaction performance and, based on this, developing user–interface design guidelines are more significant work, which would be universally beneficial for designing hand interaction techniques and related applications.

2.3. Hand Interaction Performance and Evaluation Methods

In human–computer interaction (HCI) evaluations, researchers generally adopt Fitts’ Law [36] to model and assess target acquisition efficiency. It was first proposed by Paul Fitts in 1954 but was extended and updated by other researchers. In the Fitts’ Law model, target acquisition efficiency is usually reported in its reciprocal form—movement time (MT), which is stated as a log-linear function of target width (W) and movement distance (D). The Fitts’ Law formula is usually expressed as:

$$MT=a+b\times {log}_2\left(\frac{D}{W}+1\right)$$

(1)

where a and b are two empirically determined constants. Equation (1) provides an efficient and convenient method to evaluate target acquisition efficiency; however, it is incapable of measuring accuracy. To have this shortcoming remedied, researchers suggested that the scatter of the user’s movement end-points should be additionally gathered [37]. Given the end-point scatter data, a standard deviation (SD) of the end-point positions can be calculated, and the effective target width (W_e) can be defined as below:

$$W_e=4.133\times SD$$

(2)

where 4.133 is an empirical value derived from numerous experiments and has been widely accepted by related researchers. According to Equation (2), a larger value of W_e implies a more scattered distribution of the end-points, namely a lower precision of target acquisition. In our experiment, an Oculus Touch controller was used to complete target acquisition tasks, thus distinct end-points of controller position can be gathered. Therefore, the calculation method of end-point scatter can be used to measure target acquisition accuracy.

Apart from acquisition accuracy, interaction persistence is another important metric in hand interaction evaluations. It refers to the endurance or duration of the working arm in mid-air hand interaction tasks, without relaxing. A higher persistence or a longer operational duration represents hand interaction being more coherent, with fewer alternations between two arms. According to Tanii et al. [13], free hand operation in mid-air generated arm fatigue easily, which caused spontaneous alternations of the arm per several minutes. As the authors measured, a more tiring arm posture resulted in a shorter operational duration or a higher frequency of arm alternations. From this perspective, the interaction persistence can be used as an indicator of fatigue level in hand-controller interaction.

3. Hypotheses Development

According to the preceding review, we learnt that (a) the handedness feature, biomechanical structure of the arm and its kinematic features were of less concern in hand interaction evaluations and (b) due to the intrinsic fatigue problem in hand operation tasks, two hands are often used alternately in interaction tasks. From this perspective, the left hand interaction and the right hand interaction should be evaluated and compared; however, this was often neglected in earlier research. To fill the knowledge gap, this study compared the left- and the right-hand interaction results, and based on these, summarized the hand choice effect on hand-controller interaction performance.

In a VR environment, the user performs a bent arm posture to acquire targets that are located at a close distance, but performs a more stretched arm posture to acquire targets at a farther distance relative to the body. In user-perceived fatigue measurement, neurophysiological research revealed that hand interaction in the lower field of the user’s vision is more efficient than that in the upper field [38]. Danckert and Goodale [39] and Po et al. [40] also found that arm movement in the downward direction is more accurate than that in the upward direction. Based on these, we propose Hypotheses 1 and 2 as follows:

Hypothesis 1 (H1).

In hand-controller interaction, a bent arm posture is more comfortable, which generates a higher interaction accuracy than a stretched arm posture. A bent arm also interacts more persistently than a stretched arm.

Hypothesis 2 (H2).

Hand interaction at a low position, e.g., waist height, is more accurate and persistent than that at the higher position of shoulder or overhead heights.

Human arms, including both the dominant arm and the non-dominant arm, are bilaterally symmetrical. The biomechanical structure of the arm and its kinematic features determine that arm movement and hand operation in some spaces are more flexible and accustomed than in other spaces [10,11]. Handedness is a general characteristic in human beings, which reflects the motor ability difference between the dominant hand and the non-dominant hand. Handedness is also an individual characteristic in hand-operational behaviors, which implies possible influences on hand-use frequency and hand interaction performance at different positions. Based on these, we raise another two hypotheses:

Hypothesis 3 (H3).

Arm and hand-use choice determines hand-controller interaction performance: the left arm interacts more accurately on the left side than on the right side of the user’s body; however, the right arm interacts more accurately on the right side than on the left side.

Hypothesis 4 (H4).

The user-preferred or dominant hand interacts more accurately and persistently than the non-dominant hand.

4. Methods

4.1. Participants

Male (n = 22) and female (n = 8) volunteer students and staffs from a local university were recruited in the experiment; their ages ranged from 21 to 36 (M = 24.8, SD = 3.65). All participants were familiar with VR technology and had experienced VR applications, including, but not limited to, VR games and VR-based training assistive systems. All the participants had normal or corrected-to-normal vision, without color blindness or body impairments. We adopted the Edinburgh Inventory [41] to measure the participants’ handedness and verified that all participants were right-handers. Through the measurement tool of the HMD configuration utility, the inter-pupillary distance (IPD) of the participants was measure and it ranged from 6.7 cm to 7.9 cm (M = 7.2, SD = 0.51). Given the IPD data, the HMD was tuned to provide a correct perspective and stereoscopic rendering for each participant. The arm length of the participants, i.e., the distance from the shoulder joint to the palm, was also measured. Results ranged from 66.2 cm to 73.2 cm (M = 68.8, SD = 4.67).

4.2. Apparatus

This study was conducted in a VR laboratory. An Oculus Rift (Developer kit) was used as the HMD, with a Touch controller connected. The HMD was connected to a workstation computer (Win10, 32 GB memory, and 4.0 GHz Intel 32-core processor) through a 5-m long cable. The computer was also connected to a 40-inch monitor, which was provided to supervise the visual field in Oculus. Oculus Rift offers a nominal diagonal field of vision of approximately 100 degrees at a resolution of 1920 × 1080 pixels (960 × 1080 for each eye). Two IR cameras were mounted on a desk to track spatial positions and movements of the Oculus device and the Touch controller, respectively, with a precision of 2.0 mm. Additionally, a motion-sensing camera (ASUS Xtion PRO^TM) was also mounted on the desk to identify and track the user’s skeleton joints. Figure 3 shows the experimental setting and scenario.

4.3. Procedure

A specific target-acquisition VR program was developed using the Unity3D engine in C#. In this program, a stereoscopic scene was rendered and the participant’s operating arm shoulder joint was tracked through the ASUS camera; a virtual arm was rendered to indicate the arm movement and the interactive target was randomly placed at one of pre-defined locations, as shown in Figure 3. In this experimental program, the interactive context, tasks and independent variables were specifically designed for an important reason: the study objective was to explore hand-controller interaction performance at different positions around the body, not only in front of the eyes, but also over the head and in front of the waist, and on the left side and the right side of the body. To our knowledge, such interaction tasks and independent variables have seldom been explored in earlier research. Therefore, no appropriate benchmark test could be adopted in this experiment.

Each participant was required to complete 6 task blocks, 3 of which were initiated by the left hand and the other 3 blocks initiated by the right hand. Each block consisted of 300 target acquisition trials for an important reason: one key objective of this study was to evaluate hand-controller interaction persistence and explore its influential factors. Here, the interaction persistence was measured by calculating the hand-operational duration, which referred to the interval time between two hand alternations. In this regard, a task block should be persistent enough for at least a single hand alternation event. However, a block consisting of too many trials might exhaust the participant which could then cause a potential influence on interaction performance. Through a pilot test, we found that a sum of 300 trials is an appropriate amount in a block. For each target acquisition trial, a single orange-colored sphere was presented. When the participant moved the controller to touch the target sphere, it became blue to give a visual cue. Next, the participant pressed the button on the controller to acquire the target. After an acquisition trial was completed, the target disappeared and a new one showed at another position. From one target to the next, the participant was required to continuously complete all acquisition trials until the program terminated automatically. During this procedure, the participant was permitted to alternate the operating arm for perceived arm fatigue. While acquiring a target, if the participant triggered the acquisition button without touching the target, an acquisition error was counted and the target position was recorded by the program. Task performance data, including the hand alternation moments, 3D coordinates of the end-points and the shoulder joint, as well as the error times of different target positions were recorded and stored in a log file.

At the beginning of this study, the participants were given an introduction to the research purpose and the task requirement. They were instructed to practice until they were observed to be sufficiently familiar with interaction tasks. In formal task blocks, the participants were required to stand at a specific position to initiate their interaction, and they were not permitted to move or rotate their body during tasks. Between two blocks, the participants were given 5 min to relax their eyes and arms. At the end of each block, the participant was asked to rate the fatigue level in terms of 6 metrics: Mental Demand, Physical Demand, Temporal Demand, Effort, Performance, and Frustration Level, through a 5-point Likert scale. These 6 metrics were derived from the NASA Task Load Index (NASA-TLX) which is a subjective, multi-dimensional workload assessment tool [42]. In this study, to help the participants better understand them, each metric was described through a detailed statement based on the experimental tasks in this study (see in Appendix A—

Table A1. Perceived arm-fatigue assessment questionnaire metrics and corresponding statements.

Assessment Metric	Explanatory Statement (Rated on a 5-Point Likert Scale)
Mental demand	How mentally demanding was the interaction task you completed?
Physical demand	How physically demanding was the interaction task you completed?
Temporal demand	How suffering and hard was the pace of the interaction task you completed?
Performance	How successful were you in completing the interaction task?
Effort	How hard did you have to work to accomplish your level of performance?
Frustration	How insecure, discouraged, irritated, stressed, and annoyed were you?

). More information about the subjective fatigue assessment is presented in Section 5.3. After completing all task blocks, the participant was also encouraged to comment on the experimental design, VR program, task procedures, perceived arm fatigue and overall user experience; this information was recorded by the experimenter.

4.4. Design

In this study, a 2 × 3 × 15 repeated measures within-participant design was adopted. The independent factors included hand-use choice (left, right), arm-bending degree (wholly-bent, half-bent, stretched) and hand-operational height (3 target heights × 5 horizontal angles = 15 different positions).

According to the experimental design, targets were located at different horizontal angles and vertical heights surrounding the shoulder joint. Earlier studies on kinematics have found that people can flexibly perform arm movements in body-front space, but can hardly perform any arm movements in body-behind space [43]. Because of this, in the horizontal direction, all target positions were located within 180 degrees of the visual field and were distributed at 5 different angles: −90°, −45°, 0°, 45°, and 90°, respectively, from the left side to the right side, as shown in Figure 4a. In the vertical direction, target positions were located at the overhead height, the shoulder height and the waist height, respectively, as shown in Figure 4c. There were 3 × 5 = 15 target positions in the experiment.

In terms of hand-use choice, the participant used the left hand and the right hand, respectively, to complete one task block. The order of hand use was counterbalanced across the participants: one half used the left hand first, while the other half used the right hand first. Arm-bending degree included wholly-bent, half-bent and stretched arm postures. As illustrated in Figure 4b, three bending degrees of the arm corresponded to three interaction distances between the shoulder joint and the target. To make all targets reachable, the interaction distances were defined as 30 cm, 45 cm and 60 cm. Each participant completed 2 hands × 3 bending degrees of the arm × 15 target positions = 90 task blocks.

Table 1. A summary of the independent variables in experiment.

Independent Variables
1. Hand-use choice	Left; Right;
2. Arm-bending degree:     interaction distance	Wholly-bent: 30 cm; Half-bent: 45 cm; Stretched: 60 cm;
3. Hand-operational position	Horizontal angles (5): −90°, −45°, 0°, 45°, 90°; Vertical heights (3): overhead, shoulder, waist;

presents a summary of independent variables in this experiment.

Target size, i.e., the diameter of the spherical target shown in the experiment program was defined as 2.0 cm. The presentation order of the target positions, as well as the order of the arm used, were counterbalanced across the participants.

4.5. Data Processing and Analysis

Each participant completed 90 task blocks, and each block generated one task log file, thus a total of 2700 log files were collected by the end of the experiment. Each log file contained a 3D-coordinate record of the arm shoulder joint, all hand alternation moments, 300 records of the end-point scatter and the error times of each target position. The end-point scatter data were presented as ‘[−45°], [overhead], −623, 1763, 612’. The first and second values indicated the target position, and the following three values were X, Y and Z coordinate values in millimeters. Using the hand alternation moments, we calculated the hand-working durations in different conditions of target position and the arm’s bending degree. Records that were more than 3 standard deviations from the mean values (Z-score > 3.0) were treated as outliers and excluded. Finally, 800,361 (98.81%) records were preserved. Performance data for the acquisition error rate and coordinate records were measured to be normally distributed, thus they were analyzed by repeated-measures ANOVA with a post-hoc Tukey HSD test, and a two-tailed dependent T-test for paired comparisons. However, the questionnaire results were analyzed by a Kruskal–Wallis test with a post-hoc Dunn’s test, and a Mann–Whitney test for paired comparisons. All reported results below were significant at least at the p < 0.05 level.

5. Results

5.1. Target Acquisition Error Rate

Using the error times that were recorded in task log files, we calculated error rates in percentage (%). For example, there were a total of forty-eight error times recorded at the horizontal angle of 0° at the shoulder height in 90 log files (30 participants × 3 hand-operational distances) of the left-hand interaction; the error rate of the left-hand acquisition at this position was 48/(90 × 10) = 0.0533.

Table 2. Target acquisition error rates at different positions by either hand.

Hand Use Choice	Arm-Bending Degree	Hand-Operational Height	Acquisition Error Rate at Horizontal Angles
			−90°	−45°	0°	45°	90°
Left hand	Stretched	Overhead	0.063	0.053	0.060	0.100	0.130
		Shoulder	0.067	0.043	0.057	0.083	0.117
		Waist	0.050	0.040	0.043	0.067	0.127
	Half-bent	Overhead	0.053	0.040	0.050	0.073	0.113
		Shoulder	0.047	0.033	0.053	0.057	0.090
		Waist	0.037	0.030	0.040	0.047	0.103
	Wholly-bent	Overhead	0.040	0.027	0.033	0.053	0.073
		Shoulder	0.033	0.023	0.030	0.047	0.070
		Waist	0.027	0.020	0.027	0.040	0.063
Right hand	Stretched	Overhead	0.110	0.093	0.053	0.050	0.057
		Shoulder	0.107	0.080	0.050	0.033	0.053
		Waist	0.093	0.067	0.033	0.030	0.050
	Half-bent	Overhead	0.100	0.063	0.047	0.037	0.050
		Shoulder	0.083	0.047	0.047	0.027	0.040
		Waist	0.097	0.040	0.030	0.023	0.047
	Wholly-bent	Overhead	0.070	0.047	0.030	0.023	0.033
		Shoulder	0.067	0.040	0.027	0.027	0.030
		Waist	0.083	0.037	0.027	0.017	0.027

presents a summary of error rate statistical results.

Figure 5 visualizes the target acquisition error rates at distinct horizontal angles by left and right hands. In the left-hand interactions, it can be observed that target acquisition error rates on the left side (at angles of −90° and −45°) were significantly lower than those on the right side (at angles of 45° and 90°); however, in the right-hand interactions, the result was converse. By comparing the error rates of the left-hand interactions and the right-hand interactions, no significant difference was found (t(1349) = 0.36, p = 0.63). This indicates that the user-preferred or dominant hand has no advantage in interaction accuracy than the non-dominant hand.

A repeated-measured ANOVA of Hand-use choice × Arm-bending degree showed that the effect of the Hand-use choice is not significant (F(1, 14) = 0.47, p = 0.505), indicating that although all participants were right-handers, there is little difference between error rates made by the dominant hand and the non-dominant hand. However, there was a significant effect of the Arm-bending degree (F(2, 28) = 126.42, p < 0.05). Paired comparisons using a two-tailed T-test showed that the wholly-bent arm posture results in the lowest error rate, while the stretched arm posture generates the highest error rate (see in

Table 3. Target acquisition error rate in different conditions of the arm-bending degree and the hand-operational height, and pairwise comparisons.

Arm-Bending Degree	Mean Error Rate	Std. Dev.	Post-Hoc Pairwise Comparisons
Stretched	0.069	0.013	Stretch—Half-bent: p < 0.05, Cohen’s d = 1.12
Half-bent	0.055	0.011	Stretch—Wholly-bent: p < 0.05, Cohen’s d = 2.38
Wholly-bent	0.040	0.009	Half-bent—Wholly-bent: p < 0.05, Cohen’s d = 1.48
Hand-Operational Height	Mean Error Rate	Std. Dev.	Post-Hoc Pairwise Comparisons
Overhead	0.061	0.011	Overhead—Shoulder: p < 0.05, Cohen’s d = 0.64
Shoulder	0.054	0.010	Overhead—Waist: p < 0.05, Cohen’s d = 1.04
Waist	0.049	0.012	Shoulder—Waist: p < 0.05, Cohen’s d = 0.45

).

Figure 6 visualizes the error rates at different heights by either hand. One-way ANOVA of Hand-Operational Height showed that the target acquisition at different heights results in different error rates (F(2, 89) = 3.98, p = 0.022). Post-hoc Tukey HSD comparisons showed that target acquisition overhead have a significantly higher error rate than that at the shoulder height, and the target acquisition at the shoulder height generates a significantly higher error rate than that at the waist height (see in Table 3).

5.2. End-Point Scatter

As previously mentioned, the task log files recorded 3D coordinates of the fifteen target actual positions and the end-points. Using the coordinate records, we calculated the end-point scatter from the target’s actual position, as illustrated in Figure 7. The calculation method of the end-point scatter in different directions can be expressed as shown below:

$$\mathrm{scatter}\left(x,y, z\right)=\sqrt{{\left(x_2-x_1\right)}^2+{\left(y_2-y_1\right)}^2+{\left(z_2-z_1\right)}^2}$$

(3)

According to Equation (3), the end-point scatter reflected the integrated offset in three directions. We used a visualized diagram to present the end-point scatter result at different positions by either hand, shown in Figure 8. The diameter of the circles reflects the standard deviation of scatters, the center of circles refers to the averaged position of end-points and the black line indicates the offset from the end-points’ average position to the target’s actual position. A larger circle represents the end-points being more scattered, and the target acquisition being less accurate.

A repeated-measures ANOVA of Hand-use choice × Arm-bending degree × Hand-Operational Height × Horizontal Angle showed that Hand-use choice has no obvious effect on the end-point scatter (F(1, 291) = 1.86, p = 0.137), indicating that two hands generate little difference in end-point scatter. Meanwhile, there are significant effects of the Arm-bending degree (F(2, 582) = 8.97, p < 0.05), Hand-Operational Height (F(2, 582) = 25.48, p < 0.05) and Horizontal Angle (F(4, 1164) = 207.16, p < 0.05). There is also a significant interaction effect of Hand-use choice × Hand Horizontal Angle (F(1, 291) = 51.14, p < 0.05).

Table 4. Statistical analysis on the end-point scatter and pairwise comparisons.

Hand-Use Choice		Mean	Std. Dev.	F-Value	p-Value	Post-Hoc Pairwise Comparison(s)
Left hand		7.25 mm	2.31	F(1, 291) = 1.68	=0.137	-
Right hand		7.06 mm	2.26
Arm’s Bending Degree		Mean	Std. Dev.	F-Value	p-Value	Post-Hoc Pairwise Comparison(s)
Stretched		8.41 mm	2.19	F(2, 582) = 8.97	<0.05	Stretched—Half-bent: p < 0.05, Cohen’s d = 0.54
Half-bent		7.21 mm	2.23			Stretched—Wholly-bent: p < 0.05, Cohen’s d = 0.91
Wholly-bent		6.42 mm	2.16			Half-bent—Wholly-bent: p < 0.05, Cohen’s d = 0.36
Operational Height		Mean	Std. Dev.	F-Value	p-Value	Post-Hoc Pairwise Comparison(s)
Overhead		8.38 mm	2.43	F(2, 582) = 25.48	< 0.05	Overhead—Shoulder: p < 0.05, Cohen’s d = 0.37
Shoulder		7.50 mm	2.29			Overhead—Waist: p < 0.05, Cohen’s d = 0.97
Waist		6.15 mm	2.14			Shoulder—Waist: p < 0.05, Cohen’s d = 0.61
Horizontal Angle(s)		Mean	Std. Dev.	F-Value	p-Value	Post-Hoc Pairwise Comparison(s)
Left side	−90°, −45°	8.33 mm	2.49	F(2, 582) = 207.16	<0.05	Left—Middle: p < 0.05, Cohen’s d = 2.09
Middle	0°	4.04 mm	1.48			Left—Right: p = 0.84, Cohen’s d = 0.12
Right side	45°, 90°	8.02 mm	2.71			Middle—Right: p < 0.05, Cohen’s d = −1.82
Hand-Use Choice × Horizontal Angle(s)		Mean	Std. Dev.	F-Value	p-Value	Post-Hoc Pairwise Comparison(s)
Left hand × left side		5.51 mm	1.72	F(1, 291) = 51.14	<0.05	Left side—Right side: p < 0.05, Cohen’s d = −1.81
Left hand × right side		10.63 mm	3.61
Right hand × left side		11.14 mm	3.26			Left side—Right side: p < 0.05, Cohen’s d = 2.18
Right hand × right side		5.40 mm	1.81

presents the detailed analysis result of the pairwise comparisons. In summary, a stretched arm posture generated a more obvious end-point scatter than a half-bent arm posture. A wholly-bent arm posture generated the least end-point scatter; hand-controller interaction at the waist height resulted in a smaller end-point scatter than that at the shoulder height, but end-point scatter at the overhead height was the most obvious. In left hand-controller interaction, end-point scatters on the right side (at angles of 45° and 90°) were larger than that on the left side (at angles of −45° and −90°); however, in right hand-controller interaction, the result was converse.

We additionally analyzed the scatter level in the depth direction (Z-axis in Figure 7) and found that the end-point coordinate values in the depth direction were generally smaller than the actual value of the target, as shown in Figure 9. In other words, the majority of the end-points were distributed at a closer position than the target. This implies that errors in distance estimation are a prevalent problem in computer-simulated virtual environments and should be of particular concern in a virtual environment.

5.3. Hand Operation Persistence and Perceived Arm Fatigue Assessment

According to a measurement by Tanii et al. [13], a more tiring arm posture leads to a shorter duration and more frequent alternations between two arms, suggesting that the hand-operational task is less coherent. In this regard, hand-operational persistence can be used as an independent metric in hand-controller interaction evaluations. The hand alternation moments were recorded in the task log files and the interaction duration times of each hand in different conditions of Hand-use choice × Arm’s Bending Degree × Hand-Operational Height × Horizontal Angle were calculated, as presented in

Table 5. Hand durations (in seconds) in different conditions of hand-use choice, hand-operational position and arm-bending degree.

Hand-Use Choice	Arm-Bending Degree	Mean Duration Time (in Seconds), Std. Dev.
		Hand-Operational Height			Horizontal Angle
		Overhead	Shoulder	Waist	−90°	−45°	0°	45°	90°
Left hand	Wholly-bent	175.4,	191.7,	233.6,	146.4,	226.1,	196.6,	160.1,	141.3,
		27.92	30.61	37.94	23.35	36.43	37.72	25.64	22.48
	Half-bent	150.2,	163.0,	216.2,	155.2,	189.9,	174.6,	132.2,	127.7,
		23.77	28.87	33.27	26.62	23.48	30.02	22.23	23.82
	Stretched	127.3,	135.3,	188.4,	130.3,	170.7,	142.5,	119.6,	101.2,
		20.48	25.49	27.63	19.49	26.77	25.31	17.72	20.22
Right hand	Wholly-bent	202.2,	215.0,	258.2,	157.4,	177.4,	213.9,	238.4,	164.8,
		32.23	37.48	37.78	30.46	30.82	41.22	43.37	29.21
	Half-bent	167.3,	188.5,	234.1,	140.8,	148.7,	186.4,	202.5,	173.2,
		30.45	30.69	30.34	24.66	23.39	33.14	30.43	30.33
	Stretched	139.8,	152.4,	208.6,	115.5,	130.8,	158.5,	181.6,	143.9,
		25.77	27.41	30.12	20.02	19.24	28.91	28.35	22.75

. A two-tailed T-test showed that the right hand interaction generates a significantly longer duration time than the left hand interaction (mean duration time of the left hand: 162.3 s; mean duration time of the right hand: 179.2 s; t(1349) = −2.59, p < 0.05). Given all participants were right-handed users, this indicates that the dominant hand has a higher persistence than the non-dominant hand in completing the interaction tasks.

A repeated-measures ANOVA was further conducted to analyze the hand duration result. It revealed a significant effect of the Arm-bending degree (F(2, 118) = 212.4, p < 0.05), a significant effect of Hand-Operational Height (F(2, 118) = 130.7, p < 0.05), and a significant effect of Horizontal Angle (F(4, 236) = 99.8, p < 0.05) on the hand duration result. Post-hoc Tukey HSD pairwise comparisons between different arm-bending degrees showed that a stretched arm generates the shortest duration time, while a wholly-bent arm generates the longest duration time (M(stretched) = 146.7 s, M(half-bent) = 171.9 s, M(wholly-bent) = 193.7 s; all p < 0.05), as plotted in Figure 10a. Pairwise comparisons between hand-operational heights showed that the hand interaction at a higher position generates a shorter duration time (M(overhead) = 160.37 s, M(shoulder) = 174.32 s, M(waist) = 223.18 s; all p < 0.05). More specifically, hand-controller interaction with the arm raised above the head is more tiring than that at the shoulder height, which is more tiring than that at the waist height, as plotted in Figure 10b.

There was also a significant interaction effect of Hand-use choice × Horizontal Angle on the hand duration result (F(4, 236) = 182.1, p < 0.05), as shown in Figure 10c. Further statistical analyses were conducted for each hand. In the left hand interaction, hand duration on the left side of the body (at horizontal angles of −90° and −45°) was significantly longer than that on the right side (at horizontal angles of 45° and 90°) (M(−90°, −45°) = 169.77, M(45°, 90°) = 130.35; t(539) = 7.37, p < 0.05). However, in the right hand interaction, the result was converse: hand duration on the left side of the body was significantly shorter than that on the right side (M(−90°, −45°) = 145.10, M(45°, 90°) = 184.07; t(539) = −10.22, p < 0.05).

In this experiment, a subjective assessment on perceived arm fatigue was conducted to verify the hand duration result. As interpreted in the section of the experiment procedure, the subjective assessment was measured through a post-block questionnaire, which focused on six metrics: Mental Demand, Physical Demand, Temporal Demand, Effort, Performance, and Frustration Level. Each metric consisted of a detailed statement, as presented in Appendix A—Table A1. Participants rated the statements on a 5-point Likert scale from ‘strongly disagree’ to ‘strongly agree’. Figure 11 shows the perceived arm fatigue result in boxplots, in different conditions of hand-use choice, arm-bending degree and hand-operational height. A Mann–Whitney test showed a significant effect of Hand Choice on Physical Demand (U = 78.92, p < 0.05), on Temporal Demand (U = 145.06, p < 0.05), on Performance (U = 60.43, p < 0.05) and on Frustration Level (U = 96.70, p < 0.05). More specifically, (i) the left-hand interaction was perceived to be more physically demanding than the right-hand interaction; (ii) the former was perceived to be more suffering and burdensome than the latter in completing the task; (iii) the left-hand interaction was also perceived to be more successful and efficient with less frustration than the right-hand interaction.

Next, a Kruskal–Wallis analysis on perceived arm-fatigue level was conducted. It shows a significant effect of the Arm-bending degree on Mental Demand (${\chi }_2^2$ = 15.81, p < 0.05), Physical Demand (${\chi }_2^2$ = 27.28, p < 0.05), Temporal Demand (${\chi }_2^2$ = 52.02, p < 0.05), Performance (${\chi }_2^2$ = 11.14, p < 0.05) and Frustration Level (${\chi }_2^2$ = 20.61, p < 0.05). Post-hoc Dunn’s comparisons show that (i) the bent arm posture was perceived to be more mentally and physically demanding than the stretched arm posture; (ii) the stretched arm posture was more tiring and suffering than the wholly- and half- bent arm postures; (iii) the bent posture was perceived to be more accustomed by and efficient in completing the interaction tasks with less frustration than the stretched arm posture. It also shows that the Hand-Operational Height has a significant effect on perceived arm-fatigue level, in terms of Mental Demand (${\chi }_2^2$ = 21.09, p < 0.05), Physical Demand (${\chi }_2^2$ = 35.11, p < 0.05), Temporal Demand (${\chi }_2^2$ = 19.74, p < 0.05), Effort (${\chi }_2^2$ = 40.02, p < 0.05) and Frustration Level (${\chi }_2^2$ = 45.30, p < 0.05). Post-hoc Dunn’s tests show that (i) hand-controller interaction over the head was more mentally and physically demanding than that at the shoulder and waist heights; (ii) hand-controller interaction at a higher position was perceived to be more tiring and painful than that at a lower position; moreover, (iii) hand-controller interaction at the shoulder or waist height was perceived to be more relaxed and comfortable with less confusion than that at the head height. Overall, the subjective assessment results on perceived arm fatigue level are highly consistent with the hand duration results.

Post-study comments from the participants were collected and analyzed. In general, the comments conformed to the task performance and the questionnaire assessment results. Two participants noted that they felt strained while acquiring targets at an arm-length distance. However, when acquiring targets at a closer distance, hand interaction became more relaxed and durable. This was also reported by the majority of the other participants. Three other participants pointed out that they were more accustomed to using the right hand; five participants added that hand interaction at the waist height and right side of the body was more efficient and comfortable than at other positions; this was also acknowledged by nine other participants. Additionally, two participants commented that the target presented in the experimental program was large enough to be selected precisely at a 30 cm distance; however, when the target was shown at an arm-length distance, target selection became less precise. Their interaction performance objectively reflected this comment, and the other twenty-eight participants also reported this point.

In summary, the experiment revealed that arm posture, in particular arm-bending degree, and the hand-operational position are two significant variables influencing hand- controller interaction accuracy in VR. It was found that a stretched arm posture generates more target acquisition errors, at the same time resulting in more scattered end-points, in comparison with the bent or half-bent arm postures. In addition, it was found that a stretched arm posture has a more persistent interaction with fewer hand alternations throughout the interaction task, in comparison with a wholly- or half-bent arm posture. Thus, Hypothesis 1 (H1) is completely supported. It was also found that targets located at the waist height are acquired more accurately than those at the shoulder and overhead heights, while hand operation at a lower position was also found to be more relaxed and to generate a longer duration with fewer hand alternations throughout the task. Based on these findings, Hypothesis 2 (H2) is also supported. Additionally, the hand-use choice effect on hand-controller interaction performance was observed in the experiment. In particular, in the left-hand interaction, acquiring targets on the left side of the body is more accurate than that on the right side; however, in the right-hand interaction, the result is converse. From this perspective, Hypothesis 3 (H3) is also supported. Apart from the hand-use choice effect, the individual handedness characteristic was found to impact interaction persistence, the dominant hand having a longer duration than the non-dominant hand. However, the dominant hand and the non-dominant hand generate no obvious difference on hand-controller interaction accuracy. Hypothesis 4 (H4) is not supported.

6. Discussion

Through a hand-controller target-acquisition experiment in a VR environment, this paper proves that the arm’s biomechanical structure and its kinematic features are critical factors, having a close correlation with hand-controller interaction performance, in terms of both overall task efficiency, accuracy and user-perceived comfort. On the basis of earlier research, such as Hincapié-Ramos et al. [14] and Lou et al. [21], this study extends and gains a more comprehensive understanding about arm posture change and its dependent changes in perceived arm fatigue and user experience, which is beneficial for developing a more scientific metric for quantitatively measuring user comfort and overall user experience quality. Neurophysiological studies revealed that human visual perception had a close correlation and a dominant effect on visually guided hand operations, which were widely believed to explain the hand interaction difference in upper and lower visual fields [38,39]. However, in this study, we suppose that it is the more obviously perceived arm fatigue in an upraised arm posture which generates a lower hand interaction accuracy than that through a downward arm posture. From this perspective, this study provides new evidence for explaining and predicting hand interaction differences at distinct heights. From an ergonomical evaluation perspective, this study also provides a new view of the evaluation method of hand-controller interaction. Although Fitts’ law [36] is the most classical and universally accepted theoretical model in earlier HCI evaluation practices [21,22,28,35], it is little concerned with the handedness characteristic and hand choice and their influences on the interaction performance that have been revealed in this study. Therefore, the newly achieved findings will also enlighten subsequent researchers to propose an updated Fitts’ law model that will be more applicable to hand-controller interaction evaluation in a 3D space. Given all the experimental findings, several featured contributions of this study are discussed, including suggestions and strategies that are universally applicable for designing more efficient and user-satisfying interactive spaces in VR, as well as implications for developing more effective and comprehensive metrics in hand-controller interaction evaluation.

First, biomechanical structure and kinematic features of the user’s arm, e.g., the arm’s movement accuracy, comfort level and perceived fatigue level at different arm postures, were insufficiently considered in earlier ergonomical evaluations in a VR environment. However, in this study, all these characteristics were considered and measured. From this perspective, when designing 3DUIs in VR applications, more ergonomical factors and evaluation metrics should be considered simultaneously. In particular, (a) the handedness characteristic and the hand-use choice effect should be taken into consideration in hand-controller interaction evaluations; (b) apart from the traditional metrics of interaction efficiency and accuracy, the arm’s operational persistence, or task duration without hand alternations, is another performance metric which should also be measured in hand-controller interaction evaluations; moreover, (c) interaction naturalness, user immersion, physical effort and comfort level are all important metrics which should be comprehensively considered.

Second, this study revealed that hand-controller interaction in some spaces is more flexible and accurate than in other spaces. In general, hand-controller interaction at waist height has a higher accuracy than that at shoulder height; hand-controller interaction at shoulder height generates a higher accuracy than that above head height. We named this result the ‘hand-operational height effect’. Moreover, the left arm and hand interact more accurately on the left side of the body, but the right arm and hand interact more accurately on the right side of the body. We named this contrastive performance of two hands the ‘hand-use choice effect’ which was rarely investigated in earlier research. Based on the hand-operational height effect and the hand-use choice effect, we suggest applying a hand-tracking technique in VR applications and identifying the operating hand in real time and, based on this, organizing the interactive space and the user–interface layout in the VR environment.

Third, this study introduced a new evaluation metric—hand-operational persistence or task duration—to evaluate the hand-controller interaction more comprehensively. A higher persistence or a longer duration suggests that the interaction task is completed more coherently with fewer hand alternations. In this regard, the hand interaction task duration can be used as a metric for measuring overall task efficiency. In addition, the duration result was verified to be highly consistent with the user-perceived arm fatigue result. In this regard, the duration measurement can also be used as a metric in user-perceived comfort evaluation.

The experiment was conducted in a virtual environment with an Oculus HMD for important reasons. First, the most fundamental purpose of this research was to investigate the hand-controller interaction performance and relevant influencing factors; however, there are multiple fields in which hand-controller interaction techniques can be applied, such as motion-sensing-based video games, VR and AR applications. Among these, VR is particularly a prevalent application in which hand-controller interaction is already widely integrated. Findings and inspirations gained from the current experiment can benefit many other similar applications. Second, the interactive space in a VR environment is far larger than those in other applications, and VR is the most representative application supporting motion-sensing interaction within a global space in which interactive objects can be placed at any position surrounding the body and the user performs different arm postures to adapt to the different objects. From this perspective, a VR environment is an appropriate setup for measuring the effect of arm posture and hand-operational position on hand-controller interaction.

The participants recruited in the experiment were persons with basic knowledge of VR technology and hand-controller operation methods, or at least had experienced VR applications before. These participants were selected for specific reasons: according to our survey of the most representative empirical studies in the HCI domain in the last five years, the number of the participants ranged from six to thirty. Based on this, we suppose that a group size of thirty is appropriate and sufficient in this experiment. In addition, novice users need a long time to practice and become familiar with the VR environment before they can complete the interaction tasks successfully. However, experienced users can directly participate in the interaction tasks and generate a stable performance. Experimental results gained from experienced users can objectively reflect the reality of the target user group, thus this is a better choice. The experimenter surveyed three hundred volunteers through a questionnaire and found that sixty-nine of them qualified as participants. Most of these participants (>90.0%) were young users, aged from 21 to 36, indicating that users younger than 21 or older than 36 were uncommon users of VR technologies. Among these sixty-nine participants, there were eighteen females (26.2%), while the others were males, indicating that male users had a far larger percentage than female users of VR applications. Based on these, twenty-two males and eight females were selected as participants in the experiment. Nevertheless, it must be acknowledged that the gender factor and its potential effect have not been considered in this paper. ‘Whether males and females have different interaction results’ and ‘how the VR system should respond based on individual gender’ are worthwhile issues, but have been little explored to date; however, they have been planned in our future work. Since all the participants were right-handed, findings about the handedness effect on interaction performance were drawn from right-handed but not from left-handed users, this is also a limitation of this paper. We plan to undertake a more comprehensive comparison between left-hand interaction and right-hand interaction in future work. Adolescents and elders are less common users of the VR technology and hand-controller interaction by these users has not been evaluated, which is another limitation of this paper. We plan to make a deeper investigation into a broader age group and, based on this, we intend to summarize the age effect and its implications on developing hand-controller interaction techniques for specific users.

7. Conclusions

This study revealed that the user’s arm posture and the hand-operational position have significant influences on hand-controller interaction accuracy and hand-operational persistence in VR. Compared with the non-dominant hand, the user-preferred or dominant hand was found to interact more persistently, though it has no advantage in interaction accuracy. Hand-controller interaction through a wholly-bent or half-bent arm posture was found to be more accurate than that through a stretched arm posture, and a bent arm posture was also assessed to be more relaxed with less perceived arm fatigue than a stretched arm posture. Additionally, interaction at the waist height was found to more accurate and persistent than that at the shoulder or head above height. Compared with earlier research, this study provides a clearer understanding about arm posture change and its effect on hand-controller interaction performance. The experimental findings suggest guidelines and strategies for designing more efficient and user-satisfying interactive spaces in VR, at the same time providing a more scientific and comprehensive standard for evaluating hand-controller interaction. More complicated interaction cases, including the young and older users engaged in tasks and two-hand cooperative performance were not considered in the current study; however, they will be included in our future work.