Cross-Modal Effect of Presenting Visual and Force Feedback That Create the Illusion of Stair-Climbing

Abstract

Image therapy, which creates illusions with a mirror and a head mount display, assists movement relearning in stroke patients. Mirror therapy presents the movement of the unaffected limb in a mirror, creating the illusion of movement of the affected limb. As the visual information of images cannot create a fully immersive experience, we propose a cross-modal strategy that supplements the image with sensual information. By interacting with the stimuli received from multiple sensory organs, the brain complements missing senses, and the patient experiences a different sense of motion. Our system generates the sense of stair-climbing in a subject walking on a level floor. The force sensation is presented by a pneumatic gel muscle (PGM). Based on motion analysis in a human lower-limb model and the characteristics of the force exerted by the PGM, we set the appropriate air pressure of the PGM. The effectiveness of the proposed system was evaluated by surface electromyography and a questionnaire. The experimental results showed that by synchronizing the force sensation with visual information, we could match the motor and perceived sensations at the muscle-activity level, enhancing the sense of stair-climbing. The experimental results showed that the visual condition significantly improved the illusion intensity during stair-climbing.

1. Introduction

When the motion-related region of the brain is damaged by a stroke, the body cannot execute its desired movements. In many stroke-paralyzed patients, the neural network controlling the paralyzed limb shrinks because the limb is not used, causing further functional depression. This scenario constitutes a vicious circle [1,2]. Current rehabilitation methods passively repeat simple repetitive movements, which are unlikely to reconstruct the neural network of the damaged brain. Some studies have reported that voluntary movements and their expression in the brain are important for the recovery of motor function [3,4]. The sense of motion is usually perceived during a motion, but can be generated in the absence of motion. A visually induced kinesthetic illusion instills the sense of body movement when the body is moving neither voluntarily nor passively. Such an illusion increases the excitability of corticospinal tracts during voluntary movement and activity in the premotor and supplementary motor areas of the brain [5]. The stimulus immediately increases the quantity of electromyograms and improves the motion range of the joint [6]. As a stroke patient cannot easily recall the correct image for relearning a motion, some therapists generate an illusionary image using a mirror and a head mount display. Mirror therapy presents the movement of the unaffected limb in a mirror, creating the impression that the limb on the affected side is moving. Illusion-based rehabilitation is applied to dynamic motions such as walking [7].

This paper presents an illusionary system that induces the sense of stair-climbing without walking on physical stairs. In the present VR technology, the immersive feeling to the virtual environment can be improved by using a head-mounted display (HMD). The following literature is cited as research on VR. There are kinesthetic presentation systems by VR [8], kinesthetic presentation systems by VR with a tactile interface [9], and kinesthetic presentation systems by visual and force sense [10]. The visual information from images alone is not immersive enough, we incorporate walking movements into the virtual environment. Space-perception operations in a virtual reality environment allow redirected walking [11] and an infinite walking locomotion interface [12] even in a narrow space. Users walking in real space can recognize the ground and space situation in both virtual reality and real space environments. This cross-modal phenomenon improves the illusion intensity. When the brain interacts with stimuli received from multiple sensory organs, it complements missing senses and generates a different sense of motion. Ishikawa et al. induced the weight sensation of a virtual object colliding with a hand by stimulating the skeletal hand muscles with electrical signals and vibrations [13]. Nagao et al. proposed a cross-modal system that imparts the sensation of ascending and descending stairs [14]. In this conventional system, a sense of stair-climbing is passively achieved when the user steps on physical protrusions. This system exploits the tendency of humans to unconsciously step on the edges of stairs, recalling the sense of stair-climbing [15]. However, the protrusions in this stair-climbing system must be installed in the appropriate position, and are spatially restricted. A system that induces a sense of stair-climbing in a wearable configuration is expected to be more widely applicable. Our system generates an illusion of stair-climbing by reproducing the muscle sense corresponding to the stair-climbing motion in the virtual environment. The user receives a visual presentation and wears a force-feedback device (see Figure 1). The force feedback is provided by actuators that behave as artificial muscles. The generation of muscle activity by force feedback, which provides the illusion of stair-climbing while walking on a flat level, is the main contribution of this paper. The force feedback compensates the difference in the muscle-exertion patterns between level walking and stair-climbing. Accordingly, the muscle activity of the lower limbs during level walking approaches that of stair-climbing.

The remainder of this paper is organized as follows. Section 2 describes the artificial muscles in the wearable force-feedback device and the cross-modal system that induces the sense of stair-climbing. Section 3 presents the experimental procedure and the results, and Section 4 presents the effects of the constructed system. Section 5 draws the conclusions and suggests a perspective for future research based on the current knowledge.

2. Materials and Methods

2.1. System Overview

An overview and configuration of the developed system are shown in Figure 2. The constructed system consists of two main parts: a visual display part using a HMD, and a force display part using a low-pressure pneumatic actuator called a pneumatic gel muscle (PGM). The details of each part are described in the following subsections.

2.2. Visual Display

The visual display is configured as shown in Figure 3. This part presents a virtual environment through the HMD (VIVE PRO HMD, HTC Co., Ltd., New Taipei City, Taiwan) and captures foot motions through trackers (Vive Tracker) attached to the shoes of the user. The real-time presentation of foot movement in the virtual environment creates an immersive and realistic experience for the user. The image displayed on the HMD is shown in Figure 3b.

2.3. Force Display

2.3.1. Overview of the Force Display Part

The force display part is configured as shown in Figure 4. This part creates the muscle sensation through PGMs worn on the lower limb of the user. The hip joint angle was acquired by a 9-axis acceleration sensor (M5Stack Co., Ltd., Shenzhen, China) installed on the femoral side. The intensity and timing of the force feedback were controlled by solenoid valves based on the hip joint angle. Compressed CO${}_2$ was supplied to the PGMs from a portable CO${}_2$ tank through the valves. Figure 5 shows the process of contraction of the PGM.

2.3.2. Pneumatic Gel Muscle

The PGM actuator is a specially designed low-pressure artificial muscle that generates high forces at low air pressure. The actuator is a low-pressure McKibben-type pneumatic artificial muscle (PAM). PAMs are used in various systems such as master–slave arm rehabilitation equipment [16], power-assist gloves for support and fatigue reduction of fingers [17], and power-assist suits for walking support of hemiplegic patients [18]. The PGMs shown in Figure 6 were developed by DAIYA Industry, Japan. Like other PAMs, PGMs have two layers of materials: an outer mesh composed of a commercially available plastic mesh and an inner tube formed from a customized styrene-based thermoplastic elastomer. Owing to its customized inner tube, a PGM can generate high forces at very low air pressures. The inner tube is entirely tied and concealed by the outer mesh. When compressed air is fed to the actuator, the inner tube inflates like a balloon until the inflation is checked and reversed by the crossed pattern of the mesh. The resultant shrinking action generates a linear force when the endpoints of the PGM are fastened to a base.

2.3.3. Force Characteristics

The PGM contracts linearly when compressed air enters through one of the ends. At zero or very low air pressure, the actuator behaves like a spring. The stiffness increases at higher air pressures. The characteristics of PGMs also depend on their lengths. We measured the force characteristics of a 300 mm natural-length PGM as functions of input air pressure and displacement. One end of the PGM was fixed to a force sensor and the other was fixed to a metal frame. The force sensor was attached firmly to a vertical linear guide for changing the displacement or level of stretch in the PGM. The displacement directly inferred the amount of external force applied to both ends of the actuator. Figure 7a shows the contraction force characteristics of the PGM as the air pressure changes from 50 to 300 kPa. The contraction force also depended on the rest length of the PGM. Figure 7b plots the contraction force versus air pressure in PGMs of length 260, 280, and 300 mm.

2.4. Motion Analysis

The muscle exertion patterns differ between stair-climbing and level walking. Stair-climbing requires more muscle activity in the quadriceps and vastus lateralis muscles than level walking [19]. By placing a greater physical load on the lower extremities during level walking, we can generate a muscle exertion pattern similar to that of stair-climbing. For this purpose, we attached the PGMs above the knee of the user, as shown in Figure 2a. The contraction force to be exerted by the PGMs was designed from inverse dynamics calculations in a body model. The aim was to provide a muscle sensation corresponding to the muscle exertion pattern during stair-climbing and level walking. The motions were recorded by a motion analysis system (Optitrack, Acuity Co., Ltd., Tokyo, Japan) and a floor-reaction force meter (Bertec Co., Ltd., Columbus, OH, USA). The desired knee joint torque was calculated based on the two-dimensional skeleton model. Joint torque and muscle activity pattern were calculated by the musculoskeletal model, which more accurately reflects the human musculoskeletal system. By these, it can be expected to increase the type of the kinesthesia that can be presented and to increase the accuracy. The inverse dynamics were computed using SKYCOM analysis software. The body torque was determined from the three-dimensional position coordinates, acceleration, and angular acceleration obtained by the motion capture system, and the external force obtained by the floor-reaction force meter. When wearing the PGM, the knee joint torque in the bending direction was defined as $\tau$, the torque predicted by PGM was ${\tau }_M$, and the actual knee joint torque obtained by the three-dimensional motion analysis was ${\tau }_W$. These three torques are related as

$$\begin{array}{c}\hfill \tau ={\tau }_M+{\tau }_W.\end{array}$$

(1)

Figure 8a presents the kinematic model for calculating $\tau$. Here, the force generated by the PGM is F and the knee joint angle is $\theta$. From the rotation center, the distances to the knee joint and PGM mounting position are denoted as $L_1$ and $L_2$, respectively. The length L of the PGM is calculated from $\theta$ as follows:

$$\begin{array}{c}\hfill L=\sqrt{l_1^2+l_2^2-2l_1{l}_{2}cos\theta }.\end{array}$$

(2)

The contraction-force characteristics of the PGM are controlled by the air pressure P. Figure 7 plots the relationship between the generated force and the provided air pressure. The knee joint torque ${\tau }_M$ accompanying the force from the PGM is a function of the lengths, the contraction force $F_p$, and the number of PGMs k:

$$\begin{array}{c}\hfill {\tau }_M=\frac{k{F}_p\sqrt{4L^2{l}_1^2-{(L^2+l_1^2-l_2^2)}^2}}{2L}.\end{array}$$

(3)

The torques ${\tau }_W$ and ${\tau }_M$ during the walking cycle are plotted in Figure 8b,c, respectively. Substituting ${\tau }_W$ and ${\tau }_M$ into Equation (3), we get the total knee joint torque $\tau$. Figure 8d shows the total knee joint torque during the walking cycle when the PGMs are operated at different air pressures (0–300 kPa).

The knee joint torque peaked at approximately 40% of the walking cycle. Figure 9a,b plot the peak knee-joint torques as functions of air pressure during level walking and of stair height during stair-climbing (also calculated by inverse dynamics analysis), respectively. Meanwhile, Figure 9c relates the air pressure P to the stair height H, derived from the linear regression lines of Figure 9a,b. From this figure, the linear relationship between the air pressure in the PGM and the stair height was determined as

$$\begin{array}{c}\hfill P=920H-21.\end{array}$$

(4)

Equation (4) gives the air pressure at which the PGM can generate the sense of stair-climbing in the virtual environment. For example, when the stair height in the virtual environment is set to 200 mm, the PGM should receive an air pressure of 200 kPa.

3. Experiments and Results

The performance of the developed system was measured in experiments. Experiment 1 evaluated the muscle activity based on EMG measurements and Experiment 2 evaluated the illusion intensity based on the questionnaire responses of the participants.

3.1. Experiment 1: EMG Measurements

By measuring surface myoelectricity, we determined whether the force feedback in the PGM suit creates the desired muscle activity.

3.1.1. Method

Seven males (age 21–23 yo) participated in the experiment. The subjects consisted of people who had no muscle aches, fatigue, or malaise and who had no trouble in walking. Prior to the experiments, each participant answered verbal questions on their health and gave informed consent to participate. Participants with nonideal health were rescheduled to a later date. The EMG measurements were performed under the following conditions:

• Level walking without feedback;
• Stair-climbing without feedback (physical stair-climbing);
• Level walking with visual and force feedback (virtual stair-climbing).

During level walking, the participants walked along a straight path of approximately 4 m. During physical stair-climbing, the participants climbed 10 steps, each with a height of 200 mm and a tread of 250 mm. In the virtual environment, the virtual stair height was set to 200 mm, and the air pressure supplied to the PGM was 200 kPa. All tasks were carried out three times at the natural walking speed of the participant. The motions of the vastus lateralis, biceps femoris, tibialis anterior, and gastrocnemius muscles of the dominant leg were measured. The EMG signals were measured by a wireless surface EMG measurement device (DELSYS Trigno, Natick, MA, USA). The sampling frequency was 2000 $\mathrm{Hz}$. The EMG waveforms were evaluated after bandpass filtering at 20– 450 $\mathrm{Hz}$ and a smoothing process. At the end of the task, the EMG of each participant was normalized by the maximal voluntary contraction (MVC) of the associated muscle, which was measured in isometric contraction for each participant.

3.1.2. Results

Figure 10 displays representative EMGs during a stable walking cycle. The EMG of the gastrocnemius muscle was activated when the foot touched the rear, and that of the tibialis anterior muscle was activated in the swing leg. Under both physical and virtual stair-climbing conditions, the EMG of the vastus lateralis muscle greatly increased when the foot touched the floor. Figure 11 summarizes the peak values of the EMG in each muscle, averaged among the three trials of each subject. Differences among the three groups were tested by one-way ANOVA and Tukey–Kramer multiple comparisons with a 0.05 significance level. The results of the vastus lateralis muscle significantly differed between level walking and physical stair-climbing (p = 0.000156), and between level walking and virtual stair-climbing (p = 0.00046). The EMG of the vastus lateralis muscle peaked at 15.3% and 49.0% during level walking and physical stair-climbing, respectively. Meanwhile, the EMG peaked at 45.8% during virtual stair-climbing, clarifying that the muscle activity generated in the developed system was similar to that of physical stair-climbing. The gastrocnemius muscle peaked at 26.0%, 49.0%, and 24.8% during level walking, physical stair-climbing, and virtual stair-climbing, respectively. The activities of the biceps femoris and tibialis anterior muscles showed no significant differences among the three groups, but the activities of the muscles significantly differed between physical and virtual stair-climbing.

3.2. Experiment 2: Questionnaire Survey

3.2.1. Method

The seven subjects that participated in the first experiment participated in the second experiment on a different day. The conditions of the second experiment were as follows:

• Visual feedback;
• Visual and haptic feedback (conventional method [14]);
• Visual and force feedback (proposed method).

Tests on the conventional system [14] revealed that when users stepped on protrusions installed on the floor at positions corresponding to the edges of virtual stairs, which were presented via a head-mounted display, the users acquired a sense of stair-climbing. In this study, we evaluated the cross-modal effect of the visual and force feedback, and compared it with that of visual feedback alone. Figure 12 shows the experimental scenes under the conventional and visual+force conditions, respectively. To eliminate the order effect, the order of the experimental conditions was randomly decided. At the end of each task, the subjects were presented with ten questions related to the immersion experience and the intensity of the stair-climbing illusion. The immersion questions included six items based on the Immersive Virtual Environment Questionnaire (IVEQ) [20]. The questions on illusion intensity included four items. All answers were given on a five-point Likert scale. On the basis of the ‘0’ of the neutral response to the question, we set a minus score for negative responses and a plus score for positive responses.

3.2.2. Results

The questions and their answer rates are shown in Figure 13. The red and blue sides represent positive and negative responses, respectively, and the gray sections are neutral responses. Under the visual+force condition, more than half of the answers to all questions were positive. Figure 14 shows the responses to the questions on immersion and illusion intensity under the three conditions, and

Table 1. Results of a three-group test using one-way analysis of variance at the 0.05 significance level: (a) immersive experience; (b) illusion intensity. *: p value <= 0.05.

(a)
Condition	Difference	Lower	Upper	pValue
B-A	0.195	−0.829	1.22	0.879
C-A	0.514	−0.510	1.54	0.423
C-B	0.319	−0.705	1.34	0.711
(b)
Condition	Difference	Lower	Upper	pValue
B-A	0.964	−0.411	2.34	0.201
C-A	1.93	0.552	3.30	0.00579 *
C-B	0.964	−0.411	2.34	0.201

summarizes the statistical results analyzed by one-way ANOVA and the multiple comparisons made by the Tukey–Kramer method at the 0.05 significance level. The immersion responses were slightly higher under the conventional and visual+force conditions than under the visual condition, but the difference was not statistically significant. In contrast, the illusion intensity was significantly different between the visual+force condition and the visual condition.

4. Discussion

As a technique for presenting the target torque to the knee joint, it is possible to use the torque motor, and the effective result may be obtained. However, exoskeletal suits using torque motors are not easy to adjust for differences in wearer body size. PGM is one of the most suitable actuators for the presentation of the sense of motion, which is the purpose of this research, because it is lightweight, flexible, and wearability is high because movement is not interrupted.

The experiments verified whether presenting a muscle sense by force feedback can induce the perception of stair-climbing. As shown in Figure 11a, the activity level of the vastus lateralis muscle in the illusionary system is equivalent to that of physical stair-climbing. This result confirms that force feedback by PGM effectively promotes the muscle activity of the vastus lateralis muscle. However, the muscle activities of the gastrocnemius muscle were significantly different between physical and virtual stair-climbing. One reason for this difference is the limited motion analysis considered in the kinematic model. When walking on stairs, the body’s center of gravity is lifted by a kicking force, which increases the activity of the gastrocnemius muscle. In this study, the illusion intensity in the exertion force design was limited to the knee joint torque, which does not sufficiently account for activity of the gastrocnemius muscle. By placing additional PGMs that measure the ankle joint torque, we might generate similar muscle activity to that of actual stair-climbing over a wider range of lower-limb muscles.

The questionnaire responses to the immersion perception were positive under all three conditions, indicating that the visual display in the virtual environment greatly contributed to the sense of immersion. The immersion perception scores were not significantly different among the three conditions, but were largest under the visual+force condition. The intensity of the illusion differed between the visual condition and the visual+force condition, implying that the force display enhanced the illusionary effect.

Kinesthesia is a conscious awareness of body position and movement, maintained by afferent inputs from peripheral receptors (peripheral sensation) and centrifugal input from the center (motor command) [21]. In this experiment, the subject’s toes were displayed on the steps of the virtual stairs, so their height position differed from their actual position. Therefore, the kinesthesia generated by the actual joint motion did not always match the visual information presented by the HMD. In contrast to the conventional method, which uses the haptic stimuli of the protrusions, the proposed method attempts to match the muscle activities of the actual and virtual stair-climbing activities, thus compensating for the discomfort caused by the disparity between what the user is watching and perceiving. Although the experiments revealed no significant difference between the conventional and proposed methods, the proposed system improved both the immersion perception and illusion intensity, indicating that the cross-modal effect of vision and force effectively generated the kinesthesia of stair-climbing in the virtual space. In this study, the height of the virtual stairs was fixed at 200 mm. Increasing the stair height is expected to enlarge the difference between the kinesthesia and visual information, thus enhancing the effectiveness of the force sensation presented by the proposed method. However, if the actual motion excessively differs from the virtual motion, the joint-angle differences between the actual and virtual motions cannot be compensated by force presentation alone. In such situations, users would be discomfited by the proposed system. In future work, we will investigate the extent to which visual and force displays can bridge the difference between the actual and virtual body movements.

5. Conclusions

We have proposed a method that generates motion illusions by presenting a force sensation through artificial muscles that promote muscle activity. The force presentation is supplemented by visual information. In the verification experiment, our system generated the sensation of stair-climbing in subjects walking on a level floor. The force sensation was presented by a pneumatic gel muscle (PGM). Based on the motion analysis of a human lower-limb model and the force exerted by the PGM, the appropriate air pressure for the PGM was set. The proposed system was evaluated by surface EMG measurements and a questionnaire. The experimental results showed that when the force sensation was synchronized with the visual information, it could be matched to the motor sensation at the muscle-activity level, thus enhancing the stair-climbing sensation.

The proposed system is intended for the initial neurorehabilitation stages of patients with hemiplegia or other conditions that inhibit stair-climbing. To achieve this goal, we must increase the number of kinesthesia variations that can be generated. It is necessary to conduct the experiment considering subjects of various ages and gender. The simultaneous display of visual and force sensations should also be supplemented by touch and auditory modalities.