Wearable Drone Controller: Machine Learning-Based Hand Gesture Recognition and Vibrotactile Feedback

Abstract

We proposed a wearable drone controller with hand gesture recognition and vibrotactile feedback. The intended hand motions of the user are sensed by an inertial measurement unit (IMU) placed on the back of the hand, and the signals are analyzed and classified using machine learning models. The recognized hand gestures control the drone, and the obstacle information in the heading direction of the drone is fed back to the user by activating the vibration motor attached to the wrist. Simulation experiments for drone operation were performed, and the participants’ subjective evaluations regarding the controller’s convenience and effectiveness were investigated. Finally, experiments with a real drone were conducted and discussed to validate the proposed controller.

1. Introduction

Nowadays, multicopter drones have been widely used because of their simple mechanism, control convenience, and hovering feature [1]. Drones are important in surveillance and reconnaissance, aerial photography and measurement, search and rescue missions, communication relay, and environmental monitoring [2,3]. To complete such applications and missions, highly sophisticated and dexterous drone control is required. Autonomous control has been partly used in their applications, as in waypoint following and programmed flight and mission, because of limited autonomy [4,5,6]. However, in the autonomous flight of drones, sometimes the autopilot is switched to manual control by a human operator according to the flight phase, such as landing, and in unexpected circumstances. The human role is necessary in the control loop when the system cannot fully reach an autonomous state.

Therefore, natural user interfaces for human operators have been studied extensively. An early study reported a novel user interface for manual control based on gestures, haptics, and PDA [7]. Moreover, multimodal natural user interfaces, such as speech, gestures, and vision for human-drone interaction, were introduced [8,9]. Recently, hand gesture-based interfaces and interactions using machine learning models have been proposed [10,11,12,13,14,15]. Hand gestures are a natural way to express human intent, and their effectiveness for applications of human–machine/computer interface/interaction has been reported in previous works. Some of the applications were focused on the control of drones based on deep learning models [16,17,18,19]. They used vision, optical sensors with infrared lights, and an inertial measurement unit (IMU) to capture the motion of the hand. The IMU attached to the user’s hand senses the motion of the hand robustly compared to conventional vision systems, which are easily affected by light conditions and require tedious calibrations.

In contrast, to interact with a machine/computer, tactile stimulation has been adopted as a feedback means to humans for a long time [20,21,22]. Tactile stimulation is an additional channel to visual information for providing necessary information to humans. In particular, tactile feedback is important for visually impaired persons in terms of representing surrounding circumstances and the environment instead of having to rely on visual information [23,24]. Even for sighted people, tactile feedback helps improve their understanding of the environment when visual information is insufficient or blocked. Some previous work related to tactile and force feedback for drone control was reported [25,26,27]. Tsykunov et al. [25] studied a tactile pattern-based control glove for cluster flight of nano-drones. Cluster flights were intuitive and safe, in which the user’s position had to move continuously to guide the drone. Rognon et al. [26] addressed the flyjacket to control the drone with torso movements using IMU. The cable-driven haptic feedback system was mounted on the elbows of the user, who navigated the direction of waypoints. The flyjacket was demonstrated to control drones as much as joysticks and tablets, while the torso of the user was restricted. Duan et al. [27] developed a human–computer interaction system for improving realistic VR/AR. The tactile feedback device was used for estimating environments and objects. The gesture-based control was implemented with Leap Motion and Neural Network, which tend to be affected by light conditions.

We studied and implemented a wearable drone controller with hand gesture recognition and vibrotactile feedback, and the basic idea of the interface was presented as a preliminary version of this paper [28]. The hand gestures are sensed by the IMU, having robust features for motion acquisition compared to conventional vision placed on the back of the hand. The dominant parameters of the motion are extracted through sensitive analysis, and then machine learning-based classifications are performed. The classified and recognized hand gestures are commanded to the drone, and the distance to obstacles in the heading direction of the drone measured by the ultrasonic sensor is represented by activating the vibration motor attached to the user’s wrist. The IMU-based hand motion capture is relatively free in the distance between the drone and the user, and the drone does not need to make the pose to acquire the hand motion visually. Vibrotactile feedback is an effective way to obtain obstacle information around a drone, especially when the visual information is limited or blocked during operation of the drone.

The remainder of this paper is organized as follows. The system configuration of the wearable drone controller is presented in Section 2, and hand gesture recognition based on machine learning models is analyzed in Section 3. The simulation experiment with subjective evaluation of participants is performed in Section 4, and a real drone experiment for validation is presented in Section 5. Finally, Section 6 presents the conclusion.

2. System Configuration

The controller developed here comprised vibrotactile feedback and hand gesture-based drone control. An ultrasonic sensor was mounted on the drone head to detect obstacles. A vibrator in the controller was stimulated according to the distance between the drone and its obstacles, as shown in Figure 1a. When the operator recognizes the vibration, the operator commands the gesture to avoid its obstacle (Figure 1b). The gestures were sensed by an IMU attached to the back of the hand. The gestures were categorized into two groups to control the drone appropriately. One of the control methods, called the direct mode, was defined to control drones directly for movements such as roll, pitch, and up/down. The cruise velocity of the drone was determined from the inclination of the IMU attached to the hand. Another control method, called the gesture mode, was used to control the drone with hand gestures more easily but may not perform quantitatively, as in the direct mode. The patterns of hand gestures used in the gesture mode were defined by imitating hand motions for helicopter guidance. Gesture pattern analysis and classification were conducted through machine learning to recognize these gestures, which are more sizable than the direct mode.

All sensors were connected to the Raspberry Pi 3b (Raspberry Pi Foundation, Cambridge, UK), as shown in Figure 2. The signal of the IMU was processed at 50 Hz and transmitted to the PC for gesture classification using MATLAB/Simulink. The classified gestures were also delivered to the AR drone control module of LabVIEW to command the drone. The ultrasonic sensor measured the distance, with the signal at 40 kHz. The distance information is transmitted wirelessly to the Raspberry Pi of the controller and converted to vibration intensity. The overall shape of the controller was fabricated using 3D printing.

2.1. Vibrotactile Feedback

In general, manual drone control is based on visual feedback. However, obstacles in the operating environment might interfere with visual feedback, which occasionally causes drone accidents. Hence, other sensory feedback, such as tactile and/or audio, is required to transfer obstacle information effectively.

In this study, vibrotactile feedback was adopted to represent obstacle information. An ultrasonic sensor (HC-SR04, SparkFun Electronics, Colorado, USA) was mounted on the drone head with a Raspberry Pi to operate the sensor so as to detect obstacles in front. The reliable detection region of the sensor was determined to be 0.03 to 2 m within ±15 °. The measured distance is transmitted to the Raspberry Pi of the controller to generate vibrotactile stimulation.

A coin-type vibration motor (MB-1004V, Motorbank Inc., Seoul, Korea) was used as the actuator for tactile stimulation. The motor’s size (diameter × height) was 9.0 × 3.4 mm, and its small size is suitable for wearable devices. The motor is attached to the wrist, which allows free hand motion for the intended gesture without any movement restriction. To effectively deliver vibration to the user, a tactile stimulator was designed as a bracelet, which comprised a spring and hemispherical plastic under the vibration motor, as shown in Figure 3.

Based on previous related studies [29,30,31], an amplitude of 100 µm and a frequency of 96.2 Hz were fixed as the stimulation conditions to satisfy the vibrotactile threshold of detection. The vibration intensity changes according to the distance between the drone and the obstacle. A preliminary experiment was conducted to determine appropriate stimulation conditions for the vibration motor. The vibration intensity was controlled by modulating the pulse width (PWM) applied to the vibration motor. The duty rate was changed by two levels with a step change because a minimum stimulation difference is required for human tactile perception [20]. At first, the duty rate of 50% is given as an initial vibration intensity when the measured distance to an obstacle is within 1.0 m, and then the duty rate is raised to 100% for strong vibration intensity when the measured distance is within 0.5 m and the drone is getting closer to the obstacle.

2.2. Geustre-Based Drone Control

Gestures facilitate the delivery of intentions, emotions, thoughts, and nonverbal communication. In particular, hand gestures have been widely used as effective and easy communication methods [32,33]. Hand gesture-based control provides intuitive and portable drone control. There have been studies on hand gesture recognition using vision-based sensors, such as RGB and depth cameras, and Kinect sensors [34,35]. However, these systems have limitations in terms of light, angle, and position of the environment. To overcome this problem, we propose a wearable sensor-based gesture recognition system. The wearable sensor-based method includes electromyography (EMG) and IMU, etc. [36,37]. We used an IMU (SEN-14001, SparkFun Electronics, Colorado, USA) which measures the 3-dimensional angle, angular velocity, and linear acceleration.

As mentioned above, gestures were categorized into two groups corresponding to control purposes. The direct mode was defined to match drone movements such as roll, pitch, and up/down, as shown in Figure 4. The roll, pitch, and z-axis acceleration of the user’s hand were calculated using an inertial sensor and mapped to the drone’s posture and speed. The available hand motion range and its rate were considered to avoid the recognition of unintended hand gestures. A roll and pitch motion of less than ±30° and linear acceleration of less than ±10 m/s are disregarded for the stable recognition of hand gestures.

The gesture mode shown in Figure 5 is defined by imitating the hand signal used for helicopter guidance from the ground operator [38,39]. The gesture mode has difficulty controlling the drone quantitatively, as in the direct mode, but can be used more easily to control flight direction. These hand gestures are generated naturally with individual deviations; therefore, the patterns should be analyzed and classified for accurate recognition. This study adopted machine learning to learn and classify hand gestures.

3. Hand Gesture Recognition

The signal processing scheme in hand gesture mode is illustrated in Figure 6.

In Figure 6, the signals of the hand movements are obtained from the inertial sensor of the controller. Subsequently, the key factors of gestures were determined using sensitivity analysis to reduce the calculation time of signal processing. Based on the analysis, key signals were segmented using sliding windows and filtered using a low-pass filter to eliminate the gravity component of the accelerometer. The features were extracted from the processed data, such as the mean, RMS, autocorrelation, power spectral density (PSD), and spectral peak. These features were input into the machine learning algorithm. Classification performance was evaluated based on accuracy, precision, recall, and the F-1 score.

3.1. Dataset

Large-scale motion data are required to classify hand gestures using a machine learning algorithm. Hence, the participants (male = 10, female = 2; right-hand dominant = 12; 26.6 ± 6, 24–30 years old) were asked to perform the motions of gesture mode (forward, backward, right, left, stop, up, and down). The signal was recorded at 50 Hz to obtain quaternion, acceleration, and angular velocity along the three axes. To prevent imbalanced data, the number of each gesture data was 3300, and the overall number of data was 23,100.

3.2. Sensitivity Analysis and Preprocess

A sensitivity analysis was conducted with reduced processing time to ensure that key parameters dominantly influence hand gestures. The results of the sensitivity analysis are shown in Figure 7. Based on the results, the acceleration elements of x, y, z, and the angular velocity of y and z are commonly influenced as key parameters of hand gestures.

Raw signals were also processed to remove the gravity component forced on the accelerometer using a low-pass filter. These signals were segmented with fixed sliding windows of 2.56 s and a 50% overlap (128 readings/window) [40,41,42,43].

3.3. Feature Extraction

Feature extraction is generally conducted to improve efficiency of the algorithm computation. The features are determined as time-domain and frequency-domain features [40,41,42,43,44,45]. We computed features such as the mean, RMS, autocorrelation, power spectral density (PSD), and spectral peak. The mean, RMS, and autocorrelation $r_k$ are included in the time-domain features. The characteristics are as follows:

$$mean=\frac{1}{N}{{\displaystyle \sum }}_{i=1}^N{a}_i,$$

(1)

$$RMS=\sqrt{\frac{1}{N}\left(a_1^2+a_2^2+\cdots +a_N^2\right)}$$

(2)

$$r_k=\frac{{{\displaystyle \sum }}_{i=k+1}^N\left(a_i-\overline{a}\right)\left(a_{i-k}-\overline{a}\right)}{{{\displaystyle \sum }}_{i=1}^N{\left(a_i-\overline{a}\right)}^2}$$

(3)

where $a_i$ denotes the components of the acceleration and angular velocity, and N denotes the window length. Equation 3 computes autocorrelation, which is the correlation between $a_i$ and the delayed value $a_{i+k}$, where k = 0, ⋯, N, $\overline{a}$ denotes the mean of $a_i.$ PSD and spectral peaks were included in the frequency-domain features. PSD was calculated as the squared sum of the spectral coefficients normalized by sliding window length. PSD is described as follows [44,45]:

$$PSD=\frac{1}{N}{\displaystyle \sum }_{i=0}^{N-1}{x}_i^2+y_i^2$$

(4)

with $x_i=z_i\cos \left(\frac{2\pi fi}{N}\right)$ and $y_i=z_i\sin \left(\frac{2\pi fi}{N}\right)$, where $z$ denotes the discrete data for the frequency spectrum, and $f$ represents the Fourier coefficient in the frequency domain. The spectral peak was calculated as the height and position of PSD [44,45].

3.4. Classification Model

To employ an appropriate classification algorithm, we compared the classification results of the ensemble, SVM, KNN, naive Bayes, and trees. The classifiers were used in MATLAB, and the training and test data were used in 9 to 1.

The performance of the learned model was evaluated based on the following expressions:

$$Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$$

(5)

$$Precision=\frac{TP}{TP+FP}$$

(6)

$$Recall=\frac{TP}{TP+FN}$$

(7)

$$F1-score=2\ast \frac{recall\ast precision}{recall+precision}$$

(8)

where TP is a true positive, TN is a true negative, FP is a false positive, and FN is a false negative. Figure 8a illustrates the comparison results of performance. Figure 8b shows the detailed results of each classifier’s accuracy. According to

Table 1. Comparison result of classifiers (%).

	Ensemble	SVM	KNN	Naive Bayes	Trees
accuracy	97.9	95.1	92.8	84.0	83.6
precision	97.9	95.1	93.0	84.1	83.8
recall	97.9	95.1	92.9	84.1	83.8
F1-score	97.9	95.1	92.9	84.1	83.8

, the ensemble method exhibited adequate performance in terms of classification accuracy.

4. Simulation Experiment

4.1. Gesture-Based Control

4.1.1. Simulation Setup

To assess the effectiveness of the proposed gesture control, the participants (male = 10, female = 2; right-hand dominant = 12; 26.6 ± 6, 24–30 years old) performed a virtual flight simulation with hand gestures while equipped with a wearable device. The participants did not have experience flying drones or were beginners. A schematic of the simulation mission is shown in Figure 9. The participants executed the appropriate hand gesture to maneuver the drone through the 12 waypoints described by the passing windows. These waypoints were placed according to the yellow guideline to pass, and one waypoint was required to adjust the flight altitude of the drone. The position of the drone was calculated at 50 Hz through quadrotor dynamics from MATLAB/Simulink. The simulation experiment comprised three preliminary sessions and the main experiment with a given mission. All participants were given sufficient time to complete a flight mission with direct and gesture modes.

Average trial duration, gesture accuracy, and number of gesture repetitions were calculated to evaluate the learning effect and adaptability of gesture-based drone control. Average trial duration, meaning the required average time for task completion, was used to evaluate the learning effect in the subjects. The average gesture accuracy was used to evaluate the performance of the classification of hand gestures. The average number of gesture repetitions, counting the frequency of motion changes, was used to evaluate adaptability. Participants were also asked to respond to a questionnaire regarding convenience of the control operation and physical discomfort after the experiment. Questions were answered in the form of a 7-point Likert scale.

4.1.2. Results

The results of the first and last trials were analyzed to investigate the controller’s efficiency and the learning effect. Figure 10 and Figure 11 indicate the drone trajectory and the frequency of motion changes during the first and last attempts using direct mode control. Compared with the first trial, the trajectory stabilized in the last trial. In addition, the frequency of motion changes decreased because of the adaptation of direct mode control. Average accuracy also increased as the experiment progressed, and there was a larger reduction in trial duration and repetition, as shown in Figure 12.

Similarly, Figure 13 and Figure 14 show drone trajectory and frequency of motion changes in the first and last trials with gesture mode control. In the first trial, the drone trajectory was unstable, especially when the user executed the gesture in the left direction. The frequency of motion changes also decreased compared with the first trial. Across all trials, average accuracy tended to increase, as did a larger reduction in trial duration and repetition, as shown in Figure 15. These results indicate that gesture-based control has learning effects and adaptability.

4.2. Vibrotactile Feedback

4.2.1. Simulation Setup

An obstacle avoidance simulation experiment was conducted to demonstrate the performance of vibrotactile feedback. The participants, who were the same as mentioned above, were asked to avoid obstacles distributed in front of the drone using gesture control. All the participants performed the simulation experiments three times with a fixed field of view, as shown in Figure 16a. In the first trial, participants controlled the drone without vibrotactile conditions. In the other trials, the drone was controlled using vibrotactile feedback. To assess the performance of the vibrotactile stimulation, the evaluator measured the collision status through the top view of the flight environment, as shown in Figure 16b.

4.2.2. Result

Without vibrotactile feedback, the participants could not manipulate the drone appropriately because of a lack of visual feedback, as shown in Figure 16a. The drone successfully avoided all obstacles with vibrotactile feedback, as illustrated in Figure 17.

The results of the vibrotactile feedback simulation are presented in

Table 2. Results of obstacle avoidance using vibrotactile feedback. Without vibrotactile feedback, the drone failed to avoid obstacles; meanwhile, collision avoidance was completed successfully with vibrotactile feedback.

Participant No.	First Trial (without Feedback)	Second Trial (with Feedback)	Third Trial (with Feedback)
Participant 1	2/4 *	4/4	4/4
Participant 2	1/4	4/4	4/4
Participant 3	1/4	4/4	4/4
Participant 4	2/4	4/4	4/4
Participant 5	1/4	4/4	4/4
Participant 6	2/4	4/4	4/4
Participant 7	2/4	4/4	4/4
Participant 8	1/4	4/4	4/4
Participant 9	2/4	4/4	4/4
Participant 10	2/4	4/4	4/4
Participant 11	2/4	4/4	4/4
Participant 12	2/4	4/4	4/4

* Successful/avoidance trials.

. All the participants experienced crashes with obstacles at least once. When the vibrotactile stimulator was not used, the participants did not recognize the obstacles well with limited visibility. Therefore, the drone crashed an average of 2.2 times. However, with vibrotactile feedback, each participant detected obstacles via vibrotactile stimulation in the wrist and effectively commanded the drone to avoid obstacles.

4.3. Subjective Evaluation of Participants

Subjective evaluations were conducted to assess the user-friendliness and effectiveness of the controller after the experiment. Participants were asked to respond to the experience of use based on a 7-point Likert scale. The participants gave assessment points for the user-friendliness and effectiveness of the direct and gesture modes. Similarly, participants were asked to respond to the control with and without vibrotactile feedback.

Table 3. Results of subjective evaluation by participants in gesture-based drone control.

Question	Direct Mode	Gesture Mode
1. The proposed gesture was natural for me.	6.4 ± 0.6	6.0 ± 0.7
2. I felt physical discomfort while controlling.	1.6 ± 0.6	2.0 ± 0.9
3. My hand and arm were tired while controlling.	2.0 ± 0.6	2.4 ± 0.8
4. The proposed gesture was user-friendly.	6.5 ± 0.9	6.3 ± 1.4
5. I felt the convenience of controlling a drone with one hand.	6.5 ± 0.6	6.6 ± 0.5
6. It was interesting to fly a drone with a gesture.	6.5 ±1.0	6.9 ± 0.3

shows the subjective evaluation of gesture-based drone control. The participants agreed to the convenience and effectiveness of the controller by more than 6 points and did not agree on discomfort and fatigue by less than 3 points. There are no significant differences between the direct and gesture modes, confirming that the proposed controller is easy to use and effective.

Table 4. Results of subjective evaluation by participants in vibrotactile feedback.

Question	Mean ± SD
1. The vibrotactile feedback was helpful for obstacle avoidance.	6.9 ± 0.3
2. The vibration intensity was appropriate.	6.6 ± 0.6
3. My hand and wrist were tired by vibrotactile feedback.	1.4 ± 0.8
4. The obstacle avoidance was difficult without the vibrotactile feedback condition.	6.5 ± 0.7
5. If I flew a drone in real life, vibrotactile feedback would be helpful.	6.5 ± 0.3

shows the subjective evaluations of the vibrotactile feedback. All the participants responded that there were substantial differences between the control with vibrotactile feedback and the control relying on only limited visual information. They expected that vibrotactile feedback would be helpful for drone control in the field.

5. Experimental Validation

5.1. Gesture-Based Drone Control

5.1.1. Setup

The implemented controller was tested to validate its performance. The flight test was conducted using a quadrotor (AR.Drone 2.0, Parrot, Paris, France). In the mission scenario, the user flew the drone through four gates using gesture control, as shown in Figure 18. The drone cruised at 0.4 m/s and hovered at a height of about 0.8 m. The user was positioned within gray dotted lines (Figure 18a) to ensure visibility for drone control. Three gates were placed on the guideline, and one was installed on the table to control up/down movements. An appropriate gesture was assigned for each trajectory, and the user’s gesture was recognized and compared with the assigned gesture. To evaluate the performance, gesture accuracy was evaluated using drone states, classified gestures, and aerial videos during the experiment.

5.1.2. Result

The entire recognition rate in the direct mode is shown in Figure 19a. The average accuracy of the direct mode was 96.4%, and the mission duration was 103 s. The classification rate in the gesture mode condition is shown in Figure 19b. The average accuracy of the gesture mode was 98.2%, and the mission duration was 119 s.

5.2. Vibrotactile Feedback

5.2.1. Setup

Vibrotactile feedback was tested for efficiency in a real environment, as shown in Figure 20. The user controls the drone using gesture mode through gates to avoid obstacles. To demonstrate the effectiveness of vibrotactile feedback, the user’s visibility was limited by the blinding of the panel, which blocked part of the obstacle and the gate to pass. The first trial was performed without a vibrotactile condition. The second trial was conducted with vibrotactile feedback, representing the distance between the drone and the obstacle.

5.2.2. Result

The drone failed to avoid collisions without vibrotactile feedback, as shown in Figure 21a. The user cannot estimate the distance between the drone and the obstacle owing to limited visual information. However, collision avoidance was achieved successfully with vibrotactile feedback, as shown in Figure 21b. In addition, the distance to the obstacle in front of the drone was inferred through the intensity of the vibrotactile stimulation, and the mission was carried out by maintaining a certain distance from the obstacle.

5.3. Discussions

The presented drone controller exhibited user-friendly and intuitive features, which can be used more easily by an inexperienced user for drone maneuvering. Through simulation experiments on drones, an effective interface for drone control was revealed by estimating control accuracy and mission duration. An experiment on a real drone was carried out to validate the effectiveness of the proposed controller. Indeed, vibrotactile feedback was helpful in detecting obstacles compared to using only visual information. The participants also felt more comfortable with and interested in the proposed controller.

In a further study, the addition of yawing control of the drone was considered to implement a supplementary sensor to measure the hand motion’s orientation accurately. Furthermore, implementing additional ultrasonic sensors on a drone enables the omnidirectional distance measurement of obstacles. The aforementioned approaches are expected to complete a wearable drone control system with a natural user interface.

6. Conclusions

This study proposed a wearable drone controller with hand gesture recognition and vibrotactile feedback. The hand motions for drone control were sensed by an IMU attached to the back of the hand. The measured motions were classified based on machine learning using the ensemble method with a classification accuracy of 97.9%. Furthermore, the distance to obstacles in the heading direction of the drone was fed back to the user stimulating the vibration motor attached to the wrist, known as vibrotactile feedback. In the simulation experiment by the participant group, the hand gesture control showed good performance, and the vibrotactile feedback helped the user be aware of the operation environment of the drone, especially when limited visual information was available. A subjective evaluation of the participants was performed to assess the convenience and effectiveness of the proposed controller. Finally, an experiment with a real drone was conducted, confirming that the proposed controller could be applicable for drone operation as a natural interface.