Optimization of Link Length Fitting between an Operator and a Robot with Digital Annealer for a Leader-Follower Operation

Abstract

In recent years, the teleoperation of robots has become widespread in practical use. However, in some current modes of robot operation, such as leader-follower control, the operator must use visual information to recognize the physical deviation between him/herself and the robot, and correct the operation instructions sequentially, which limits movement speed and places a heavy burden on the operator. In this study, we propose a leader-follower control parameter optimization method for the feedforward correction necessitated by deviations in the link length between the robot and the operator. To optimize the parameters, we used the Digital Annealer developed by Fujitsu Ltd., which can solve the combinatorial optimization problem at high speed. The main objective was to minimize the difference between the hand coordinates target and the actual hand position of the robot. In simulations, the proposed method decreased the difference between the hand position of the robot and the target. Moreover, this method enables optimum operation, in part by eliminating the need for the operator to maintain an unreasonable posture, as in some robots the operator’s hand position is unsuitable for achieving the objective.

1. Introduction

During the last few decades, the demand for robot teleoperation in various workspaces has been increasing across a number of fields: from industry to service robotics [1,2]. Teleoperation is useful in complicated environments, including surgical procedures in humans, because it is still difficult for a robot to recognize the environment, avoid collisions with some furniture and humans, and perform tasks automatically.

Several studies have been conducted to improve the teleoperation quality of robots. Some researchers have used controllers to measure the target values of the robot. Sian et al. proposed a whole-body motion generator with joysticks for commanding walking [3]. Shi et al. proposed inverse kinematics-based trajectory planning of a serial robotic arm with a simplified leader robot for measuring surgical operation [4]. Gonzalez et al. proposed a teleoperation method with augmented reality and a haptic feedback device for a car body surface treatment [5]. More precise and operable teleoperation requires more inputs from an operator. Therefore, a leader-follower controller (previously known as a master-slave controller) is used in various teleoperation. With a leader-follower controller, link positions, orientation or joint angles of the operator were measured and used as commands for the robot. However, with the conventional leader-follower controller, an operator must pay careful attention to the movement of the robot and must sequentially visually recognize and modify the robot’s movement. Wang et al. investigated the reaction times between joint space and task space inputs with an exoskeletal motion-capture device, and confirmed that most subjects adapted to task space mapping faster than joint space mapping after similar amounts of practice [6]. Without practice it was impossible to increase the movement speed above a certain level due to delays both in the operation of the robot and in updating the camera image to allow the operator to recognize the movement of the robot. A leader-follower controller especially requires visual-feedback if there is a difference between the robot and the operator in the number of joints, size of links and/or movable speed. Humanoid robots with the same number of joints and size as humans are easy for the operator to operate due to feeling similar to his/her own body, but these robots are very complicated and expensive, and are thus difficult to implement in the real world. Therefore, a technique for correcting the structural difference between the operator and the robot is required. One method that does not consider structural differences calculates the robot joint angle by performing inverse kinematics calculations based on the position and posture of the controller as the operator’s hand sends command values to the robot [7]. This focuses on the operability of the hand position, which is important during work and is easy to implement. However, since only the hand position and posture can be manipulated, moving only the elbow position to avoid obstacles in front of the work target, for example, is not possible. In order to improve operability, it is necessary to consider the movements of some of the operator’s links and joints. The use of optical motion capture technology to measure the position of each part of the operator is a general method for acquiring more physical information [8]; infrared reflection markers are attached to each part of the operator’s body and reflect infrared rays provided in the surrounding environment; detection of the reflected rays allows the determination of the position of each marker. Recent developments also include attaching an inertial measurement unit to each part of the body in order to measure posture and estimate a human pose by analyzing an image taken from the outside [9,10]. Digo et al. used optical motion capture and inertia measurement sensors to evaluate the motion of upper limbs during typical industrial gestures of pick and place [11]. In either method, it is important to associate a part/joint of the operator with the part/joint of the robot whose structure is different from that of humans. Regarding the correction of the difference in the number of joints between the operator and the robot, there is a matching technique that reflects only the movements of the dominant joints of a human to the movements of the robot joints [12]. This has also been studied in the operation of virtual-reality characters [13].

Furthermore, if the link size of the operator and the robot is different, a movement such as extending the arm to a certain target also causes the operator and the robot to have different hand positions, which reduces operability. Nakamura et al. proposed a minimal-delay, self-body image update method for augmented reality, and investigated how humans learned the body link length [14]. There is a method for measuring and accounting for size differences between the operator and the robot in advance and accounting for them at the time of operation, but it takes time and effort to implement [15]. Optimization techniques have been used in teleoperation for high operability, however, most of them focused on trajectory optimization rather than joint operation. Gomes et al. proposed a humanoid whole-body movement optimization method with motion retargeting from the human to the robot according to the link-length ratio between them [16].

In this study, we propose a parameter optimization method for feedforward correction of the deviation of the link length between the robot and the operator in the leader-follower controller; our method synchronizes the robot with the movement of the operator without using visual feedback and/or practice. Specifically, the correction parameters are obtained by having the operator perform several types of target movements prepared in advance, and then using these data to solve the optimization problem in order to minimize the deviation from target hand coordinates. By applying the optimized correction parameter set during real-time teleoperation, it is possible for the operator to perform the expected operation by natural movement and without requiring visual feedback, and it will decrease the mental burden and/or proficiency required for operation.

This study has two main components: The first is the proposal of an operating method that compensates for the difference in link length between the operator and the robot, and accomplishes the intended operation by measuring only the joint angle of the operator. This method optimizes the pose that the operator wants to take when trying to perform a certain movement. Therefore, the operator does not have to be in the same pose as the robot, and operation can be made easy by considering the operator’s own physical movement. Second, regarding robot teleoperating technology, this study is the first, as far as we know, to use quantum annealing technology. Although it is rarely used in the robotics field, it is a technology that should be promoted in the future because it can perform large-scale optimization at high speed. This paper aims to set the precedent.

The remainder of this paper is organized as follows: in Section 2, we describe a parameter optimization method for feedforward correction of the deviation in link length using quantum annealing; in Section 3, we present the simulation results; Section 4 is the discussion; finally, in Section 5, we present our conclusions.

2. Materials and Methods

2.1. Approach

In a general leader-follower controller, the robot controls the joint angle using the joint angle of the operator as the target value; in this research, the compensation parameter to be multiplied by each of the operator’s joint angles is used as a framework during operation. In order to determine this compensation parameter prior to operation, the operator performs a sample motion to create a dataset of the operator joint angle with respect to the target hand position coordinates. Parameters are identified by minimizing the difference between the target and the robot hand position coordinates. As a result, the robot can be operated by measuring only the joint angle of the operator. As a simplified robot model, consider a two-link robot arm that operates on a horizontal plane. Using general forward kinematics calculations, the hand position horizontal plane coordinates (x, y) of the 2-link robot arm are expressed by Equations (1) and (2) (Figure 1):

$$x=L_{upperarm}\cos \left({\theta }_{shoulder}\right)+L_{forearm}\cos \left({\theta }_{shoulder}+{\theta }_{elbow}\right)$$

(1)

$$y=L_{upperarm}\sin \left({\theta }_{shoulder}\right)+L_{forearm}\sin \left({\theta }_{shoulder}+{\theta }_{elbow}\right)$$

(2)

where $L_{upperarm}$ is the length of the upper arm, $L_{forearm}$ is the length of the forearm, ${\theta }_{shoulder}$ is the joint angle of the shoulder in the horizontal plane and ${\theta }_{elbow}$ is the joint angle of the elbow in the horizontal plane.

By modifying Equations (1) and (2) with compensation parameters, the horizontal plane coordinates ($x_{comp}$, $y_{comp}$) can be expressed by Equations (3) and (4) (Figure 2):

$$x_{comp}=L_{upperarm}\cos \left(K_{shoulder}{\theta }_{shoulder}\right)+L_{forearm}\cos \left(K_{shoulder}{\theta }_{shoulder}+K_{elbow}{\theta }_{elbow}\right)$$

(3)

$$y_{comp}=L_{upperarm}\sin \left(K_{shoulder}{\theta }_{shoulder}\right)+L_{forearm}\sin \left(K_{shoulder}{\theta }_{shoulder}+K_{elbow}{\theta }_{elbow}\right)$$

(4)

where $K_{shoulder}$ is the compensation parameter of the shoulder joint and $K_{elbow}$ is the compensation parameter of the elbow joint.

2.2. Implementation of Digital Annealer

There are various optimization methods, but in this research, we considered optimization of the combinatorial problem of the joint angle compensation parameters of some joints and used a Digital Annealer [17] to solve the combinatorial optimization problem at high speed. The Digital Annealer hardware was developed by Fujitsu Ltd., and was inspired by quantum annealing, specifically for solving evaluation function quadratic unconstrained binary optimization (QUBO). It enables faster computation of QUBO than conventional methods, such as simulated annealing [18]. Digital Annealer has been used to solve large-scale combinatorial optimization problems. Maruo et al. used Digital Annealer to optimize a 2-D magnetic array to maximize induced voltage in coils placed above the array [19]. Rahman et al. proposed an Ising model formulation in Wi-Fi-based positioning [20]. Digital Annealer has not yet been used in the robotics field.

The QUBO model for describing the combinatorial optimization problem is expressed in Equation (5):

$$E_x={\displaystyle \sum }_{i=1}^n{\displaystyle \sum }_{j=1}^i{c}_{ij}{x}_i{x}_j$$

(5)

where $c_{ij}$ is the coefficient and x is the binary. In this study, since the compensation parameter for each joint of the robot should be set within a finite range, the error between the target value of the hand position and the robot’s current value should be minimized. To achieve this, the optimized compensation parameters from the range were selected. As a simplified model to calculate the compensation parameters of the joint angles, the QUBO model was created based on an objective function and constraints. The objective function $H_{objective}$ is formulated as the expression in Equation (6):

$$H_{objective}={\left(q_{target}-q_{comp}\right)}^2$$

(6)

where $q_{target}$ are the targeted hand position coordinates and $q_{comp}$ are the hand position coordinates of the robot with the compensation parameters. Based on Equations (3) and (4), $q_{comp}\left(x_{comp}, y_{comp}\right)$ is formulated with the binary ${\alpha }_{jk}$ in Equations (7) and (8):

$$x_{comp}={\displaystyle \sum }_{i=1}^{num\_link}{l}_i{\displaystyle \sum }_{j=1}^{num\_joint}{\displaystyle \sum }_{k=0}^{range}{\alpha }_{jk}\cos (rk{\theta }_j)$$

(7)

$$y_{comp}={\displaystyle \sum }_{i=1}^{num\_link}{l}_i{\displaystyle \sum }_{j=1}^{num\_joint}{\displaystyle \sum }_{k=0}^{range}{\alpha }_{jk}\sin (rk{\theta }_j)$$

(8)

where $l_i$ is the link length of i-link, ${\theta }_j$ is the joint angle of j-joint and $r$ is the resolution of the compensation parameters in the range. For example, when the resolution of the compensation parameters is 0.1 and the order of binary is 5, the compensated joint angle is half of the operator’s joint angle. In addition, it should include a constraint function $H_{constraint}$ as in Equation (9):

$$H_{constraint}={\displaystyle \sum }_j^{joint\_num}{\left(1-{\displaystyle \sum }_k^{range}{\alpha }_{jk}\right)}^2$$

(9)

The constraint function means that only one compensation parameter should exist for each joint. The energy function $H$, for Hamiltonian, should be prepared from the objective function and the constraint function as in Equation (10):

$$H=H_{objective}+w{H}_{constraint}$$

(10)

where $w$ is a weighting factor between the objective function and the constraint function. Translation from the optimization problem to the QUBO model consists of several steps: (1) Formulate the energy function; (2) Encode the integer variables with binary variables; (3) Obtain the QUBO matrix from the coefficients. Programming this translation may be challenging when a complicated energy function, Equation (10), must be translated. We used the Python library PyQUBO to easily move from Equation (10) to Equation (5) to enable the building of the QUBO model [21]. The library completes steps (2) and (3) above; therefore, users only need to prepare the objective function. After that, Digital Annealer repeats for a number of iterations to minimize the energy of the model, thus finding the compensation parameters as corresponding binaries.

2.3. Experimental Setup

We verified whether operating a fitting that does not require prior measurement of the operator’s link length was possible in simulation. The average human upper- and forearm lengths are 300 mm and 240 mm [22], respectively, with the forearm length about 80% of that of the upper arm. If the robot’s upper arm to forearm length ratio is the same as that of the operator, there is no operation error due to the difference in link size; based on this, a 2-link horizontal arm robot model with different parameters was prepared as the operation target. The link length of the robot’s upper arm was twice as long as that of its forearm. Arm movement objectives Goals A and B were set (

Table 1. Hand position Goals A and B.

Goal of Hand Position		X	Y
Goal A	Ratio compared to whole arm length %	30	50
	Human mm	162	270
	Robot mm	135	225
Goal B	Ratio compared to whole arm length %	−30	50
	Human mm	−162	270
	Robot mm	−135	225

).

As a simulated operator model, the inverse kinematics calculation was performed assuming that the operator reached for the goal of a hand position, and the calculated angles of the shoulder and elbow joints were the command values. Using the joint angle command value for each target hand position as a data set, we searched for joint angle correction parameters using Digital Annealer (the parameters for using Digital Annealer were summarized in

Table 2. Parameters for Digital Annealer in the simulation.

Parameters	Values
Robot upper arm length mm	300
Robot forearm length mm	150
Resolution of compensation parameters r	0.03
Max. number of steps	100
Weighting factor w	10,000
Iteration number	1,000,000

). The number of iterations affects the time for solving and finding the global optimal solution without falling into a local optimal solution. In this simulation, the number of iterations was set as 1,000,000 for finding the global optimal solution. After that, the obtained correction parameters were given to the robot model to correct the joint angle command value, and the change in the hand coordinates was verified.

3. Results

The following results are shown in

Table 3. Results of compensation parameters and joint angles.

		Shoulder	Elbow
Goal A	Compensation parameters	2.40	1.20
	Joint angles of an operator rad	0.23	1.92
	Compensated joint angles rad	0.55	2.30
Goal B	Compensation parameters	1.14	0.96
	Joint angles of an operator rad	1.31	1.92
	Compensated joint angles rad	1.49	1.84

and

Table 4. Results of simulations with and without compensation.

		X mm	Y mm
Goal A	Target hand position of a robot	135	225
	Robot hand position without compensation	211	194
	Robot hand position with compensation	112	201
Goal B	Target hand position of a robot	−135	225
	Robot hand position without compensation	−72	277
	Robot hand position with compensation	−124	271

: parameters obtained, compensated joint angles, target hand coordinates, and uncompensated and compensated robot hand coordinates. Figure 3 and Figure 4 show the simulation result for Goal A and B.

From the results of the target hand position experiment for the total length of the robot arm under each condition, it was confirmed that the parameters obtained in advance could be used to approach the target hand coordinates simply by giving the joint angle command to the robot.

4. Discussion

It is possible to control the robot’s hand coordinates with high accuracy by only measuring the angle of each joint of an operator, and without measuring the link length or the operator’s hand position. In this study, the operator and the robot were fitted, but the arm movement was limited to the horizontal plane. Since most actual robot operations are performed in a three-dimensional space, it is necessary to extend the proposed method to three dimensions—this can change the problem depending on how the objective function is determined. In order to extend to three-dimensional space, instead of calculating the hand coordinates using geometric kinetics, a simultaneous transformation matrix can be used to make a robot model with more degrees of freedom. When operating the arm for several tasks, it is thought that more accurate and easier operation can be achieved by changing the compensation parameters according to the target hand position. When applying this method to actual operation, it is necessary to deal with the fact that human movements are generally not as reproducible as robots. The proposed method would be made more effective by measuring movements several times and using the average value instead of only measuring the sample pose once.

Also noteworthy, we asked the operators to move naturally and easily when performing a target pose in the simulation experiment. Therefore, the operator’s hand position coordinates did not exactly match the target hand position. In the proposed method, the operator being in a geometrically correct pose to achieve the target hand position does not affect the hand position coordinates of the robot. Since compensation with the proposed method is performed using a data set consisting of a certain target hand position coordinate and the joint angle of the operator, the operator can assume the operating posture that he/she finds most comfortable. This makes it easier to feedforward the operation so that the operator can achieve the goal without using visual feedback.

In this paper, the optimization of link length fitting with Digital Annealer was proposed, and it was confirmed that the global optimal solution could be found. The proposed fitting method performs a parameter search before teleoperation. The current compensation method takes too long to make it practical for real-world implementation; however, optimizing for minimal or real-time parameter search would increase usability. As mentioned above, the number of iterations affects the calculation time and helps avoid falling into a local optimal solution. Moreover, the Digital Annealer can be used to increase the probability of finding the optimum solution. If no bit-flip candidate is found, the escape from a local minimum state is facilitated by adding the positive offset to the energy [17]. For decreasing the calculation time, we will improve the number of iterations and implementation of the Digital Annealer and compare the performance with other methods.

5. Conclusions

A parameter optimization method for feedforward correcting the deviation of the link length between the robot and the operator in the leader-follower controller method, without the requirement of visual feedback and/or practice was proposed. This method can compensate not only for the differences in link length between an operator and a robot, but also the differences between a target hand position and the operator’s pose.

In the future, we will compare the calculation time and accuracy to other methods, expand to robot operation in three-dimensional space, proceed with verification of operability, and attempt to compensate for operator’s own physical operation habits to determine operator burden with the proposed method.