Trajectory Extrapolation for Manual Robot Remote Welding

Abstract

This article describes an algorithm for the online extrapolation of hand-motion during remote welding. The aim is to overcome the spatial limitations of the human welder’s arms in order to cover a larger workspace with a continuous weld seam and to substantially relieve the welder from strain and fatigue. Depending on the sampled hand-motion data, an extrapolation of the given motion patterns is achieved by decomposing the input signals in a linear direction and a periodic motion component. An approach to efficiently determine the periodicity using a sampled autocorrelation function and the subsequent application of parameter identification using a spline function are presented in this paper. The proposed approach is able to resemble all practically relevant motion patterns and has been validated successfully on a remote welding system with limited input space and audio-visual feedback by an experienced welder.

1. Introduction

To decrease programming effort, improve ergonomics, and reduce hazards to the welder, remote welding applications are pursued [1,2,3,4,5,6,7,8,9]. The typical setup consists of a motion input device and a standard industrial robot with a welding torch attached. The input devices range from complex applications of VR technology [10] to simpler methodologies using infrared optical motion sensors [11,12]. A comprehensive overview of different approaches is given in Section 3 of the review paper [13]. The common problem with those approaches is the limited tracking region of the input device, whereas longer, and even more important, continuous welding seams are to be manufactured [14]. To achieve these long continuous welding seams, solutions have been presented [15,16,17,18] that apply 2D laser-scanning before the welding in order to pre-calculate a trajectory. However, it is still beneficial to directly control the torch’s motion to save time and to respond to unexpected variations in material properties, as they are frequently present in repair applications [19].

Recent work performed by the authors of the paper [20] introduces a promising approach to tackle the problem of the limited workspace of the motion input devices, as illustrated in Figure 1. When the motion of the input device, which is tracked by the robot, reaches the boundary of the input device’s acquisition space, the robot’s motion is continued by extrapolation of the input trajectory for a period of time. In the meantime, the input device can be re-positioned into a convenient position and the control is resumed to the tracking of the input device thereafter. The extrapolation algorithm uses a discrete Fourier transformation (DFT) approach to isolate the dominant spatial frequency and is able to generate the continuous motion pattern for the weaving motion in concave and convex shapes by estimating parameters of a sine template function. As a result, spatial motions must be composable by a single sine function per degree of freedom. Thus, the practically relevant Christmas-tree-shaped motion pattern is extrapolated with limited fidelity only and the algorithms proposed in the paper [20] still have to be validated in practical welding experiments.

In this paper, we introduce an approach based on a sampled autocorrelation function and a spline parameter estimation that is applicable to the practically relevant straight line, weaving, and Christmas-tree-shaped (see the publication [21]) motion patterns. Successful experimental results using real-life input motion data and practically performed remote welding validations are presented and discussed.

2. Problem Description

For the input device, the coordinate system depicted in Figure 2 is applied. A dummy torch tracked by a stereoscopic vision system is used and the main motion is performed in the y-z-plane. Details of the signal processing and the integration into the robot control are given in Section 4.1 of this article.

A Christmas-tree-shaped input trajectory obtained from a practical weld is depicted in Figure 3.

This is the trajectory to be extrapolated in order to achieve a continuous weld and the control by the input device has to be resumed afterward without causing any discontinuities in torch position and velocity. The proposed algorithm has been successfully applied to straight-line and weaving motion patterns, but the discussion in the following section is based on the most complex motion pattern shape, the Christmas tree shape. It can be seen from the data that compared to synthetic signals a lot of hand tremor is present, which is quite typical for manual welding [22,23,24].

The following goals must be met:

• The overall direction of the welding seam must be kept. Therefore, the linear motion components must be isolated from periodic ones.
• The macroscopic shape of the weld must be kept so that the outer geometry of the seam still connects the workpieces properly.
• The motion velocity along the path must be maintained in the particular parts of the shape so that the metallurgic properties of the weld seam are sound. However, it is beneficial to the weld quality to suppress a certain amount of tremor-induced motion [24,25].
• Tremor-induced microscopic motion must not be mistaken for a periodic macroscopic motion pattern.

3. Extrapolation Algorithm

The extrapolated robot trajectory is calculated for the motion in the y-z-plane, whereas the last measured angles $\alpha$, $\beta$, and $\gamma$, as well as the last measured x-coordinate, are kept constant during extrapolation. Equations are given for the y-coordinate of the plane only; the motion in the z-direction is calculated analogously.

The first step of the algorithm determines the overall direction of the weld and distinguishes between linear motion with tremor present and an intended periodic welding motion pattern.

3.1. Motion Linear Direction and Periodic Component Frequency Estimation

The input signal captured by the vision system at the sample times $t_k$ is split in its Cartesian components and modeled according to Equation (1) for each component in order to estimate the linear motion parameters ${\pi }_1$ (axis intercept) and ${\pi }_2$ (slope).

$$\underset{\mathbf{y}}{\underbrace{\left[\begin{array}{c}{y}_{t_0}\\ y_{t_1}\\ y_{t_2}\\ ⋮\end{array}\right]}}=\underset{\mathsf{\Xi }}{\underbrace{\left[\begin{array}{cc}1& t_0\\ 1& t_1\\ 1& t_2\\ ⋮& ⋮\end{array}\right]}}·\underset{\pi }{\underbrace{\left[\begin{array}{c}{\pi }_1\\ {\pi }_2\end{array}\right]}}+\underset{{\mathit{r}}_y}{\underbrace{\left[\begin{array}{c}{r}_{y,t_0}\\ r_{y,t_1}\\ r_{y,t_2}\\ ⋮\end{array}\right]}}$$

(1)

In Equation (1), $\mathit{y}$ is the vector containing the y-component of the input signal over time. ${\mathit{r}}_y$ denotes a vector representing the residuals of the parameter estimation problem, which are unknown as well as the parameters $\pi$. The affine linear parameters $\pi$ are determined by QR-decomposition of the regressor matrix $\mathsf{\Xi }$ using the Armadillo math package [26,27] over 15 s of measurement data and denoted as $\tilde{\pi }$. After this parameter estimation, the residual signal vector ${\mathit{r}}_y$ is calculated by

$${\mathit{r}}_y=\mathit{y}-\mathsf{\Xi }·\tilde{\pi },$$

(2)

which removes the linear components from the signal and the RMS value of the residual signal ${\mathit{r}}_y$ is used to determine if tremor motion only or a significant intended periodic motion are present in the signal. A threshold value is chosen empirically from the measured motion patterns as shown in Figure 3. The periodic welding pattern shows a motion amplitude that is significantly larger than the tremor component, which is typically lower than a threshold of 3 mm. This value is determined by letting welders draw a straight line by hand at a typical welding speed of 10 mm/s and measuring the RMS value of the tremor.

If the threshold is not exceeded, the extrapolation is performed by

$$y_k={\tilde{\pi }}_1+{\tilde{\pi }}_{2}kT$$

(3)

where ${\tilde{\pi }}_1$ and ${\tilde{\pi }}_2$ are the elements of the estimated vector $\tilde{\pi }$, k is the index of the extrapolation signal and T is the sample time period.

If the threshold for the tremor-only motion is exceeded and thus, a significant periodic motion pattern is present in the residual vector ${\mathit{r}}_y$, the period of the most prominent frequency is estimated by calculating the autocorrelation function of the signal according to Equation (4).

$$\Psi \left(j\right)=\sum _m{r}_m{r}_{m-j}$$

(4)

Figure 4 shows the plot of the autocorrelation function (ACF) calculated from the measured welding motion depicted in Figure 3. The highest maximum is always found at zero time delay, resulting from a perfect correlation of the signal with itself. The second-highest maximum represents the most significant periodic motion frequency determined by its time delay $jT$.

To identify the second-highest maximum efficiently, Equation (4) is calculated at a discrete number of equally spaced probe points first and then evaluated for a possible region in which to find it. Candidate regions are determined by all probe points that have two lower neighbor probe points. Then, from these points, the candidate with the highest value is chosen. As a result, the left index l and the right index r of the neighbors are used and the dominant motion frequency $\tilde{\omega }$ is therefore determined by a linear search according to Equation (5).

$$\tilde{\omega }=\frac{1}{\left[\arg \underset{j,l<j<r}{max}\Psi \left(j\right)\right]·T}$$

(5)

This approach reduces the computational effort of the dominant motion frequency identification, since the computationally expensive autocorrelation function has to be calculated for the probe points and the samples contained in the interval $l<j<r$, only.

3.2. Periodic Motion Generation

In the case that the input motion is not determined as a straight line, a periodic motion pattern has to be continued by the extrapolation algorithm as a second step. After estimating the frequency of the periodic motion, the last signal period is used to fit $n-1$ spline segments to n equidistant supporting points in order to resemble the periodic motion signal. The spline function used is given by Equation (6).

$$\begin{array}{cc}\hfill S_{i,k}=& a_i+b_i·(k-{\nu }_i)+c_i·{(k-{\nu }_i)}^2+d_i·{(k-{\nu }_i)}^3,i=0,\cdots ,n-1\hfill \end{array}$$

(6)

In this equation, ${\nu }_i$ is the starting sample index for the respective spline segment $S_i$ and $a_i$, $b_i$, $c_i$ and $d_i$ are the spline parameters to become identified. The spline should match all supporting points $r_{{\nu }_i}$ of the residual signal exactly except the first and the last ones, where the arithmetic mean is matched in order to achieve a smooth periodic signal. This condition yields the first set of coefficients of the spline function:

$$\begin{array}{cc}\hfill a_0& =\frac{r_{{\nu }_0}+r_{{\nu }_n}}{2}\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill a_i& =r_{{\nu }_i},i=1,\cdots ,n-2\hfill \end{array}$$

(8)

$$a_{n-1} = a_0$$

(9)

Introducing h as the number of signal samples between two spline-supporting points, the $c_i$-coefficients are determined by

$$\left(\begin{array}{c}{c}_1\\ c_2\\ ⋮\\ c_{n-2}\\ c_{n-1}\\ c_0\end{array}\right)={\mathit{A}}^{-1}·\frac{1}{h}·\left(\begin{array}{c}3(r_{{\nu }_2}-2r_{{\nu }_1}+a_0)\\ 3(r_{{\nu }_3}-2r_{{\nu }_2}+r_{{\nu }_1})\\ ⋮\\ 3(r_{{\nu }_{n-1}}-2r_{{\nu }_{n-2}}+r_{{\nu }_{n-3}})\\ 3(a_0-2r_{{\nu }_{n-1}}+r_{{\nu }_{n-2}})\\ 3(r_{{\nu }_1}-2a_0+r_{{\nu }_{n-1}})\end{array}\right)$$

(10)

with

$${\mathit{A}}^{-1}={\left(\begin{array}{cccccc}4h& h& 0& \cdots & 0& h\\ h& 4h& h& \ddots & 0& 0\\ 0& h& \ddots & \ddots & \ddots & ⋮\\ ⋮& \ddots & \ddots & 4h& h& 0\\ 0& 0& \ddots & h& 4h& h\\ h& 0& \cdots & 0& h& 4h\end{array}\right)}^{-1},$$

(11)

which is constant because of the equidistant supporting points and thus, can be precalculated. The remaining coefficients of the spline function are determined by

$$\begin{array}{cc}\hfill b_0=& \frac{r_{{\nu }_1}-a_0}{h}-\frac{h}{3}(c_1+2c_0)\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill b_i=& \frac{r_{{\nu }_{i+1}}-r_{{\nu }_i}}{h}-\frac{h}{3}(c_{i+1}+2c_i),i=1,\cdots ,n-2\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill b_{n-1}=& \frac{a_0-r_{{\nu }_i}}{h}-\frac{h}{3}(c_0+2c_{n-1})\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill d_i=& \frac{(c_{i+1}+2c_i)}{3h},i=0,\cdots ,n-2\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill d_{n-1}=& \frac{(c_0+2c_{n-1})}{3h}\hfill \end{array}$$

(16)

To correct for small-amplitude errors in order to maintain the outer geometry of the weld seam, the maxima and the minima of the spline are adjusted by a scaling factor g and an offset o according to the median values for the maxima ${\mu }_{\max }$ and minima ${\mu }_{\min }$ of the last three periods of the input signal:

$$g=\frac{{\mu }_{\max }-{\mu }_{\min }}{\underset{i,k}{max}{S}_{i,k}-\underset{i,k}{min}{S}_{i,k}}$$

(17)

$$o={\mu }_{\max }-\underset{i,k}{max}{S}_{i,k}$$

(18)

The extrapolation signal is then composed by

$$y_{i,k}={\tilde{\pi }}_1+{\tilde{\pi }}_{2}kT+g·(S_{i,k}+{ϵ}_k)+o$$

(19)

introducing the blending term

$${ϵ}_k=(r_{{\nu }_0}-a_0)·e^{-\tau k},$$

(20)

which is required for a smooth transition from the measurement to the extrapolation function, because Equations (7) and (9) are applied to determine the spline coefficients (the last and the first spline segments share a common sample point). Figure 5 shows the algorithm applied to the measured input signal (compare Figure 3) for $n=10$ spline sample points per residual motion period.

It can be seen that the choice of the number of sample points, as well as the amplitude adjustment by the median amplitude of the last 3 periods of the input signal, lead to a suitably fitting shape of the extrapolated signal. With all spline coefficients determined, Equation (19) can be calculated online for an unlimited period of time.

3.3. Resumption of the Control by the Input Device

At the point in time when the operator chooses to take back manual control, the relative translational motion with reference to the starting point of the input trajectory is added to the actual translatory position of the extrapolated signal directly. However, it turns out by practical experience that the rotational motion has to be blended in from the new input signal ${\alpha }_{\mathrm{ref}}$ according to

$${\alpha }_k=(1-\delta )·{\alpha }_{k-1}+\delta ·{\alpha }_{\mathrm{ref}}$$

(21)

(${\beta }_k$ and ${\gamma }_k$, respectively), with $\delta$ used for tuning the blending speed. This allows the operator to adjust for orientation misalignment during the next motion patterns while the algorithm provides a continuous trajectory for the robot. When the orientation error is sufficiently small, i.e.,

$${(\alpha -{\alpha }_{\mathrm{ref}})}^2<e_{\alpha ,\max }$$

(22)

is met for all orientation angles, the output is switched back to the input device again.

4. Experimental Validation

To validate the proposed extrapolation algorithm for its practical feasibility, single V-grooves were welded. The 8 mm metal sheets (EN 10025-2 material: S235JR + AR) were prepared with a 30-degree edge and were pre-positioned and tacked to each other. The seams to be welded were at least 150 mm (in length) and ranged up to 200 mm in length. All seams were welded in a flat position (ISO 6947: PA/ASME, Sec. IX: 1G) using a 1 mm diameter wire (ISO 14341-A: G42 4 M21 3Si1) and argon (ISO 14175: I1) as the shielding gas. The wire feed rate and the used current settings were varied over the tests. A trained and experienced welder operated the system. After welding a straight-line root seam, the top bead was used to activate the extrapolation algorithm and to resume manual control afterward to finish the weld seam. The operator performed the manual welding process of the top-bead for at least 30 s with a Christmas-tree-shaped hand motion before he activated the extrapolation algorithm. The extrapolation algorithm was applied for at least 30 s before resuming the manual control. The parameters of the equations used in the experiments are given in

Table 1. Values used in experimental validation.

Variable	Symbol	Value
Sample time	T	$0.004$ s
Spline supporting points	n	10
ACF horizon	m	2000 Samples
Orientation error threshold	$e_{\alpha ,\max }$	$0.2$${}^{\circ }$
Decay (translational blending)	$\tau$	$0.02$
Blending coefficient (rotational)	$\delta$	$6.623·{10}^{-3}$

. Focus is put on the quality impact of the hand tremor suppression and if visible differences in the weld seam’s surface pattern occur, especially at the beginning of the extrapolation.

4.1. Input Device and Robot Working Cell Setup

The experiments were carried out on a research setup for remote robot welding. To capture the hand-motion of the operator, a stereoscopic sensor system was used that consists of two DMK33GP2000e cameras [28], as can be seen in Figure 6a. During the welding process, visual information was provided to the operator from underneath a transparent ground plate by a computer display. The workspace provided a $527\times 296\times 300$ mm cube to operate in. The robot used is a KR6 Agilus R900, which was operated in the same room because the welding robot was mounted in an individual welding cell with an exhaust system and a height-adjustable protection glass. The workspace of the robot had a defined working plane of $1500\times 570$ mm. To use different tools during system operation, a fully automated pneumatic tool-change process was integrated.

A modified welding-torch-gun was used as an input device, which was tracked optically utilizing an attached optical marker. The relative pose between the tool center point and the marker was referenced by a dedicated holder. Additionally, the orientation of the working plane of the robot was aligned to the input device’s coordinate system with a 2D laser scanner.

The interface to the robot for direct motion control, the KuKa RSI [29] module, demands a deterministic response of the trajectory generation module every 4 ms. Due to the non-deterministic image-data processing time and due to the Windows 10 operating system, which is needed to take the full advantage of the used cameras, hard real-time cannot be achieved. Furthermore, the cameras of the stereoscopic vision system operate with a framerate of $f_{\mathrm{fps}}=50$ Hz. Therefore, the quantized data from the stereoscopic system must be either filtered by a second-order linear filter or shaped into a two-times continuously differentiable trajectory by other means. In the paper [30] the authors propose and validate an interpolation algorithm consisting of a linear Kalman filter and a PT-2 error observer, which generates an appropriate trajectory with low latency. This algorithm is calculated on a dedicated real-time PC running Linux with the PREEMPT_RT [31] patch and the resulting trajectory was used as the input signal to the extrapolation algorithm covered in this paper.

A footswitch was used to activate the extrapolation algorithm and to resume the manual control of the robot again afterward. Figure 7 gives an overview of the complete system.

4.2. Results

Figure 8 shows a selected representative result from the tests. Section 1 marks the area in which the operator started the seam. He ignited the arc and created the required fused material. Furthermore, the Christmas tree motion pattern was initiated visibly by the structure of the bead. The variations in the seam’s outer geometry and in the motion pattern show the adjustment process of the operator in order to achieve a proper and sound result. After a short period of time, the welder mastered the required motion and traveling speed and entered Section 2, where he continued the seam in steady-state periodic motion. Overall, the process (Section 1 and Section 2) took 44 s with a length of 60 mm.

At the end of Section 2, the extrapolation algorithm was activated and was deactivated again at the end of Section 3. Section 1 had a total length of 60 mm and the algorithm presented in this article was activated for 41.2 s to complete it. During this period, the operator could relax, oversee the welding by the robot and reposition his dummy welding torch in the working area. Deviations in the outer geometric dimensions of the seam are not recognizable without closer examination. The fused mass in combination with the extrapolation algorithm created a homogeneous weld seam. At the beginning of Section 4, the control was resumed by the operator at the holding position of the Christmas-tree-shaped motion pattern. Section 4 was 28 mm long and small differences in the seam’s outer geometric dimensions can be seen.

4.3. Discussion

The results show the performance of the presented algorithm. Without close visual inspection, a changeover from the manually performed weld to the extrapolation algorithm is not detectable. The extrapolated section of the weld is only distinguishable by its precise periodic pattern. Process parameters, such as the feed-wire rate and the flow rate of the protection gas, can be kept constant over the whole process.

The spline resembles the geometric motion pattern as well as the motion speed adequately because the visual properties of the weld do not change. The choice that the seam width is determined by the median of the extreme values of the last three periods of the input signal leads to the best outer geometry fidelity. When the number of motion periods considered is decreased, the results become too prone to geometric mistakes by the operator, which cause falsely estimated dimensions for the algorithm. Inclusion of all motion periods captured in the data set can also lead to inferior results because on some occasions the operator has to cover up small welding errors. That would mean that the operator would have to weld for 30 s close to perfection before the extrapolation algorithm could be activated. With the described solution, the operator can judge the quality of the last three periods and activate the extrapolation accordingly. When averaging instead of the median filter is used, the outer geometry fidelity decreases, because the influence of the hand-tremor is too strong.

The choice of 10 spline nodes is a good trade-off between geometric fidelity and tremor suppression. However, the optical impression of a sound weld seam is prone to subjective judgment. As a result, this parameter was tuned with the help of an experienced welder, who gave feedback on what geometric shape he intended to achieve. Fewer nodes decrease the alignment of the spline to the original signal. Simulation yields that 5 nodes are the minimum required for the reconstruction of the original signal. The variance of the error decreases strongly with an increased number of nodes until reaching 10 nodes. More nodes will further decrease the variance in small amounts but the acquisition of the tremor increases, which results in undesired spiky spline segments. The optical impression of a sound weld seam is prone to subjective judgment. As a result, we tuned this parameter with the help of an experienced welder.

The motion speed was kept adequately as well because no heightening or thinning of the seam can be observed. However, at the beginning of Section 4 a heightening frequently occurs since the operator has to resume the motion pattern and speed without aid. The filtered resumption of the orientation according to Equation (21) allows the operator to concentrate on the translatory motion first. The experiments show that this needs to be trained by the welder and leads to improved results over time. The result depicted in Figure 8 was obtained after the 5th try by an experienced welder and was the first he was satisfied with. Further aiding by visual guides on the GUI can be provided but is regarded as distracting by that particular welder. It is not possible to judge from the motion signal alone if a weld is successful or not.

To evaluate, if the signal contains periodic characteristics, the recorded manual input signal is freed from affine linear components by Equation (2) and examined by its standard deviation. Periodic signals differ strongly from tremor-loaded lines. The welder who performed the experiments showed a typical hand tremor of 2.5 mm. With an appropriate margin, the threshold was set to 3.0 mm. It is easily adjustable to any operator but proved to be an appropriate value for untrained users as well.

The pose of the welding torch (orientation and distance to the y-z-plane) in Section 2 of the discussed result was close to constant due to the trained operator’s skill. To freeze these values during the active extrapolation algorithm is important to create sufficient results. However, a referencing of this plane by the 2D laser scanner is required. If structures such as corrugated sheet metal or spherical objects such as pipes require the inclusion of the x-axis, the algorithm can easily be extended, since the motion axes are analyzed individually and decoupled from each other. However, for spherical objects, a circular component has to be added to Equation (19).

5. Conclusions and Future Work

A computationally efficient extrapolation algorithm for manual remote welding has been presented. It was experimentally validated as applicable for straight-line, weaving, and Christmas-tree-shaped motion patterns. Because of the spline approach, it is able to provide motion patterns that better resemble the geometry of a Christmas-tree-shaped weld than those based on the composition of sine-templates as proposed by the authors of the paper [20]. The base frequency estimation by using a sampled autocorrelation function proved to be suitable as well.

The experimental proof-of-concept for extrapolation, re-positioning of the input device, and subsequent resumption of control to overcome the spatial limitations of a geometrical input device was given. The experiments revealed that even long distances can be welded by the presented algorithm as long as the overall linear direction remains constant. Moreover, there is no theoretical limitation to the extrapolation length, as long as the material properties do not change. The operator is allowed to relax, supervise the process and occasionally resume manual control to adjust the overall direction or the motion pattern and then hand the control over to the algorithm again. This leads to substantial relief of strain from the operator.

A trained and experienced operator has assessed the weld quality of the samples as promising. As a result, in the near future, a metallography analysis of the weld seam will be performed to ensure that the extrapolation of the inner geometry leads to flawless welds. Afterward, a certification for the system for professional welding operations will be pursued. An improved input device and a dedicated robot tool are currently in fabrication. An extension of the algorithm incorporating distance control and an online adaptation of the overall direction during extrapolation are being considered for future work.