Automatically Tightening Tiny Screw Using Two Images and Positioning Control

Abstract

This paper describes how to tighten M1.4 screws by controlling a manipulator. The whole process is based on a human–machine interface designed using Visual Studio C++ to run image processing algorithms and control the position of a manipulator. Two charge-coupled device cameras are used. One is fixed on the stationary frame above screw holes and used to take pictures of the holes. The positions of the holes are determined using image processing algorithms and then transformed into the coordinate system of the manipulator by using coordinate transformation. The other camera, installed on the end effect of the manipulator, photographs the screw hole to fine-tune the position of the manipulator, improving positioning control. The image processing methods including grayscale, Gaussian filter, bilateral filter, binarization, edge detection, center of gravity, and minimum circumcircle are used to find the center coordinates of the target holes. Experimental study shows that M1.4 screws can be tightened into the target holes with the manipulator.

1. Introduction

With the development of automation in the manufacturing and production industries, increasing types of automation equipment have been designed to replace human labor and improve performance [1]. Automation equipment can be used for many purposes such as spray-painting [2], laser-cutting [3], welding [4], polishing [5], and the handling of objects [6].

Pitipong et al. [7] used two cameras to investigate screw tightening. The two cameras were placed equal distances from the screw and photographed from different perspectives. Image processing was then used to measure the angle of the screw and detect whether the screw was perpendicular to the screw hole. However, the accuracy of the coordinate position of the screw hole was not considered. The screw used by Pitipong et al. [7] was much larger than the one used in our research. In our research, two cameras are placed at different distances from the screw hole; one camera photographs the screw hole from afar, while the other does so from a close distance. The screw used is an M1.4 screw, which is much smaller than that used in [7]. Image processing is used in our research to detect the accurate position of the screw hole.

This study designed a human–machine interface (HMI) for controlling the manipulator that tightens an M1.4 screw. An electric screwdriver is modified and installed on the end of a manipulator. One charge-coupled device (CCD) camera is fixed on the stationary frame above the screw holes to detect their positions. Another CCD camera is installed on the end effect of the manipulator. Once the screw has been positioned above the chosen screw hole, the second camera photographs the screw hole to fine-tune the position of the manipulator.

The visual system [8,9] and HMI were designed using Microsoft Visual Studio C++ (Microsoft Co., Redmond, WA, USA). By clicking the buttons on the interface, different commands can be performed such as image processing, selecting the screw hole, starting the manipulator, and starting the electric screwdriver.

The center coordinates of a screw hole, calculated using image processing, are transformed into the objective coordinates of the manipulator through coordinate transformation. Once the screw has arrived at a fixed height above the intended screw hole, the camera installed on the end effect of the manipulator photographs the screw hole. The system then processes the image, detects the position of the screw hole, and transforms it into the coordinate system of the manipulator to fine-tune the manipulator, increasing the positioning control.

Finally, when the screw is in the appropriate position, touching the screw hole, the user clicks the button to start the motor in the screwdriver, finishing the process of screw tightening.

2. Image Processing

After the camera photographs screw holes on the workpiece, the picture must be processed to detect the center coordinates of the screw holes. Images are processed using OpenCV, which involves the following processing methods: minimum circumcircle, binarization [10], edge detection [11], grayscale [12], center of gravity [13], bilateral filter [14], and Gaussian filter [15].

3. Coordinate Transformation

The coordinates of the screw hole, determined using image processing, must be transformed into the coordinate system of the manipulator through coordinate transformation.

Consider a point P and two coordinate systems {A} and {B}, as shown in Figure 1. Accordingly, ^AP represents the point P described in coordinate system {A}, ^BP represents the point P described in coordinate system {B}, and ${\widehat{X}}_A$ represents the unit vector of the X-direction of {A}.

[$\begin{array}{ccc}{}^A\widehat{X}_B& {}^A\widehat{Y}_B& {}^A\widehat{Z}_B\end{array}$] denotes the three unit vectors of {B} described in the coordinate system {A}. The rotation matrix between {A} and {B} is composed of an inner product of these unit vectors:

$$\begin{array}{ll}{}_B{}^{A}R=\left[{}^A\widehat{X}_B{}^A\widehat{Y}_B{}^A\widehat{Z}_B\right]& =\left[\begin{array}{ccc}{\widehat{X}}_B·{\widehat{X}}_A& {\widehat{Y}}_B·{\widehat{X}}_A& {\widehat{Z}}_B·{\widehat{X}}_A\\ {\widehat{X}}_B·{\widehat{Y}}_A& {\widehat{Y}}_B·{\widehat{Y}}_A& {\widehat{Z}}_B·{\widehat{Y}}_A\\ {\widehat{X}}_B·{\widehat{Z}}_A& {\widehat{Y}}_B·{\widehat{Z}}_A& {\widehat{Z}}_B·{\widehat{Z}}_A\end{array}\right]=\left[\begin{array}{ccc}{r}_{11}& r_{12}& r_{13}\\ r_{21}& r_{22}& r_{23}\\ r_{31}& r_{32}& r_{33}\end{array}\right]=\\ & =\left[\begin{array}{ccc}{\widehat{X}}_B& {\widehat{Y}}_B& {\widehat{Z}}_B\end{array}\right]·\left[\begin{array}{c}{\widehat{X}}_A\\ {\widehat{Y}}_A\\ {\widehat{Z}}_A\end{array}\right]=\left(\left[\begin{array}{ccc}{\widehat{X}}_A& {\widehat{Y}}_A& {\widehat{Z}}_A\end{array}\right]·\left[\begin{array}{c}{\widehat{X}}_B\\ {\widehat{Y}}_B\\ {\widehat{Z}}_B\end{array}\right]\right)={}_A{}^{B}R^T\end{array}$$

(1)

The prime of the matrix denotes the transpose matrix. The equation representing the transformation between ^AP and ^BP is as follows,

$${}^{A}P={}_B{}^{A}R{}^{B}P$$

(2)

The displacement between the origins of {A} and {B} is denoted by ^AP_BORG, as shown in Figure 2. The equation representing the transformation between ^AP and ^BP is then as follows,

$${}^{A}P={}_B{}^{A}R{}^{B}P+{}^{A}P_{\mathit{BORG}}$$

(3)

$$\left[\begin{array}{c}{}^{A}P\\ 1\end{array}\right]=\left[\begin{array}{cccc}& {}_B{}^{A}R& & {}^{A}P_{\mathit{BORG}}\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{c}{}^{B}P\\ 1\end{array}\right]$$

(4)

where ^AP_CORG denotes the translation vector from the origin of the Camera 1 coordinate system to the origin of the manipulator coordinate system, i.e., ($X_0,Y_0,Z_0$) and ${}_C{}^{A}R$ denotes the rotation matrix between the two coordinate systems, consisting of r₁₁~r₃₃.

A 4 × 4 transformation matrix can be used to describe the transformation of P,

$${}^{A}P={}_B{}^{A}T{}^{B}P$$

(5)

4. Experiment Platform and Equipment

As shown in Figure 3, a manipulator with six degrees of freedom and Camera 1 are fixed on the stationary frame of the platform; thus, the position of the manipulator relative to that of Camera 1 does not change. Camera 1 is fixed above the screw holes of the workpiece. Camera 2 is installed on the end effect of the manipulator and photographs the screw hole. The resolution of Camera 1 and 2 is 1280 × 960 and 1024 × 768, respectively. The focal length and aperture of the lenses are fixed during the experiment. An Arduino UNO microcontroller board is used to control a dc motor in the electric screwdriver located on the manipulator and an L298N motor controller is used to drive the motor to tighten the screw. The actual diameter of the M1.4 screw hole is 1.1 mm. Thus, the system has to identify a 1.1 mm hole using image processing. Some fixing mechanisms were designed to fix the cameras.

Table 1. Equipment specifications.

Equipment	Specification/Description
Manipulator	MITSUBISHI RV-2SD
CCD Camera 1	SONY DFW-SX910
CCD Camera 2	SONY XCD-X710
Camera lenses	6~60 mm, F1.4
Image capture card	NI PCI-8254R
Microchip	Arduino UNO
Drive circuit	L298N
Electric screwdriver	Neopower, 6V, 200 rpm
Screw	M1.4 glasses screw
Screw hole workpiece	Twelve M1.4 screw holes
Lamp	LED
Fixing mechanisms	Made in house

summarizes the equipment used in this study.

5. Transformation between Camera 1 and Manipulator

The coordinates of the screw hole, calculated using image processing, must be transformed into the coordinate system of the manipulator through coordinate transformation. Therefore, the equation representing the transformation between Camera 1 and the manipulator is required. The coordinate systems of the manipulator and Camera 1 are defined as {A} and {C}, respectively, as shown in Figure 4. The origin of {A} is at the center of the robot’s base, and the unit of measurement used in the coordinate system is millimeters. The origin of {C} is defined at the upper left corner of an image taken by Camera 1 (Figure 5). One length unit in the X-direction multiplied by one length unit in the Y-direction equals one pixel.

5.1. Transformation Formula

$$\left[\begin{array}{c}{}^{A}P\\ 1\end{array}\right]=\left[\begin{array}{cccc}& {}_C{}^{A}R& & {}^{A}P_{\mathit{CORG}}\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{c}{}^{C}P\\ 1\end{array}\right]$$

(6)

$$\left[\begin{array}{c}{X}_A\\ Y_A\\ Z_A\\ 1\end{array}\right]=\left[\begin{array}{cccc}{r}_{11}& r_{12}& r_{13}& X_0\\ r_{21}& r_{22}& r_{23}& Y_0\\ r_{31}& r_{32}& r_{33}& Z_0\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{c}{X}_C\\ Y_C\\ Z_C\\ 1\end{array}\right]$$

(7)

where ^AP_CORG denotes the translation vector from the origin of the Camera 1 coordinate system to the origin of the manipulator coordinate system, i.e., ($X_0,Y_0,Z_0$).

A 4 × 4 matrix is used as the transformation matrix, with 12 unknown elements required by calculation. The objective Z_A is set as 253 and Z_C is set at 0; thus, the formula can be simplified to the following equation with six unknown elements,

$$\left[\begin{array}{c}{X}_A\\ Y_A\\ 1\end{array}\right]=\left[\begin{array}{ccc}{r}_{11}& r_{12}& X_0\\ r_{21}& r_{22}& Y_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{X}_C\\ Y_C\\ 1\end{array}\right]$$

(8)

5.2. Measurement Data

At least six ($X_A,Y_A$) and ($X_C,Y_C$) data points are required to solve Equation (8) for the six unknowns. In the present experiment, the manipulator was manually controlled and the screw was aligned above the screw holes vertically to determine the ($X_A,Y_A$) data. Each screw hole was aligned three times and the average coordinates were determined. For the same screw hole, if the error was larger than 0.1 mm, the datum was deleted and measurement was repeated. The parameter 0.1 is a custom value for the range of error. Tightening the M1.4 screw with that parameter is allowed. Image processing was used to solve for ($X_C,Y_C$).

It should be noted that increasing the number of measurement data can increase the accuracy of the least square approach. However, greater accuracy is not required for the calibration of Camera 1 since fine tuning will be performed for Camera 2. If it is necessary to speed up the experimental process, the least square method can be omitted to reduce the number of screw hole detections by half. Furthermore, due to the fact that the elements r₁₁~r₂₂ of the rotation matrix ${}_C{}^{A}R$ only depend on angle, there are only three unknowns in Equation (8). The calibration of Camera 1 can be performed with just about 3 points measured on the workpiece.

Table 2. Camera 1 and manipulator measurement data.

No. of Screw Hole	XC	YC	XA	YA
1	921.3380	772.7296	455	30.66
2	195.9257	749.3015	454.85	−35.30
3	635.7407	676.7407	446.46	5.05
4	1116.6790	603.7736	438.80	48.70
5	399.4202	544.6489	434.55	−16.70
6	122.8602	463.9617	427.50	−42.10
7	642.1467	411.4099	421.36	5.30
8	932.9725	385.6583	418.31	31.58
9	328.2440	275.1014	409	−23.21
10	460.1701	213.3160	402.95	−11.51
11	785.4467	198.0349	400.80	17.86
12	1091.8260	152.6689	395.57	46

lists the measurement data of twelve screw holes.

5.3. Calculation

The 12 data points in Table 2 are substituted into Equation (8) as follows,

$$\left[\begin{array}{ccc}{X}_{C1}& Y_{C1}& 1\\ X_{C2}& Y_{C2}& 1\\ ⋮& ⋮& 1\\ X_{C12}& Y_{C12}& 1\end{array}\right]\left[\begin{array}{c}{r}_{11}\\ r_{12}\\ X_0\end{array}\right]=\left[\begin{array}{c}{X}_{A1}\\ X_{A2}\\ ⋮\\ X_{A12}\end{array}\right]$$

(9)

$$\left[\begin{array}{ccc}{X}_{C1}& Y_{C1}& 1\\ X_{C2}& Y_{C2}& 1\\ ⋮& ⋮& 1\\ X_{C12}& Y_{C12}& 1\end{array}\right]\left[\begin{array}{c}{r}_{21}\\ r_{22}\\ Y_0\end{array}\right]=\left[\begin{array}{c}{Y}_{A1}\\ Y_{A2}\\ ⋮\\ Y_{A12}\end{array}\right]$$

(10)

The equations are simplified as follows,

$$C\left[\begin{array}{c}{r}_{11}\\ r_{12}\\ X_0\end{array}\right]=A_X$$

(11)

$$C\left[\begin{array}{c}{r}_{21}\\ r_{22}\\ Y_0\end{array}\right]=A_Y$$

(12)

Here, C is the matrix of the {C} data and A_X and A_Y are the matrices of the {A} data. Using the 12 data points, the six unknowns are calculated using the least square method,

$$\left[\begin{array}{c}{r}_{11}\\ r_{12}\\ X_0\end{array}\right]={{\displaystyle \left(C^{\prime }C\right)}}^{-1}{C}^{\prime }{A}_X$$

(13)

$$\left[\begin{array}{c}{r}_{21}\\ r_{22}\\ Y_0\end{array}\right]={{\displaystyle \left(C^{\prime }C\right)}}^{-1}{C}^{\prime }{A}_Y$$

(14)

Next, the value of the transformation matrix T can be obtained,

$$T=\left[\begin{array}{ccc}{r}_{11}& r_{12}& X_0\\ r_{21}& r_{22}& Y_0\\ 0& 0& 1\end{array}\right]=\left[\begin{array}{ccc}-0.0022& 0.0955& 383.1165\\ 0.0914& 0.0013& -54.4697\\ 0& 0& 1\end{array}\right]$$

(15)

Due to the influence of the preset z axis, the elements of the third row of the matrix is not symmetrical as in Equation (1). However, r₁₂ is not equal to r₂₁, which is due to the existence of measurement errors of data ($X_A,Y_A$) and ($X_C,Y_C$). By using an automatic process, (firstly, randomly move the screw hole workpiece) the coordinates of the screw hole can be determined using image processing. These coordinates can then be substituted into Equation (8) and automatically transformed into coordinates in the manipulator’s coordinate system.

6. Transformation Equation between Camera 2 and Manipulator

Camera 2 photographs the screw hole from a fixed height. The coordinates of the screw hole, determined using image processing, must be transformed into the coordinate system of the manipulator through coordinate transformation. Therefore, the equation representing transformation between the coordinate systems of Camera 2 and the manipulator must be calculated. The coordinate system of the manipulator is defined as {A}, whereas that Camera 2 is defined as {C₂}, as shown in Figure 6.

Coordinate system {C₂} moves with the manipulator. The coordinate system {D} is defined as having the same $\widehat{X}$ and Ŷ values as those of {C₂}, but a different origin, as shown in Figure 7. From its position, the manipulator descends vertically to touch the workpiece with the screw. The point at which the screw touches the workpiece is the origin of {D} and thus also has equal (x, y) to the coordinates of the manipulator. If the origin of {D} is at the center of the screw hole, the screw is considered to be aligned over the screw hole.

6.1. Transformation Formula

$${}^{A}P={}_D{}^{A}R{}^{D}P+{}^{A}P_{\mathit{DORG}}$$

(16)

$$\left[\begin{array}{c}{X}_A\\ Y_A\\ Z_A\end{array}\right]=\left[\begin{array}{ccc}{\widehat{X}}_D·{\widehat{X}}_A& {\widehat{Y}}_D·{\widehat{X}}_A& {\widehat{Z}}_D·{\widehat{X}}_A\\ {\widehat{X}}_D·{\widehat{Y}}_A& {\widehat{Y}}_D·{\widehat{Y}}_A& {\widehat{Z}}_D·{\widehat{Y}}_A\\ {\widehat{X}}_D·{\widehat{Z}}_A& {\widehat{Y}}_D·{\widehat{Z}}_A& {\widehat{Z}}_D·{\widehat{Z}}_A\end{array}\right]\left[\begin{array}{c}{X}_D\\ Y_D\\ Z_D\end{array}\right]+\left[\begin{array}{c}x\\ y\\ z\end{array}\right]$$

(17)

The 3 × 3 matrix is a rotation matrix, for which nine unknowns must be calculated. The objective Z_A is fixed, with Z_D set to 0; thus, the formula can be simplified to the following equation, which has four unknown elements. In the equation, (x, y) indicates the coordinates of the manipulator.

$$\left[\begin{array}{c}{X}_A\\ Y_A\end{array}\right]=\left[\begin{array}{cc}{\widehat{X}}_D·{\widehat{X}}_A& {\widehat{Y}}_D·{\widehat{X}}_A\\ {\widehat{X}}_D·{\widehat{Y}}_A& {\widehat{Y}}_D·{\widehat{Y}}_A\end{array}\right]\left[\begin{array}{c}{X}_D\\ Y_D\end{array}\right]+\left[\begin{array}{c}x\\ y\end{array}\right]$$

(18)

In Figure 8, ${\widehat{X}}_D$ is perpendicular to the dashed line and ${\widehat{Y}}_D$ is collinear with the dashed line. Therefore, Equation (18) can be rewritten as the following equation, with unknowns (a, b, c, d),

$$\left[\begin{array}{c}{X}_A\\ Y_A\end{array}\right]=\left[\begin{array}{cc}\frac{-y}{\sqrt{x^2+y^2}}a& \frac{x}{\sqrt{x^2+y^2}}b\\ \frac{x}{\sqrt{x^2+y^2}}c& \frac{y}{\sqrt{x^2+y^2}}d\end{array}\right]\left[\begin{array}{c}{X}_D\\ Y_D\end{array}\right]+\left[\begin{array}{c}x\\ y\end{array}\right]$$

(19)

6.2. Measurement Data

In the present experiment, the manipulator was manually controlled and the screw manually aligned over each screw hole. Camera 2 took 12 pictures of each screw hole. The coordinates (X_C2, Y_C2) of the screw holes were determined using these 12 pictures (

Table 3. Measurement data from 12 pictures taken by Camera 2.

No. of Screw Hole	XC2	YC2
1	348.5488	241.7073
2	349.4948	232.7661
3	357	235.5000
4	354.3511	250.7480
5	357.8855	234.8419
6	360	223
7	358.5000	239
8	355.8795	245.0656
9	349	240.5000
10	348.5000	242.5000
11	347.3996	245.8686
12	337.9721	246.1950
Average	352.0443	239.8077

). The average values indicate the correct position of the screw hole.

If it needs speedy time, we can reduce the number of detections. At least four ($X_A,Y_A$) and ($X_D,Y_D$) data points were required to solve for the four unknowns. The manipulator was manually controlled to align the screw at the No. 6 screw hole, after which it was moved 1 mm forward, backward, left, and right. Camera 2 took a picture of the manipulator in each of these positions. The coordinates of the screw hole were calculated using these five pictures. The same steps were performed for the No. 12 screw hole.

Table 4. Measurement data for No. 6 screw hole.

Position	x	y	XC2	YC2	XD	YD
Aligned	427.5	−42.1	360	223	0	0
1 mm forward	428.5	−42.1	356	184.5	−4	−38.5000
1 mm backward	426.5	−42.1	364.4524	264.9232	4.4524	41.9232
1 mm left	427.5	−43.1	398.2689	218.4446	38.2689	−4.5554
1 mm right	427.5	−41.1	315	229.5	−45	6.5

and

Table 5. Measurement data for No. 12 screw hole.

Position	x	y	XC2	YC2	XD	YD
Aligned	395.57	46	337.9721	246.1950	0	0
1 mm forward	396.57	46	341.4503	210.5288	3.4782	−35.6662
1 mm backward	394.57	46	336	287	−1.9721	40.8050
1 mm left	395.57	45	378.6491	249.5791	40.6770	3.3841
1 mm right	395.57	47	292.6097	243.6904	−45.3624	−2.5046

list the measurement data for these two screw holes. For the No. 6 screw hole, (x, y) = (427.5, −42.1). For the No. 12 screw hole, (x, y) = (395.57, 46).

6.3. Calculation

Variables u and v are defined as follows,

$$u=\frac{x}{\sqrt{x^2+y^2}}$$

(20)

$$v=\frac{y}{\sqrt{x^2+y^2}}$$

(21)

The 10 data points in Table 4 and Table 5 were substituted into Equation (19),

$$\left[\begin{array}{cc}{X}_{D1}& Y_{D1}\\ X_{D2}& Y_{D2}\\ ⋮& ⋮\\ X_{D12}& Y_{D12}\end{array}\right]\left[\begin{array}{c}-va\\ ub\end{array}\right]=\left[\begin{array}{c}{X}_{A1}-x_1\\ X_{A2}-x_2\\ ⋮\\ X_{A12}-x_{12}\end{array}\right]$$

(22)

$$\left[\begin{array}{cc}{X}_{D1}& Y_{D1}\\ X_{D2}& Y_{D2}\\ ⋮& ⋮\\ X_{D12}& Y_{D12}\end{array}\right]\left[\begin{array}{c}uc\\ vd\end{array}\right]=\left[\begin{array}{c}{Y}_{A1}-y_1\\ Y_{A2}-y_2\\ ⋮\\ Y_{A12}-y_{12}\end{array}\right]$$

(23)

Variables u and v were moved to the left matrix:

$$\left[\begin{array}{cc}-v_1{X}_{D1}& u_1{Y}_{D1}\\ -v_2{X}_{D2}& u_2{Y}_{D2}\\ ⋮& ⋮\\ -v_{12}{X}_{D12}& u_{12}{Y}_{D12}\end{array}\right]\left[\begin{array}{c}a\\ b\end{array}\right]=\left[\begin{array}{c}{X}_{A1}-x_1\\ X_{A2}-x_2\\ ⋮\\ X_{A12}-x_{12}\end{array}\right]$$

(24)

$$\left[\begin{array}{cc}{u}_1{X}_{D1}& v_1{Y}_{D1}\\ u_2{X}_{D2}& v_2{Y}_{D2}\\ ⋮& ⋮\\ u_{12}{X}_{D12}& v_{12}{Y}_{D12}\end{array}\right]\left[\begin{array}{c}c\\ d\end{array}\right]=\left[\begin{array}{c}{Y}_{A1}-y_1\\ Y_{A2}-y_2\\ ⋮\\ Y_{A12}-y_{12}\end{array}\right]$$

(25)

The equations were simplified,

$$D_1\left[\begin{array}{c}a\\ b\end{array}\right]=A_X$$

(26)

$$D_2\left[\begin{array}{c}c\\ d\end{array}\right]=A_Y$$

(27)

The {D} data matrices are represented by D₁ and D₂. The {A} data matrices are represented by A_X and A_Y. With the 10 data points, the four unknowns were solved using the least square method.

$$\left[\begin{array}{c}a\\ b\end{array}\right]={\left({D_1}^{\prime }{D}_1\right)}^{-1}{D_1}^{\prime }{A}_X$$

(28)

$$\left[\begin{array}{c}c\\ d\end{array}\right]={\left({D_2}^{\prime }{D}_2\right)}^{-1}{D_2}^{\prime }{A}_Y$$

(29)

The elements of rotation matrix R were then calculated,

$$\begin{array}{c}R=\left[\begin{array}{cc}{\widehat{X}}_D·{\widehat{X}}_A& {\widehat{Y}}_D·{\widehat{X}}_A\\ {\widehat{X}}_D·{\widehat{Y}}_A& {\widehat{Y}}_D·{\widehat{Y}}_A\end{array}\right]\\ =\left[\begin{array}{cc}-0.0244\frac{-y}{\sqrt{x^2+y^2}}& 0.0254\frac{x}{\sqrt{x^2+y^2}}\\ 0.0234\frac{x}{\sqrt{x^2+y^2}}& 0.0196\frac{y}{\sqrt{x^2+y^2}}\end{array}\right]\end{array}$$

(30)

The matrix is not completely symmetrical, and its slight error is caused by the fact that the inside of the robot arm is not an absolute coordinate system.

Through the automatic process, the coordinates of the screw hole are solved using image processing. These coordinates and the current coordinates of the manipulator (x, y) can be substituted into Equation (19) and automatically transformed into coordinates in the manipulator’s coordinate system.

During real-time implementation, the parameter values of mathematical equations are given in Equations (15) and (30). Using these equations, we can execute the real-time implementation in experiments.

7. Program and Experiment Process

The camera photographs the screw hole. The image is processed and the coordinate value is then transformed through the HMI. The manipulator controller and the HMI communicate through the RS232. The dc motor in the electric screwdriver was controlled in this study by an Arduino UNO. The Arduino and HMI communicated through a Universal Serial Bus connection.

Figure 9 displays the HMI, which has buttons and text boxes as well as an image of all the screw holes. First, the user randomly moves the screw hole workpiece. The workpiece should be in the shooting range of Camera 1. After Camera 1 photographs the workpiece, the image is processed using the methods described in Section 2: gray scale, Gaussian filter, bilateral filter, binarization, edge detection, and center of gravity. The system detects 12 screw holes. If the user chooses the No. 8 screw hole, the system calculates its coordinates and then transforms them into the coordinate system of the manipulator, with the resulting coordinates displayed by the HMI. When the user presses the button “Manipulator Start,” the manipulator moves to the intended position automatically, which in this case is a fixed height above the No. 8 screw hole.

After the manipulator has arrived at the calculated position, Camera 2 photographs the No. 8 screw hole. The image is processed, and the system detects the screw hole and calculates its coordinates, transforming them into the coordinate system of the manipulator. The HMI displays the coordinates, as shown in Figure 10. When the user presses the button “Manipulator Start,” the manipulator moves to the determined position automatically, so that the screw touches the No. 8 screw hole vertically. When the user presses the button “Screwdriver Start,” the dc motor in the electric screwdriver starts rotating, tightening the screw into the screw hole (Figure 11). When the user presses the button “Manipulator Return,” the manipulator returns to its initial position (Figure 12).

8. Conclusions

The proposed approach tightens an M1.4 screw automatically. The cost for Digital Image Processing on a general computer is about 0.5 s, for the manipulator with slow motion is about 5 s, and for the fast motion is about 2 s instead. In contrast with manual labor, automatically tightening screw manipulators is faster, and would not affect the success rate from fatigue. The success rate is about 80% in this study. The failure is mainly due to the fact that the screw is not properly be positioned in the screwdriver head since the screw is secured to the screwdriver head by magnetizing the screw head. Since there is no other research related to tightening tiny screws, we cannot provide performance indicators to compare with other studies. Future work will continue to explore the extension of the scheme to curved 3-D Car-body-like-surfaces.