A Cross-Line Structured Light Scanning System Based on a Measuring Arm

Abstract

The measurement system proposed in this paper, using a measuring arm and line structured light, has a wide range of applications. To improve the scanning efficiency, the system outlined in this paper uses two single-line structured lights to form crosshair structured light, which we combine with a measuring arm to form a comprehensive scanning measurement system. The calibration method of Zhengyou Zhang and a calibration board are used to complete parameter calibration of the sensors and cameras, as well as hand–eye calibration of the measuring arm. For complex curved-surface objects, this system extracts the cross-line structured light optical center location, which suffers from ambiguity. Therefore, we introduce the use of periodic control of the two line structured light sources in order to resolve the light extraction polysemy. Our experimental results indicate that the proposed system can effectively satisfy the function of crosshair structured light scanning of large, complex surfaces.

1. Introduction

With the rapid development of automation technology, increasingly higher machine accuracy of complex parts is required, which has put pressure on modern measurement technology in terms of developing the necessary production components and equipment. Structured light scanning measurement technology has high speed, is highly flexible and highly accurate, and is barely influenced by the surface material of the measured object. It has been widely used for surface contour extraction, reverse engineering, and surface defect detection of product, as well as in other fields [1,2]. As the structured light and camera cannot scan all of an object alone, a coordinate measuring instrument, manipulator, and turntable must be used to obtain 3D information about the object [3]. Measuring arms have been widely used for the measurement of workpiece size, due to their flexibility, light weight, and wide measuring range [4]. Combining the advantages of a measuring arm and structured light measurement, a measuring arm structured-light scanning system was constructed by combining structured light and a camera fixed at the upper end of a measuring arm.

Structured-light scanning systems can be divided into single-line structured light systems and crosshair structured light systems according to the type of light. A single-line structured light emitter only shoots out a laser line, and images are aligned in the camera. The extraction of light strips is not polysemous and is easy to achieve, and this type of approach is widely used [5,6,7] along the orthogonal direction of the structured light. The digital modulus accuracy of the complex surface is not high, and so, it needs to be measured several times along different directions. Crosshair structured light can solve the problem of the measured edge being parallel to the light bar, as its two light planes are perpendicular to each other. However, due to the ambiguity of the plane where the light strips extracted from the camera are located, 3D reconstruction is difficult. Many scholars have conducted research into how to solve the polysemy problem. Liguo Zhang studied the application of cross-line structured light in searching for and tracking electric welds using a template-matching algorithm [8]. Yu Zhejiang used the light plane constraints of a two-sided camera to match the extracted light bar center points [9]. Michael Bleier considered the use of a different color to distinguish between the two light planes, that is, by distinguishing different colors of laser light [10].

In this paper, the method used to solve the ambiguity of light strip extraction was to periodically control two line laser emitters. Specifically, first, only one laser was turned on, which was then turned off, at which point the other laser was turned on. Then, the two lasers were turned on at the same time. This pattern was continued periodically. The advantages of this method are as follows: It can scan complex surfaces, it has low cost, and it can improve the scanning efficiency.

2. Methodology

To achieve rapid and accurate acquisition of the indicated contour of the measured object, we designed a set of crosshair structured-light scanning systems based on the structured light 3D vision measurement principle. Figure 1 shows a schematic diagram of the scanning system. For the single-line structured light A and the single-line structured light B, the light planes are perpendicular to each other, thus forming the cross-line structured light. The cross-line structured light and 2D camera are fixed together onto mounting plate A, and the whole system is installed at the end of the measuring arm. The measuring arm and camera laser are connected to a computer through a USB interface, enabling unified control.

2.1. Model

Each part of the measuring arm cross-line structured light system must be calibrated to work normally.

2.2. 3D Reconstruction

As shown in Figure 2, the projection of point P in space on the image plane is $p_u={\left[\begin{array}{cc}{\mathrm{x}}_{\mathrm{u}}& {\mathrm{y}}_{\mathrm{u}}\end{array}\right]}^{\mathrm{T}}$. Due to the lens distortion, the actual imaging point is $p_d={\left[\begin{array}{cc}{\mathrm{x}}_{\mathrm{d}}& {\mathrm{y}}_{\mathrm{d}}\end{array}\right]}^{\mathrm{T}},$ and the world coordinate of point P is:

$${\mathrm{P}}_{\mathrm{w}}={\left[\begin{array}{ccc}{\mathrm{x}}_{\mathrm{w}}& {\mathrm{y}}_{\mathrm{w}}& {\mathrm{z}}_{\mathrm{w}}\end{array}\right]}^{\mathrm{T}}.$$

(1)

This is converted to the camera frame ${\mathrm{O}}_{\mathrm{c}}{\mathrm{X}}_{\mathrm{c}}{\mathrm{Y}}_{\mathrm{c}}{\mathrm{Z}}_{\mathrm{c}}$ by:

$${\mathrm{P}}_{\mathrm{c}}={\left[\begin{array}{ccc}{\mathrm{x}}_{\mathrm{c}}& {\mathrm{y}}_{\mathrm{c}}& {\mathrm{z}}_{\mathrm{c}}\end{array}\right]}^{\mathrm{T}}={\mathrm{RP}}_{\mathrm{w}}+\mathrm{t},$$

(2)

where R and t represent the rotation and translation, respectively, of the world coordinate system and the camera coordinate system. The relationship between $p_u{\mathrm{and}\mathrm{p}}_{\mathrm{c}}$ is as follows:

$$\left[\begin{array}{c}{x}_u\\ y_u\end{array}\right]=\left[\begin{array}{c}\frac{X_c}{Z_c}\\ \frac{Y_c}{Z_c}\end{array}\right].$$

(3)

Considering only the radial and tangential distortion of the lens, the relationship between the actual imaging point $p_d$ and the theoretical point $p_u$ is:

$$\left[\begin{array}{c}{x}_d\\ y_d\end{array}\right]=\left(1+k_1{r}^2+k_2{r}^4+k_5{r}^6\right)\left[\begin{array}{c}{x}_u\\ y_u\end{array}\right]+\left[\begin{array}{c}2k_3{x}_u{y}_u+k_4\left(r^2+2x_u{}^2\right)\\ 2k_4{x}_u{y}_u+k_3\left(r^2+2y_u{}^2\right)\end{array}\right],$$

(4)

where ($k_1$, $k_2$, $k_5$) are the radial and ($k_3$, $k_4$) are the tangential distortion parameters and $r^2=x_u{}^2+y_u{}^2$. The pixel coordinates on the image $p={\left[\begin{array}{cc}\mathrm{u}& \mathrm{v}\end{array}\right]}^{\mathrm{T}} \mathrm{are}$:

$$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}{f}_x& 0& c_x\\ 0& f_y& c_y\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{x}_d\\ y_d\\ 1\end{array}\right],$$

(5)

where f_x and f_y are the respective focal lengths and c_x and c_y are the principal points. In the case of known pixels, through Equation (5), Equation (4) can be used to calculate p_u. However, we cannot calculate Pc, as the constraints are not repeated; in this case, additional constraints are required.

Suppose the equation of the light plane of the laser in the camera coordinate system is:

$$aX+bY+cZ+D=0.$$

(6)

Putting Equation (3) into Equation (6), we obtain:

$$a{x}_u{Z}_c+b{y}_u{Z}_c+c{Z}_c+D=0$$

(7)

P is solved for the camera coordinate system as follows:

$$Z_c=-\mathrm{D}/(a{x}_u+b{y}_u+c),X_c=x_u{Z}_c,Y_c=y_u{Z}_c.$$

(8)

In this paper, the method in [11] was used to calibrate the structured light plane of a line.

2.3. Laser Line Extraction

The control of the two single-line lasers was carried out as explained next.

Figure 3 depicts the manner in which the two lasers were turned off and on periodically. “Time” denotes the time required for the camera to acquire a frame, which is aligned and synchronized in the hardware. Figure 4 shows the pictures captured by the camera when laser A, B and AB shoot on the plane.

The direction of the line in the 2D image was estimated locally by computing the eigenvalues and eigenvectors of the Hessian matrix. The response of the ridge detector given by the value of the maximum absolute eigenvalue is a good indicator of the saliency of the extracted line points. It should be noted that the algorithm for the method is not able to run on an FPGA due to the complexity of the algorithm.

For the extraction of laser lines, we used the gray center method. When laser A is turned on, the current frame is denoted by N and the center of the light bar was extracted from left to right and top to bottom. In the j-th column of the image, the light bar region is S. Then, the abscissa is calculated as:

$$u_{jN}^c=\frac{{{\displaystyle \sum }}_{y\in s}gray\left(j,y\right)\ast y}{{{\displaystyle \sum }}_{y\in s}gray\left(j,y\right)}.$$

(9)

All the collected light bars are collectively denoted as:

$$U_N^C=\left\{u_{jN}^c\right\}.$$

(10)

Similarly, when laser B is turned on, the current frame is N + 1 and the light bar in the j-th column is located in region S. Similarly, the abscissa is:

$$u_{jN+1}^c=\frac{{{\displaystyle \sum }}_{x\in s}gray\left(j,y\right)\ast y}{{{\displaystyle \sum }}_{x\in s}gray\left(j,y\right)}.$$

(11)

The coordinates of all the collected light bars are denoted as follows:

$$U_{N+1}^C=\left\{u_{jN+1}^c\right\}.$$

(12)

When lasers A and B are opened at the same time, light bars in all column directions are found. At this time, the frame is N + 2 and the collected light bars are denoted as:

$$U_{N+2}^C=\left\{u_{jN+2}^c\right\}.$$

(13)

The position of the light bar recorded by $U_{N+2}^C$ in Equation (13) is polysemous and needed to be distinguished, for which the following equation can be used:

$$\left|u_{jN+2}^c-u_{jN}^c\right|\le T,$$

(14)

and

$$\left|u_{jN+2}^c-u_{jN+1}^c\right|\le T.$$

(15)

In the comparison, $T$ is the allowable threshold for extracting the optical bar position. If it matches Equation (14), it is determined that the light strip belongs to laser A, and if it matches Equation (15), then the strip belongs to laser B. If three pictures are captured by the camera, we can only obtain four sets of points with the cross-line laser compared to three sets of points obtained with the single-line laser. Through this method to improve the frame rate of the 3D points, the scanning frame rate increased to 1.33 times that with a single-line laser ideally.

2.4. Hand–Eye Calibration

Hand–eye calibration is required to establish a relationship between the measuring arm and the camera. In the system shown in Figure 1, assuming that there is a fixed point P_board on the calibration board, the coordinate $P_{base}^i$ in the base coordinate under the condition that the measuring arm takes pose i satisfies the following relationship:

$$P_{base}^i={}_{tool}{}^{base}T{}^i {}_{cam}{}^{tool}T{}^i {}_{word}{}^{cam}T{ }^i{P}_{board},$$

(16)

where ${}_{tool}{}^{base}T{}^i$ is the transformation matrix between the tool coordinate system at the end of the measuring arm and the attitude of base i, which is directly fed back by the measuring arm; ${}_{cam}{}^{tool}T{}^i$ is the transformation between the camera coordinate system and the tool coordinate system at the end of the measuring arm, which is the target relationship required to be solved in the hand–eye calibration; and ${}_{word}{}^{cam}T{ }^i$ is the relationship between the points on the calibration plate and the camera coordinate system, which can be solved using the calibration method of Zhengyou Zhang [12]. Additionally, in the j-th pose:

$$P_{base}^j={}_{tool}{}^{base}T{}^j {}_{cam}{}^{tool}T{}^j {}_{word}{}^{cam}T{ }^j{P}_{board}.$$

(17)

As the calibration plate is in a stationary state with respect to the measuring arm base:

$$P_{base}^j=P_{base}^i.$$

(18)

Combining Equations (16)–(18), we have:

$${}_{tool}{}^{base}T{}^i {}_{cam}{}^{tool}T{}^i {}_{word}{}^{cam}T{ }^i={}_{tool}{}^{base}T{}^j {}_{cam}{}^{tool}T{}^j {}_{word}{}^{cam}T{ }^j.$$

(19)

Obviously, A and B are the same in positions i and j.

From Transformation (19), we have:

$${\left({}_{cam}{}^{tool}T{}^j \right)}^{-1}{}_{tool}{}^{base}T{}^i X=X{}_{word}{}^{cam}T{ }^j{\left({}_{word}{}^{cam}T{ }^i\right)}^{-1}.$$

(20)

As

$$A={\left({}_{cam}{}^{tool}T{}^j \right)}^{-1}{}_{tool}{}^{base}T{}^i ,$$

(21)

$$B={}_{word}{}^{cam}T{ }^j{\left({}_{word}{}^{cam}T{ }^i\right)}^{-1},$$

(22)

Equation (20) can be written as:

$$AX=XB.$$

(23)

For Equation (23), the method in [13] was used to solve the relationship X between the camera coordinate system and the tool coordinate system at the end of the measuring arm.

2.5. Laser Plane Extraction

The relationship R and t in Equation (2) between the circle grid board and the camera can be determined from the known camera parameters in Equations (4) and (5). Thus, we obtained the laser line positions (height A) in the camera coordinate system with R and t. Fixing the camera and moving the circle grid board to another position, we obtained the laser line positions (height B) in the same manner as height A. The laser line plane was then calculated using heights A and B.

3. Experiment

3.1. The Hardware System

Figure 5 shows the hardware system used in this experiment. The system mainly consisted of a flexible 7-axis measuring arm, laser A, laser B, and a 2D camera. The hardware parameters were as follows:

(1)

Precision of the measuring arm: 0.02 mm

(2)

For the 2D camera:

(a)

Field of view: 62–72 mm

(b)

Acquisition rate: 80 frames per second (FPS)

(c)

Resolution: 3072 × 2048 pixels

(d)

Working resolution: 2658 × 800 pixels

(3)

Linear structured light 3D camera based on camera and laser

(a)

Z direction resolution: 0.01–0.012 mm

(b)

Laser line resolution (near–far): 0.025–0.028 mm

(c)

Laser wavelength: 405 nm

Figure 6 shows the calibration camera and the circular array calibration plate used for hand–eye calibration. In Figure 7, the circle spacing was 3.75 mm, the circle diameter was 1.875 mm, and the accuracy was 0.001 mm. The material was ceramic. The metal gauge block used to verify the calibration result is also shown.

3.2. Calibration of the Camera and Laser Light Plane

First, the camera was calibrated, following which the calibrated camera parameters were used to calibrate the optical plane of AB and lasers A and B. As shown in Figure 8, 15 pictures were collected from various angles, and the internal parameters of the camera were obtained using the calibration method of Zhengyou Zhang, where fx = 5145.362, fy = 5153.664, cx = 1517.375, and cy = 1023.363. In the obtained distortion parameters, for the radial distortion, $k_1=-0.09903$, $k_2=0.195427$, and $k_5=0.41673$; meanwhile, for the tangential distortion, $k_3=-0.000065974$ and $k_4=-0.0000978477$.

As shown in Figure 9, the upper and lower graphs on the left were used to calibrate the optical plane of laser B, while the upper and lower graphs on the right were used to calibrate the optical plane of laser A. The calibrated equation of the light plane of laser A was as follows:

$$\left[A_{\mathrm{a}}{B}_{\mathrm{a}}{C}_{\mathrm{a}}{D}_{\mathrm{a}}\right]=\left[-0.659905-0.5517320.51-68.7215\right],$$

while that for laser B was:

$$\left[A_{\mathrm{b}}{B}_{\mathrm{b}}{C}_{\mathrm{b}}{D}_{\mathrm{b}}\right]=\left[0.76417-0.5431560.34787-49.616\right].$$

The accuracy of the calibration was verified by measuring the height of 20 mm and the statistical height distribution, as explained next.

As shown in Figure 10, a measurement block with a height of 20 mm was used for verification. The abscissa represents the measured height (in mm), and the ordinate represents the height value statistics (in units). The mean height was 19.989 mm, and the standard deviation(std) of height was 0.006 mm.

3.3. Hand–Eye Calibration

As shown in Figure 11, the joint arm with the camera photographed the calibration plate in different postures and recorded the coordinates of the joint arm feedback. A total of five sets were captured.

Using Equation (22), we obtained:

$$X=\left[\begin{array}{cccc}-0.42684& 0.66270& -0.61532& 4.1373\\ -0.904314& -0.3163& 0.286604& 21.393\\ -0.004728& 0.678785& 0.73432& 112.962\\ 0& 0& 0& 1\end{array}\right]$$

3.4. Scanning Test

The calibration result X and laser light plane parameters were used for scanning and testing.

Figure 12 shows a scan of the calibration plate (note that only a single scan could be completed), Figure 13 shows a scan of a complex surface that is correct, and Figure 14 shows the scanning of a metal pellet.

Figure 14 shows a white plate with a scanning flatness of 0.015 mm.

Figure 15 (right) shows a point cloud obtained from a white flat board. All points were fit to a plane, and the distance from all points to the plane was calculated. The statistical histogram shown in Figure 16 was obtained. The abscissa represents the distance (in mm), while the ordinate is the number of points in the distribution, with 96% distributed within +/−0.05 mm. The main deviation was due to the hand–eye calibration.

In the scan, the determination threshold T of Equation (14) was set as 10 pixels.

If Equations (14) and (15) both met the conditions, the one with the smallest difference was chosen.

4. Conclusions

In this paper, we presented a cross−line laser scanning system with an articulated arm. We showed how a single-line laser scanning technique can be extended to a cross−line laser scanning system in order to improve the scanning efficiency. Moreover, we provided our implementation details, through which an object point cloud can finally be obtained, and showed how to differentiate the cross-line in one frame by comparing the pixel position extracted throughout the frame sequence. We experimentally demonstrated that high-quality, accurate scans can be achieved, while reducing the time required to scan a complex surface; in particular, the frame rate of 3D points was 1.33 times faster than that of single-line structured light with the same 2D camera. However, it should be noted that further work is necessary to improve the ratio of AB opening simultaneously, which may increase the efficiency of scanning by controlling the periodicity of the two lasers.