Single-Shot Three-Dimensional Reconstruction Using Grid Pattern-Based Structured-Light Vision Method

Abstract

Structured-light vision methods are widely employed for three-dimensional reconstruction. As a typical structured light pattern, grid pattern is extensively applied in single-shot three-dimensional reconstruction. The uniqueness of the grid feature retrieval is critical to the reconstruction. Most methods using grid pattern utilize the epipolar constraint to retrieve the correspondence. However, the low calibration accuracy of the camera–projector stereo system may impact the correspondence retrieval. An approach using grid pattern-based structured-light vision method is proposed. The grid pattern-based structured-light model was combined with the camera model and the multiple light plane equations. An effective extraction method of the grid stripe features was investigated. The system calibration strategy, based on coplanar constraint, is presented. The experimental setup consisted of a camera and an LED projector. Experiments were carried out to verify the accuracy of the proposed method.

1. Introduction

Three-dimensional reconstruction technologies are widely applied in many fields, such as reverse engineering, robot vision, medical surgery, intelligent manufacturing, and in the entertainment industry [1,2,3,4,5]. The technologies could be classified into three categories: time-of-flight (TOF), stereo vision, and structured light (SL) [6]. SL techniques have many advantages over the other methods in terms of high accuracy, robustness, simplicity and low cost [7,8]. The available projection patterns for SL techniques could be the single-line [3,9], multi-line [10,11] or surface pattern [12,13]. By projecting surface pattern onto the object and capturing the corresponding deformed patterns modulated by the object profile, 3D information could be reconstructed more effectively than with the other two patterns [6,12].

According to the codification strategies, the surface patterns can be roughly categorized into two classes: temporal-based and spatial-based schemes [14]. The temporal-based approaches have advantages in terms of high accuracy and robustness. These approaches adopt the binary code method [7,15] or the phase shifting method [16,17,18]. However, the approaches have to conduct multi-shots to capture the patterns in a time series. So, the temporal-based techniques are not suitable for dynamic scenes. In contrast, the spatial-based approaches only implement a single-shot, which is favorable for 3D reconstruction of dynamic scenes [19]. Among spatial-based methods, the encoding scheme, such as the De Bruijn sequence [20], M-arrays [21] and pseudo-random codes [22], is a commonly used strategy [7] These methods encode the features by different heights, widths, orientations, or compactness for a unique identity. However, the uniqueness of the local code-word retrieval is hard to guarantee, due to perspective distortion [23]. Differing from spatial-coded patterns, coding-free patterns are always simple and periodic. The methods using coding-free patterns are more robust, insensitive to distortion, and low-cost [24].

Grid pattern is a typical surface pattern. Some researchers have studied methods with a grid pattern in recent decades [6,24,25,26,27,28,29,30]. Salvi et al. [25] employed a chromatic grid pattern with six different colors to identify lines, where three colors are assigned for horizontal lines, and the other three for vertical lines. The De Bruijn sequence with order determines the three colors of three adjacent lines. Sagawa et al. [26] used a periodic color-encoded grid pattern to achieve more accurate retrieval correspondence. Furukawa et al. [31] proposed a grid pattern with two colors to distinguish the vertical and horizontal lines. Then coplanarity constraint and epipolar constraint are utilized to retrieve the correspondence. All of these methods require projecting a chromatic pattern for correspondence retrieval. Shi et al. [24] investigated the depth sensing method by using a coding-free binary grid pattern based on a camera and a DLP projector stereo platform. A coarse-to-fine line detection strategy was applied for extracting the grid. A graph-based topological labeling algorithm was proposed to determine the topological coordinates of the intersections of the grid. By using the topology of the grid and the epipolar constraint, the corresponding pixels between the projected and captured pattern were matched. Then the 3D reconstruction was implemented, based on the triangular stereo model. However, low calibration accuracy of the camera–projector stereo system may occur and influence the correspondence determination with epipolar constraint. The sub-grid pixel matching is realized by solving an over-determined system of equation. The least square solution should be calculated in every sensing process.

This paper aimed to provide a simple and effective single-shot 3D reconstruction approach using the grid pattern-based structured-light vision method. Without the use of the epipolar constraint, the system model was directly combined with the camera model and multiple light plane equations, which avoids the error propagation due to the low calibration accuracy of the stereo vision model. Thus, the uniqueness of the grid feature retrieval is the only critical task in the calibration and reconstruction. We investigated the extraction method of grid stripe features. The calibration method, based on coplanar constraint, is proposed. The experimental setup consisted of a camera and an LED projector. Experiments were carried out to verify the accuracy of the proposed method.

The remainder of this paper is organized as follows. In Section 2, we introduce the system model. And the extraction method of the grid stripe features is explained. Furthermore, the system calibration procedure is also elaborated in Section 2. Experimental results are presented in Section 3. We make a conclusion in Section 4.

2. Methods

In this section, the system model is first elaborated, and the model coefficients introduced. Then, the extraction method of the grid stripe features is investigated. Finally, the calibration procedures are proposed.

The extraction method of the grid stripe features guaranteed unique retrieval during measurement and calibration. The disturbance to the grid stripe extraction during calibration was addressed. The calibration procedures could be accomplished by collecting only one set of images.

2.1. System Model

Generally, an SL system, using grid pattern only, has two components: a camera and a projector (Figure 1). The binary grid pattern is shown in Figure 2. The value O_w-X_wY_wZ_w is the world coordinate system, wherein the measured object is located. The value O_c-X_cY_cZ_c is the camera coordinate system, O_I-X_IY_I is the physical image coordinate system, and o-UV is the pixel coordinate system corresponding to O_I-X_IY_I. The projector casts the grid pattern onto the object. The camera captures the grid feature image indicating the object topography. Point M is one intersection of the grid pattern and the object surface. According to the pinhole model, the point m is the ideal (distortion-free) pixel point of the object point M. The measuring coordinate system is usually consistent with O_c-X_cY_cZ_c. Thus, the conversion function between m(u, v) and M(x_c, y_c, z_c) can be expressed as [32]:

$$z_c\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}{f}_x& 0& u_0\\ 0& f_y& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{cc}{R}_c^w& T_c^w\end{array}\right]\left[\begin{array}{c}{x}_w\\ y_w\\ \begin{array}{c}{z}_w\\ 1\end{array}\end{array}\right]=\left[\begin{array}{ccc}{f}_x& 0& u_0\\ 0& f_y& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{x}_c\\ y_c\\ z_c\end{array}\right]$$

(1)

where (x_w, y_w, z_w) represents the coordinates of point M in O_w-X_wY_wZ_w; f_x and f_y are the camera pixel focal lengths in the horizontal ordinate direction, respectively; (u₀, v₀) are the coordinates of the principal point in o-UV; $R_c^w$ and $T_c^w$ represent the rotation matrix and the translational vector from O_w-X_wY_wZ_w to O_c-X_cY_cZ_c.

The projected grid pattern could be defined as multiple orthogonal light planes in O_c-X_cY_cZ_c. The equations of the planes are as follows [33]:

$$\{\begin{array}{c}{\Pi }_{Hi}: a_{Hi}{x}_c+b_{Hi}{y}_c+c_{Hi}{z}_c+d_{Hi}=0, \left(i=1,2,\dots ,n\right)\\ {\Pi }_{Vj}:a_{Vj}{x}_c+b_{Vj}{y}_c+c_{Vj}{z}_c+d_{Vj}=0, \left(j=1,2,\dots ,n\right)\end{array}$$

(2)

where ${\Pi }_{Hi}$ is the horizontal light plane; ${\Pi }_{Vj}$ is the vertical light plane; [a_Hi, b_Hi, c_Hi, d_Hi] and [a_Vj, b_Vj, c_Vj, d_Vj] represent the four coefficients of each ${\Pi }_{Hi}$ and ${\Pi }_{Vj},$ respectively; n is the number of the planes.

The pinhole model and the multiple light plane constraint, (x_c, y_c, z_c) combined can be calculated using the following formula:

$$\{\begin{array}{c}\hspace{1em}{x}_c=\frac{\left(u-u_0\right)·z_c}{f_x}\\ \hspace{1em}{y}_c=\frac{\left(v-v_0\right)·z_c}{f_y}\\ z_c=\frac{-d_i}{a_i·\frac{\left(u-u_0\right)}{f_x}+b_i·\frac{\left(v-v_0\right)}{f_y}+c_i}\end{array}$$

(3)

where [a_i, b_i, c_i, d_i] is [a_Hi, b_Hi, c_Hi, d_Hi] or [a_Vj, b_Vj, c_Vj, d_Vj].

In addition, the distortion caused by the lens is inevitable. Considering the tangential distortion and radial distortion, the relationship between the actual (distorted) pixel coordinates (u_d, v_d) and ideal pixel coordinates (u, v) of point m can be expressed as [34]:

$$\left[\begin{array}{c}{u}_d\\ v_d\end{array}\right]=\left(1+k_1{r}^2+k_2{r}^4\right)\left[\begin{array}{c}u\\ v\end{array}\right]+\left[\begin{array}{c}2p_{1}uv+p_2\left(r^2+2u^2\right)\\ 2p_{2}uv+p_1\left(r^2+2v^2\right)\end{array}\right]$$

(4)

where r is the distance between (u, v) and (u₀, v₀); (k₁, k₂) are the radial distortion coefficients; (p₁, p₂) are the tangential distortion coefficients.

2.2. Extraction of the Grid Stripe Features

The grid pattern consists of a number of intersecting light stripes. The common algorithms for stripe center extraction are inaccurate [35,36]. The intersections of the grid lines could be precisely extracted based on our previous work [37]. Thus, an effective extraction method of the grid stripe features is proposed. The procedures of the method are as follows:

(1)

The corner points of the grid lines are detected by using the Shi-Tomasi algorithm [38]. The green points in Figure 3b indicate the corner of the intersecting lines.

(2)

Then the corner points are clustered into m groups by using the DBSCAN algorithm [39]. The points in the same group wrap one of the intersecting regions R_i (i = 1, 2, … m) as shown in Figure 3c.

(3)

After that, the center points of the regions can be easily obtained and sorted to be the markers (Figure 3d).

(4)

The gray centroid method is used to extract the strip centers between adjacent markers (Figure 3e). Finally, the horizontal stripes, L_Hi, and vertical stripes, L_Vj, are obtained and topologically labeled (Figure 3f).

During the calibration, the grid pattern should be projected on the calibration board (Figure 4). The white circles on the board can disturb the corner extraction (Figure 5b). To solve the problem, the gray value of the ellipse region on the input image is properly attenuated. The ellipses are extracted during the calibration. The maximum gray value in each ellipse is used as the threshold of gray filtering to remove the interference of the white circles. After the image gray attenuation, the corner detection of the output image (Figure 5d) can achieve the correct result (Figure 5e). Hence, the extraction of grid stripe features in a captured image is guaranteed by using the proposed algorithm. In Figure 6, the red lines tag the extracted horizontal stripes, while the yellow lines tag the vertical ones.

2.3. Calibration Procedures

According to the measuring principle of the SL system using a grid pattern, there are three categories of parameters which need to be calibrated: intrinsic parameters (f_x, f_y, u₀, v₀), distortion coefficients (k₁, k₂, p₁, p₂) and multiple light planes (${\Pi }_{Hi}$, ${\Pi }_{Vj}$). By obtaining different pose views of a 2D coplanar point calibration board under several angles, all the parameters can be calibrated with a simple operation (Figure 7).

The procedures of the system calibration are shown in Figure 8. The calibration board as a target are placed in different poses (Figure 7). The grid pattern is projected on the board at each position. A set of target images and corresponding grid pattern images are acquired first.

The elliptic centers on each target image are extracted to provide the data for the camera calibration. By adopting Zhang’s method [40], the intrinsic parameters and distortion coefficients can be obtained. In addition, the rotation matrix R_i and translation vector T_i of each target pose can be calculated. The equations of the target planes in O_c-X_cY_cZ_c can be formed as:

$$a_{ti}{x}_c+b_{ti}{y}_c+c_{ti}{z}_c+d_{ti}=0$$

(5)

where $\overrightarrow{n}=$ [a_ti, b_ti, c_ti] is the third column of R_i, d_tn = -$\overrightarrow{n}·T_i$. n = 1, 2,…, N. N is the total number of the positions.

Meanwhile, the stripe centers extraction is performed on the grid pattern images. The actual (distorted) pixel coordinates of the centers are labelled according to the topological relationship of the grids. The values $m_{Vij}^{\prime n}$ and $m_{Hij}^{\prime n}$ are the actual pixel points on the vertical and horizontal stripes, respectively. Subsequently, the ideal (distortion-free) pixel points $m_{Vij}^n$ and $m_{Hij}^n$ are calculated by substituting the actual pixel coordinates and calibrated distortion coefficients into Equation (4). By combining Equations (1) and (5), the 3D coordinates of the ideal pixel points in O_c-X_cY_cZ_c can be obtained. The values $M_{Vij}^n$ and $M_{Hij}^n$ are the 3D grid feature points on the target planes and also the points on the grid light planes. Finally, the equations of the vertical and horizontal light planes can be fitted with the corresponding $M_{Vij}^n$ and $M_{Hij}^n$.

3. Experiments and Discussion

The experimental platform was constructed as shown in Figure 9. The camera was a Hikvision MV-CA013-21UM, with a 1280 × 1024 resolution. The focal length of the lens was 16 mm. Instead of utilizing a DLP projector, a low-cost LED projector was employed, which minimized the cost, reducing it by about 60%. The projector was the OPT structured light source OPT-SL03-R, with 16 × 16 grid stripes. The camera and the projector were aligned with an angle. The system parameters were calibrated in advance. An electric sliding table (Zolix TSA200) and a grating ruler with 1 µm accuracy were used to verify the precision of measurements. The algorithms and software were developed using Visual C# 2019 and opencv.

It should be noted that the grid pattern-based structured-light vision method suffers from a monotonicity constraint, even though it has many advantages in terms of higher accuracy and lower cost, compared with time-of-flight (ToF) and stereo vision [24]. The method is not capable of measuring large depth changes due to the monotonicity constraint. There is an upper bound of depth change. When all the local depth changes in a scene were less than the upper bound, the monotonicity constraint was satisfied, and, thus, all the light stripes could be obtained in the image (Figure 10a). The monotonicity constraint was violated when measuring depth changes over the upper bound, resulting in loss of light stripes (Figure 10b).

The upper bound of depth change, by employing the method is given by the following formula [24]:

$$\Delta z=\frac{z_0^2\Delta d}{f_{p}B+z_0\Delta d}$$

(6)

where f_p denotes the focal length of the projector, B denotes the length of the baseline, z₀ denotes the reference depth, and Δd denotes the line interval in the projected grid. In our system, z₀ = 450 mm, Δd = 0.45 mm, B = 250 mm, f_p = 35 mm. Thus, the Δz = 10.179 mm.

3.1. System Calibration

The 2D coplanar calibration board for the calibration had plenty of circular features (Figure 11). The specification of the target is shown in

Table 1. The specification of the target.

Specifications	Values
Boundary dimensions (mm)	180 × 150
Circle diameter (mm)	7 (L), 3.5 (S)
Center distance (mm)	15
Array number	11 × 9
Precision (mm)	±0.01

.

The target was placed at six different positions in the viewing field of the camera. The contour points of the white circles were extracted and fitted to the ellipses, as shown in Figure 12a. The ellipse centers were regarded as the feature points for calibration. The ellipse centers were sorted and marked according to topological ordering, as shown in Figure 12b.

By utilizing Zhang’s method [40], the intrinsic parameters, distortion coefficients and relative poses R_i, T_i were obtained. The intrinsic parameters and distortion coefficients are listed in

Table 2. The intrinsic parameters and distortion coefficients.

Parameters	Results
fx, fy (pixel)	3354.0982, 3354.8173
u0, v0 (pixel)	697.163, 478.755
k1, k2	−0.041, −1.344
p1, p2	−0.00039, −0.00041

. The rotation matrix R_i was transformed into a vector formation rvec_i, consisting of Euler angles. The extrinsic parameters of the six pose views are listed in

Table 3. The extrinsic parameters of the six pose views.

Pose Views	rveci	Ti
1	[−3.020, 0.046, −0.760]	[1.616, 5.987, 514.204]
2	[−3.124, 0.083, −0.186]	[−8.074, 5.674, 511.550]
3	[−3.048, 0.017, −0.677]	[−18.359, 5.856, 500.676]
4	[−3.073, −0.015, −0.548]	[−25.7326, 0.505, 549.236]
5	[−3.021, 0.031, −0.023]	[−7.029, 5.590, 469.236]
6	[−2.990, 0.909, −0.355]	[−9.006, 2.654, 464.551]

. The reprojection errors were the distances from the measured 2D points and their respective projections obtained by applying the calibrated camera model to the 3D points [41]. The mean reprojection errors of the six images were below 0.103 pixel, as shown in Figure 13.

There were sixteen horizontal and sixteen vertical planes on the grid pattern. The parameters of the light planes are listed in

Table 4. The parameters of the light planes.

ID	[Aver, Bver, Cver, Dver]	[AHor, BHor, CHor, DHor]
1	[0.807, 0.009, 0.59, −224.019]	[−0.041, 0.996, 0.807, 6.870]
2	[0.814, 0.009, 0.579, −224.667]	[−0.035, 0.997, 0.071, 6.137]
3	[0.821, 0.008, 0.569, −225.443]	[0.029, −0.998, −0.061, −5.225]
4	[0.829, 0.008, 0.558, −225.711]	[−0.023, 0.999, 0.05, 4.704]
5	[0.836, 0.007, 0.548, −226.566]	[−0.016, 0.999, 0.039, 4.195]
6	[0.843, 0.007, 0.537, −226.901]	[−0.01, 0.999, 0.028, 3.694]
7	[0.849, 0.007, 0.527, −227.53]	[−0.003, 0.999, 0.017, 3.244]
8	[0.856, 0.006, 0.516, −227.949]	[0.004, 1.000, 0.006, 2.82]
9	[0.863, 0.005, 0.505, −228.439]	[−0.01, −0.999, 0.005, −2.447]
10	[0.869, 0.006, 0.493, −228.702]	[−0.017, −0.999, −0.015, 1.838]
11	[0.875, 0.004, 0.483, −229.223]	[0.023, −0.999, −0.028, 1.98]
12	[0.881, 0.004, 0.471, −229.522]	[0.029, 0.998, −0.037, 1.143]
13	[0.897, 0.004, 0.46, −229.653]	[−0.036, −0.998, 0.049, −1.109]
14	[0.893, 0.004, 0.449, −230.287]	[0.043, 0.997, −0.059, 0.085]
15	[0.898, 0.003, 0.438, −230.369]	[0.048, 0.996, −0.069, −0.395]
16	[0.903, 0.003, 0.428, −231.273]	[−0.054, −0.995, 0.078, 1.719]

. The values [A_ver, B_ver, C_ver, D_ver] represent the coefficients of the vertical planes, while the values [A_Hor, B_Hor, C_Hor, D_Hor] represent the coefficients of the horizontal planes. The planes in the camera coordinate system are shown in Figure 14.

3.2. Measurement Verification

Displacement measurement experiments were conducted to evaluate the performance of the grid pattern-based structured-light vision method. A plane board was placed on the electrical sliding table and carried to several positions in front of the system. The electrical sliding table was equipped with a grating ruler (1 µm accuracy). The moving distances measured by the grating ruler were regarded as the ground truth. Three-dimensional coordinates of the grid pattern points on the board were obtained by the proposed method and fitted to planes at each location. The plane at the first position was regarded as the base plane. The distances from the 3D points on the other planes to the base plane were calculated sequentially. The measurement results and the corresponding ground truth values are listed in

Table 5. Accuracy verification experiment.

Ground Truth (mm)	Measurement Results (mm)	Errors (mm)	Relative Errors
3.862	3.869	0.007	0.181%
6.838	6.824	−0.014	−0.205%
9.045	8.969	−0.076	−0.840%
12.524	12.475	−0.049	−0.391%
15.601	15.514	−0.087	−0.558%
50.190	20.039	−0.151	−0.748%
24.573	24.558	−0.015	−0.061%
28.892	28.879	−0.013	−0.045%
32.511	32.550	0.039	0.120%
61.571	61.748	0.177	0.287%
113.541	113.880	0.339	0.299%

. The maximum relative error was 0.840%.

The 3D reconstruction of a gypsum sphere object was implemented in single-shot by using the grid pattern-based structured-light vision system. The Intel Realsense D455 is a popular RGB-D camera, using the speckle-based structured-light stereo vision method. As shown in Figure 15, a D455 camera was employed to reconstruct the same gypsum sphere. The approximate distance from the sphere to our system and the RGB-D camera was 450 mm.

The experimental result using our approach is shown in Figure 16, while the result using the D455 camera is shown in Figure 17.

The nominal radius of the gypsum sphere was 85 mm. The 3D reconstruction data from our system and from the D455 camera were fitted to the sphere equation to calculate the measured radius. The measured radius of our approach and of that using D455 are listed in

Table 6. The measured radius of our approach and D455.

Nominal Radius (mm)	Our Approach (mm)	D455 (mm)
85.000	84.652	88.937

.

4. Conclusions

In this paper, a simple and effective single-shot 3D reconstruction approach, using the grid pattern-based structured-light vision method, was proposed. The system consisted of a camera and a low-cost LED projector.

The grid pattern-based structured-light model was combined with the camera pinhole model and the multiple light plane constraint. The tangential distortion and radial distortion were compensated for. Without the use of the epipolar constraint, the error propagation of the low calibration accuracy of the stereo vision model was avoided. The grid pattern consisted of a number of intersecting light stripes. The common algorithms for stripe center extraction are inaccurate. However, the intersections of the grid lines could be precisely extracted. Following the topological distribution of the intersections, an effective extraction method was investigated to gather the sub-pixel centers on the grid strips. The white circles on the calibration board could disturb the extraction. To solve the problem, the gray value of the ellipse region on the input image was properly and automatically attenuated. A calibration method, based on the coplanar constraint, was presented, which could be implemented in one step.

Displacement measurement experiments were conducted to evaluate the performance of the proposed method. The displacements measured by the grating ruler with 1 µm accuracy were regarded as the ground truth. The maximum relative error between the measurement results and the corresponding ground truth values was 0.840%. The 3D reconstruction of a sphere object was implemented in single-shot by using the grid pattern-based structured-light vision system. By comparing this with the reconstruction of the sphere from the commercial RGB-D camera, the proposed method was verified. Following the development of deep learning techniques, the detection and correspondence retrieval techniques of the grid pattern could achieve several innovations in the future.