Development of Quick Three-Dimensional Shape Measurement Projection Mapping System Using a Whole-Space Tabulation Method

Abstract

A grating projection method is often used as a highly accurate 3D shape measurement method. A real-time 3D shape measurement system can also be applied to measure a wide and smooth curved surface, such as in sheet metal processing. In this case, operators take much effort to recognize the positions of some problem areas on an object from a measured result displayed on a monitor. This study develops a projection mapping system projecting an evaluation image, such as height, displacement, gradient, curvature factor, and area of defect, onto an object. These evaluation results are obtained from the measured 3D shape. The evaluation image should be deformed according to the 3D shape of the object because the camera and projector positions are different. Therefore, this study proposes a method to quickly produce a deformed evaluation image using a whole-space tabulation method. A coordinate transform table allowed the conversion of a camera pixel coordinate into a projector pixel coordinate by using reference planes to apply deformation to the evaluation image according to the measured 3D shape. The quick coordinate transformation from a camera pixel coordinate into a projector pixel coordinate was realized using the coordinate transform table. This is a key idea of this study. It was confirmed that the coordinate transformation from the camera pixel coordinate to the projector coordinate could be performed in 4.5 ms using the coordinate transform table. In addition, 3D shape measurement projection mapping was applied to a curved sheet metal with small deformation, and the deformation part was clearly shown by projecting the height distribution. The architecture and the experimental results are shown herein.

1. Introduction

Three-dimensional shape measurement systems that use contactless methods are necessary in many industrial fields [1]. A grating projection method is often used as a high accuracy, high speed, and real-time 3D shape measurement method [2,3,4]. It can also be effectively applied to a wide and smooth curved surface object, such as a deformed sheet, a press formed object, and a space deployable structure.

The evaluated distribution such as height distribution, deformation distribution, and an image of contour lines can be obtained from 3D shape measurement results. They are generally displayed on a monitor as a 2D distribution or a 3D view. However, it is difficult for an operator to recognize the correspondence between a point on the monitor and the corresponding point on the actual measured object. Therefore, there is a demand for technology that supports the visual inspection of flat objects such as sheet metal. Projection mapping is an effective method used to show the evaluated distribution. An operator can easily recognize the point to be noticed if the evaluated distribution is projected onto the measured object. The object is wider, and the effect becomes greater.

A real-time projection mapping system with a shape measurement was developed in 1999 [5]. Here, height distribution was measured using a phase-shifted grating projection method with an infrared light. The height distribution was analyzed with a real-time phase analysis board that employed a field-programmable gate array [6]. The measured height distribution was projected onto the actual measured object by a digital light processing (DLP) projector with visible lights. In this system, each camera pixel and the corresponding mapping projector pixel must be adjusted to the same optical axis using a half mirror. This is one of the difficulties in producing the device.

Figure 1 shows an overview of our projection mapping system. In the first phase, a grating pattern is projected onto an object with a measurement projector to measure the 3D shape, as shown in Figure 1a. In the second phase, the measured results and the evaluation values are projected onto an object from a projector different with a measurement projector, as shown in Figure 1b. In this configuration, the evaluation image should be deformed according to the 3D shape of the object because the camera and projector positions are different. Many studies of projection mapping combined with the 3D shape measurement have been performed in the fields of art, augmented reality, and 3D displays [7,8,9,10,11,12]. In conventional studies, the deformed projected image is produced with a coordinate transform calculation.

In contrast, in this study, a method used to produce a deformed evaluation image using a coordinate transform table to convert a camera pixel coordinate into a projector pixel coordinate is proposed. This is a key idea of this study. The computing time necessary to produce the deformed evaluation image can be reduced with the tabulation. The table can produced at the same time when a phase and 3D coordinate table can be produced based on a whole-space tabulation method (WSTM) [13,14]. The WSTM makes it possible to produce a real-time 3D shape measurement system. The authors also developed a light-source-stepping method (LSSM) using linear light-emitting diode (LED) arrays [13,14,15,16]. A real-time and wide-range 3D shape measurement unit for a large object using the WSTM in combination with the LSSM was developed [16]. This method has an advantage of stability because of its lack of mechanical moving parts for phase-shifting as compared to several individualistic fringe projection methods that have been proposed by other researchers [17,18,19].

An image deformed according to a 3D shape projected by a mapping projector can be quickly generated using this table. We developed a quick 3D shape measurement projection mapping system (hereinafter referred to as the 3DM-PM system) using the coordinate transform table. This system can project the evaluation image, such as height, displacement, gradient, curvature factor, and area of defect, onto an object immediately after measuring the 3D shape of the object. An experiment to measure the 3D shape of the step sample using the prototype 3DM-PM device and to project the height distribution was performed. An experiment to confirm whether the 3DM-PM can be applied to a curved sheet metal was also performed. The architecture and the experimental results are shown herein.

2. 3D Shape Measurement Methods

2.1. Measurement Methods

Calibration was required in advance to realize 3DM-PM using the WSTM. Figure 2 shows the relationship between the point on the reference plane and the phase of the projected grating (i.e., projected grating from light sources for measurement and projected grating from a projector for mapping). Figure 2a shows the relationship between the phase of the grating projected from the measurement projector and the z coordinates at a pixel. Figure 2b shows the relationship between the phase of the grating projected from the mapping projector and the z coordinates at the same pixel.

One pixel of the camera captures the P₀, P₁, P₂ … P_N points according to the R₀, R₁, R₂ … R_N positions of the reference plane, respectively. Each θ_m0, θ_m1, θ_m2 … θ_mN phase can be determined by the projected grating of the measurement projector, while each Φ_p0, Φ_p1, Φ_p2 … Φ_pN phase can be determined by the projected grating of the mapping projector. The relationship between the 3D coordinates and the phase of the grating projected by the measurement projector is calibrated using the WSTM.

The reference plane is placed perpendicular to the z-axis. A parallel movement is possible in the z-axis direction. The grating is projected from the projector onto the reference plane.

In this way, at each pixel, the θ_mN phase of the grating is projected from the measurement projector, the Φ_pN phase of the grating is projected from the mapping projector, and the 3D coordinates are obtained at the position of each reference plane.

2.2. Coordinate Transformation

The measurement or evaluation results in this system are projected onto the object surface from a mapping projector. In this case, the projected image should be produced with deformation according to the 3D shape of the object because the position, orientation, and image size of the mapping projector and the camera are different. This study proposes a method to generate the mapping image using a coordinate transformation table in a short time. Figure 3 shows the change in the coordinates of the measured and projected images. The measurement image is obtained using the measurement projector, as shown in Figure 3a. The projection image projected from the mapping projector is produced using a coordinate transformation of the measurement image according to the projection destination shape, as shown in Figure 3b.

Therefore, the phase value of the projector can be obtained from the measurement coordinates. The coordinates (i_p, j_p) of the image for mapping can be determined from Equations (1) and (2) once the phase value of the projector is determined. The phase value of the mapping projector in the x direction after the coordinate transformation is denoted as Φ_ip, while that in the y direction is denoted by Φ_jp. The grating pitches in the x and y directions projected by the projector are Pip and Pjp, respectively.

$$i_p=\frac{P_{ip}}{2\pi }{\Phi }_{ip}$$

(1)

$$j_p=\frac{P_{jp}}{2\pi }{\Phi }_{jp}$$

(2)

The phase values of the projector are transformed into the coordinate values of the projected image based on Equations (1) and (2). A table of 3D and projected coordinates is then created. The mapping coordinates are obtained by the coordinate transformation of the z coordinate of the measurement value at each pixel. Subsequently, mapping the measurement and analysis results to the correct position on the object becomes possible. In this way, the coordinate values can be obtained at a high speed during 3DM-PM. The high-speed projection mapping can be realized using this table. Figure 4 shows the relationship between the z coordinate and the projection coordinate (i_p, j_p) in 1 px of the camera.

3. Experimental Setup and Accuracy Evaluation of the 3D Shape Measurement

3.1. Experimental Setup

Figure 5 shows the prototype of the 3DM-PM device. A light source, a grating plate, and a camera were fixed as a unit to realize the 3D shape measurement. A projector was mounted as a unit for projection mapping. Figure 5a shows a photograph of the 3DM-PM device. Figure 5b presents the configuration diagram of the 3DM-PM device. The measurement projector and the mapping projector were located 350 and 80 mm apart from the camera.

Table 1. Parameter configuration of the 3DM-PM system.

Parameter	Value
Camera pixel resolution (pixels)	552 × 320
Lens focal length (mm)	3.5
Projector pixel resolution (pixels)	1280 × 720
Grating pitch (mm)	2.0
Distance between grating and light source A (mm)	150
Distance between grating and light source B (mm)	130

shows various parameters of the 3DM-PM device at the time of measurement.

Table 2. Equipment list for the 3DM-PM system.

Equipment	Description
Computer CPU	Intel Core i5-7200U 2.50–3.10 GHz
Computer RAM	32 GB
Camera	IDS, UI-3060CP-M-GL Rev.2 (monochrome)
LED source	Philips, LXZ1-PM01 (dominant wavelength: 530 nm)
Mapping projector	Optoma ML750STS1

presents the hardware information of the 3DM-PM device.

Figure 6 displays the light-source-stepping method [13,14,15,16], where the grating patterns were projected by switching the lighting position of the light source. The prototype device used this method to realize a high-speed phase shift by controlling the lighting position of the LED device. A signal was sent from the PC to the microcomputer. The timing of the LED lighting and that of the camera capture were synchronized by a microcomputer. A high-speed phase shift was realized by electrically switching the lighted line.

The measurement depth in the z direction can be expanded projecting two types of grating with different pitches [16]. Figure 6 shows how grating with different pitches were projected using two light sources installed at different distances from the grating plate.

The pitch of a grating plate can be denoted as $p_0$. The distance between a plane z = $z_0$ and the grating plate is b. The distance between light source A and the grating plate is $a_A$. The distance between light source B and the grating plate is $a_B$. The pitches $P_A$ and $P_B$ of the projected grating at the plane z = $z_0$ are calculated based on Equation (3).

$$P_A=\frac{a_A + b}{a_A}{P}_0$$

(3)

3.2. Accuracy Evaluation of the 3D Shape Measurement

The plane object was measured using the 3D shape measurement device. Figure 7 shows a photograph and configuration diagram of the experimental setup. The 3D shape measurement device was mounted on a linear stage. The positioning accuracy of the linear stage was 0.0001 mm. The device could be moved to any position in the z direction. A liquid crystal display (LCD) was used as a reference plane. A light diffusion sheet was pasted onto the LCD surface. The grating plate of the 3D shape measurement device was placed 700 mm away from the reference plane position at z = 0 mm. The 3D shape measurement device was moved 130 times at intervals of 1.000 mm in the z direction, and calibration was performed using the WSTM. The image resolution size was set to 552 × 320 pixels.

The reference plane was regarded as a plane of the measurement object after the calibration. The plane object was measured to avoid the z positions, where the reference planes were obtained, and was measured from z = 0.500 mm to z = 120.500 mm at intervals of 10.000 mm. Figure 8 shows the cross-sectional measurement results of one pixel of the 3D measurement result of the flat plane. The average value, measurement error, and standard deviation were calculated at each position (

Table 3. Experimental results of the plane measurement.

Position z (mm)	Average (mm)	Error (mm)	Standard Deviation (mm)
0.500	0.505	0.005	0.045
10.500	10.520	0.020	0.043
20.500	20.519	0.019	0.042
30.500	30.516	0.016	0.041
40.500	40.511	0.011	0.040
50.500	50.506	0.006	0.039
60.500	60.507	0.007	0.039
70.500	70.505	0.005	0.039
80.500	80.500	0.000	0.039
90.500	90.500	0.000	0.039
100.500	100.509	0.009	0.039
110.500	110.509	0.009	0.040
120.500	120.503	0.003	0.041

). The standard deviation of the measured value was less than 0.05 mm. The measurement error was less than 0.02 mm. The measured value was not distorted by the influence of lens aberration using the WSTM.

4. Experimental Results of Projection Mapping

4.1. Height Distribution Projection Mapping

The height distribution obtained by the 3D shape measurement was projected on the measurement object as an experiment of the 3DM-PM system. Figure 9 shows the step sample with five 10 mm steps used as the measurement object. The step sample was made of five 10 mm acrylic plates and all the surfaces were painted white. Figure 10 shows a photograph and configuration diagram of the measurement environment and the 3DM-PM device. The measurement objects were installed 700 mm apart and in between the measurement camera and measurement projector. All the surfaces of the step sample were located parallel to the reference plane. The measuring device could move any distance in the z direction. It was installed on a linear stage with 0.0001 mm accuracy.

When measuring, two grating patterns with different pitches were projected to expand the measurement range for the z direction [15]. Figure 11a shows a grating image of a fine pitch grating projected onto a step sample, and Figure 11b shows a grating image of a large pitch grating projected onto a step sample. By analyzing the phase of these images, the two phase images shown in Figure 12a,b were obtained. Figure 13a is a height distribution image obtained using the WSTM based on this phase information as the 3D shape measurement result. This height distribution image was transformed into a projection image, as shown in Figure 13b using the transformation table. Figure 14 shows a photograph of the height distribution projected onto the measurement object. Based on the 3D measurement result, the range of 75–120 mm in height is depicted as colors from blue to red.

Table 4. Required times of the 3DM-PM system.

Element	Time (ms)
3D measurement	259.1
Generating mapping image (height distribution)	45.2
Details: Coordinate transformation	4.5
Interpolation of generated image	35.2
Other processing	5.5
Total time	304.3

shows the contents of the time spent by the 3DM-PM system. When the 3DM-PM of the height distribution projection was used on the step sample, the time for the 3D shape measurement was 259 ms, and the time for the generation of the mapping image of the height distribution was 45 ms. The coordinate transformation from the measurement coordinates to the projection coordinates was performed in 4.5 ms. The time to generate a projection image included the time of coordinate transformation. This system could perform quick 3D shape measurement and projection mapping in real time at 304 ms intervals.

4.2. Evaluation of the Projection Accuracy

An experiment was performed to verify the projection accuracy of the developed 3DM-PM device. Figure 15 illustrates a triangular prism prepared as the measurement object. Contour lines 20 mm apart from the hypotenuse were printed. Projection accuracy can be evaluated through the difference between the position of the projected contour lines and the position of the printed contour lines. The triangular prism was made of acrylic plate, and it had sufficient flatness. On the surface of the triangular prism, paper with printed contour lines was evenly attached. The projection images with contours of 20 mm intervals were mapped to verify the projection position accuracy. The size of one pixel of the mapping projector for the x-direction at the object position was about 0.70 mm. The thickness of the generated contour lines was more than 0.70 mm.

Figure 16 shows the arrangement of the experimental equipment. The triangular prism to be measured was placed with its hypotenuse parallel to the x-axis. In addition, the state of the mapped triangular prism was photographed with an industrial monochrome camera installed perpendicular to the 208 × 100 mm square surface of the triangular prism.

Figure 17a shows the height distribution image obtained by the measurement. This measurement image was coordinate-transformed into a projection image using a transformation table (Figure 4). Figure 17b shows the contour lines projected onto the measurement object. The contour lines were generated at intervals of 20 mm in the z direction based on the measured height distribution. Figure 18 shows a photograph of the triangular prism onto whose surface the contour lines were projected. Figure 19 shows a photograph taken from the front face of the triangular prism. Figure 19a is an image before projecting the contour lines. Figure 19b depicts an image after the projection of the contour lines. Figure 19b shows that the positions of the contour lines drawn in advance and the projected contour lines coincided, confirming that the image was projected to the correct position. This was sufficient as a projection for supporting visual inspection.

4.3. Application to Height Measurement of a Curved Sheet Metal

An application of the proposed method to the height measurement of a curved sheet metal was performed. The specimen was an aluminum sheet with a size of 500 × 500 × 2 mm. Figure 20 shows photographs of the measured specimen. The specimen was deformed by shot peen forming process [20,21]. Shot peening was applied to the obverse side of the specimen with vertical scanning and the back side was applied with horizontal scanning to form a saddle shape. The specimen was slightly curved for the horizontal direction, as shown in Figure 20a, and also slightly deformed for the vertical direction on the shot areas due to the horizontal scanning, as shown in Figure 20b. However, the deformations for the vertical direction were too small to recognize the deformation with the visual contact. Several markers were attached on the obverse side to determine the shot areas from the obverse side, as shown in Figure 20c.

Figure 21 shows photograph of the developed 3D measurement projection mapping system and the specimen. The measured height distribution was projected onto the specimen. It can be recognized from the projected height distribution that the vertical center area (red area) of the specimen was bulged, and the marked horizontal areas were slightly concaved.

5. Conclusions

This study developed a method for measuring a 3D shape and mapping the measurement results. The developed method uses a table created in advance using a WSTM to transform the measurement coordinates into the projection coordinates for the 3DM-PM. A projection image can quickly be generated because a complex coordinate calculation is not required at the time of measurement.

First, a device using the proposed method was prototyped. The 3D shape measurement was performed on a plane object using the prototyped device. The standard deviation of the measured value was less than 0.05 mm, and the measurement error was less than 0.02 mm. Next, the 3DM-PM of the height distribution was performed on the step sample, and the image colored according to the height was projected. At this time, the coordinate transformation of a camera pixel coordinate into a projector pixel coordinate was performed using the coordinate transformation table. It was confirmed that the coordinate transformation from the camera pixel coordinate to the projector coordinate was performed in 4.5 ms. Finally, the contour lines generated based on the measurement values were projected onto the triangular prism, onto which the contour lines were drawn in advance, to confirm that the prototype 3DM-PM device could be projected to the correct position. Consequently, the match of the contour line positions was confirmed. The experiment showed that the 3DM-PM system could be projected to the correct position. An experiment to confirm whether the 3DM-PM device using the proposed method could be applied to a curved sheet metal was performed. It was confirmed that the deformed areas could be easily recognized by projecting the height distribution onto the specimen using the prototype device. We will apply this method to larger objects as a future work.