T-Spline Based Unifying Registration Procedure for Free-Form Surface Workpieces in Intelligent CMM

Abstract

With the development of the modern manufacturing industry, the free-form surface is widely used in various fields, and the automatic detection of a free-form surface is an important function of future intelligent three-coordinate measuring machines (CMMs). To improve the intelligence of CMMs, a new visual system is designed based on the characteristics of CMMs. A unified model of the free-form surface is proposed based on T-splines. A discretization method of the T-spline surface formula model is proposed. Under this discretization, the position and orientation of the workpiece would be recognized by point cloud registration. A high accuracy evaluation method is proposed between the measured point cloud and the T-spline surface formula. The experimental results demonstrate that the proposed method has the potential to realize the automatic detection of different free-form surfaces and improve the intelligence of CMMs.

1. Introduction

The coordinate measuring machine (CMM) is a powerful tool in the modern manufacturing industry [1,2]. With the development of modern society, CMMs must be increasingly intelligent, and the next generation of CMMs is the intelligent CMM [1]. Intelligent CMMs should be able to recognize position and orientation, plan safe paths, and evaluate test results. Free-form surface parts that have pleasing appearance and texture have been increasingly used with the development of modern manufacturing. Due to the complex shape and expression of free-form surface, the intelligent detection is still difficult, including finding model rapidly and calculating form error. The study on the intelligent detection free-form surface is significantly important to improve the intelligence of CMMs.

A number of previous papers have discussed intelligent CMMs, which are mainly focused on position recognition for conventional parts and path planning. Only a few studies have focused on the position and orientation recognition of free-form surface workpieces. In relevant work on intelligent CMMs [1,2,3], a single camera is used to identify the position and orientation of the parts and a path planning method is proposed, but not for free-form surface parts [4]. Various methods, such as the ant colony algorithm and the genetic algorithm, have been proposed for path planning in the literature [5,6,7,8,9,10]. A method of path planning for free-form surface scanning has been presented [11]. An intelligent interface for computer aided design (CAD)-CMM [3] and an off-line interactive CMM [12] have been proposed. Free-form surfaces have been detected and the error has been calculated based on non-uniform rational B-spline curve (NURBS) and the CMM [2,13,14,15].

The registration of measured data and the surface model and the recognition of free-form surface parts have similar principles. The registration method can be divided into two categories: one is the matching of the scanned point cloud and the designed surface point cloud [16,17,18,19,20,21,22]; the other is associating the mathematical formula of the designed surface and the scanned point cloud [2,13,14,15,23]. Where “designed surface point cloud” is the point cloud generated by the design surface model, it is usually generated with the help of the software as Unigraphics NX or NURBS sampling. “mathematical formula of the designed surface” is NURBS formula of surface, which can be obtained from CAD files. The point cloud registration is realized based either on local characteristics [16,17] or global features [19]. A greedy search algorithm has been used for point cloud matching [13]. The fruit fly optimization algorithm has also been used [20]. The iterative closest point (ICP) algorithm and its improved counterpart have also been used [21,22]. The accuracy of the first matching class is relatively low. If high accuracy matching is realized, the more intensive point cloud must be extracted. This intensive point cloud will occupy a large amount of storage space and is not conducive to the system software model database. The second class can achieve high accuracy [13,23], but the current literature has generally focused on the NURBS surface and detection point cloud. When a complex free-form surface cannot be fully expressed by NURBS, it is impossible to achieve a unified registration and evaluation. A sub-evaluation must be performed using human intervention.

The T-spline has many advantages not afforded by NURBS, such as seamless splicing, a unified formula, allow partial modification and fewer control points. The control points of T-spline no need to be aligned in the entire row of rows as NURBS [24,25,26]. Numerous studies [24,25,26,27,28,29] have focused on T-spline modeling technology. The concept, rules and seamless splicing method of T-spline modeling have been proposed [24]. A local interpolation and modification method for the T-spline surface has also been proposed [25]. A T-spline surface obtained using the Z-map fitting method has been proposed [27]. A previous study noted that a T-spline surface can be converted into NURBS patches, and T-spline surfaces can then be spliced into an overall surface [28]. The T-spline surface has also been fit using periodic parameterization and automatic interaction [26]. A new algorithm was proposed to transform a triangular mesh into a T-spline surface [29]. An adaptive T-spline surface fitting method based on quad-tree segmentation was proposed [30]. Another study transformed an irregular quadrilateral mesh into a T-spline surface and proposed a mesh generation method around a single point [31,32]. Another study proposed a method for fitting the T-spline surface with curvature guidance [33]. The T-spline can be used to fit large datasets [34]. A previous study performed isogeometric analysis on an arbitrary T-spline [35,36]. T-spline modeling technology is still being further developed, and an arbitrary order of T-spline has been analyzed in the parameter space [37].

To develop a more intelligent CMM, the current study designs a new visual inspection system. A new visual system is designed based on the characteristics of CMMs. A unifying accurate registration procedure using T-spline representation is proposed for various free-form surfaces. The practical application of the procedure is to automatically detect the positions and orientations of free-form workpieces in intelligent CMM. A discretization method of the T-spline surface formula model is proposed, where the maximum distance between adjacent points can be set. Position and orientation can be determined by the registration of the discretization of the T-spline surface formula model and the extracted point cloud, which reduces the storage space. Using the point cloud registration results as the initial solution, an overall high accuracy registration of the measured point cloud and the T-spline surface is proposed. This approach reduces human intervention and achieves free-form surface workpiece intelligent detection and error evaluation.

2. System Architecture

An intelligent CMM should be able to recognize the workpiece’s position and orientation, plan the detection path, and evaluate the error automatically. This paper focuses on the automatic recognition of the position and orientation of free-form surface parts, accurate registration and error evaluation.

2.1. Overall System Procedure

In Figure 1, the workpiece unified models refer to the fact that the free-form surface parts are all represented by T-spline surfaces. A number of T-spline surface artifacts can form a model database, which is the basis of rapid recognition and accurate registration, shown in the green and red dashed boxes. In the rapid recognition box, the extracted point cloud from the actual measured workpiece is acquired by the visual system. The discrete model is the point cloud generated by the T-spline surface formulas. In the rapid recognition procedure, the workpiece and the pose are recognized by the registration between the extracted and generated point cloud, whose result is set as the initial position of the high accuracy registration. The CMM performs accurate registration between the T-spline surface formulas and the measured point cloud. Finally, based on the high accuracy registration, the distance between the measured point and workpiece surface model is calculated as the error of the T-spline generated model and the measured points.

2.2. The New Visual System

There are two methods for extracting the surface point cloud: contact and non-contact. This paper adopts a non-contact visual detection method to extract the point cloud rapidly. The probe of the CMM can rotate around two perpendicular axes. The CMM can detect points on a discrete surface larger than a hemisphere. Based on the characteristics of the CMM, the visual inspection system is designed as shown in Figure 2. A laser is fixed on one side of the bodies of rotation such that the visual system can determine each position in the CMM measurement space. The system can also be used to detect the same workpiece from different positions and angles, which provides relatively large flexibility.

The conversion relation between the CMM coordinate system and the vision coordinate is obtained by calibration [38]. Finally, the coordinates of the extracted point cloud are converted into the CMM coordinate system.

3. Technology of the System

In 2003, Sederberg et al. extended the tensor product surface and proposed T-splines and T-NURCCs [24]. Compared with NURBS, a T-spline provides a uniform expression for the entire spline surface, with seamless splicing. The following three parts correspond to the unified workpiece model, discrete model, high accuracy registration and error evaluation in Figure 1.

3.1. Establishment of a Unified Model

Based on the advantages of the T-spline, this paper establishes a unified T-spline surface model for the entire measured free-form surface workpiece. The T-spline surface is a new representation method for a parameter surface that allows T-junctions. The T-mesh is formed by all of the control points. A T-spline surface is defined by the following formula [24,25].

$$S(s,t)=\frac{{\displaystyle \sum _{i=1}^m{P}_i{w}_i{B}_i(s,t)}}{{\displaystyle \sum _{i=1}^m{w}_i{B}_i(s,t)}}$$

(1)

where P_i and w_i are the control points and weights, respectively, m is the number of control points. (s,t) is the parameter field coordinates. The corresponding T-spline blending function is

$$B_i(s,t)=N[s_i](s)N[t_i](t)$$

(2)

where N[s_i](s) and N[t_i](t) are the cubic B-spline basis functions associated with the knot vector, and they have the similar form.

$$s_i=[s_{i0},s_{i1},s_{i2},s_{i3},s_{i4}],t_i=[t_{i0},t_{i1},t_{i2},t_{i3},t_{i4}]$$

$$\begin{array}{l}N[s_i](s)=N_{0,3}(s)=\frac{s-s_{i0}}{s_{i3}-s_{i0}}{N}_{0,2}(s)+\frac{s_{i4}-s}{s_{i4}-s_{i1}}{N}_{1,2}(s)\\ =\{\begin{array}{l}\frac{{\left(s-s_{i0}\right)}^3}{\left(s_{i3}-s_{i0}\right)\left(s_{i2}-s_{i0}\right)\left(s_{i1}-s_{i0}\right)},s_{i0}\le s<s_{i1}\\ \frac{{\left(s-s_{i0}\right)}^2\left(s_{i2}-s\right)}{\left(s_{i2}-s_{i1}\right)\left(s_{i3}-s_{i0}\right)\left(s_{i2}-s_{i0}\right)}+\frac{\left(s-s_{i0}\right)\left(s_{i3}-s\right)\left(s-s_{i1}\right)}{\left(s_{i2}-s_{i1}\right)\left(s_{i3}-s_{i1}\right)\left(s_{i3}-s_{i0}\right)}\\ +\frac{\left(s_{i4}-s\right){\left(s-s_{i1}\right)}^2}{\left(s_{i2}-s_{i1}\right)\left(s_{i4}-s_{i1}\right)\left(s_{i3}-s_{i1}\right)},s_{i1}\le s<s_{i2}\\ \frac{\left(s-s_{i0}\right){\left(s_{i3}-s\right)}^2}{\left(s_{i3}-s_{i2}\right)\left(s_{i3}-s_{i1}\right)\left(s_{i3}-s_{i0}\right)}+\frac{\left(s-s_{i1}\right)\left(s_{i3}-s\right)\left(s_{i4}-s\right)}{\left(s_{i3}-s_{i2}\right)\left(s_{i4}-s_{i1}\right)\left(s_{i3}-s_{i1}\right)}\\ +\frac{{\left(s_{i4}-s\right)}^2\left(s-s_{i2}\right)}{\left(s_{i3}-s_{i2}\right)\left(s_{i4}-s_{i2}\right)\left(s_{i4}-s_{i1}\right)},s_{i2}\le s<s_{i3}\\ \frac{{\left(s_{i4}-s\right)}^3}{\left(s_{i4}-s_{i3}\right)\left(s_{i4}-s_{i2}\right)\left(s_{i4}-s_{i1}\right)},s_{i3}\le s<s_{i4}\\ 0,s<s_{i0}ors>s_{i4}\end{array}\end{array}$$

(3)

Based on the literature [30,33], the T-spline surface fitting is divided into the steps shown in Figure 3.

To unify the representation of all free-form surface workpieces, the T-spline surface model database is established by fitting the T-spline surface formula for all of the free-form surfaces based on Figure 3. First, the triangular mesh is extracted from the designed model. There are many methods for parameterizing triangular meshes [39,40,41]. Conformational parameterization [42] is used in this paper due to its speed and effectiveness. The mean value parameterization method [43] is adopted after the conformal parameterization to facilitate the construction of the pre-image of the T-mesh. The parameter domain is split by the quad-tree. The result of the quad-tree is used to fit the T-spline surface as the pre-image of the T-mesh. Finally, the accuracy of the surface fitting is determined based on the needs of different applications. In this manner, a unified model is established.

The basic principle of T-spline surface fitting is to minimize the distance between the triangular mesh and the surface S, and the f equation can be obtained

$$E_{fit}({\mathrm{P}}_1,{\mathrm{P}}_2,\cdots ,{\mathrm{P}}_m)={\displaystyle \sum _{i=1}^n{\left(S(s_i,t_i)-{\mathrm{q}}_i\right)}^2}$$

(4)

where q_i is the vertex of the triangular mesh and (s_i, t_i) are the parameter domain coordinates corresponding to the vertex of the triangular mesh. S_ss(s_i, t_i), S_st(s_i, t_i) and S_tt(s_i, t_i) are the second order derivatives of S(s_i, t_i).The following energy function is used to smooth the surface

$$E_{fair}({\mathrm{P}}_1,{\mathrm{P}}_2,\cdots ,{\mathrm{P}}_{\mathrm{m}})={\displaystyle \sum _{i=1}^n\left(S_{ss}^2(s_i,t_i)+S_{st}^2(s_i,t_i)+S_{tt}^2(s_i,t_i)\right)}$$

(5)

The overall objective function is

$$E({\mathrm{P}}_1,{\mathrm{P}}_2,\cdots ,{\mathrm{P}}_{\mathrm{m}})=E_{fit}+\sigma E_{fair}$$

(6)

where the coefficient σ is generally between 0.01 and 0.1. The control point and knot vector of the surface are obtained by solving the above objective function. This point and vector form the complete T-spline surface expression formula. The T-spline surface model of a girl’s face is shown in Figure 4a as the original girl’s face model. The blue dot is the parameterized vertex distribution of the girl’s face triangular mesh, and the red block diagram is the result of the quad-tree division in (b). The blue points in (c) are control points, which form the T-mesh.

3.2. Discretization of the Model

The position and orientation recognition of the free-form surface workpiece is an important function provided by the intelligent CMM. The specific requirement is to be able to quickly recognize the different free surface workpieces in the detection space, including their position and orientation. The unified model established in Section 3.1 is discretized to reduce the storage space required. A point cloud is generated based on the T-spline surface formula. The recognition is realized by registration of the generated point cloud and extracted point cloud. The center of gravity coordinates and eigenvector direction are aligned via principle component analysis (PCA). Point cloud registration is achieved using the ICP algorithm.

This paper uses the T-spline surface formula to generate the point cloud whose adjacent distance is not larger than a specified value. The conversion between the two coordinate systems is achieved via point cloud registration. After the registration, if the maximum distance (D_max) between the corresponding points is less than a certain value, the measured workpiece and the surface model are considered to be the same.

The following formula provides the mapping of surface points to control points

$$E_{map}({\mathrm{p}}_i)=S(s_i,t_i)-{\mathrm{p}}_i,i=1,2,3\cdots$$

(7)

where (s_i,t_i) are the coordinates of the pre-image point in the parameter field to the corresponding control point and p_i is the surface mapping point. p_i is solved for using the least squares method to minimize the energy function.

As shown in Figure 5, a knot is inserted based on the quad-tree split. When the distance between the surface mapping points, which correspond to knots P₁’P₂’P₃’P₄’ in the parameter domain, is greater than a certain value, the parameter space is split and knots P₅’P₆’P₇’P₈’P₉’ are inserted. Then, the corresponding surface mapping point and adjacent distance are calculated. When the adjacent distance is less than a certain value, the quad-tree division is stopped, and finally, all of the mapping points are saved as the final matching point cloud.

The adjacent distance of the generated points is less a certain value, which ensures a certain degree of accuracy and saves storage space by releasing dynamic data. Compared with other methods based on software, such as Unigraphics NX (Siemens PLM software, Plano, TX, USA) generating the point cloud, this method can set the D_max to the nearest point, the overall generation, without human intervention, reducing the software storage space required for the entire system.

3.3. Overall Registration and Error Evaluation

This section is the last step of the intelligent CMM. The rotation translation matrix RT between two different coordinate systems can be obtained as detailed in Section 3.2. The measured surface, position and orientation are obtained by matching the point cloud. Then, a safe path is planned, and the workpiece is detected. The error is calculated through the registration of the point cloud collected by the CMM and the T-spline surface formula as follows

$$E_{dis}(s_i,t_i)={\displaystyle \sum _{i=1}^{nn}{\left(S(s_i,t_i)-{\mathrm{q}}_i^{\prime }\right)}^2}$$

(8)

where nn is the number of points collected by the CMM.

$$\left(\begin{array}{c}{x}_{\mathrm{d}}\\ y_{\mathrm{d}}\\ z_{\mathrm{d}}\end{array}\right)=R\left(\begin{array}{c}{x}_{\mathrm{w}}\\ y_{\mathrm{w}}\\ z_{\mathrm{w}}\end{array}\right)+T$$

(9)

$$\mathrm{where},R=\left(\begin{array}{ccc}\cos \beta \cos \gamma & \sin \alpha \sin \beta \cos \gamma -\cos \alpha \sin \gamma & \cos \alpha \sin \beta \cos \gamma +\sin \alpha \sin \gamma \\ \cos \beta \sin \gamma & \sin \alpha \sin \beta \sin \gamma +\cos \alpha \cos \gamma & \cos \alpha \sin \beta \sin \gamma -\sin \alpha \cos \gamma \\ -\sin \beta & \sin \alpha \cos \beta & \cos \alpha \cos \beta \end{array}\right),T=\left(\begin{array}{c}{t}_x\\ t_y\\ t_z\end{array}\right).$$

The rotation transformation between the designed free-from surface workpiece coordinate system o_d-x_dy_dz_d and the CMM coordinate system o_w-x_wy_wz_w is executed as follows: first, the rotation angle around the x_d axis is α, the rotation angle around the y_d axis is β, and the rotation angle around the z_d axis is γ.

The specific steps of accurate registration are as follows:

(1)

RT solved through the point cloud registration is substituted as the initial solution.

(2)

Based on the following formula to optimize s_i, t_i, the sum of the distance of the collected points to the T-spline surface, is calculated according to Formula (8) and (9).

(3)

Cyclically optimize RT and perform step (2) until a certain level of accuracy is achieved.

(4)

The distance data of each point corresponding to the formula surface model are retained as the result of the error evaluation through Levenberg-Marquardt method.

When the amount of data is relatively large, the point cloud must be simplified according to the relevant literature [44] to save time and in the case of error tolerance. Then, the rotation translation matrix RT is solved by optimization, and finally, the distances of all of the collected points to the T-spline surface are calculated.

4. Experiments

4.1. Workpiece Recognition

The human knee bone model, Chinese lion head model and girl’s face model are used in the following experiments, as shown in Figure 6. The physical equipment is designed as shown in Figure 6, where (a) is the overall physical Figure and (b) is the Figure partially enlarged. The Ximea MU9PM-MH camera (Ximea, Münster, Germany) used in our experiment has a resolution of 2592 × 1944 and a pixel size of 2.2 µm × 2.2 µm. The focal length of the lens is 6 mm. The CMM adopted in this paper is Global classic SR 07. 10. 07 CMM (Hexagon, Qingdao, China).The maximum error of CMM is less 3 μm. The error of the visual inspection system includes a camera and a laser is less 1 mm. The initial state of the system is camera has a certain distance to platform, and parts are photographed and detected roughly the size of the outline. And then according to the part of the outline size CMM move from a side of the workpiece to the other side in a certain interval. The interval size is generally 2–5 mm, and then if the workpiece is large enough to properly enlarge the scanning interval. In this paper, through the point cloud registration the workpiece and rough position and orientation are recognized. The application condition of the system is that the point cloud collected by the visual system is enough to represent the general shape of the workpiece. It is difficult to achieve accurate identification if the interval size is too large or only part of the workpiece is collected.

Figure 7a and Figure 8a are the human knee bone model and Chinese lion head model, respectively. Figure 7b and Figure 8b show the results of the quad-tree division of the parameter field, where the red grid corresponds to the pre-image of the T-mesh and the point cloud in blue corresponds to the vertex of the triangular mesh. The T-meshes obtained by fitting the T-spline surface are shown in Figure 7c and Figure 8c. The point in light blue is the control point of the T-spline surface. Part of the control point is in the shape, which results in an intermittent display. This is a reasonable distribution of control points. Figure 9a and Figure 10a show the point clouds generated by the T-spline surface formula. Figure 9b and Figure 10b show the partial magnification. Here, the adjacent distance to the nearest point is no more than 2 mm, as this department is only involved in the workpiece position recognition and initial registration, within the error no more than 2 mm can meet the actual needs, it can also be specified as some other value. This generated point cloud and that extracted by the visual system are then registered. Figure 11 and Figure 12 show the registration results of the point cloud collected by the visual system and the point cloud generated by the T-spline surface formula.

In this paper, the free-form surface workpiece, position and orientation are recognized via point cloud registration. In the experiment, the point clouds of the knee bone model, Chinese lion head model and girl’s face model are collected by the visual system. Then, the collected point cloud and generated point cloud are registered one by one. The results of the registration are shown in

Table 1. Recognition data (D_max [mm]).

	Chinese Lion	Girl’s Face	Knee Bone
Chinese lion	0.856	25.413	15.791
Girl’s face	25.361	0.985	15.145
Knee bone	14.832	14.107	0.913

. When the D_max between the corresponding points after the registration is greater than a certain threshold, it is not considered to be the same free-form surface workpiece. Table 1 shows that for the same workpiece, the D_max is less than 1 mm. The registration results between the different workpieces are considerably larger than this threshold.

4.2. Error Evaluation

Certain points covering the majority of the characteristics of the knee bone model and Chinese lion head model are collected by the CMM. The registration and error evaluation are performedbased on Section 3.3. The results are shown in Figure 13a,b, The curvature is relatively large and the error is relatively large, which conforms to the conventionalmachining error distribution.

Matching simulation experiments are also performed to further verify the effect of error evaluation, as shown in Figure 14. Certain vertexes (shown in Figure 14a,d) of the triangular mesh of the knee bone model and Chinese lion head model are extracted after rotation and translation and are used to be matched as the extracted point cloud. These vertexes are used to simulate the point cloud collected by the CMM, as shown in Figure 14b,e. The results of accurate registration are shown in Figure 14c,f.

The accurate registration results are shown in

Table 2. Matching simulation results of knee bone model and Chinese lion model.

RT	α (rad)	β (rad)	γ (rad)	tx (mm)	ty (mm)	tz (mm)
Solved RT (lion)	3.1413	0.7853	0.7856	50.026	40.021	29.975
Precise RT (lion)	3.1416	0.7854	0.7854	50	40	30
Error of RT	0.0003	0.0001	0.0002	0.026	0.021	0.025
Solved RT (knee)	1.5710	0.5236	1.0471	60.022	69.972	79.983
Precise RT (knee)	1.5708	0.5236	1.0472	60	70	80
Error of RT	0.0002	0.0	0.0001	0.022	0.028	0.017

The meaning of each parameter in Table 2 is given in Equation (9).

. There is only a slight error between the solved rotation translation matrix RT and the ideal value, mainly because there is some degree of error between the T-spline surface and triangular mesh. Therefore, the matching error and will be further reduced as the T-spline surface fitting error decreases, as shown in Figure 15a. In Figure 15a, the horizontal axis is the error threshold of the T-spline surface fitting and the vertical axis represents the average distance of the matching point cloud to the T-spline surface. Figure 15a illustrates that the average distance is also reduced with decreases in the T-spline surface fitting error. Figure 15b the horizontal axis is the error threshold of the T-spline surface fitting and the vertical axis shows the standard deviation of the distance of the matching point cloud to the T-spline surface, which shows small standard deviation. As the T-spline fitting error increases, the standard deviation also increases slightly. It can be seen from Figure 15 the proposed method has good performance and can realize high accuracy. The average distance is solved as follows:

$$\mathrm{Er}=\frac{E_{dis}(s_i,t_i)}{nn}=\frac{{\displaystyle \sum _{i=1}^{nn}\sqrt{{\left(S(s_i,t_i)-q_i^{\prime }\right)}^2}}}{nn}$$

(10)

5. Summary

This study designs a new visual inspection system to improve the intelligence of the CMM for free-form surface workpiece detection. The new visual inspection system can extract more comprehensive 3D feature information for the free-form surface workpiece and has high flexibility. A unified model for all free-form surfaces is proposed based on T-splines. A discretization method of the T-spline surface formula model is proposed, where the maximum distance between adjacent points can be set. Position and orientation can be recognized by the registration of the discretized point cloud and extracted point cloud, which reduces the storage space. The overall high accuracy registration of the measured point cloud and the T-spline surface is proposed based on the point cloud registration results as the initial solution. This registration process reduces the amount of human intervention required, realizes the free surface workpiece intelligent detection and error evaluation and has high accuracy.