A Model Development Approach Based on Point Cloud Reconstruction and Mapping Texture Enhancement

Abstract

To address the challenge of rapid geometric model development in the digital twin industry, this paper presents a comprehensive pipeline for constructing 3D models from images using monocular vision imaging principles. Firstly, a structure-from-motion (SFM) algorithm generates a 3D point cloud from photographs. The feature detection methods scale-invariant feature transform (SIFT), speeded-up robust features (SURF), and KAZE are compared across six datasets, with SIFT proving the most effective (matching rate higher than 0.12). Using K-nearest-neighbor matching and random sample consensus (RANSAC), refined feature point matching and 3D spatial representation are achieved via antipodal geometry. Then, the Poisson surface reconstruction algorithm converts the point cloud into a mesh model. Additionally, texture images are enhanced by leveraging a visual geometry group (VGG) network-based deep learning approach. Content images from a dataset provide geometric contours via higher-level VGG layers, while textures from style images are extracted using the lower-level layers. These are fused to create texture-transferred images, where the image quality assessment (IQA) metrics SSIM and PSNR are used to evaluate texture-enhanced images. Finally, texture mapping integrates the enhanced textures with the mesh model, improving the scene representation with enhanced texture. The method presented in this paper surpassed a LiDAR-based reconstruction approach by $20\%$ in terms of point cloud density and number of model facets, while the hardware cost was only $1\%$ of that associated with LiDAR.

1. Introduction

A digital twin is an accurate virtual replica of a physical asset or system, capable of simulating and predicting various scenarios, encompassing the geometric structure, behavior, and performance of the entity [1]. The rapid advancement of the digital twin concept has led to its application in diverse industrial fields, such as manufacturing, construction, and energy. A critical requirement for digital twins is accurate real-time representation of the target entity through high-quality 3D models. However, the current geometrical model development approaches exhibit the following limitations: the modeling of specialized equipment is expensive, the modeling process is complex, the costs associated with modeling are high [2], and the accuracy of the models still falls short of the required standards [3], creating significant barriers to widespread adoption. Developing a cost-effective and fast modeling method is essential for the progress of the digital twin industry, facilitating broader application and yielding substantial industrial benefits.

To achieve a model development method that is low-cost and efficient for rapid development in digital twin scenarios, while accurately preserving the geometric feature information and texture details of the reconstructed objects and improving the display of textures on the model surface, a pipeline of methods is proposed in this paper that combines SFM-based point cloud data acquisition, surface reconstruction, style-transfer-based texture enhancement, and texture mapping. In Section 3, the methodology is explained. Experimental results and discussions are given in Section 4. Finally, an analysis of the results and future improvements are provided in Section 5.

2. Related Work

2.1. There-Dimensional Reconstruction

Current methodologies for geometric model development rely on a variety of costly hardware to acquire point cloud data for the purpose of the three-dimensional reconstruction of objects. Point cloud data are a widely used 3D representation, typically containing six dimensions, 3D coordinates $(x,y,z)$, and RGB values or other information for each point, which represent geometric relationships within complex 3D objects. The main point cloud acquisition methods available are laser-scanner-based, time-of-flight (ToF)-camera-based, structured-light-camera-based, stereo-vision-camera-based, and photogrammetry methods, whose equipment is shown in

Table 1. Comparison of point cloud data acquisition equipment ¹.

Hardware	Cost (GBP)	Accuracy (mm)	Measurement Range (m)	Outdoor Work
Lidar scanner	600+	1–3 [4]	200+	Unaffected
Structured light camera	200–4000	0.01–0.32 [5]	0.3–10 [5]	Highly affected
ToF camera	400–30,000	0.5–2.2 [6]	0.5–6.0 [6]	Minimally affected
Stereo vision camera	200+	0.05–1 [7]	10–100 [7]	Unaffected
Photogrammetry methods	N/A	1–10	10–100	Unaffected

¹ The data presented in the table without references are derived from the operational experiences of the authors.

.

LiDAR systems perceive depth information by emitting laser pulses and measuring their round-trip time, while simultaneously recording the angle of the reflected light, which relies on the speed of light and precise temporal measurements to generate high-precision three-dimensional point clouds. Pose graph clustering [8] and NeRF [9] were introduced to precisely map large-scale environments. The LiDAR-based method is ideal for large-scale reconstruction due to its high efficiency, and independence from ambient light. However, it is costly and unable to reconstruct surface color and texture, and it struggles with geometric interference and environmental occlusion.

Depth cameras are categorized into ToF cameras, structured light cameras, and stereo vision cameras. ToF cameras use an active emitter to project light onto a target and an optical sensor to collect the reflected light. The 3D information is obtained by calculating the time delay between the emission and collection of the signal. Unlike structured light and binocular vision technologies, the ToF-camera-based method does not require triangulation to compute 3D information [10]. The compact hardware system of ToF cameras facilitates their deployment in the execution of scanning tasks. Structured light camera technology employs optical encoding methods, such as projecting laser stripes, Gray codes, or sinusoidal patterns onto the object’s surface. Other patterns include point maps [11], line maps [12], and crossed line maps [13]. However, structured light cameras using optical coding methods typically employ statistical rather than precise mathematical models, limiting their accuracy to within a few millimeters. With a stereo vision camera, depth information is obtained by determining the angle between the projected rays and their baselines. To achieve robust matching of the two camera views, various stereo-matching methods have been proposed [14,15,16]. The measurement accuracy of stereo vision is comparable to structured light techniques, achieving 0.05 mm to 0.1 mm, and even 0.001 mm in microscopic applications [17].

Representative algorithms for photogrammetry methods include structure-from-motion (SfM) and multi-view stereo (MVS) [18]. Current SfM algorithms can be categorized into incremental SfM, global SfM, SfM integrated with deep learning, and combined SfM with MVS reconstruction methods. Gao et al. [19] developed a multi-view 3D reconstruction system fusing SIFT and SURF features for enhanced point detection, applying scale constraints for robust matching and using RANSAC for error removal. These enhancements significantly outperformed bundle adjustment [20] in robustness and completeness. With continuous algorithmic advancements, global SFM offers enhanced reconstruction accuracy and efficiency compared to incremental SFM, due to the simultaneous consideration of all image factors [21]. The main framework of global SFM includes global similarity estimation, feature extraction and matching, and camera pose estimation [22]. To enhance algorithm efficiency, the visual bag-of-words model is used in global similarity estimation [23]. Recent algorithms such as NAPSAC, PROSAC, and P-NAPSAC have been developed to expedite robust feature point estimation. PROSAC starts by sampling from points with the highest probability based on global prior ranking [24], NAPSAC utilizes spatial coherence by sampling from points with high inner line ratios [25], and P-NAPSAC combines local and global sampling methods for improved accuracy [22].

With advancements in deep learning, key aspects of the SFM algorithm are being enhanced by using neural networks for improved efficiency. CNN-based global descriptors are used for efficient feature matching of unordered image sequences [26]. Scholars [27] from Fudan University integrated deep learning into the bundle adjustment (BA) process, enhancing the 3D reconstruction robustness for untextured or non-Lambertian surfaces.

SFM combined with MVS can create denser point cloud data, computing the depth of each pixel in a 2D image by identifying corresponding points in multiple images and generating dense 3D point clouds based on these depth maps. Some open-source algorithms integrate SfM and MVS, such as COLMAP [28], OpenMVS [29], MVE [30] and VisualSFM [31].

2.2. Surface Reconstruction

A mesh model with a continuous surface can be generated using surface reconstruction technology. Due to optical reflections, environmental noise, and other factors, point cloud data often contain outliers and noise [32], necessitating denoising before surface reconstruction. Additionally, to effectively address the issue of point cloud data discreteness and provide normal information for the reconstructed surface, surface reconstruction of the point cloud is required [33,34,35]. Wu et al. [36] proposed a skip-attention-based correspondence filtering network (SACF-Net) for point cloud filtering and point cloud registration, Lu et al. [37] achieved automatic prediction for filtering by training a classification network model based on patch samples, and Zhang et al. [38] used graph-based denoising to capture geometric details by treating the point cloud as a graph signal. Deep learning effectively filters point clouds through supervised methods like PointNet [39], which uses MLPs for feature learning, and PointNet++ [40], which captures geometric relationships using a hierarchical network. Unsupervised methods like TotalDenoising [41] use Monte Carlo convolution-based encoder–decoders to reduce noise by leveraging spatial locality and bilateral appearance, without needing ground truth samples. Liu et al. [42] used partial differential equations and IMQ radial basis functions for 3D surface reconstruction and repair of scattered data. Dai et al. [43] introduced a novel point-based representation termed Gaussian surfels, which integrates the flexible optimization of 3D Gaussian points with the surface alignment properties of surfels, enhancing the optimization stability and surface alignment. Researchers have accelerated surface solving with deep learning, exemplified by Comi et al. [44], who improved DeepSDF, a neural network representing shapes with a continuous signed distance function for high-quality shape representation and interpolation.

2.3. Style Transfer

A mesh model exhibits a uniform coloration, lacking a reproduction of the surface textures characteristic of the target object. Style transfer is a technical way of extracting the texture of a style image and combining it with core features from the content image to change the texture of the content image, which can be applied to enhance the texture of mesh models. With the increasing influence of convolutional neural networks in image processing, it has become the consensus that shallow convolutional layers extract texture and deep convolutional layers extract image features [45]. Zhang et al. [46] proposed an inversion-based style transfer method (InST) that efficiently and accurately extracted the pivotal information from images, thereby capturing and transferring the artistic style of a painting. Lin et al. [47] used a generative adversarial neural network (GAN) and added a convolutional layer to extract the features of the content image for image-to-image style conversion, as shown below. Feedforward networks allow for fast synthesis, but these methods often lack diversity and quality. To improve the diversity of the texture features learned by network models, Gatys et al. [48] introduced control over the spatial location, color information, and cross-spatial scaling, to achieve high-resolution controlled stylization. In recent research [49,50,51], network models with complex and large model architectures have been replaced by efficient encoder–decoder architectures. Zhang et al. [52] introduced an edge loss function within the transformer model, which enhances the content details and prevents the generation of blurred results due to the excessive rendering of style features, while simultaneously mitigating the issue of content leakage. To improve the architecture, Zhu et al. [53] proposed a novel all-to-key attention mechanism that integrates distributed attention and progressive attention, matching each position in content features to stable key positions in style features, demonstrating exceptional performance in preserving semantic structures and rendering consistent style patterns.

2.4. Texture Mapping

Through texture mapping technology, texture images can be combined with mesh models to create realistic 3D models, which can provide visual details like texture and material. Texture mapping, which uses a 2D image to represent surface appearance, is widely used. Neural scene representations, such as NeRF [54] and deep reflectance volumes [55], use volume rendering but fail to separate geometry from appearance, limiting surface editing. Thies et al. [56] addressed this by optimizing neural textures on 3D mesh proxies. Xiang et al. [57] further improved on this with the neutex model, which uses volumetric representation for geometry and 2D texture maps for surface appearance, enabling scene reconstruction and traditional texture editing from multi-view images.

3. Methodology

3.1. Dataset

A public dataset (hereinafter referred to as Dataset 1) was used to evaluate the application of the reconstruction algorithm in large-scale scenes, such as the external environment of factories, and contains 200 images of building structures in eight categories, with corresponding internal camera parameter files. In addition, a private dataset (hereinafter referred to as Dataset 2) was established, which contains 48 high-definition images of two kinds of robotic arms and two types of CNC machining centers, as representatives of common factory equipment. Examples of the two kinds of datasets are shown in Figure 1.

Figure 2 shows the dataset (hereinafter referred to as Dataset 3) collected for the style transformation task in deep learning, containing six different style images and the corresponding data sources.

Photogrammetry algorithms require a camera’s intrinsic parameter matrix for calculations. The calibration process, results, and intrinsic parameters for the images in Dataset 2 are detailed in Appendix A.

3.2. Point Cloud Data Acquisition

The SFM algorithm is realized through feature point extraction and matching, camera pose estimation, triangulation, and global optimization, as illustrated in Figure 3.

The feature points, uniquely identifying the scene, are extracted from the image by calculating feature descriptors.

$$D(x,y,\sigma )=\left[G(x,y,k\sigma )-G(x,y,\sigma )\right]\ast I(x,y)=L(x,y,k\sigma )-L(x,y,\sigma ).$$

(1)

Equation (1) is the computational formula for the image pyramid before SIFT feature detection, where the Gaussian scale space L is first constructed by continuously varying the scale by Gaussian kernel function convolving with the image I, $L(x,y,\sigma )$ denotes the Gaussian scale space, and $D(x,y,\sigma )$ is the differential Gaussian image. The differential Gaussian (DoG) response image is obtained by subtracting the images of two neighboring Gaussian spaces, and the location of feature points is detected by the DoG. A gradient histogram of the image is used to find the stable orientation of the feature points. Then, the SIFT feature point location, scale, and orientation information can be obtained. Speed-up robust features (SURF) [58] and KAZE [59] feature extraction are also used to compare with SIFT.

Feature points are first matched by KNN matching [60], then refined by RANSAC [19] to enhance the accuracy.

The camera pose estimation is realized by estimating rotation and translation matrices, which is based on the camera imaging model and coplanarity condition in photogrammetry.

As shown in Figure 4, the coordinate system transformation is described in the following equation,

$$Z{\mathit{P}}_{uv}=Z\left[\begin{array}{c}\hfill u\hfill \\ \hfill v\hfill \\ \hfill 1\hfill \end{array}\right]=\mathit{K}\left(\mathit{R}{\mathit{P}}_w+\mathit{t}\right)=\mathit{K}\mathit{T}{\mathit{P}}_w,$$

(2)

where the matrix consisting of $3\times 3$ vectors is named the camera intrinsic matrix K, with the camera coordinate system rotation matrix R and translation matrix t being called the extrinsic matrix of the camera.

Figure 5 describes the geometric correspondence relationship of a point-in-world coordinates when using two cameras. The mathematical expression of the coplanarity condition in photogrammetry is described in Equation (3).

$${\mathit{p}}_2^T{\mathit{K}}^{-T}{\mathit{t}}^{\wedge }\mathit{R}{\mathit{K}}^{-1}{\mathit{p}}_1=0,$$

(3)

where $p_1$ and $p_2$ are pixel locations of point P in the pixel planes of two cameras. Given the intrinsic parameters of the camera, the concept of an essential matrix is introduced here [61], simplifying the equation as

$${\mathit{p}}_2^T\mathit{E}{\mathit{p}}_1=0.$$

(4)

Thus, the problem of solving the camera position involves calculating the camera essential matrix E. By expanding the essential matrix E and writing it in the form of a vector and selecting eight pairs of feature points, Equation (5) can be obtained:

$$\left(\begin{array}{ccccccccc}{u}_1^1{u}_2^1& u_1^1{v}_2^1& u_1^1& v_1^1{u}_2^1& v_1^1{v}_2^1& v_1^1& u_2^1& v_2^1& 1\\ u_1^2{u}_2^2& u_1^2{v}_2^2& u_1^2& v_1^2{u}_2^2& v_1^2{v}_2^2& v_1^2& u_2^2& v_2^2& 1\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& \\ u_1^8{u}_2^8& u_1^8{v}_2^8& u_1^8& v_1^8{u}_2^8& v_1^8{v}_2^8& v_1^8& u_2^8& v_2^8& 1\end{array}\right)\left(\begin{array}{c}{e}_1\\ e_2\\ e_3\\ e_4\\ e_5\\ e_6\\ e_7\\ e_8\\ e_9\end{array}\right)=0.$$

(5)

The R, t matrix can be obtained by decomposing the essential matrix E using singular value decomposition (SVD), and finally determined according to the orientation of the camera positive depth.

The point-in-world coordinate is computed based on the coordinate transformation matrix P, as follows:

$$P=K[R\mid t].$$

(6)

Decomposing P into three vectors,

$$\begin{array}{c}{P}_i=\left[\begin{array}{c}\hfill P_{i1}\hfill \\ \hfill P_{i2}\hfill \\ \hfill P_{i3}\hfill \end{array}\right]\end{array}$$

(7)

The relationship between the world coordinates and the pixel coordinates of a point can be expressed as follows,

$$\left[\begin{array}{cc}{P}_{i1}& \tilde{X}\\ P_{i2}& \tilde{X}\\ P_{i3}& \tilde{X}\end{array}\right]=\left[\begin{array}{c}{x}_i\\ y_i\\ 1\end{array}\right],$$

(8)

where $\tilde{X}$ is the location of the point-in-world coordinate, and $x_i$ and $y_i$ are the locations in pixel coordinates. Since $x_1$, $x_2$, $P_1$, and $P_2$ are known, a transformation of the above equation yields Equations (9)–(11).

$$\begin{array}{c}A=\left[\begin{array}{c}\hfill A_1\hfill \\ \hfill A_2\hfill \end{array}\right],\\ A\tilde{X}=0.\end{array}$$

(9)

$$A_1=\left[\begin{array}{c}\hfill x_1{P}_{13}-P_{11}\hfill \\ \hfill y_1{P}_{13}-P_{12}\hfill \end{array}\right],$$

(10)

$$A_2=\left[\begin{array}{c}\hfill x_2{P}_{23}-P_{21}\hfill \\ \hfill y_2{P}_{23}-P_{22}\hfill \end{array}\right].$$

(11)

Using the least squares method to solve the system of equations, the spatial coordinates of point $\tilde{X}$ can be obtained, while solving the coordinates of the point cloud.

Finally, bundle adjustment [28] is used as a global optimization method to improve the accuracy of the triangulation process by iterative refinement of the camera position parameters.

3.3. Surface Reconstruction

Poisson surface reconstruction is realized by a process involving constructing and solving Poisson’s equation and generating an equivalent surface, as shown in Figure 6.

First constructing the indicator function X, setting the target M inside to 1 and the target M outside to 0, and its exponential function ${\chi }_M$, where the point cloud normal vector with ${\chi }_M$ is shown in Figure 7.

The indicator function is smoothed using a smoothing function $\tilde{F}$. For any point $p\in \partial M$, define $N\partial M\left(p\right)$ as the inward surface normal vector and ${\tilde{F}}_p\left(q_0\right)$ as a smoothing filter.

$$\nabla \left({\chi }_M\ast \tilde{F}\right)\left(q_0\right)={\int }_{\partial M}{\tilde{F}}_p\left(q_0\right){\overrightarrow{N}}_{\partial M}\left(p\right)dp.$$

(12)

According to Gauss’s divergence theorem, the vector space and the indicator function satisfy the constraints, $\nabla \tilde{\chi }=\tilde{V}$. Since vector fields cannot be integrated, the derivation of both sides of the above equation yields the Laplace equation. Morphing the equation yields

$$\Delta \chi \equiv \nabla ·\nabla \chi =\nabla ·\overrightarrow{V}.$$

(13)

By solving this Poisson equation, the indicator function can be solved. The function $F_0\left(q\right)$ corresponding to a node o in the octree is

$$F_0\left(q\right)\equiv F\left({\displaystyle \frac{q-o.c}{o.w}}\right){\displaystyle \frac{1}{{(o.w)}^3}},$$

(14)

where $o.c$ is the center of o and $o.w$ is the width of o. The vector space $\overrightarrow{V}\left(q\right)$ can be approximated as

$$\overrightarrow{V}\left(q\right)\equiv \sum _{s\in S}\sum _{o\in Ng{b}_0\left(s\right)}{\alpha }_{o,s}{F}_o\left(q\right)s.\overrightarrow{N}.$$

(15)

Defining $\tilde{\chi }={\sum }_0{x}_0{F}_0$, then solving for $\chi$ is the same as solving for $x_o$. Let the number of nodes in the octree be $\mathrm{N}$, and the mesh can be built by calculating value of the $\mathrm{N}\times \mathrm{N}$ matrix L at the position $\left(0,O^{\prime }\right)$

$$L_{o,0^{\prime }}\equiv 〈{\displaystyle \frac{{\partial }^2{F}_o}{\partial x^2}},F_{o^{\prime }}〉+〈{\displaystyle \frac{{\partial }^2{F}_o}{\partial y^2}},F_{o^{\prime }}〉+〈{\displaystyle \frac{{\partial }^2{F}_o}{\partial z^2}},F_{o^{\prime }}〉.$$

(16)

3.4. Style-Transfer-Based Texture Enhancement

Style transfer is achieved by combining the VGG-19 network and the Gram matrix. The VGG19 network (Figure 8) is capable of advanced image feature extraction, synthesis, and manipulation. It can extract content features from images. The network comprises 16 convolutional layers, 5 pooling layers, 3 fully connected layers, and a final softmax layer.

The VGG model is divided into two parts: features (containing convolutional and pooling layers) and a classifier (containing fully connected layers). The output of each convolutional layer from the features model is used to compute the content and style loss. To ensure consistency, the .eval() function sets the network to evaluation mode. Additionally, before feeding an image into the VGG network, it must be normalized using $mean=[0.485,0.456,0.406]$ and $std=[0.229,0.224,0.225]$ for each channel.

Define the input content image as $\overrightarrow{p}$, the final generated image as $\overrightarrow{x}$, and the convolutional layer used in the style migration process as l. $F_{ij}^l$ denotes the activation value of the image generated l on the ith convolutional kernel of the first layer located at j, and similarly $P_{ij}^l$ denotes the activation value of the content image on the No. i convolutional kernel of the first layer located at j. Therefore, the loss function of the content of a single layer is

$$Lcontent(\overrightarrow{\mathrm{p}},\overrightarrow{x},l)={\displaystyle \frac{1}{2}}\sum _{i,j}{\left(F_{ij}^l-P_{ij}^l\right)}^2.$$

(17)

The derivative of the content loss function concerning the activation of layer l is equal to

$${\displaystyle \frac{\partial L_{\mathrm{content}}}{\partial F_{ij}^l}}=\left\{\begin{array}{cc}\hfill {\left(F^l-P^l\right)}_{ij}\hfill & \hfill \mathrm{if}{F}_{ij}^l>0\hfill \\ \hfill 0\hfill & \hfill \mathrm{if}{F}_{ij}^l<0\hfill \end{array}\right..$$

(18)

The style of an image is represented by a style matrix. To capture the texture representation of the input image, a Gram matrix [62], as shown in Figure 9, uses a symmetric matrix formed by the pairwise inner products of k vectors in an n-dimensional Euclidean space, representing these vectors’ texture features.

The feature space of the Gram matrix consists of the correlations between the different filter responses, where the expectation accounts for the spatial extent of the feature mapping. Defined as the activation value of an image $F_{ij}$ located at j on the ith convolution kernel of the first layer, the corresponding Gram matrix is

$$G_{ij}^l=\sum _k{F}_{ik}^l{F}_{jk}^l.$$

(19)

Let the height and width of the convolutional layer l be $N_l$ and $M_l$, respectively, with the Gram matrix $A_{ij}^l$ corresponding to the stylized image $\overrightarrow{p}$, and the Gram matrix corresponding to the generated image $\overrightarrow{a}$, $G_{ij}^l$. So the stylized loss function for a single layer is

$$E_l={\displaystyle \frac{1}{4N_l^2{M}_l^2}}\sum _{i,j}{\left(G_{ij}^l-A_{ij}^l\right)}^2.$$

(20)

The derivative of the style loss function $E_l$ concerning the layer l activation can be computed analytically, as follows:

$${\displaystyle \frac{\partial E_l}{\partial F_{ij}^l}}=\left\{\begin{array}{cc}{\displaystyle \frac{1}{N_l^2{M}_l^2}}{\left({\left(F^l\right)}^T\left(G^l-A^l\right)\right)}_{ji}\hfill & \mathrm{if}{F}_{ij}^l>0\hfill \\ 0\hfill & \mathrm{if}{F}_{ij}^l<0\hfill \end{array}\right..$$

(21)

The total style loss can be calculated as follows:

$$Lstyle(\overrightarrow{a},\overrightarrow{x})=\sum _{l=0}^L{w}_l{E}_l.$$

(22)

The loss function $L_{\mathrm{total}}$ for style migration consists of the content loss function and the style loss function described above, with the weighting factors $\alpha$ and $\beta$ controlling the balance between content and style reconstruction, as shown in the following equation:

$$L_{\mathrm{total}}(\overrightarrow{p},\overrightarrow{a},x)=\alpha L_{\mathrm{content}}(\overrightarrow{p},\overrightarrow{x})+\beta L_{\mathrm{style}}(\overrightarrow{a},\overrightarrow{x}).$$

(23)

The Figure 10 illustrates the architecture of style transfer. The style image $\overrightarrow{a}$ is processed to compute and store its style representation $A^L$ across all layers. The content image $\overrightarrow{p}$ is processed to store its content representation $P^L$ in one layer. A random noise image $\overrightarrow{x}$ is then passed through the network, computing its style feature $G^L$ and content feature $F^L$. The style loss $L_{style}$ is calculated as the mean squared difference between $G^L$ and $A^L$ for each layer. The content loss $L_{content}$ is the mean squared difference between $F^L$ and $P^L$. The total loss $L_{total}$ is a linear combination of these losses. Using error backpropagation, the gradient updates the image $\overrightarrow{x}$ iteratively to match both the style features of the style image and the content features of the content image.

The hyperparameters for style transfer deep learning training based on VGG networks are shown in

Table 2. Hyperparameters for style transfer training.

Content Weight	Style Weight	Content Layer	Style Layer	Optimizer	Learning Rate	Ephochs	Iteration
1	1000	$\prime block4-conv{2}^{\prime }:0.5,$ $\prime block5-conv{2}^{\prime }:0.5$	$\prime block1-conv{1}^{\prime }:0.2,$ $\prime block2-conv{1}^{\prime }:0.2,$ $\prime block3-conv{1}^{\prime }:0.2,$ $\prime block4-conv{1}^{\prime }:0.2,$ $\prime block5-conv{1}^{\prime }:0.2$	Adam	0.03	20	100

.

3.5. Texture Mapping

Mathematically, a projector function is applied to spatial points to obtain parameter space values, which are then converted to texture space using corresponding functions. The algorithm’s flow is illustrated in Figure 11.

• Step one. A set of parameter space values are obtained by applying the projector function to points in space, transforming 3D points into texture coordinates. The relationship between the points in the world coordinate system and the points in the pixel coordinate system are shown as follows:

  $$\left[\begin{array}{c}{\mathit{P}}_{uv}\\ 1\end{array}\right]=\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]={\displaystyle \frac{1}{z}}\left[\begin{array}{ccc}{f}_x& 0& c_x\\ 0& f_y& c_y\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}\hfill x\hfill \\ \hfill y\hfill \\ \hfill z\hfill \end{array}\right]={\displaystyle \frac{1}{z}}\mathit{K}{\mathit{P}}_c.$$

  (24)
• Step two. Before accessing the texture with these new values, corresponding functions convert the parameter space values to the texture space. The image appears at position $(u,v)$ on the object’s surface with uv values in the normal range of [0,1). Textures outside this range are displayed according to the corresponder function.
• Step Three. These texture space locations are used to obtain the corresponding color values from the texture. The built-in functions bilinear and trilinear interpolation sampling are used to map spatial points between the UV space points.
• Step Four. The value transform function is used to transform the retrieved results, and finally the new values $L_d$ are used to change the surface properties $k_d$, such as material or coloring normals, and so on.

  $$L_d=k_d\times {\displaystyle \frac{I}{2}}\times max(0,\overrightarrow{l}·\overrightarrow{n}),$$

  (25)

  where the vector n represents the normals at the coloring points. Changes in vertex normals alter the coloring results, creating different shades that provide a sense of depth and texture. With the vertices’ original positions unchanged, altered vertex normals are used to generate artificial shading effects, enhancing the model’s realism.

4. Result and Discussion

In this section, Section 4.1 encompasses feature point detection, feature point matching, feature point triangulation, and point cloud generation, using the structure-from-motion algorithm to obtain the point cloud. Section 4.2 presents the surface reconstruction results based on the point cloud data, producing a mesh model composed of triangular facets. Section 4.3 demonstrates the results of the deep-learning-based texture mapping enhancement method. In Section 4.4, the experimental results of the reconstruction model, which integrated texture-enhanced mapping with the mesh model, are exhibited and compared with models generated using the LiDAR-based method. The experiments were conducted in a 64-bit Windows environment, using the Pycharm compilation environment and the Computer Vision toolbox in MATLAB2024a.

4.1. SFM-Based Point Cloud Data Acquisition

4.1.1. Feature Point Detection

The positional distribution of feature points obtained using SIFT, SURF, and KAZE descriptors is shown in Figure 12. It is evident that the KAZE feature descriptor detected significantly more feature points compared to SIFT and SURF. The dense distribution of KAZE feature points across all datasets suggests that KAZE was heavily influenced by environmental geometric features rather than the target object. Conversely, the SIFT feature points effectively detected the geometric edges of the target object across all six dataset types, with fewer points, reducing the likelihood of duality in the geometric information representation.

4.1.2. Feature Point Matching

The feature points obtained from the three image descriptors were coarsely matched through a K-nearest neighbor search (KNN), and the number of matched pairs and the matching rate were counted according to the spatial distance of the feature points as the filtering condition, as shown in

Table 3. Statistics of SIFT feature point matching.

Threshold	0.65		0.70		0.75		0.80		0.85
	Rate	Quantity	Rate	Quantity	Rate	Quantity	Rate	Quantity	Rate	Quantity
CNC1	0.0596	95	0.0734	117	0.0891	142	0.1350	215	0.1965	313
CNC2	0.0519	93	0.0664	119	0.0859	154	0.1233	221	0.2015	361
ROBOTS	0.0109	16	0.0157	23	0.0321	47	0.0560	82	0.1051	154
STATUE	0.0548	91	0.0759	126	0.1048	174	0.1452	241	0.2133	325
FOUNTAIN	0.1247	328	0.1433	377	0.1699	447	0.1984	522	0.2535	667
CASTLE	0.1777	513	0.2040	589	0.2442	705	0.2910	841	0.3543	1023

,

Table 4. Statistics of SURF feature point matching.

Threshold	0.65		0.70		0.75		0.80		0.85
	Rate	Quantity	Rate	Quantity	Rate	Quantity	Rate	Quantity	Rate	Quantity
CNC1	0.0902	326	0.0999	361	0.1151	416	0.1300	470	0.1494	540
CNC2	0.0506	347	0.0573	393	0.0642	440	0.0713	489	0.0791	542
ROBOTS	0.0288	93	0.0349	113	0.0470	152	0.0591	191	0.0724	234
STATUE	0.0617	103	0.0707	118	0.0779	130	0.0845	141	0.0911	152
FOUNTAIN	0.1173	160	0.1254	171	0.1334	182	0.1400	191	0.1481	202
CASTLE	0.1696	573	0.1835	620	0.1995	674	0.2175	735	0.2362	798

and

Table 5. Statistics of KAZE feature point matching.

Threshold	0.65		0.70		0.75		0.80		0.85
	Rate	Quantity	Rate	Quantity	Rate	Quantity	Rate	Quantity	Rate	Quantity
CNC1	0.0552	2470	0.0652	2918	0.0764	3418	0.0890	3981	0.1030	4609
CNC2	0.0367	2151	0.0432	2543	0.0510	2993	0.0595	3492	0.0697	4090
ROBOTS	0.0225	1185	0.0298	1566	0.0380	1996	0.0485	2549	0.0604	3177
STATUE	0.0533	738	0.0599	800	0.0652	870	0.0710	947	0.0767	1024
FOUNTAIN	0.1685	5716	0.1712	5806	0.1736	5889	0.1759	5967	0.1784	6051
CASTLE	0.1866	5174	0.1947	5379	0.2020	5601	0.2100	5823	0.2170	6017

.

Table 3, Table 4 and Table 5 illustrate that, as the threshold increased, both the matching rate and the number of matched point pairs rose. However, the above tables show that for the same target object, there was no direct connection between the number of feature points and the point matching rate, which also means that more feature points will not have a direct positive impact on 3D reconstruction. Among the three feature point detection methods, as shown in Figure 13, the SIFT descriptor detected the smallest number of feature points, but its point-matching rate was the highest. Notably, the SIFT-detected feature points were more sensitive to matching threshold variation.

As shown in the above figure, the feature points obtained by the SIFT descriptor achieved a higher matching rate compared to the SURF and KAZE descriptors across all datasets. This suggests that matching the feature points obtained by KAZE and SURF was more challenging, and these points did not cluster well around key geometric elements. Thus, SIFT proved to be the best feature detection and extraction method for the datasets in this paper. The results of feature point extraction using SIFT and matching with KNN are shown in Figure 14 and Figure 15 as examples.

In Figure 14 and Figure 15, the shooting angles of the two images used for matching are similar, so, ideally, each pair of feature point lines should be nearly parallel. However, as the matching threshold decreased, the cross confusion of the feature point connecting lines improved, proving that increasing the matching threshold can help reduce the occurrence of false matches. It is evident that Dataset 1 (building class), shown in Figure 15, had significantly fewer point-matching line crossings under the same matching threshold conditions compared to Dataset 2 (industrial equipment class), shown in Figure 14. This difference can be attributed to the fact that the building class dataset has more surface texture features, whereas the industrial equipment class dataset lacks surface texture.

In Figure 16b, the blue box represents the projection of the left image plane onto the right image plane, visualizing the homography matrix between the camera planes. It can be observed that false matches were corrected by RANSAC, and each pair of feature points is correctly positioned in the images.

4.1.3. Triangulation of Feature Points

The results of the matching points triangulation, from solving the homography matrix to obtain a camera pose matrix, are shown in Figure 17 and Figure 18.

The points in Figure 17b and Figure 18b are denser than those in Figure 17a and Figure 18a suggesting that the spatial points obtained through multiple RANSAC matches were significantly more than those from a single RANSAC match. The inherent randomness of the RANSAC algorithm means that multiple iterations increased the matching range of feature points.

4.1.4. Point Cloud Data

A comparison of Figure 19 and Figure 20 shows that the reconstruction result of Dataset 1 was superior to that of Dataset 2. This difference is attributed to Dataset 1’s composition of large buildings, which offer richer surface textures, more detectable feature points, and more extractable geometric information. In contrast, the objects in Dataset 2 have simple geometric contours and larger untextured planes, making it difficult to detect feature points at the center of these planes.

4.2. Surface Reconstruction

The results of the normal vector solution for Poisson surface reconstruction are shown below.

The results in Figure 21 and Figure 22 indicate that the normal vectors were uniformly distributed across the entire point cloud, with no significant outliers or anomalies. In Dataset 2, which primarily consists of flat surfaces, the normal vectors were consistent in the smooth regions. In Dataset 1, composed of complex geometric elements, the normal vectors exhibited smooth transitions in areas with more edges and curvature. The lengths of all normal vectors were consistent across all distribution plots for each point cloud set. The computed normal vectors were accurate in terms of directional consistency, distribution pattern, and length uniformity.

As shown in Figure 23 and Figure 24, the visual reconstruction effect of Dataset 1 (architecture) was superior to Dataset 2 (workshop equipment). In Figure 23a, the statue’s head has an extraneous tubular extension due to an upward acquisition angle, resulting in a lack of top data and an incomplete reconstructed surface. In Figure 24b, the surface transition of the small pool in the fountain’s center is uneven, failing to restore the real water surface. This is due to the acquisition angle causing the water surface to almost overlap from the proximal to distal ends, making it difficult to extract geometric profile information, leading to significant reconstruction errors.

4.3. Texture Enhancement

The style transfer results are shown in Figure 25, Figure 26, Figure 27, Figure 28, Figure 29 and Figure 30.

In the figures above, the number of training epochs increased from 0 to 20 (iterations from 0 to 2000), causing the edges in the generated image to gradually disintegrate and form local contours consistent with the style image. This indicates a deepening fusion of content and style with more training epochs. Comparing Figure 25, Figure 26, Figure 27, Figure 28, Figure 29 and Figure 30, when the texture features in stylized images exhibit recurring complex structures and these structures are relatively large in scale, the stylization process is more perceptible to the human eye. In contrast, the scale ratios of the stainless steel texture and carbon fiber texture in Figure 29 and Figure 30 are relatively small, resulting in minimal visual differences in the stylization results from epochs 6 to 20. Additionally, the stainless steel texture in Figure 29 mainly varies in the lateral direction, leading to blurred lateral features and an unclear preservation of contours in the content images.

The loss function during the training process exhibited a consistent pattern of variation across the six categories of objects. The training loss of the experiment carried on the CNC1 object, as an example, is shown in Figure 31.

The subfigure on the left in Figure 31 depicts the style and content loss over the iterations, with the red line representing style loss and the green line content loss. The content loss did not increase with more training rounds, indicating that the geometric contours of objects in the generated images and content images remained similar as perceived by the convolutional neural network, preserving the details of the content image. The style loss decreased rapidly within the first five training rounds, indicating a significant reduction in texture disparity between the generated and style images, suggesting the model quickly learned and integrated the style image’s texture.

The subfigure on the right in Figure 31 demonstrates the total loss over the iterative process, where total loss is a weighted sum of style and content loss. As detailed in Table 2, the style loss weight was substantially higher than that of the content loss. Consequently, the total loss primarily followed the trend of style loss, gradually decreasing with additional training epochs.

The image quality assessments for these six datasets exhibit a uniform pattern, with CNC1 serving as an example, as shown in Figure 32 The result for each epoch was evaluated using both PSNR and SSIM methods, with the content image as the reference image.

The SSIM and PSNR indices generally decreased with increasing training epochs, as these indices evaluate the similarity between the images generated by style migration and the content images. The image quality evaluation of the six objects exhibited an interesting phenomenon at the 12th training epoch, where both indices first increased and then rapidly decreased. However, this change is not perceptible to the human eye, as the results from the 12th epoch are almost indistinguishable from those of the 10th and 14th epochs in Figure 25, Figure 26, Figure 27, Figure 28, Figure 29 and Figure 30. Because of the poor interpretability of CNN models, this interesting change is hard to convincingly explain and avoid. Nonetheless, as style transfer experiments prioritize the visualization of results, this phenomenon did not affect the application of the style transfer.

4.4. Texture Mapping

The results of texture mapping are shown in Figure 33 and Figure 34, whose upper parts show the surface reconstruction results, with the lower part displaying the effect after texture mapping.

As shown in Figure 33, the enhanced texture mapping exhibited a precise alignment with the reconstructed surface, markedly improving the outcomes, such as covering the highlights on the Statue model and enhancing the metal texture. The Fountain object’s water surface holes are obscured by the red brick texture, while the integration of the white brick texture with the Castle model offers a novel visual impact. The application of texture mapping to Dataset 1 was deemed successful, with an accurate spatial alignment and an improvement in model defects, thus enhancing the overall visual effect.

Figure 34 shows that the CNC1 in Dataset 2 has optimal mapping. Its geometric edges align closely with the model’s contours, respecting CNC1’s geometric constraints, without offset or contour loss. For the Robot objects, the complex spatial contours, with the main lines on various planes, resulted in texture maps that struggle to perfectly match the device’s edges to the model’s contours.

The comparisons in the

Table 6. Statistics of surface reconstruction on Dataset 2 using LiDAR-based method ¹.

	Vertices	Faces	Time Consumption	Equipment	Equipment Cost
Robot	95,556	115,736	29 min	Leica RTC 360	92,800 GBP
CNC1	175,175	204,458	40 min	Leica RTC 360	92,800 GBP
CNC2	195,788	210,396	34 min	Leica RTC 360	92,800 GBP

¹ The solution based on LiDAR involved the use of LiDAR for point cloud acquisition, point cloud filtering, and Poisson reconstruction.

and

Table 7. Statistics of surface reconstruction on Dataset 2 using proposed method.

	Vertices	Faces	Time Consumption	Equipment	Equipment Cost
Robot	172,875	145,702	33 min	Iphone 13 pro	949 GBP
CNC1	216,893	234,135	37 min	Iphone 13 pro	949 GBP
CNC2	221,040	200,205	38 min	Iphone 13 pro	949 GBP

show that the method proposed in this paper outperformed the LiDAR solution by $20\%$ in terms of both point cloud quantity and the number of generated mesh models, while the hardware cost was only $1\%$ of that of the LiDAR solution. However, the modeling time consumption of the proposed method was slightly higher than that of the LiDAR solution.

5. Conclusions and Future Work

5.1. Conclusions

The paper has presented a cost-effective and rapid 3D modeling technique that leverages point cloud acquisition, surface reconstruction, and texture processing to create visually striking models from mobile or DSLR images, with a demonstrated high visualization performance on architectural datasets, accurately capturing geometric details and constraints without requiring specialized hardware.

This study advances 3D reconstruction by indicating SIFT as a superior feature descriptor for enhancing feature point matching and SfM efficiency. Furthermore, the application of deep learning for texture enhancement in model mapping is identified as a promising area for future research, with the provided data offering guidance on selecting optimal textures for style image enhancement.

5.2. Future Work

Although this study yielded promising outcomes, it was not without limitations. Notably, the initial phase of the systematic modeling approach, particularly the point cloud 3D reconstruction, could benefit from optimization to reduce environmental clutter and enhance geometric details, as observed in Dataset 2. Additionally, the final texture mapping stage, which involves direct application of images to the model’s surface, may lead to alignment issues that are challenging to rectify manually.

Future research should aim to refine the point cloud reconstruction and the visual fidelity of the proposed modeling system. This could be accomplished by leveraging deep learning to confine feature points to an object’s geometric contours and to optimize their distribution for uniformity, thereby improving the reconstruction quality. Moreover, integrating depth information from the dataset and aligning specific geometric features to precise spatial locations on the model could effectively address mapping discrepancies.