From Single Shot to Structure: End-to-End Network-Based Deflectometry for Specular Free-Form Surface Reconstruction

Abstract

Deflectometry is a key component in the precise measurement of specular (mirrored) surfaces; however, traditional methods often lack an end-to-end approach that performs 3D reconstruction in a single shot with high accuracy and generalizes across different free-form surfaces. This paper introduces a novel deep neural network (DNN)-based approach for end-to-end 3D reconstruction of free-form specular surfaces using single-shot deflectometry. Our proposed network, VUDNet, innovatively combines discriminative and generative components to accurately interpret orthogonal fringe patterns and generate high-fidelity 3D surface reconstructions. By leveraging a hybrid architecture integrating a Variational Autoencoder (VAE) and a modified U-Net, VUDNet excels in both depth estimation and detail refinement, achieving superior performance in challenging environments. Extensive data simulation using Blender leading to a dataset which we will make available, ensures robust training and enables the network to generalize across diverse scenarios. Experimental results demonstrate the strong performance of VUDNet, setting a new standard for 3D surface reconstruction.

1. Introduction

In recent years, rapid advancements in optical-based industries have significantly influenced sectors such as microelectronics [1,2,3] and automotive design [4,5,6], leading to a strong demand for highly accurate methods for measuring specular surfaces [7,8]. Deflectometry [9,10,11] is an optical measurement technique which relies on the reflection of structured light patterns from an unknown surface to infer its shape. Deflectometry is highly effective for measuring specular surfaces due to its accuracy, speed, and non-contact approach, making it particularly suitable for capturing detailed surface variations [12]. Additionally, free-form surface reconstruction [13,14] is crucial for modeling complex, irregular shapes found in advanced optical systems [15]. Together, these methods enhance precision and innovation in optical manufacturing.

The more specific context of single-shot deflectometry [16,17,18], performing deflectometry based on only a single image, is crucial for several reasons, particularly for high-speed imaging of fast dynamically changing surfaces. This includes applications such as fast-moving or morphing specular surfaces like liquids (water, metals), where capturing multiple images is impractical due to the rapidly changing nature of the surface. Additionally, in industrial scanning, single-shot deflectometry enhances efficiency by eliminating the need for multiple images, important for quality control of reflective objects [9]. For example, in General Fusion’s Magnetized Target Fusion technology scheme, measurement of the 3D shape of an imploding specular liquid metal vortex to compress plasma to generate nuclear fusion with microsecond precision is required [19]. Traditional methods [10] require multiple captures and complex setups to obtain surface-gradient information in perpendicular directions, which can be time-consuming and problematically inefficient for dynamic applications.

To further enhance the efficiency and accuracy of single-shot deflectometry, recent research has explored the use of composite fringe patterns in deep learning-based 3D surface measurement techniques [17,20]. These composite patterns are particularly effective in addressing the challenges associated with single-shot deflectometry, such as encoding complex surface details in a single capture. Historically, in the use of fringe patterns for surface measurement, vertical and horizontal patterns are often combined into a single composite pattern. The separation of these orthogonal fringe patterns into their individual components is crucial for accurate analysis and has traditionally been achieved through methods like the 2D Fourier algorithm [21] which, despite its utility, often results in phase errors and spectrum aliasing [22].

In contrast, in this paper, we propose a novel deep neural network (DNN)-based approach designed for the end-to-end 3D reconstruction of free-form surfaces. Our network, VUDNet, processes 2D images captured from a surface onto which an orthogonal fringe pattern is projected (Figure 1). This discriminative–generative hybrid network excels in depth estimation. To the best of our knowledge, VUDNet represents the first DNN-based approach in deflectometry, pioneering the use of generative methods for depth estimation.

The integration of both generative and discriminative components in our network is pivotal for achieving high-fidelity 3D surface reconstruction. The discriminative part of the network is essential for accurately identifying and interpreting the orthogonal fringe patterns projected onto the specular surface. This involves extracting meaningful features from an image, which are crucial for determining the gradient information necessary for depth estimation. The discriminative model’s ability to parse complex visual patterns with precision directly impacts the accuracy of the subsequent 3D reconstruction.

On the other hand, the generative component plays a crucial role in translating the extracted gradient information into a coherent 3D representation. The generative model excels in learning and representing the underlying data distribution through embedding representations [23,24]. These embeddings capture the intricate details and variations of the surface, allowing the reconstruction of a highly detailed and accurate 3D shape. The learned embeddings are vital because they encapsulate the spatial relationships and depth cues that are not readily apparent in the raw 2D images [24]. This deep representation allows the generative model to fill in gaps and correct any inconsistencies in the depth information, resulting in a precise 3D reconstruction. This synergy between the two models not only enhances the overall performance of the network but also ensures that the 3D reconstructions are both geometrically precise and visually coherent.

Paper Contributions

• Novel DNN-Based Approach for Deflectometry: We introduce VUDNet, the first deep neural network (DNN) designed specifically for end-to-end 3D reconstruction of free-form specular surfaces using single-shot deflectometry. VUDNet leverages a hybrid architecture that combines the strengths of both generative and discriminative models, ensuring high accuracy and generalization.
• Dataset Simulation: To train and evaluate our model, we simulated an extensive dataset. This dataset, which includes a variety of deformed specular surfaces and their corresponding depth maps, will be made publicly available to support future research in this field.
• Robust Performance in Challenging Environments: Experimental results demonstrate that VUDNet significantly outperforms existing methods in reconstructing 3D surfaces from single-shot 2D images, particularly in challenging environments. Our network demonstrates the ability to generalize across diverse scenarios.

2. Related Work

Deflectometry has high accuracy, speed, and is non-contact, making it suitable for a variety of industrial applications [9]. However, traditional deflectometry methods, such as phase-shifting techniques, require multiple images to be captured in order to obtain surface gradients in two perpendicular directions [9,17]. This process can be time-consuming and inefficient, particularly for dynamic or real-time applications.

Single-shot deflectometry was developed to overcome these limitations [9,17,18,25,26]. Seo et al. developed a single-shot freeform surface profiler based on spatially phase-shifted lateral shearing interferometry [17]. This approach, while simplifying the measurement process, still struggles with complex surface geometries. Phase measuring profilometry has also seen advancements, with deep learning methods enhancing measurement capabilities [27,28]. Working from a single image significantly reduces the time required for surface measurement; however, there are resulting challenges in interpreting the complex patterns reflected by specular surfaces, especially those with low reflectivity or intricate geometries.

The application of deep learning in optical metrology, including deflectometry, has shown significant promise [12,17,26,29]. Deep learning models can learn complex patterns and provide robust phase retrieval from noisy or low-quality data. DYNet++ [17] utilizes deep learning for single-shot deflectometry, capable of retrieving phase information from composite patterns even in challenging conditions with both closed- and open-loop fringe patterns.

Wang et al. proposed a deep learning-based deflectometric method for freeform surface measurement [26]. Their approach involves devising a deep neural network specifically for freeform surface reconstruction. This method offers a solution for the general measurement of freeform surfaces while minimizing measurement errors caused by noise and system geometry calibration.

3. Method

The architecture of our proposed network represents a sophisticated blend of specialized components designed to tackle the unique challenges of reconstructing 3D specular surfaces from single-shot 2D images. Central to this architecture are two main components: the discriminative and generative branches which form a novel hybrid depth estimator network. Our proposed model is an ensemble of a Variational Auto Encoder (VAE) [30,31,32] and a modified U-Net [33] architecture, adapted here to enhance depth map refinement. This section details each component’s role and functionality within our network, elucidating how they collectively contribute to 3D surface reconstruction. Figure 2 illustrates the comprehensive architecture of our network.

3.1. Problem Formulation

In this work, we address the challenge of reconstructing a 3D surface from a single-shot 2D image of an orthogonal sinusoidal fringe pattern reflected on a specular surface. To formulate the problem, we introduce the following notation:

• $I=\{I_k\mid k=1,\cdots ,n\}$: The 2D reflected image of a pattern $\mathcal{P}$ reflected by surface $\mathcal{S}$.
• $\mathcal{D}=\{{\mathcal{D}}_k\mid k=1,\cdots ,n\}$: The ground truth depth map corresponding to surface $\mathcal{S}$.
• $\widehat{\mathcal{D}}$: The estimated depth map produced by our network.
• $f_{\mathrm{MLP}}:{\mathbb{R}}^{H\times W}\times {\mathbb{R}}^{H\times W}\to {\mathbb{R}}^{H\times W}$: The MLP function that combines the outputs of the VAE and U-Net to produce $\widehat{\mathcal{D}}$.
• $\mathcal{C}$: camera parameters.
• $\mathcal{F}:{\mathbb{R}}^3\to {\mathbb{R}}^2$: The projection to render a 2D image I from surface $\mathcal{S}$.
• $\psi :{\mathbb{R}}^3\to \mathbb{R}$: The function mapping a 3D point to depth.

The objective is to develop a hybrid depth estimator network, VUDNet, that integrates both generative and discriminative modeling approaches. To train and evaluate our model, we simulate a comprehensive dataset that includes 2D images I and corresponding ground truth depth maps $\mathcal{D}$. The generated dataset captures the complex interactions of light with specular surfaces, providing a robust foundation for training our hybrid depth estimator network. This approach ensures that our model can generalize well to real-world scenarios involving specular reflections.

3.2. Architecture Overview

The VUDNet architecture integrates a Variational Auto Encoder (VAE) and a modified U-Net, both of which are provided with an input 2D image I, the reflection of pattern $\mathcal{P}$ on some unknown surface $\mathcal{S}$. The VAE is a generative model that aims to learn the underlying distribution of the data, providing a coarse estimation of the depth map. In contrast, the U-Net is a discriminative model designed to refine this estimation, yielding a finer and more precise depth map. The hybrid nature of our architecture leverages the strengths of both generative and discriminative approaches, creating a comprehensive depth estimation model.

The VAE component of our architecture consists of an encoder function $E_{\mathrm{enc}}:{\mathbb{R}}^{H\times W\times c}\to {\mathbb{R}}^Z$ that compresses the input image into a latent space representation $z=E_{\mathrm{enc}}\left(I\right)$, followed by a decoder function $E_{\mathrm{dec}}:{\mathbb{R}}^Z\to {\mathbb{R}}^{H\times W}$ that reconstructs the depth map from this latent representation, producing depth estimate ${\widehat{\mathcal{D}}}_k^{\mathrm{V}}=E_{\mathrm{dec}}\left(z\right)$. This process allows the VAE to capture and model the complex variations in specular surfaces. On the other hand, the modified U-Net architecture, tailored for depth estimation, employs a series of downsampling and upsampling layers specifically structured to refine the depth map, producing estimate ${\widehat{\mathcal{D}}}_k^{\mathrm{U}}$. Unlike the traditional U-Net, our modified version diminishes the effect of skip connections to prevent the direct transfer of the texture information present in the patterning of input image I, thereby ensuring that the output is a detailed depth map rather than a combination of depth and texture features.

The output depth maps from both the VAE and the modified U-Net are then fed into a fully connected Multi-Layer Perceptron (MLP) that serves as an ensemble method. We let $f_{\mathrm{MLP}}:{\mathbb{R}}^{H\times W}\times {\mathbb{R}}^{H\times W}\to {\mathbb{R}}^{H\times W}$ denote the MLP function that combines the depth maps from the two networks:

$${\widehat{\mathcal{D}}}_k=f_{\mathrm{MLP}}({\widehat{\mathcal{D}}}_k^{\mathrm{V}},{\widehat{\mathcal{D}}}_k^{\mathrm{U}})\approx \psi \left({\mathcal{F}}^{-1}\left(I\right)\right)$$

(1)

Such an ensemble approach is beneficial for several reasons:

• Reduced Overfitting: Ensemble methods inherently reduce the risk of overfitting [34]. They are known for their ability to improve generalization by leveraging the strengths of multiple models and mitigating individual model biases [34]. In our work, this is especially true as we combine one discriminative model and one generative model. Each model captures different aspects of the data distribution, and their combination leads to a more generalizable solution.
• Complementary Strengths: The VAE’s ability to model complex data distributions complements the U-Net’s strength in preserving spatial details, making the ensemble approach particularly effective for depth estimation tasks.
• Improved Performance: Hybrid models that incorporate both generative and discriminative components have been shown to outperform single-method models in various tasks [35,36,37]. This combination harnesses the power of both methodologies, leading to improved performance in reconstructing 3D surfaces from 2D images.

3.3. Loss Function

The loss function used in our network is a composite of reconstruction loss and regularization terms. This combination is designed to ensure that the network not only accurately reconstructs the desired output but also maintains generalization capabilities and prevents overfitting [38]. The reconstruction loss directly measures the difference between the predicted and true values, ensuring precise output generation. Meanwhile, the regularization terms add constraints that promote smoother and more robust model predictions, enhancing the overall performance and stability of the network [39]. The VAE component of the loss

$${\mathcal{L}}_{\mathrm{V}}={\parallel {\widehat{\mathcal{D}}}_k^{\mathrm{V}}-{\mathcal{D}}_k\parallel }_2^2+\beta \mathrm{KL}(q\left(z\right|\mathbf{I})\parallel p\left(z\right))$$

(2)

consists of a Mean Squared Error (MSE) reconstruction loss (left) and a Kullback–Leibler (KL) divergence for regularization (right). Here, $\beta$ is a weighting factor, $q\left(z\right|I)$ is the posterior distribution, and $p\left(z\right)$ is the prior distribution.

For the modified U-Net component,

$${\mathcal{L}}_{\mathrm{U}}={\parallel {\widehat{\mathcal{D}}}_k^{\mathrm{U}}-{\mathcal{D}}_k\parallel }_2^2+{\lambda }_{\mathrm{TV}}\sum _{i,j}\sqrt{{({\widehat{\mathcal{D}}}_{i+1,j}^{\mathrm{U}}-{\widehat{\mathcal{D}}}_{i,j}^{\mathrm{U}})}^2+{({\widehat{\mathcal{D}}}_{i,j+1}^{\mathrm{U}}-{\widehat{\mathcal{D}}}_{i,j}^{\mathrm{U}})}^2}$$

(3)

there is again an MSE term (left), but additionally a Total Variation (TV) regularization [40] term (right), controlled by weighting factor ${\lambda }_{\mathrm{TV}}$ which encourages spatial smoothness in the predicted depth map by penalizing large gradients [41] and reducing noise [42], leading to improved generalization and robustness [43].

The final output from the MLP ensemble is optimized using a combined loss function from both VAE and U-Net losses:

$${\mathcal{L}}_{\mathrm{final}}=\alpha {\mathcal{L}}_{\mathrm{V}}+\gamma {\mathcal{L}}_{\mathrm{U}}$$

(4)

Statistically, the integration of these loss terms ensures that the model benefits from both global structural understanding (via VAE) and local detail refinement (via U-Net). The MLP ensemble leverages this dual optimization, effectively balancing bias and variance trade-offs [44]. This approach mitigates the risk of overfitting through regularization while maintaining high reconstruction accuracy, resulting in a model that performs well across varied and unseen data distributions [45].

3.4. Vudnet Discussion

Ensemble methods combine multiple models to improve overall performance by leveraging the strengths of each model [46]. In the context of this work, our results showcase a compelling strategy by ensembling a discriminative model (U-Net) with a generative model (VAE) through an MLP, harnessing the complementary advantages of both types of models. Drawing on the theoretical foundations provided by Bishop and Lasserre’s work [47], we can explore the benefits of such an ensemble.

Ensembling helps to mitigate overfitting by combining models that have different error patterns [48]. We let ${ϵ}_{\mathrm{U}}$ and ${ϵ}_{\mathrm{V}}$ be the errors of the U-Net and VAE, respectively. The combined error ${ϵ}_{\mathrm{ensemble}}$ can be expressed as

$${ϵ}_{\mathrm{ensemble}}=\beta {ϵ}_{\mathrm{U}}+(1-\beta ){ϵ}_{\mathrm{V}}$$

(5)

where $0\le \beta \le 1$ is a weight learned by the MLP. By appropriately weighting the errors of U-Net and VAE, the ensemble can reduce the overall error variance. If the errors of U-Net and VAE are uncorrelated, the covariance term drops out, leading to a reduced overall variance. In our work, this is the case because VAE and U-Net approach the problem from different angles: VAE focuses on capturing the global structure and underlying distribution of the data, while U-Net emphasizes local details and spatial relationships. Consequently, the sources of their errors are fundamentally different and likely unrelated. Since ${ϵ}_{\mathrm{V}}$ and ${ϵ}_{\mathrm{U}}$ are not strictly independent, we assume

$$\mathbb{E}\left[{ϵ}_{\mathrm{V}}{ϵ}_{\mathrm{U}}\right]=\delta ,\mathrm{where}\delta \mathrm{is}\mathrm{small}\mathrm{but}\mathrm{not}\mathrm{necessarily}\mathrm{zero}$$

(6)

If $\delta$ approaches zero, the covariance term becomes negligible, and the ensemble error approaches the minimum possible value based on the variances of the individual models. The optimal weight $\beta$ balances the contributions of the VAE and U-Net, leveraging their complementary strengths to reduce the overall error. Even with a non-zero $\delta$, the error ${ϵ}_{\mathrm{ensemble}}$ of the ensemble model still has the potential to be lower than the individual errors:

$$\mathbb{E}\left[{ϵ}_{\mathrm{ensemble}}^2\right]\le min(\mathbb{E}\left[{ϵ}_{\mathrm{V}}^2\right],\mathbb{E}\left[{ϵ}_{\mathrm{U}}^2\right])$$

(7)

Given an adequate learning process to learn the optimal weights, the ensemble model combines the strengths of both VAE and U-Net, reducing the overall error through weighted averaging.

4. Dataset Development

To the best of our knowledge, there is no public dataset available for specular free-form surface reconstruction; therefore, we generate our own synthetic dataset.

By creating a virtual environment that approximates key optical aspects of real-world conditions for a reflective textured surface according to physical laws of reflection embedded in the Blender Cycles ray-tracing engine, we can systematically explore the nuances of specular surface reconstruction. This section details the precise setup and execution of our simulation environment, ensuring that every aspect, from lighting and camera positioning to surface properties, is meticulously configured to generate reliable and varied datasets necessary for robust training.

4.1. Setting up the Simulation Environment

The simulation environment was created using Blender, the Cycles ray-tracing engine, for its ability to produce realistic, physically accurate simulations. This level of realism ensures that the synthetic data closely mimic real-world conditions, thereby improving model robustness and reliability. Simple reflection models often fail to account for optical phenomena such as diffraction, scattering, and the intricate interplay of light with micro-surface textures. In contrast, Blender’s comprehensive simulation environment allows for the accurate replication of these effects, providing a more robust and realistic dataset.

The Cycles engine has been validated in numerous scientific studies [49,50,51] for its ability to simulate complex optical phenomena. Additionally, the open-source nature of Blender, its extensive online support, and the convenience of programming through Python make it an ideal choice for automated dataset generation. Despite some limitations with caustics and computational cost, Cycles provides a balanced trade-off between usability and precision, making it suitable for our purposes. By leveraging the advanced features of Blender’s Cycles engine, we ensure that the synthesized data appear realistic.

A fixed camera, fixed pattern and surface settings were configured to replicate a realistic scenario for deflectometry, including the use of an orthogonal sinusoidal fringe pattern. This pattern is projected onto a specular object, and the reflected fringes are recorded by a camera. A substantial number of images were captured under these settings, providing a robust foundation for training based on fringe patterns. Figure 3 illustrates this environment for visual clarity.

4.2. Dataset Generation

To generate the dataset, 2D images were rendered from the simulated environment using Blender’s Cycles ray-tracing engine. For each image, the surface was deformed (detailed in Section 4.4) to ensure variability in the dataset with an associated depth map $\mathcal{D}$. Each 2D image was rendered based on the specific surface reflectance characteristics, camera settings, and surface topology. The optical ray tracing setup in Cycles was configured with a maximum sample count of 4096, ensuring high-quality image generation. Total bounces were set to 12, diffuse bounces to 4, glossy bounces to 4, transmission bounces to 12, and transparent bounces to 8. In this work, the measurement range is defined as the relative depth of the reconstructed surface, which was normalized to the range $[0,1]$ during preprocessing. This normalization ensures consistency and stability across varying surface geometries and enables the network to generalize to different scales. As a result, the proposed method can handle surfaces with different depth ranges without requiring additional re-calibration or re-scaling [52].

4.3. Pattern

We utilize a fixed orthogonal sinusoidal fringe pattern $\mathcal{P}$ which remains constant throughout the measurement process. A fixed pattern design is particularly advantageous for industrial applications where the pattern needs to be forged, printed, or otherwise permanently established. Unlike dynamic systems, such as in [22], where an LCD changes the pattern with every shot, our fixed pattern ensures consistency and reliability in scenarios where altering the pattern is impractical or impossible.

The fixed orthogonal sinusoidal fringe pattern $\mathcal{P}$ is represented as

$$\mathcal{P}(x,y)={\mathcal{I}}_0+{\mathcal{I}}_{m}sin\left({\displaystyle \frac{2\pi }{T_x}}x+{\phi }_x\right)+{\mathcal{I}}_{m}sin\left({\displaystyle \frac{2\pi }{T_y}}y+{\phi }_y\right)$$

(8)

for mean intensity ${\mathcal{I}}_0$, modulation amplitude ${\mathcal{I}}_m$, periods $T_x,T_y$, and phase shifts ${\phi }_x,{\phi }_y$.

This pattern is projected onto the surface, and the reflected image is captured. The orthogonal nature of the pattern ensures that gradient information is obtained in both x and y directions, which is crucial for 3D reconstruction. By maintaining a fixed pattern, we ensure that our system can be reliably deployed in industrial applications without the need for complex pattern generation mechanisms, providing a stable and efficient solution for 3D surface measurement.

4.4. Shape Deformations

To have a diverse array of geometric features, we systematically introduced deformations to a plane surface $\overline{S}\in {\mathbb{R}}^2$:

$$\Pi :\overline{S}\times \Theta \to \mathcal{S},{\mathcal{S}}_{\theta }=\{(x,y,z)\in {\mathbb{R}}^3\mid (x,y,z)=\Pi (u,v;\theta ),(u,v)\in \overline{S},\theta \in \Theta \}$$

(9)

Here, $\Theta$ represents the set of parameters that conditions the projection. These deformations were designed to simulate realistic scenarios and to increase the likelihood of network generalizability from training data to real-world applications.

4.4.1. Surface Generation

The surface generation process involved modifying a base planar surface by integrating various shapes, such as hemispheres, to resemble geometric structures and by applying parametric deformations to create deformation forms. Each surface was assigned a specular material to mimic realistic lighting interactions, crucial for representing real-world scenarios.

• Shape Integration: To generate the Geometric Surfaces set, multiple convex shapes, primarily hemispheres, were superimposed onto a base planar surface $\overline{S}$ at randomized positions and scales.
• Parametric Deformations: To generate the Deformation Surfaces set, the planar surface was deformed using a set of parametric functions $\Pi$, such as parabolic and sinusoidal transformations. These functions control the depth and curvature of the deformation, sculpting the surface into various forms.
• Randomization: To enhance diversity, the parameters involved in both geometric shape integration (size ${\theta }_1$, position ${\theta }_2$, and the number of hemispheres ${\theta }_3$) and deformations were randomized, resulting in a wide range of surface topologies ${\mathcal{S}}_{\theta }$, featuring variations from subtle indentations to pronounced curvatures.
• Specular Material Assignment: All generated surfaces $\mathcal{S}$ were assigned a specular material to ensure that the generated data accurately represented real-world scenarios, where surface reflectivity significantly impacts depth perception.

4.4.2. Data Capture

Following the creation of these surfaces, we rendered 2D images and captured corresponding depth maps. This process involved two steps:

• Rendering 2D Images: Each deformed surface was rendered under controlled lighting conditions to capture the specular reflections characteristic of real-world environments.
• Depth Map Acquisition: Depth maps $\mathcal{D}$ were generated for each configuration, providing the precise geometric ground-truth information essential for training.

This methodical approach to generating a diverse and realistic dataset enables our deep learning model to learn robust features applicable across a wide range of real-world surfaces.

4.5. Depth Map Standardization

Depth maps generated from simulations often require standardization to ensure consistency and accuracy in subsequent processing steps. Algorithm 1 illustrates the process we employ for this standardization. The algorithm begins with the calculation of the centroid of the depth map, providing a central reference point for further transformations. Principal Component Analysis (PCA) is then used to determine the primary axes of variance in the data, identifying the principal components that describe the orientation of the surface in the 3D space. The PCA eigenvectors lead to a rotation matrix, ensuring that the surface data are oriented with the XY plane, with the Z-axis representing height. The transformation aligns the centroid of the depth map to the origin, standardizing the position and orientation. Finally, the depth values are normalized to a predefined range of $[0,1]$.

Algorithm 1 Normalize and Align Depth Map

1: procedure NormalizeDepthMap($DepthMap$)   2:       Centroid Calculation:   3:       $C\leftarrow {\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{\mathcal{D}}_k^i$        ▹ Calculate the centroid C of the depth map $\mathcal{D}$   4:       Principal Component Analysis (PCA):   5:       $\Sigma \leftarrow {\displaystyle \frac{1}{N-1}}{\sum }_{i=1}^N({\mathcal{D}}_k^i-C){({\mathcal{D}}_k^i-C)}^T$    ▹ Compute the covariance matrix $\Sigma$   6:       $[{\overrightarrow{v}}_1,{\overrightarrow{v}}_2,{\overrightarrow{v}}_3]\leftarrow \mathrm{Eigenvectors}(\Sigma )$        ▹ Eigenvectors ${\overrightarrow{v}}_1,{\overrightarrow{v}}_2,{\overrightarrow{v}}_3$ of $\Sigma$   7:       Rotation Calculation:   8:       Normalization and Vector Operations:   9:       Normalize the normal vector $\overrightarrow{n}={\displaystyle \frac{\overrightarrow{n}}{\parallel \overrightarrow{n}\parallel }}$ 10:       $\overrightarrow{v}=\overrightarrow{n}\times [0,0,1]$                 ▹ Cross product with Z-axis 11:       $s=\parallel \overrightarrow{v}\parallel$, $c=\overrightarrow{n}·[0,0,1]$             ▹ Sine and cosine of the angle 12:       Skew-Symmetric Matrix and Rodrigues’ Formula: 13:       $\mathrm{Skew}\left(\overrightarrow{v}\right)=\left[\begin{array}{ccc}0& -v_3& v_2\\ v_3& 0& -v_1\\ -v_2& v_1& 0\end{array}\right]$ 14:       $R=I+\mathrm{Skew}\left(\overrightarrow{v}\right)+\mathrm{Skew}{\left(\overrightarrow{v}\right)}^2{\displaystyle \frac{1-c}{s^2}}$      ▹ Compute the rotation matrix 15:       Transformation Application: 16:       ${\mathcal{D}}_k^{i^{\prime }}\leftarrow R({\mathcal{D}}_k^i-C)$   ▹ Apply the rotation and translation to each point in the depth map 17:       Normalization: 18:       $a,b\leftarrow \mathrm{range}\mathrm{values}(0,1)$ 19:       ${\mathcal{D}}_k^{i^{″}}\leftarrow a+{\displaystyle \frac{({\mathcal{D}}_k^{i^{\prime }}-min\left({\mathcal{D}}_k^{i^{\prime }}\right))(max\left({\mathcal{D}}_k^{i^{\prime }}\right)-min\left({\mathcal{D}}_k^{i^{\prime }}\right))}{b-a}}$  ▹ Normalize the depth values to the range $[a,b]$ 20:       return $NormalizedMap\leftarrow {\mathcal{D}}_k^{i^{″}}$ 21: end procedure

4.6. Data Preparation

The dataset used in our study consists of a diverse set of depth maps generated from simulated environments, totaling 400 images. We include 250 deformation surface samples and 150 geometric ones, as our research indicated that deformation surfaces present more complexity, and so were emphasized in our simulations. Each image is $320\times 240$ pixels in size and is rendered in greyscale. The dataset is divided into training, validation, and test sets, with $80\%$ used for training, $10\%$ for validation, and $10\%$ for testing. This division ensures a robust evaluation of the model’s performance across different data splits and helps in assessing generalization.

5. Results and Discussion

All experiments were conducted on a system equipped with an NVIDIA GeForce RTX 3060 GPU, 12th Gen Intel(R) Core(TM) i7-12700 CPU, and 32 GB of RAM. The training process for the VUDNet was implemented using TensorFlow (version 2.12.0) and Keras (version 2.12.0). The implementation of VUDNet is available on GitHub at https://github.com/hadious/VUDNet, accessed on 16 November 2024. The entire training phase took approximately 4 h to complete. Inference times are reported in

Table 1. Performance comparison of different models on our generated dataset. VUDNet has reconstruction errors approximately one fourth that of competing methods, under all three criteria of Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Log Error, and with an inference time (computational complexity) similar to or less than competing methods. Bolded numbers indicate the best performance on the General data set.

Model	MAE	RMSE	LogError	Inference Time (ms)
VUDNet (On Deformation)	0.0443	0.0600	0.0122	19
VUDNet (On Geometric)	0.0168	0.0218	0.0038	19
VUDNet (General)	0.0355	0.0470	0.0090	19
DYNet++ * (General)	0.1607	0.1987	0.0790	26
D-UNet ** (General)	0.2052	0.2245	0.0451	21

* DYNet++ plus reconstructing the surface by zonal integration for reconstructing the surfaces as mentioned in [17]. Since there are no publicly available data on the DYNET++ dataset, we were unable to test on their dataset, so we implemented their method on our dataset. ** For D-UNet [29], as there are no publicly available data or code, we implemented it based on [29] and tested it on our proposed dataset.

.

Figure 4 qualitatively demonstrates the success of our model. The predicted depth map exhibits a high degree of smoothness, indicating effective regularization and the absence of noise or artifacts. The image shows no visible outliers, suggesting consistent accuracy across the surface. Most significantly, there is no discernible presence of the orthogonal fringe pattern that would have been strikingly present in the input image, highlighting VUNet’s ability to reconstruct the underlying surface geometry without retaining or passing through the pattern information.

Figure 5 presents a comparison between the ground truth and predicted depth maps for a randomly selected sample. The left images display the ground truth depth map, while the right images show the depth map predicted by our VUDNet model. The predicted depth map is smoother than the original, due to the regularization term. Nevertheless, the estimates still retain a high degree of fine detail. The visual comparison suggests that the model effectively captured the essential features of the surface, aligning well with the expected characteristics observed in the ground truth data. This smoothing effect helps to manage the noise and artifacts typically associated with specular surfaces.

The t-SNE (t-distributed Stochastic Neighbor Embedding) method [53] is a powerful dimensionality reduction technique commonly used to visualize high-dimensional data in a two- or three-dimensional space. By applying t-SNE, we can effectively observe how well-separated the latent space is, learned by the VAE. Figure 6 demonstrates such a separation of the representation space, whereby the clustering effect indicates that the VAE successfully formed a coherent representation space where intrinsic features of different surfaces are well captured and recognizable. In this context, a cluster in the VAE’s representation space refers to a grouping of data points (input images) that share similar underlying features. Ideally, cluster separation is evidence of meaningful feature learning, as it implies that the VAE is able to effectively distinguish between various surface characteristics.

In Figure 6, the well-defined clusters suggest that the VAE can identify and learn the underlying patterns in the data, leading to potent feature extraction. This capability is essential for generalizing the network’s performance across diverse and complex surface geometries.

Figure 7 builds on Figure 6, but now based on a network trained exclusively on the deformation data as opposed to the entire dataset in Figure 6. The scattering of the samples based on their features shows that similar data points (similar surface shapes) are located near to each other in the latent space. This proximity indicates that the VAE can effectively group similar surfaces, enhancing the network’s ability to perform accurate depth estimation. The close grouping of similar samples demonstrates the VAE’s proficiency in feature extraction, ensuring that surfaces with similar characteristics are represented similarly.

The VAE’s ability to develop such clusters and similarity behaviours signifies its strength in capturing and representing the essential features of the input data. This representation plays a critical role in the subsequent stages of depth estimation, where accurate and reliable feature extraction is paramount.

Although our method employs an ensemble of a VAE and U-Net, we focus on visualizing the latent representation of the VAE. As highlighted in prior research [32], VAEs are designed to learn compact, informative latent spaces [54,55,56] that capture the underlying structure of the data. In contrast, U-Net primarily performs pixel-level refinement, working directly in the image space without explicitly learning a latent representation [33]. Therefore, the U-Net’s contribution is more appropriately assessed through its impact on the final depth maps, rather than through latent space visualization.

Table 1 demonstrates the remarkable performance of VUDNet. The quantitative metrics presented in the table demonstrate the network’s capability to achieve high accuracy and consistency in depth estimation. Specifically, the VUDNet model shows significantly lower Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and LogError compared to the comparator DYNet++ [17] and D-UNet [29] models. The lower errors across all metrics, at the same time at reduced computational complexity, highlight the superior performance of the VUDNet’s hybrid architecture in providing precise, reliable, and efficient 3D surface reconstructions.

The effective combination of VAE and U-Net components, along with the MLP ensemble, allows VUDNet to capture intricate surface details while maintaining overall structural coherence. The distinct separation in the VAE representation space indicates successful feature extraction, validating the model’s ability to generalize across diverse and complex surface geometries.

6. Conclusions

In this paper, we introduced VUDNet, an innovative deep neural network designed for the end-to-end 3D reconstruction of specular free-form surfaces using single-shot deflectometry. Our approach uniquely combines the strengths of both generative and discriminative models by integrating a Variational Autoencoder (VAE) and a modified U-Net. This hybrid architecture excels in depth estimation and detail refinement, providing high-fidelity reconstructions that surpass the performance of existing methods. Our dataset, which allows for network training on a diverse range of scenarios, will be made publicly available to support future work in this domain.

Experimental results demonstrate that VUDNet achieves incredible reconstruction accuracy from single-shot 2D images, with an RMSE error one-fourth as large (or better) and at reduced computational complexity in comparison to competing state-of-the-art methods. The integration of both MSE and TV regularization terms in the loss function promotes spatial smoothness and reduces noise, leading to visually coherent depth maps. The findings of this study set a new standard for single-shot deflectometry, showcasing the potential of deep learning in advancing optical metrology for specular surfaces.

In future work, we plan to explore the application of VUDNet to other forms of surface measurement and extend our approach to dynamic and real-time scenarios. Additionally, we aim to investigate the integration of more advanced generative models to further improve the robustness of depth estimation in diverse and complex environments.