Convincing 3D Face Reconstruction from a Single Color Image under Occluded Scenes

Abstract

The last few years have witnessed the great success of generative adversarial networks (GANs) in synthesizing high-quality photorealistic face images. Many recent 3D facial texture reconstruction works often pursue higher resolutions and ignore occlusion. We study the problem of detailed 3D facial reconstruction under occluded scenes. This is a challenging problem; currently, the collection of such a large scale high resolution 3D face dataset is still very costly. In this work, we propose a deep learning based approach for detailed 3D face reconstruction that does not require large-scale 3D datasets. Motivated by generative face image inpainting and weakly-supervised 3D deep reconstruction, we propose a complete 3D face model generation method guided by the contour. In our work, the 3D reconstruction framework based on weak supervision can generate convincing 3D models. We further test our method on the MICC, Florence and LFW datasets, showing its strong generalization capacity and superior performance.

1. Introduction

Single-view 3D face reconstruction refers to obtaining a user-specific 3D face surface model given one input face image. This is a classical and fundamental problem in computer vision [1,2,3]. It has a wide range of applications, such as 3D-assisted face recognition [1,4,5,6] and digital entertainment [7]. Existing methods mainly concentrate on reconstructing beautiful textures and ignore geometric details. At the same time, these methods can only work effectively when frontal faces are unobstructed, which makes the application of scenes very limited. When considering the occlusion of the scene, the reconstruction of the 3D face model is challenging since part of the facial features is not visible.

In recent years, due to the rapid development of deep learning methods, similar face inpainting tasks have made significant breakthroughs [8,9]. By comparison, because deep learning methods cannot be applied to 3D structures end-to-end, 3D reconstruction methods have remained far behind [10].

In 1999, Blanz and Vetter proposed early 3D morphable models (3DMM) [1,11,12], and the field of 3D face reconstruction using a single image opened. These approaches are based on automated template matching robust reproduction results within 1–5 min. However, due to the constraint space’s existence, the model’s performance is still lacking in competitiveness in terms of the expressiveness of geometric details [13]. At the same time, 3DMM and other methods also cannot deal with the situation in which faces occlude scenes robustly (especially the texture). These methods generally indiscriminately reconstruct the occluded field of face. Unlike previous arts, we propose a method designed to attain both goals: detailed 3D face reconstruction and robustness to occlusions (Figure 1). How did we do it? With the assistance of the face parsing framework, face contour map and deep learning method, we find a way to identify the occluded area and reconstruct the input image to an accurate 3D face model.

The main contributions are summarized as follows:

• We propose a novel approach that combines the face parsing approach and face contour map to generate a face with complete facial features.
• Face occlusion is a common problem. In response to the problem of an invisible face area under occluded scenes, we propose synthesizing the input face image based on GANS rather than reconstructing the 3D face directly.
• We improved the loss function of our 3D face reconstruction framework for occluded scenes. Our results (especially the face texture) are more accurate than other recent methods.

2. Related Work

2.1. Single-View 3D Face Shape Prediction

When it comes to 3D face reconstruction, the classic methods use reference 3D face models to fit the input face photo. The first step is face alignment. Face alignment, which fits a face model to an image and extracts the fiducial facial landmarks, has many solutions in the CV community. These solutions including the active appearance model [14,15,16] and the constrained local model [17,18,19]. Besides traditional models, some recent techniques use convolutional neural networks (CNNs) to regress landmark locations with the raw face image [20,21,22].

The second step is to solve the nonlinear optimization function to regress the 3DMM coefficients [1]. Some recent techniques firstly used CNNs to predict the 3DMM parameters with the input face image [2,23,24]. Some works proposed a cascaded CNN structure to regress the accurate 3DMM shape parameters [25,26,27,28,29]. Some frameworks explored the end-to-end CNN architectures to regress 3DMM coefficients directly. Each calculation usually takes a long time because the dimensionality of the data is very high [30].

2.2. Face Parsing

A face parsing map generally serves as an intermediate representation for conditional face image generation [31]. In addition, the image-to-image GAN model can learn the mapping from the semantic map to realistic RGB image [32,33,34,35]. In the pixel-level image semantic segmentation methods based on deep learning, fully convolutional networks (FCN) [36] is the well-known baseline for generic images which analyze per-pixel feature. Following this work, the DeepLab approaches [37,38,39,40] have achieved impressive results. The main feature of the series is to use dilated convolution instead of traditional convolution. However, directly applying these frameworks for face parsing may fail to map the varying-yet-concentrated facial features, especially hair, leading to poor results. A workable solution should directly predict per-pixel semantic label across the entire face photo. Wei et al. [41] proposed a novel method for regulating receptive fields with superior regulation ability in parsing networks to access accurate parsing map. MaskGan [42] contributed a labeled face dataset [43]. Zhou et al. [44] proposed an architecture that explores how to combine the fully convolutional network model and super-pixel data to model together. In order to solve the question of global image information access restriction, some methods [45] have introduced the transformer component and achieved state-of-the-art results. The semantic layout guides the location and appearance of facial features and further facilitates the training. The majority of face parsing methods work require semantic labels. Hence, these frameworks [42,46,47,48,49,50,51] usually train on the CelebA and Helen datasets, which contain labeled attributes.

2.3. Generative Adversarial Networks

Generative adversarial networks (GANs) [52] generally consist of a generator and a discriminator. The two components compete with each other. Since GANs can generate realistic images, GANs have been successfully applied to various face image synthesis tasks, such as image manipulation [53], image-to-image translation [54], image inpainting [55] and texture blend [56,57,58]. For example, the face images generated by Stylegan2 [59] can be confused with the real. With continuous improvements in regularization [60], users can control the synthesis by feeding the generator with conditioning information instead of noise. Our work was built on conditional GANs [61] with face parsing map inputs, which aims to tackle facial reconstruction under occluded scenes.

2.4. Face Image Synthesis

Deep pixel-level face generating has been studied for several years. Many methods [46,62,63,64,65] have achieved remarkable results. Context encoder [66] is the first deep learning network designed for image inpainting with the encoder–decoder architecture. Nevertheless, the networks do a poor job in dealing with human faces. Following this work, Yang et al. [35] used a modified VGG network [67] to improve the result of the context encoder by minimizing the feature difference of the photo background. Dolhansky et al. [68] demonstrated the significance of exemplar data for inpainting. However, this method only focuses on filling in missing eye regions of the frontal face, so it does not generalize well. EdgeConnect [69] shows impressive proceeds, disentangling generation into two stages: edge generator and image completion network. Contextual attention [70] takes a similar two-step approach. First, it produces a base estimate of the invisible region. Next, the refinement block sharpens the photo by background patch sets. The typical limitations of current face image generate schemes are the necessity of manipulation, the complexity of fundamental architectures, the degradation in accuracy, and the inability of restricting modification to local region.

3. Our Method

We propose a detailed 3D face reconstruction method (as shown in Figure 1) based on a single photo that consists of two stages:

• In response to the occlusion area, synthesizing the 2D face with complete facial features.
• Detailed 3D shape reconstruction module based on unobstructed frontal images.

Our goal is to realize detailed 3D face shape reconstruction under occluded scenes using our method. Given a source color face image ${\mathbf{I}}_{\mathbf{ori}}\in {\mathbb{R}}^{H\times W\times 3}$ with obstructions, we obtain the final 3D face model.

3.1. Face Mask Generation

As the first step of our 3D reconstruction framework, we need to identify the occluded area for generating the face mask image (1 for the occluded region, 0 for background) for the next task. Inspired by traditional face parsing tasks, as shown in Figure 2, given a square image ${\mathbf{I}}_{\mathbf{ori}}\in {\mathbb{R}}^{H\times W\times 3}$ of the face under occlusion, we applied the trained face mask generator ${\mathcal{N}}_{mask}$ to obtain the face mask ${\mathbf{I}}_{\mathbf{mask}}\in {\mathbb{R}}^{H\times W\times 1}$. This mask generation task is very similar to the traditional face parsing task. Our face mask generator is partly inspired by the annotated face dataset CelebAMask-HQ [42]. We trained an encoder–decoder module ${\mathcal{N}}_{mask}$ based on U-Net [71] to predict the occluded region.

3.2. Face Image Synthesis with GANs

Our face image synthesis module is guided by the contour. First, we need to predict the contours ${\mathbf{C}}_{\mathbf{syn}}\in {\mathbb{R}}^{H\times W\times 1}$ of facial features in the occluded area. We assume that the final unobstructed face image is ${\mathbf{I}}_{\mathbf{fina}}\in {\mathbb{R}}^{H\times W\times 3}$ and the ground truth image without obstruction ${\mathbf{I}}_{\mathbf{true}}\in {\mathbb{R}}^{H\times W\times 3}$. In the training set, the corresponding complete contour image and gray image are ${\mathbf{C}}_{\mathbf{true}}\in {\mathbb{R}}^{H\times W\times 1}$ and ${\mathbf{I}}_{\mathbf{gray}}\in {\mathbb{R}}^{H\times W\times 1}$. We trained the contour generator ${\mathcal{N}}_{cont}$ to predict the contour map for the occluded region.

$$\left\{\begin{array}{cc}\hfill & {\mathbf{C}}_{\mathbf{syn}}={\mathcal{N}}_{cont}\left({\widehat{\mathbf{I}}}_{\mathbf{gray}},{\mathbf{I}}_{\mathbf{mask}},{\widehat{\mathbf{C}}}_{\mathbf{true}}\right)\hfill \\ \hfill & {\widehat{\mathbf{I}}}_{\mathbf{gray}}={\mathbf{I}}_{\mathbf{gray}}\odot \left(1-{\mathbf{I}}_{\mathbf{mask}}\right)\hfill \\ \hfill & {\widehat{\mathbf{C}}}_{\mathbf{ture}}={\mathbf{C}}_{\mathbf{ture}}\odot \left(1-{\mathbf{I}}_{\mathbf{mask}}\right)\hfill \end{array}\right.$$

(1)

where ${\widehat{\mathbf{I}}}_{\mathbf{gray}}$ denotes the masked grayscale image, ${\widehat{\mathbf{C}}}_{\mathbf{ture}}$ denotes the masked contour image and ⊙ denotes the Hadamard product. We trained the discriminator of the module ${\mathcal{N}}_{cont}$ to predict which of ${\mathbf{C}}_{\mathbf{syn}}$ and ${\mathbf{C}}_{\mathbf{ture}}$ is a true contour map and which is a false contour map. The adversarial loss is defined as

$${\mathcal{L}}_{adv}={\mathbb{E}}_{\left({\mathbf{C}}_{\mathbf{ture}},{\mathbf{I}}_{\mathbf{gray}}\right)}\left[log{D}_1\left({\mathbf{C}}_{\mathbf{true}},{\mathbf{I}}_{\mathbf{gray}}\right)\right]+{\mathbb{E}}_{\left({\mathbf{I}}_{\mathbf{gray}}\right)}log\left[1-D_1\left({\mathbf{C}}_{\mathbf{syn}},{\mathbf{I}}_{\mathbf{gray}}\right)\right]$$

(2)

where $\mathbb{E}$ denotes the expected value of the function, and $D_1$ denotes the discriminator of the adversarial loss function.

In addition, we compare the feature activation maps of the discriminator. We set the face feature matching loss as

$${\mathcal{L}}_{ffea}=\mathbb{E}\left[\sum _{i=1}^K\frac{1}{N_i}∥D_1^{\left(i\right)}\left({\mathbf{C}}_{\mathbf{true}}\right)-D_1^{\left(i\right)}\left({\mathbf{C}}_{\mathbf{syn}}\right)∥\right]$$

(3)

where $N_i$ is the number of elements in the ith activation layer, K is the final convolution layer of the discriminator and $D_1^{\left(i\right)}$ is the activation in the i-th layer of the discriminator.

After obtaining the complete contour map, we design ${\mathcal{N}}_{syn}$ to generate the complete face image ${\mathbf{I}}_{\mathbf{fina}}$. The complete contour map ${\mathbf{C}}_{\mathbf{fina}}$ is formed by adding ${\mathbf{C}}_{\mathbf{syn}}$ and ${\mathbf{C}}_{\mathbf{true}}$, which follows ${\mathbf{C}}_{\mathbf{fina}}={\mathbf{C}}_{\mathbf{true}}\odot (1-{\mathbf{I}}_{\mathbf{mask}})+{\mathbf{C}}_{\mathbf{syn}}\odot {\mathbf{I}}_{\mathbf{mask}}$. In the map ${\mathbf{C}}_{\mathbf{fina}}$, we can see the contours of all facial features, especially the occluded areas. In addition, we set ${\widehat{\mathbf{I}}}_{\mathbf{true}}\in {\mathbb{R}}^{H\times W\times 3}$ to be an incomplete face picture, which follows ${\widehat{\mathbf{I}}}_{\mathbf{true}}={\mathbf{I}}_{\mathbf{true}}\odot (1-{\mathbf{I}}_{\mathbf{mask}})$. So, we utilize ${\mathcal{N}}_{syn}$ to obtain the final complete face image ${\mathbf{I}}_{\mathbf{fina}}$, with occluded regions recovered, which follows ${\mathbf{I}}_{\mathbf{fina}}={\mathcal{N}}_{syn}({\widehat{\mathbf{I}}}_{\mathbf{true}},{\mathbf{C}}_{\mathbf{fina}})$.

We trained the module ${\mathcal{N}}_{syn}$ to predict the final complete face image ${\mathbf{I}}_{\mathbf{fina}}$ over a joint loss. The adversarial loss is defined as

$${\mathcal{L}}_{sar}={\mathbb{E}}_{\left({\mathbf{I}}_{\mathbf{true}},{\mathbf{C}}_{\mathbf{fina}}\right)}\left[log{D}_2\left({\mathbf{I}}_{\mathbf{true}},{\mathbf{C}}_{\mathbf{fina}}\right)\right]+{\mathbb{E}}_{\left({\mathbf{C}}_{\mathbf{fina}}\right)}log\left[1-D_2\left({\mathbf{I}}_{\mathbf{fina}},{\mathbf{C}}_{\mathbf{fina}}\right)\right]$$

(4)

The per-pixel loss [72] is defined as follows:

$${\mathcal{L}}_{pix}=\frac{1}{{\mathrm{S}}_m}{∥{\mathbf{I}}_{\mathbf{fina}}-{\mathbf{I}}_{\mathbf{true}}∥}_1$$

(5)

where $S_m$ denotes the size of the face mask ${\mathbf{I}}_{\mathbf{mask}}$, and ${∥·∥}_1$ denotes the $L_1$ norm. Notice that we use the mask size $S_m$ as the denominator to adjust the penalty.

The style loss [73] computes the style distance between two face images as follows

$${\mathcal{L}}_{styl}=\sum _n\frac{1}{Q_n\times Q_n}{∥\frac{G_n({\mathbf{I}}_{\mathbf{fina}}\odot (1-{\mathbf{I}}_{\mathbf{mask}}))-G_n\left({\widehat{\mathbf{I}}}_{\mathbf{true}}\right)}{Q_n\times H_n\times W_n}∥}_1$$

(6)

where $G_n\left(x\right)={\phi }_n{\left(x\right)}^T{\phi }_n\left(x\right)$ denotes the gram matrix corresponding to ${\phi }_n\left(x\right)$, and ${\phi }_n(·)$ denotes the $Q_n$ feature maps with the size $H_n\times W_n$ of the n-th layer.

In summary, the contour generator network ${\mathcal{N}}_{cont}$ was trained with an objective comprised of an adversarial loss and feature-matching loss

$$\underset{G_1}{\min }\underset{D_1}{\max }{\mathcal{L}}_{G_1}=\underset{G_1}{\min }\left({\lambda }_{adv}\underset{D_1}{\max }{\mathcal{L}}_{adv}+{\lambda }_{ffea}{\mathcal{L}}_{ffea}\right)$$

(7)

The total loss function of ${\mathcal{N}}_{syn}$ follows

$$\underset{G_2}{\min }\underset{D_2}{\max }{\mathcal{L}}_{G_2}={\lambda }_{sar}\underset{D_2}{\max }{\mathcal{L}}_{sar}+{\lambda }_{pix}{\mathcal{L}}_{pix}+{\lambda }_{styl}{\mathcal{L}}_{styl}$$

(8)

where we set ${\lambda }_{adv}=1$, ${\lambda }_{ffea}=11.5$, ${\lambda }_{sar}=0.1$, ${\lambda }_{pix}=1$ and ${\lambda }_{styl}=250$, respectively. The values of these weights refer to the method of Lee et al. [42].

3.3. 3D Shape Model

A 3DMM consists of three model parts: the shape, texture and camera models. Let us denote the 3D shape and texture of an object with n vertices as a $3n\times 1$ vector:

$$\mathbf{S}=\left(x_1,y_1,z_1,\dots ,x_n,y_n,z_n\right)$$

(9)

$$\mathbf{T}=\left(r_1,g_1,b_1,\dots ,r_n,g_n,b_n\right)$$

(10)

where ${\mathbf{S}}_{\mathbf{i}}=\left(X_i,Y_i,Z_i\right)$ denotes the object-centered shape vector of the i-th vertex, and ${\mathbf{T}}_{\mathbf{i}}=\left(r_i,g_i,b_i\right)$ denotes the texture vector of the i-th vertex.

The face model to be solved can be weighted and combined by the m face model in the dataset:

$$\left\{\begin{array}{c}{\mathbf{S}}_{\mathbf{mod}}={\displaystyle \sum _{i=1}^m}{\mathit{\alpha }}_{\mathbf{i}}{\mathbf{S}}_{\mathbf{i}}\\ {\mathbf{T}}_{\mathbf{mod}}={\displaystyle \sum _{i=1}^m}{\mathit{\beta }}_{\mathbf{i}}{\mathbf{T}}_{\mathbf{i}}\\ {\displaystyle \sum _{i=1}^m}{\mathit{\alpha }}_{\mathbf{i}}={\displaystyle \sum _{i=1}^m}{\mathit{\beta }}_{\mathbf{i}}=1\end{array}\right.$$

(11)

where $\mathit{\alpha }$, and $\mathit{\beta }$ denote the weighting coefficient of the face model.

However, the basis vectors here are not orthogonally related. We normally use the following formula when building the model:

$${\mathbf{S}}_{\mathbf{mod}}=\overline{\mathbf{S}}+\sum _{i=1}^{m-1}\widetilde{{\mathit{\alpha }}_{\mathbf{i}}}\widetilde{{\mathbf{S}}_{\mathbf{i}}},{\mathbf{T}}_{\mathbf{mod}}=\overline{\mathbf{T}}+\sum _{i=1}^{m-1}\widetilde{{\mathit{\beta }}_{\mathbf{i}}}\widetilde{{\mathbf{T}}_{\mathbf{i}}}$$

(12)

where $\overline{\mathbf{S}}$ and $\overline{\mathbf{T}}$ denote the average shape and average texture, $\widetilde{{\mathit{\alpha }}_{\mathbf{i}}}\in {\mathbb{R}}^{80},\widetilde{{\mathit{\beta }}_{\mathbf{i}}}\in {\mathbb{R}}^{80}$ denote the eigenvalue of the covariance matrix arranged in descending order by the value, and $\tilde{\mathbf{S}},\tilde{\mathbf{T}}$ denote the eigenvector of the shape and texture covariance matrix.

In fact, only the first few components of $\tilde{\mathbf{S}}$ and $\tilde{\mathbf{T}}$ need to be selected to make a better approximation to the face sample. Not only can the number of parameters that needs to be estimated be greatly reduced, but the accuracy will not be significantly reduced. We describe the basic 3D face space with PCA:

$${\mathbf{S}}_{\mathbf{basi}}=\overline{\mathbf{S}}+{\mathbf{A}}_{\mathbf{id}}{\mathit{\alpha }}_{\mathbf{id}}+{\mathbf{B}}_{\mathbf{exp}}{\mathit{\beta }}_{\mathbf{exp}},\mathbf{T}=\overline{\mathbf{T}}+{\mathbf{B}}_{\mathbf{t}}{\mathit{\beta }}_{\mathbf{t}}$$

(13)

where ${\mathbf{A}}_{\mathbf{id}}$, ${\mathbf{B}}_{\mathbf{exp}}$ and ${\mathbf{B}}_{\mathbf{t}}$ denote the PCA bases of identity, expression and texture, ${\mathit{\alpha }}_{\mathbf{id}}\in {\mathbb{R}}^{80}$ and ${\mathit{\beta }}_{\mathbf{exp}}\in {\mathbb{R}}^{64}$, and ${\mathit{\beta }}_{\mathbf{t}}\in {\mathbb{R}}^{80}$ are the corresponding 3DMM coefficient vectors. We adopt the Basel Face Model (BFM) [12]. It is a publicly available 3DMM dataset for a single view face model.

3.4. Camera and Illumination Model

After the 3D face is reconstructed, it can be projected onto the image plane with the perspective projection

$$V_{2d}\left(\mathbf{P}\right)=f\times {\mathbf{P}}_{\mathbf{r}}\times \mathbf{R}\times {\mathbf{S}}_{\mathbf{mod}}+{\mathbf{t}}_{\mathbf{2}\mathbf{d}}$$

(14)

where $V_{2d}\left(\mathbf{P}\right)$ denotes the projection function that turned the 3D model into 2D face positions, f denotes the scale factor, ${\mathbf{P}}_{\mathbf{r}}$ denotes the projection matrix, $\mathbf{R}\in SO\left(3\right)$ denotes the rotation matrix, and ${\mathbf{t}}_{\mathbf{2}\mathbf{d}}\in {\mathbb{R}}^3$ denotes the translation vector.

Therefore, we approximated the scene illumination with spherical harmonics (SH) [74,75,76,77] parameterized by coefficient vector $\gamma \in {\mathbb{R}}^9$. In summary, the unknown parameters to be learned can be denoted by a vector $y=({\mathit{\alpha }}_{\mathbf{id}},{\mathit{\beta }}_{\mathbf{exp}},{\mathit{\beta }}_{\mathbf{t}},\gamma ,\mathbf{p})\in {\mathbb{R}}^{239}$, where $\mathbf{p}\in {\mathbb{R}}^6=\{\mathbf{pitch},\mathbf{yaw},\mathbf{roll},f,{\mathbf{t}}_{\mathbf{2}\mathbf{D}}\}$ denote face poses. In this work, we used a fixed ResNet-50 [78] network to regress these coefficients. We used a coarse-to-fine network based on the graph convolutional networks of Lin et al. [79] for producing the fine texture ${\mathbf{T}}_{\mathbf{fin}}$.

3.5. Loss Function of Shape Reconstruction

Given a synthetic face photo ${\mathbf{I}}_{\mathbf{fina}}$, we used the ResNet to regress the corresponding coefficient y. Because the collection of large scale high-resolution 3D texture datasets is still very costly and scarce, the ResNet was trained under weak supervision. The corresponding loss function consists of four parts:

$${\mathcal{L}}_{shape}={\lambda }_{feat}{\mathcal{L}}_{feat}+{\lambda }_{regu}{\mathcal{L}}_{regu}+{\lambda }_{phot}{\mathcal{L}}_{phot}+{\lambda }_{land}{\mathcal{L}}_{land}$$

(15)

The second term is a regularizer, and the other terms are data terms. We used fixed ${\lambda }_{•}$ values to weigh the losses. Here, we set ${\lambda }_{feat}=0.2$, ${\lambda }_{regu}=3.6\times {10}^{-4}$, ${\lambda }_{phot}=1.4$, ${\lambda }_{land}=1.6\times {10}^{-3}$, respectively, in all our experiments. The values of these weights refer to the method of Deng et al. [77].

Face Features Level Consistency [77,79,80]. Face recognition is a very mature research area. In order to measure the difference between the 3D face and the two-dimensional face, we introduced the loss at face features level. The face features level consistency measures the difference between the 2D input image ${\mathbf{I}}_{\mathbf{fina}}$ and rendered image ${\mathbf{I}}_{\mathbf{j}}$.

$${\mathcal{L}}_{feat}=1-\frac{<F\left({\mathbf{I}}_{\mathbf{fina}}\right),F\left({\mathbf{I}}_{\mathbf{j}}\right)>}{∥F\left({\mathbf{I}}_{\mathbf{fina}}\right)∥·∥F\left({\mathbf{I}}_{\mathbf{j}}\right)∥}$$

(16)

where $F(·)$ denotes the feature extraction function by FaceNet [81], and $<·,·>$ denotes the inner product.

Regularization Consistency [77]. To prevent shape deformation, we introduce the prior distribution to the parameters of the 3DMM face model. We add the regularization consistency on the regressed 3DMM coefficients.

$${\mathcal{L}}_{regu}={\omega }_{\alpha }{∥\widetilde{{\mathit{\alpha }}_{\mathbf{i}}}∥}^2+{\omega }_{\beta }{∥\widetilde{{\mathit{\beta }}_{\mathbf{i}}}∥}^2$$

(17)

here, we set ${\omega }_{\alpha }=1.0$, ${\omega }_{\beta }=1.75\times {10}^{-3}$ respectively.

Photometric Consistency [11,82,83,84]. As a common weak supervision method, it is easy to think of the dense photometric discrepancy. The rendering module renders back an image ${\mathbf{I}}_{\mathbf{j}}^{{}^{\left(i\right)}}$ to compare with the image ${\mathbf{I}}_{\mathbf{fina}}^{{}^{\left(i\right)}}$.

$${\mathcal{L}}_{phot}\left(y\right)=\frac{{\sum }_{i\in \Psi }{Z}_i·{∥{\mathbf{I}}_{\mathbf{fina}}^{\left(i\right)}-{\mathbf{I}}_{\mathbf{j}}^{\left(i\right)}∥}_2}{{\sum }_{i\in \Psi }{Z}_i}$$

(18)

where i denotes the pixel index, $\psi$ is the reprojected face region which was obtained with landmarks [85], ${∥·∥}_2$ denotes the $L_2$ norm, and $Z_i$ is the occlusion attention coefficient which is described as follows.

To gain robustness to accurate texture, we set

$Z_i=\left\{\begin{array}{c}1\mathrm{where}\mathrm{the}\mathrm{reconstructed}\mathrm{mesh}\mathrm{projects}\mathrm{to}\hfill \\ 0.1\mathrm{otherwise}\hfill \end{array}\right.$ for each pixel i.

Landmark-wise Consistency [77,86,87]. As landmarks convey the topological information of the face, we ran the faceboxes toolbox to predict 68 landmarks $\mathbf{P}\in {\mathbb{R}}^{68}$ as the reference. We compared the 2D landmarks of ${\mathbf{I}}_{\mathbf{fina}}$ with sparse vertices of the reconstruction which correspond to these landmarks. We attained the landmarks $\mathbf{L}\in {\mathbb{R}}^{68}$ from the landmark vertex.

$${\mathcal{L}}_{land}=\frac{1}{N}\sum _{k=0}^N{∥{\mathbf{P}}_{\mathbf{k}}-{\mathbf{L}}_{\mathbf{k}}∥}_2^2$$

(19)

Here, $N=68$, ${\mathbf{L}}_{\mathbf{k}}$ denotes the 2D projection of the kth landmark vertex, and ${∥·∥}_2$ denotes the $L_2$ norm.

4. Implementation Details

Our mask generation process is very similar to the traditional face parsing process. Considering the generation of the training dataset of ${\mathcal{N}}_{mask}$, we adopted the CelebA-HQ dataset, a high-quality version of CelebA that consists of $\mathrm{30,000}$ images at $1024\times 1024$ resolution, each having a segmentation mask and sketch. We designed ${\mathcal{N}}_{mask}$ with U-Net [71] as the backbone.

To obtain ${\mathbf{C}}_{\mathbf{fine}}\in {\mathbb{R}}^{H\times W\times 1}$, we generate contour maps using the Canny toolbox [88] as the training dataset. The sensitivity of the Canny toolbox is regulated by the standard deviation of the Gaussian smoothing filter $\delta$. In our work, we analytically found that $\delta \approx 1.8$ yields the best results. Our proposed network is implemented in PyTorch. We used $256\times 256$ images with a batch size of ten to train the model of ${\mathcal{N}}_{\mathrm{cont}}$. To train ${\mathcal{N}}_{syn}$, we followed the design of Pix2PixHD [35] with four residual blocks. The network is trained using $512\times 512$ images with a batch size of 12. Before training the ResNet, as an initialization, we take the weights from pre-trained R-Net [77]. We set the input image size to $224\times 224$. Our texture refinement network is designed according to the method of Lin et al. [79].

5. Experimental Results

5.1. Qualitative Comparisons with Recent Arts

Figure 3 shows our results compared with the other work. The last columns show our results. The remaining columns demonstrate the results of 3DDFA [89], PRNet [30], and DF${}^2$Net [90] (Chen et al. [91]). Our results show that our results have better handled the occlusion area than other methods. Figure 3 shows that our method can reconstruct a complete face shape with geometry details under occlusion scenes, such as glasses, food and fingers. The approach of 3DDFA was aimed at extremely large poses. Therefore, it cannot reconstruct a correct face texture under occluded scenes. Other methods focused on generating high-resolution face textures rather than distinguishing occluders. At the same time, it must also be pointed out that other methods do not set up a dedicated de-occlusion component and, therefore, do not perform well under occlusion scenarios.

5.2. Ablation Study

In this section, we define the ablation study as a scientific examination of a deep learning system by removing its loss function blocks in order to gain insight into their effects on its overall performance. Here, we present various ablation results on the MICC and FaceWarehouse datasets [92,93]. The MICC dataset contains challenging face models of 53 subjects. For the test set, we use 90 identities with various expressions from the FaceWarehouse dataset.

Table 1. Average reconstruction errors (mm) on MICC datasets [92] and FaceWarehouse datasets [93] for ResNet trained with different loss combinations.“✓” denotes employed, while “−” denotes unemployed. Our total hybrid-level loss yields considerably higher accuracy than other baselines on the two datasets.

Loss Function				MICC	Face Warehous
${\mathcal{L}}_{\mathit{feat}}$	${\mathcal{L}}_{\mathit{regu}}$	${\mathcal{L}}_{\mathit{phot}}$	${\mathcal{L}}_{\mathit{land}}$
✓	−	−	✓	$1.83\pm 0.42$	$2.29\pm 0.25$
−	✓	✓	−	$1.90\pm 0.12$	$1.92\pm 0.29$
✓	−	✓	✓	$1.88\pm 0.33$	$1.90\pm 0.28$
−	✓	✓	−	$1.78\pm 0.40$	$1.88\pm 0.77$
✓	✓	✓	✓	$1.61\pm 0.73$	$1.79\pm 0.57$

shows that our ablation study produced various reconstruction evaluation results on the two datasets. Study results demonstrate that the best reconstruction results can be achieved only when the four loss functions are used fully.

5.3. Quantitative Comparison

5.3.1. Comparison Result on the MICC Florence Datasets

We evaluate the accuracy of the shape regression on the MICC Florence dataset [92]. The dataset is a 3D face dataset that contains 53 subjects with their ground truth 3D face scans. The ground truths are provided for 52 out of the 53 people. We artificially added some occluders (i.e., eyeglasses) as input. We calculated the average $90\%$ largest error between the generative model and the ground truth model. Figure 4 shows that our method can effectively handle occlusion.

5.3.2. Quantitative Comparison

The acceptance rate of face recognition is straightforward to think of as a reconstruction effect test method. Inspired by the method of Deng et al. [77], our choice of using the ResNet-50 to regress the shape coefficients ensure robustness. The basic shape is the cornerstone of our method, and we tested our approach on the Labeled Faces in the Wild datasets (LFW) [94]. Test system setting details followed the approach of Anh et al. [6].

The left of Figure 5 shows the comparison of the method of sensitivity of our method and the approach of Sela et al. It can be easily discovered that a method of Sela et al. cannot reconstruct the occluded chin. The mistake may be because their method focuses more on local details than on the consistency of global shapes. It can be seen from Figure 5 that our method can generate a full face with the chin, which shows that this method can deal with occlusion robustly. Though 3DMM also limits the details of shape, we use it only as a foundation and add geometry details separately.

We further quantitatively verify the robustness of our method to occlusions.

Table 2. Quantitative comparison on LFW.

Method	100%-EER	Accuracy	nAUC
Tran et al.	$89.40\pm 1.52$	$89.36\pm 1.25$	$95.90\pm 0.95$
Our Shape and occlusions
Ours(w/Occ)	$83.89\pm 1.08$	$85.25\pm 0.85$	$89.75\pm 0.87$
Ours(w/o Occ)	$89.78\pm 1.21$	$90.33\pm 0.67$	$95.91\pm 0.64$

(top) reports the verification results on the LFW benchmark [2], with and without occlusions (see also ROC in Figure 5 (right)). Though occlusion does affect the accuracy, the decline of the curve is limited, demonstrating the robustness of our approach.

6. Conclusions

In this work, we describe a method capable of producing 3D face reconstructions with convincing texture from photos taken in occluded scenes. These occlusions include fingers, food that is about to enter the mouth, glasses, and so on. At the heart of our method is its weakly supervised design, which decouples the task of estimating a robust fundamental shape from the task of estimating its mid-level details, represented here as the bump maps. Comprehensive experiments have shown that our method outperforms previous methods by a large margin in terms of both accuracy and robustness. As part of our next step, we will try to use self-supervision to reconstruct the 3D face model.