3D Point Cloud Completion Method Based on Building Contour Constraint Diffusion Probability Model

Abstract

Building point cloud completion is the process of reconstructing missing parts of a building’s point cloud, which have been affected by external factors during data collection, to restore the original geometric shape of the building. However, the uncertainty in filling point positions in the areas where building features are missing makes it challenging to recover the original distribution of the building’s point cloud shape. To address this issue, we propose a point cloud generation diffusion probability model based on building outline constraints. This method constructs building-outline-constrained regions using information related to the walls on the building’s surface and adjacent roofs. These constraints are encoded by an encoder and fused into latent codes representing the incomplete building point cloud shape. This ensures that the completed point cloud adheres closely to the real geometric shape of the building by constraining the generated points within the missing areas. The quantitative and qualitative results of the experiment clearly show that our method performs better than other methods in building point cloud completion.

1. Introduction

Urban-scale real-world 3D construction is becoming an important direction in the digital era of urbanization, with widespread applications in smart cities, urban planning, and surveying [1,2,3]. 3D point cloud can map information about the geometric shape, surface features, and spatial distribution of buildings in the real world into digital data [4], making them a crucial data source for real-world 3D construction. However, laser LiDAR point clouds may have non-uniform density distribution due to laser scattering, and point clouds generated by photogrammetry can be influenced by lighting conditions, weather, equipment accuracy, and other factors, leading to potential differences in the accuracy and consistency of point cloud data. On the other hand, both of these methods can result in missing point cloud data in certain areas due to occlusion by objects, leading to incomplete geometric information about buildings. Therefore, to correctly fill in missing building point cloud data, point cloud completion has become a crucial duty as a preprocessing approach for point clouds [5].

Generative Adversarial Models (GAN) [6], Variational Auto-Encoders (VAE) [7], Normalization Flow (NF) [8], and Denoising Diffusion Probabilistic Models (DDPM) [9] have achieved significant success in content generation task. Building upon these generative models, research has explored methods for point cloud completion based on GANs [10,11], VAE models [12], flow-based models [13], and diffusion-based models [14]. While these methods have made progress in certain aspects, they also have inherent limitations. For instance, GAN-based methods can be challenging to train due to the need for training two networks, leading to unstable training processes and potential convergence issues. VAE-based models may generate building shapes that are not sufficiently smooth, partially attributed to injected noise and the use of element-wise measurements like the squared error. Training times for flow-based models can be lengthy, and they may not perform well in generating sparse point clouds. DDPM-based models, with their high stochasticity in the noise injection and denoising process during training, may struggle to deterministically control the generation of geometric point clouds in missing areas of buildings.

To deal with these problems, we propose the building contour constraint diffusion probabilistic model for complementing building point clouds. By constructing the contour-constrained shape latent codes for the missing regions, the constrained point cloud generates the diffusion probabilistic model point cloud generation process, so that the complementary point cloud of the missing regions is close to the geometric features of the original building. This method not only recovers missing parts from incomplete point cloud data but also maintains consistency with the building’s geometric characteristics and contours. Here is an overview of our approach: First, we use the RANSAC [15] to fit the building’s point cloud into multiple surface polygons. Next, we group individual buildings into one or more clusters based on predefined polygon correlation constraints. Then, we project the roof and wall polygons of clusters with missing areas onto the ground plane and use geometric transformations to construct closed polygons, forming contour-constrained regions. Finally, the extracted contour-constrained region is encoded into a latent code representing its shape distribution, which is then fused with the latent code of the incomplete building point cloud. This fused representation is used in the reverse diffusion process to complete the point cloud. Here are our primary contributions:

(1) For clustered segments with missing parts, we construct constraint regions by projecting wall contour lines and roof edge contour lines onto the ground plane.

(2) By encoding the constrained region’s point cloud and fusing the latent code of the incomplete building point cloud with the latent code of the building’s contour-constrained region, the point cloud’s missing regions are completed through the reverse process.

The rest of this essay is structured as follows: Section 2 presents related work. Section 3 introduces our model. Section 4 describes the experiments we conducted on the BuildingNet dataset for point cloud completion tasks in incomplete building point clouds. Section 5 concludes our paper.

2. Related Works

2.1. Traditional Point Cloud Completion Methods

Traditional point cloud completion methods can typically be categorized into geometry-based methods and alignment-based methods [16]. Geometry-based methods leverage the geometric features of the original point cloud for shape completion. In geometry-based point cloud completion methods, Berger et al. [5] used interpolation to fill gaps in the incomplete point cloud. Thrun et al. [17] based on regular structural repetitions, identified symmetry in the input point cloud, and predicted the missing pieces. These methods, are sometimes useful in specific situations; however, the input point cloud is required to have symmetry with a low missing rate and may result in the loss of some fine-grained details.

Alignment-based methods aim to achieve point cloud shape completion by finding matching primitives from a model database. In alignment-based point cloud completion methods, Li et al. [18] segmented building point clouds into planar patches and used an improved global fitting approach to estimate the parameters of corresponding building primitives, progressively completing the missing regions. Verdie et al. [19] employed geometric attributes and a set of semantic rules combined with Markov random fields to classify building scenes and fit corresponding planar structures to complete missing areas. Although these methods are effective, retrieving matching building primitives from a model library often requires significant computational resources, and maintaining the reliability and accuracy of retrieval necessitates regular updates to the model library to reflect new building elements and their variations.

2.2. Deep Learning-Based Point Cloud Completion Methods

Deep learning-based point cloud complementation models can learn features from the input point cloud to complement the point cloud of the complete shape of the target. They offer stronger generalization capabilities compared to traditional methods.

2.2.1. Voxel-Based Method

The point cloud complementation of voxelized point clouds can be accomplished using CNN. Wu et al. [20] introduced 3D ShapeNets, a method that utilizes Convolutional Deep Belief Networks to transform 3D geometric shapes into binary variable probability distributions on 3D voxel grids. The authors of [21] employed MRF model combined with both geometric and multi-view RGB data for point cloud completion, the authors of [22] created a complementary point cloud by combining the overall structural information and local geometric information of the input point cloud, and the authors of [23] fused 3D shape information through a volumetric deep neural network to complement the point cloud. GRNet [24] regularizes disordered point clouds into 3D meshes to better access the geometric structure and contextual information of the point clouds. SPU-Net [25] achieves the simultaneous acquisition of contextual information within local regions and between local regions by combining self-attention modules with GCN. It then utilizes a hierarchically learnable folding strategy to generate point clouds with a learnable two-dimensional grid. However, these methods usually require voxelization of the point cloud followed by a convolution operation to complement the point cloud, resulting in high computational cost.

2.2.2. Transformer-Based Methods

Transformer-based methods can determine how a point cloud’s local features and general structural characteristics relate to one another. Pointr [26] treats point cloud completion as a set-to-set transformation task by representing a point cloud as a set of unordered point proxies, employing an Encoder–Decoder structure to generate a point cloud of missing regions, and designing a geometry-aware block to explicitly model the geometric relationships. PMP-Net++ [27] introduces the Transformer network on the basis of PMP-Net [28], which makes the point moving path conform to the target structural features through the relationship between the overall structural features and local features of the point cloud, and significantly improves the performance of the point cloud complementation. However, compared to other methods, the Transformer-based method has a lot of parameters, needs expensive experimental equipment, and its attention process is hard to understand.

2.2.3. Point-Based Method

Point-based methods directly model each point in the point cloud.

Multi-layer perceptrons (MLPs) are used by PointNet [29] to learn how the distribution and relationships of the points in the point cloud relate to one another, creating new points in the missing regions to fill them, so that the filled point cloud maintains a consistent geometric shapes. This approach ensures that the completed point cloud maintains consistent geometric shapes. PointNet++ [30] introduced hierarchical layers for multi-scale feature aggregation, enabling the learning of multi-scale features of point clouds. To mitigate structural losses introduced by MLPs, AtlasNet [31] reconstructs complete outputs by evaluating a set of parameterized surface elements, achieving full point cloud generation. PCN [16] can directly perform feature learning directly in the point cloud space to complete missing point clouds. However, PCN only utilizes global features and does not consider local geometric features, making it challenging to capture the topological structures of fine-grained building details. FinerPCN [32] combines PCN and pointwise convolution to extract local information, progressively generating detailed and complete point cloud data. LAKe-Net [33] completes point clouds using a keypoint-skeleton-shape prediction approach by locating aligned keypoints. AGConv [34] enhances the flexibility of point cloud convolution by generating adaptive kernels based on dynamically learned features of points. This enables effective and precise capture of diverse relationships between points from different semantic parts, achieving adaptability within the convolution operation. Although point-based methods independently process points at the local level to maintain permutation invariance, they lose local features because they do not consider the geometric relationships between points and their positions.

The latent space representation of point clouds provides crucial feature information for 3D shape reconstruction. Cycle4Completion [35] allows models to learn how to completely complement a point cloud by learning the process of two simultaneous cyclic transformations of the potential space of a complete 3D shape and an incomplete 3D shape. ShapeInversion [36] introduces GAN into point cloud completion by searching for the latent code in a pre-trained GAN for the complete shape that best reconstructs the given input partial shape. It then uses gradient descent to backpropagate the loss function to update the latent code. PC-GAN [11] combines hierarchical and interpretable sampling with a combination of hierarchical Bayesian and implicit generative networks. This increases the model’s stability and extracts more expressive features to generate more consistent point clouds, thus improving the quality of generated point clouds. PF-Net [37] uses a multi-layer network based on feature points to generate point clouds, taking in-complete point clouds as inputs, and utilizing multi-stage point cloud complementation and adversarial loss to produce more realistic point clouds of missing regions, thus better preserving the individual features of objects. CP3 [38] introduces an Incomplete-to-Incomplete (IOI) sampling mechanism as inverse prompt function, which randomly cuts the incomplete point cloud into a new incomplete point cloud. This new incomplete point cloud is then fed into a pre-trained generative network to recover the original incomplete point cloud. While GAN architectures can utilize discriminator-guided implicit learning to estimate sets of points provided by the generator, the unordered nature of point clouds can result in differently positioned corresponding areas in 3D space, leading to incorrect completions. Additionally, training GANs on complex raw point sets (exceeding 2048 points) can be challenging due to the difficulty in handling point distribution.

VAE and DDPM are typically trained on complete 3D objects to determine the model’s weights for generating the latent representations of the corresponding point cloud data. Spurek et al. [12] proposed the HyperPocket variational autoencoder architecture, which uses the hypernetwork paradigm to fill the missing regions of the point cloud with different variants produced by the underlying representations. VRCNet [39] uses dual-path units and VAE-based relationship enhancement modules for probabilistic modeling of point clouds. S. Luo and W. Hu et al. [14] proposed a point cloud generation diffusion probability model that transforms the noise distribution resulting from multiple noisy additions to the point cloud into a target distribution Markov chain. The neural network learns this process to obtain optimal parameters for generating point clouds from Gaussian noise that conforms to the target data distribution. However, the diffusion model is a stochastic process and may result in partial deviations between the finally generated point cloud and the actual shape.

3. Methods

The structure of our proposed 3D point cloud generation diffusion probability model based on building contour constraints is shown in Figure 1. Four modules make up the model: Contour Constraint, Point encoding, Diffusion, and Reverse. The contour constraint module aims to extract the building contour constraint region $\left\{C,M\right\}$ of the missing part of the input point cloud and samples it as dense point clouds $X_C$ and $X_M$. The diffusion module involves gradually adding Gaussian noise to the input original building point cloud over time steps. The point-encoding module is divided into three parts: the first part encodes the input building point cloud into its shape latent code $h$; the second part encodes the dense point cloud $X_C$ and $X_M$ extracted from the contour constraint, which represents the building’s contour constraint region, into its shape latent codes $C$ and $M$; and the third step combines the shape latent codes obtained from the first two steps into $H$. The reverse module involves taking samples from the noise distribution $p\left(X^{\left(T\right)}\right)~\mathcal{N}\left(0,I\right)$, which is approximated to be similar to the point set $q\left(X^{\left(T\right)}\right)$. These samples are used as input to a network $\theta$ to estimate the denoising motion for each point at time step $t$, thereby reconstructing the overall building point cloud. In this context, the representation of building $Z$’s point cloud is denoted as $X^{(0)}={\left\{x_i^{(0)}\right\}}_{i=1}^N$.

3.1. Contour Constraint

Due to the complexity of the geometric structure of the building surface, extracting the geometric features of the building surface using deep learning requires a large amount of data for training, but training the deep learning model is complex and takes a long time. Therefore, we use the RANSAC [15] to fit the building’s point cloud into multiple surface polygons and divide it into one or more building blocks $\left\{\left.Z_1,\cdots ,Z_a\right\}\right.$. Then, we extract contour constraint regions for each group to address the missing parts. The extracted polygons are denoted as $V=\left\{\left.V_1,V_2,\cdots ,V_n\right\}\right.$, and the normal vectors for each polygon point outward are denoted as $\overrightarrow{n}=\left\{\left.{\overrightarrow{n}}_1,{\overrightarrow{n}}_2,\cdots ,{\overrightarrow{n}}_n\right\}\right.$ The centers of each polygon are represented as $G=\left\{\left.G_1,G_2,\cdots ,G_n\right\}\right.$. Figure 2 illustrates the process of constructing Contour Constraint.

3.1.1. Grouping of Building Surface Polygons

We define a polygon correlation constraint $E=\left[F_{Con},\rho \right]$ to group polygons $V$ based on the relationships between polygons. The polygon correlation constraint $E$ is composed of local convexity $F_{Con}$ [40] and minimum Euclidean distance $\rho$ [41]. The polygon correlation constraint $E$ is defined as follows:

$$E=\left\{\begin{array}{lll}True& ,& F_{Con\left(V_i-V_j\right)}=convex and {\rho }^{(V_i-V_j)}=0\\ False& ,& F_{Con\left(V_i-V_j\right)}=concave or {\rho }^{(V_i-V_j)}\ne 0\end{array}\right.,$$

(1)

The local convexity $F_{Con}$ and minimum Euclidean distance $\rho$ are defined as:

$$F_{Con\left(V_i-V_j\right)}=\left\{\begin{array}{c}convex\iff {\theta }_i>{\theta }_j\iff {\overrightarrow{n}}_i\cdot \overrightarrow{G_{i-j}}<{\overrightarrow{n}}_j\cdot \overrightarrow{G_{i-j}}\\ concave\iff {\theta }_i<{\theta }_j\iff {\overrightarrow{n}}_i\cdot \overrightarrow{G_{i-j}}>{\overrightarrow{n}}_j\cdot \overrightarrow{G_{i-j}}\end{array}\right.,$$

(2)

$${\rho }^{(V_i-V_j)}=\min \left\{\sqrt{{\left(x_p^{(V_i)}-x_q^{(V_j)}\right)}^2+{\left(y_p^{(V_i)}-y_q^{(V_j)}\right)}^2+{\left(z_p^{(V_i)}-z_q^{(V_j)}\right)}^2}\right\},$$

(3)

In the above two equations, $\overrightarrow{n}$ is the normal vector of the polygon, $\overrightarrow{G_{i-j}}$ is the vector in the direction from the center of the polygon $G_i$ to $G_j$, and $\theta =\arccos 〈\overrightarrow{n},\overrightarrow{G_{i-j}}〉$ is the angle between the polygon vector $\overrightarrow{n}$ and vector $\overrightarrow{G_{i-j}}$. $x_p$ and $x_q$ represent points on two different edges of the polygons. $i\ne j\in \left\{\left.1,2,\cdots ,n\right\}\right.$.

Utilizing Equation (2), we have illustrated a schematic representation of the convex-concave relationships among adjacent polygons in Figure 3.

Based on the aforementioned polygon correlation constraint $E=\left[\rho ,F_{Con}\right]$, the polygons with correlation are merged into an associated group, thereby grouping the building surface polygons $V$ into $\left\{\left.Z_1,\cdots ,Z_a\right\}\right.$.The actual incomplete building point cloud fitting results in plane grouping is shown in Figure 4.

3.1.2. Definition of Building Surface Polygon Attributes

Based on the grouping results of building surface polygons in Section 3.1.1, and in accordance with the standard Code for design of building foundation [42], which specifies that the wall inclination of buildings below 24 m should be less than 0.23°, we define the polygon attribute $F_V$ to determine whether the polygon attribute is a wall or a roof.

$$F_{V }=\left\{\begin{array}{lll}Wall& ,& \theta =\arccos \frac{{\overrightarrow{n}}_i\cdot {\overrightarrow{n}}_{horizon}}{\left|{\overrightarrow{n}}_i\right|\left|{\overrightarrow{n}}_{horizon}\right|}\in ({90}^{°}\pm {0.23}^{°})\\ Roof& ,& \theta =\arccos \frac{{\overrightarrow{n}}_i\cdot {\overrightarrow{n}}_{horizon}}{\left|{\overrightarrow{n}}_i\right|\left|{\overrightarrow{n}}_{horizon}\right|}\in \left[0^{°},{90}^{°}-{0.23}^{°}\right]\cup \left[{90}^{°}+{0.23}^{°},{180}^{°}\right]\end{array}\right.,$$

(4)

where ${\overrightarrow{n}}_i$ represents the normal vector of the building surface polygon, $i\in \left\{\left.1,2,\cdots ,n\right\}\right.$; ${\overrightarrow{n}}_{horizon}$ represents the normal vectors of the ground plane; and $\theta$ represents the angle between the building surface polygon and the ground plane.

Based on the building surface polygon attribute $F_V$, attributes are defined for each polygon contained within the individual building, and they are divided into wall polygons $\left\{\left.W^{\left(Z_1\right)},\cdots ,W^{\left(Z_a\right)}\right\}\right.$ and roof polygons $\left\{\left.R^{\left(Z_1\right)},\cdots ,R^{\left(Z_a\right)}\right\}\right.$ based on these attributes.

3.1.3. Constructing Contour Constraint Polygons

Due to the difficulty in handling wall contours in three dimensions and the mutual influence of edges of various subparts, each building subpart is sequentially projected onto the ground plane $V_{horizon}$ to generate a two-dimensional constraint outline, which is then projected onto the roof plane to construct the contour constraint polygons.

As shown in Figure 5, taking the subpart $Z_a$ with missing data as an example, the associated polygon is projected onto the ground plane as $V_{horizon}$, the wall polygon is projected as a straight line $\left\{\left.L_1^{\left(W^{\left(Z_a\right)}\right)},L_2^{\left(W^{\left(Z_a\right)}\right)},\cdots ,L_b^{\left(W^{(Z_a)}\right)}\right\}\right.$ on $V_{horizon}$, and the roof polygon is projected as a roof projection polygon on $V_{horizon}$. Regular edges $\left\{\left.L_1^{\left(R^{\left(Z_a\right)}\right)},L_2^{\left(R^{\left(Z_a\right)}\right)},\cdots ,L_p^{\left(R^{\left(Z_a\right)}\right)}\right\}\right.$ of the projection polygon are retained along with irregular curves $\left\{\left.S_1^{\left(R^{\left(Z_a\right)}\right)},\cdots ,S_q^{\left(R^{\left(Z_a\right)}\right)}\right\}\right.$, if they exist (otherwise, the roof has no missing areas). $p=\left\{\left.1,2,\cdots \right\}\right.$, $q=\left\{\left.1,2,\cdots \right\}\right.$.

First, by projecting the regular edges $L^{(R)}$ of the roof projection polygon associated with subpart $Z_a$ onto the ground plane, we generate the contour constraint polygon $U^{\left(R\right)}$. As shown in Figure 6a, within the roof projection polygon of subpart $Z_a$, the extension segments $L_1^{{(R)}^{\prime }}$ and $L_4^{{(R)}^{\prime }}$ are obtained by extending the roof regular edges $L_1^{(R)}$ and $L_4^{(R)}$ on the side intersecting with the irregular curve $S^{(R)}$ at point $P_0$. Projecting lines passing through $L_1^{{(R)}^{\prime }}$ and $L_4^{{(R)}^{\prime }}$ along the $z$-axis direction onto the roof plane results in intersecting segments $L_1^{{(R)}^{″}}$ and $L_4^{{(R)}^{″}}$, which, along with $S^{{(R)}^{\prime }}$, form the contour constraint polygon $U^{\left(R\right)}$.

Next, by projecting the wall polygon associated with subpart $Z_a$ onto the ground plane as the straight line $L^{(W)}$, we generate the contour constraint polygon $U^{\left(W\right)}$. As shown in Figure 6b, within subpart $Z_a$, the wall polygon’s projected straight lines $L_1^{(W)}$ and $L_4^{(W)}$, which are closest to the midpoints of $L_1^{{(R)}^{\prime }}$ and $L_4^{{(R)}^{\prime }}$, are translated to intersect at points $P_1$ and $P_5$, generating the contour edges $L_1^{{(W)}^{\prime }}$ and $L_4^{{(W)}^{\prime }}$ for the missing areas. Then, by projecting $L_1^{{(W)}^{\prime }}$ and $L_4^{{(W)}^{\prime }}$ along the $z$-axis direction onto the roof plane, we obtain intersecting segments $L_1^{{(W)}^{″}}$ and $L_4^{{(W)}^{″}}$, which, along with $S^{{(R)}^{\prime }}$, form the contour constraint polygon $U^{\left(W\right)}$.

3.1.4. Constructing Contour Constraint Regions

As shown in Figure 7, within subpart $Z_a$’s missing area, when $U^{\left(R\right)}\ge U^{\left(W\right)}$ is considered, the contour constraint possibility region $C^{(Z_a)}$ is formed by contour constraint polygons $U^{\left(R\right)}$ and $U^{\left(W\right)}$, while the contour constraint necessity region $M^{(Z_a)}$ is formed by the contour constraint polygon $U^{\left(W\right)}$ and the irregular roof curve $S^{{(R)}^{\prime }}$ (if $U^{\left(R\right)}<U^{\left(W\right)}$, then the opposite region is generated).

To facilitate the mapping of building contour constraints into latent space, we sample the extracted contour constraint possibility region $C^{(Z)}$ and contour constraint necessity region $M^{(Z)}$ as dense point clouds $X_C$ and $X_M$, respectively.

3.2. Diffusion

To transform the distribution of the original building point cloud into a Gaussian distribution suitable for modeling, we follow the process shown in Figure 8. Over time steps, we gradually add Gaussian noise with known parameters, dispersing the points from the original distribution into a disordered point set that approximates a Gaussian distribution.

$X^{(0)}={\left\{x_i^{(0)}\right\}}_{i=1}^N$ make up the input-constructing point cloud, where each point is independently sampled from the point distribution. The process of adding random Gaussian noise at each time step is modeled as a Markov chain:

$$q\left(X^{(1:T)}\mid X^{(0)}\right)={\displaystyle \prod _{t=1}^{T}q}\left(X^{(t)}\mid X^{(t-1)}\right),$$

(5)

$$q\left(X^{(t)}\mid X^{(t-1)}\right)=\mathcal{N}\left(X^{(t)}\mid \sqrt{1-{\beta }_t}{X}^{(t-1)},{\beta }_{t}I\right),$$

(6)

where $q\left(X^{(t)}\mid X^{(t-1)}\right)$ is the Markov diffusion kernel, ${\beta }_t$ is the hyperparameter that controls the variance scheduling of the diffusion process, and $t=\left\{1,\dots ,T\right\}$.

3.3. Point Encoding

As shown in Figure 9, to enable the fusion of contour constraint regions with the shape latent code of the original point cloud, we use the same encoder as the one used to encode the original building point cloud into shape latent codes to encode the contour constraint regions.

Inputting the original building point cloud $X^{\left(0\right)}$ into the trained encoder $\phi$, we map the point cloud from sample space to latent space, generating the corresponding shape latent code $h$. The process is described as follows:

$$q_{\phi }\left(h\mid X^{(0)}\right)=\mathcal{N}\left(h\mid {\mu }_{\phi }\left(X^{(0)}\right),{\Sigma }_{\phi }\left(X^{(0)}\right)\right),$$

(7)

where ${\mu }_{\phi }\left(X\right)$ and ${\Sigma }_{\phi }\left(X\right)$ are estimated mean and variance obtained through pre-training with PointNet [29].

To enable the fusion of building contour constraints with the shape latent code of the incomplete point cloud, we use the same encoder $\phi$ to map the contour constraint shape point clouds $X_C$ and $X_M$ into the same low-dimensional feature space, generating the corresponding shape latent codes $C$ and $M$. The process is described as follows:

$$C=\phi \left(X_C\right),$$

(8)

$$M=\phi \left(X_M\right),$$

(9)

where $\phi$ is the encoder.

By merging the shape latent codes $h$ of the incomplete building point cloud, the contour constraint shape latent codes $M$, and the intersection of shape latent codes E, we obtain the building shape latent code $H$:

$$H=h\cup M\cup \left(h\cap C\right),$$

(10)

3.4. Reverse

As shown in Figure 10, the reverse process is contrary to the diffusion process, i.e., sampling point clouds from the Gaussian distribution that approximates the forward diffusion point cloud results. It is based on the encoded shape latent code and gradually denoises through decoding, thereby completing the missing building point cloud.

Sampling points from the Gaussian noise distribution $p\left(X^{\left(T\right)}\right)~\mathcal{N}\left(0,I\right)$ that approximates the point set $q\left(X^{\left(T\right)}\right)$ as input, using the shape latent code $H$ fused with geometric features, the denoising decoding proceeds step by step from time step $T$ to time step 0. $\theta$ network A is used to estimate the denoising motion of each point at time step $t$, recursively moving towards the point distribution that approximates $q\left(X^{\left(0\right)}\right)$. This process can be modeled as a Markov chain:

$$p_{\theta }\left(X^{(0:T)}\mid H\right)=p\left(X^{\left(T\right)}\right){\displaystyle \prod _{t=1}^T{p}_{\theta }}\left(X^{\left(t-1\right)}\mid X^{\left(t\right)},H\right),$$

(11)

$$p_{\theta }\left(X^{\left(t-1\right)}\mid X^{\left(t\right)},H\right)=\mathcal{N}\left(X^{\left(t-1\right)}\mid {\mu }_{\theta }\left(X^{\left(t\right)},t,H\right),{\beta }_{t}I\right),$$

(12)

where ${\mu }_{\theta }$ is the estimated mean obtained by the neural network $\theta$.

Since each point is sampled independently, the probability of the whole point cloud in reverse precession is the product of the probabilities of each point:

$$p_{\theta }\left(X^{\left(0:T\right)}\mid H\right)={\displaystyle \prod _{i=1}^N{p}_{\theta }}\left(X^{\left(0:T\right)}\mid H\right),$$

(13)

The points sampled from the noise distribution $p\left(X^{\left(T\right)}\right)~\mathcal{N}\left(0,I\right)$ that approximates the point set $q\left(X^{\left(T\right)}\right)$ are generated into a point cloud $X={\left\{x_i\right\}}_{i=1}^N$ with the shape of the building using the reverse Markov chain $p_{\theta }\left(X^{\left(0:T\right)}\mid H\right)$.

3.5. Training Objectives

The training objective of the point cloud generation process is to maximize the log-likelihood of the point cloud. We set a target $L$ as the maximum variational lower bound of the log-likelihood with respect to $p_{\theta }\left(X^{\left(0\right)}\right)$: $\mathbb{E}\left[\log p_{\theta }\left(X^{\left(0\right)}\right)\right]$, and based on the shape latent code $H$ fused with geometric features, iteratively compute multiple times to minimize $L$ and achieve the optimal solution for the entire model.

$$\begin{array}{c}{L}_G\left(\theta ,\phi ,\alpha \right)={\mathbb{E}}_q\left[{\displaystyle \sum _{t=2}^T{\displaystyle \sum _{i=1}^N{D}_{KL}}}\left(q\left(x_i^{\left(t-1\right)}\mid x_i^{\left(t\right)},x_i^{\left(0\right)}\right)\Vert \right.\right.\\ \left.p_{\theta }\left(x_i^{\left(t-1\right)}\mid x_i^{\left(t\right)},H\right)\right)\\ \hspace{1em}-{\displaystyle \sum _{i=1}^N\log }{p}_{\theta }\left(x_i^{\left(0\right)}\mid x_i^{\left(1\right)},H\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\left.+D_{KL}\left(q_{\phi }\left(H\mid X^{\left(0\right)}\right)\Vert p_w\left(w\right)\cdot {\left|\det \frac{\partial F_{\alpha }}{\partial w}\right|}^{-1}\right)\right],\end{array}$$

(14)

where $D_{KL}\left(·\right)$ is the Kullback–Leibler divergence, $q_{\phi }\left(H\mid X^{\left(0\right)}\right)$ is the point cloud shape encoder, $F_{\alpha }$ is a trainable bijection implemented by an affine coupling layer, and $p_w\left(w\right)$ is an isotropic Gaussian distribution.

4. Experimental Section

4.1. Experimental Dataset and Processing

We utilized the BuildingNet [43] public dataset for the point cloud generation task. This dataset comprises 1849 complete building point cloud models, encompassing various architectural categories, including houses, churches, skyscrapers, city halls, libraries, and castles, among others. The dataset was randomly divided into training, testing, and validation sets in a ratio of 80%, 15%, and 5%, respectively. From each complete building point cloud, we uniformly sampled 20,480 points. We randomly selected one viewpoint from multiple viewpoints as the center and removed points within a certain radius to obtain incomplete point cloud data. We chose building point cloud data with a 5% missing rate for training and testing in the experiments. All input point cloud data were centered around the original points and their coordinates were normalized to [−1,1].

4.2. Experimental Equipment and Model Training Configuration

Experimental hardware configuration: The CPU model was an Intel Core i7-7700k, and the GPU model was a NVIDIA GeForce RTX 2080Ti. The deep learning environment for the experiments include CUDA 11.2, Python 3.6.9, and PyTorch 1.6. The training process consisted of 100 epochs with a batch size of 4, utilizing the Adam [44] optimizer with an initial learning rate of 0.001.

4.3. Evaluation Metrics

To comprehensively evaluate the quality of point cloud generation, we used CD (Chamfer Distance) and EMD (Earth Mover’s Distance) [45] as evaluation metrics. CD measures the average squared distance between the generated point cloud and the ground truth point cloud. A smaller CD distance indicates that the generated point cloud is closer to the vicinity of the building’s surface, reflecting the completeness of the point cloud shape and the smoothness of the edge contours. A smaller EMD distance is preferable, indicating that each spatial point generated by the model can be paired with real sample point cloud positions with smaller distances, reflecting the similarity in point cloud resolution and density.

Let $G$ be the generated point cloud set and $R$ be the real point cloud set. “$x$” and “$y$” are point clouds in point sets $G$ and $R$, respectively.

$$d_{CD}\left(G,R\right)=d_{CD}\left(G\to R\right)=\frac{1}{\left|G\right|}{\displaystyle \sum _{x\in G}{\min }_{y\in R}}\Vert G-R{\Vert }_2^2,$$

(15)

In the equation, for each point in set $G$, you should first find the closest point in set $R$, calculate the Euclidean distance, and then sum these distances and take the average.

$$d_{CD}\left(R,G\right)=d_{CD}\left(R\to G\right)=\frac{1}{\left|R\right|}{\displaystyle \sum _{y\in R}{\min }_{x\in G}}\Vert R-G{\Vert }_2^2,$$

(16)

The above equation follows the same calculation process as Equation (15), but with the point sets taken in reverse.

The EMD requires a one-to-one point-to-point mapping between two points.

$$d_{EMD}\left(G,R\right)=\underset{\varphi :G\to R}{\min }\frac{1}{\left|G\right|}{\displaystyle \sum _{x\in G}\Vert x-\varphi \left(x\right){\Vert }_2},$$

(17)

$$d_{EMD}\left(R,G\right)=\underset{\varphi :R\to G}{\min }\frac{1}{\left|R\right|}{\displaystyle \sum _{y\in R}\Vert y-\varphi \left(y\right){\Vert }_2},$$

(18)

In the equation, $\varphi \left(·\right)$ is the bijection between them.

4.4. Comparative Analysis of Experimental Results

To demonstrate the superiority of our model, we conducted comparative experiments between PCN [16], PF-Net [37], VRC-Net [39], and our model on the BuildingNet dataset. We used the metrics described in Section 4.3 to evaluate the point clouds generated by the above-mentioned models and summarized the results in

Table 1. We present the comparative results of d_CD and d_EMD between our approach and PCN, PF-Net, and VRC-Net on the BuildingNet dataset for complete building point cloud completion. The paired numerical values indicate d_CD and d_EMD for G→R and R→G, where lower values are preferable.

	Building 1				Building 2
Model	dCD (↓)		dEMD (↓)		dCD (↓)		dEMD (↓)
	G→R	R→G	G→R	R→G	G→R	R→G	G→R	R→G
PCN	2.896	3.112	6.121	6.099	3.405	2.887	7.335	7.686
PF-Net	2.06	1.838	4.001	3.773	2.417	2.165	4.808	4.431
VRC-Net	1.419	1.307	3.853	3.531	1.774	1.579	4.434	4.182
Ours	1.057	0.828	3.568	3.196	1.646	1.522	4.253	4.015

and

Table 2. Below are the comparison results of d_CD and d_EMD for point cloud completion in missing areas of buildings between our method and PCN, PF-Net, and VRC-Net on the BuildingNet dataset. For each model, the paired numerical values represent d_CD and d_EMD for G→R and R→G. Lower values indicate better performance.

	Building 1				Building 2
Model	dCD (↓)		dEMD (↓)		dCD (↓)		dEMD (↓)
	G→R	R→G	G→R	R→G	G→R	R→G	G→R	R→G
PCN	5.674	5.763	9.156	9.043	7.263	7.038	11.847	11.647
PF-Net	4.835	4.785	7.248	6.97	6.098	5.945	8.566	8.891
VRC-Net	4.248	4.196	6.863	6.725	5.059	4.903	7.695	7.633
Ours	3.831	3.781	6.324	6.428	4.817	4.512	7.314	7.062

. In Figure 11, we present the results of a visual comparison between the three-point cloud completion technique and our approach.

Table 1 and Table 2 show the quantified comparison results of the models. G→R refers to the error from the generated point cloud to the real point cloud, which measures the difference between the generated and real scenarios. R→G refers to the error from the real point cloud to the generated point cloud, which reflects the degree to which the ground truth surface is covered by the predicted point cloud. The d_CD score is multiplied by 10³, and the d_EMD score is multiplied by 10².

In Table 1, the overall point cloud completion error consists of two components: prediction error in the missing regions and shape variation in the original parts. Table 1 unmistakably demonstrates that our approach surpasses other currently available methods in the comprehensive completion of building point clouds.

In Table 2, the missing area error is calculated to ensure the fairness of the model evaluation. Table 2 continues to affirm that our approach is better suited for completing building point clouds, specifically in terms of filling in the gaps, when compared to other methods.

The findings presented in Table 1 and Table 2 illustrate that our approach excels in producing point clouds with increased precision and reduced distortion, addressing both overall point cloud completion and the specific task of filling in missing areas.

As can be seen from the comparison of experimental results in Figure 11, our method has better point cloud completion effect. In Figure 11, PCN focuses on global features and generates point clouds from coarse to fine during decoding. However, due to insufficient extraction of local features, it fails to completely recover these structures and exhibits uneven distribution of generated point clouds in space, as shown in ‘complete’ for Building 1 and ‘View 1.’ Additionally, PCN is trained based on the assumption of smooth features, which leads to the smoothing of the roof parts connecting different sections, as seen in ‘View 2’ for Building 1 and ‘View 1’ for Building 2. Despite PF-Net’s utilization of multiple fully connected layers to improve the smoothness of prediction outcomes and simplify the point cloud generation process, during the encoding stage, it exclusively downsamples the input point cloud for feature extraction and does not effectively capture all the local point cloud features. As a result, when it attempts to recover missing areas, the point clouds generated at the boundaries deviate from the original geometric space of the building, as shown in ‘View 1’ for Building 1 and in Building 2. VRC-Net consistently provides a smooth global shape, and the generated point clouds exhibit good spatial uniformity. It effectively retains the intricate details present in the input point cloud. However, the edges of the buildings are sharp, and excessive smoothing results in some non-realistic points at the edges, as shown in ‘View 2’ for Building 1. Similarly, for approximately curved multi-planar surfaces, they are smoothed into curves, as seen in ‘View 1’ for Building 2. Our model addresses this by using contour-constrained shape codes to constrain the diffusion probability model; the generated point clouds do not deviate from the original geometric space of the buildings. As shown in Building 1, compared to other methods, its edges closely match the geometric edges of the real building. In the case of Building 2, View 2, our method accurately completes the building point cloud in the corner area between two parts, while other methods connect these two parts together. This demonstrates the strong reconstruction capability of our method, and the predicted point cloud shape exhibits strong spatial continuity and geometric invariance, reducing the generation of non-realistic point clouds and effectively improving point cloud completion accuracy.

4.5. Completion Comparison Experiments under Different Missing Rates

Experiments were conducted to complete the same buildings with different missing rates of 10%, 15%, 20%, and 25%, respectively, to evaluate the model’s capacity to restore missing buildings at varying rates of absence. The precise quantitative outcomes are displayed in

Table 3. On the BuildingNet dataset, we present the comparative results of d_CD for point cloud completion between our method and Model [14], as well as VRC-Net, across four different missing rates: 10%, 15%, 20%, and 25%. For each model, the paired numerical values represent d_CD for G→R and R→G. Lower values indicate better performance.

Missing Ratio	VRC-Net		Model [14]		Our
	G→R	R→G	G→R	R→G	G→R	R→G
10%	2.431	2.095	2.619	2.283	1.835	1.606
15%	2.879	2.607	3.011	2.718	2.051	1.811
20%	5.154	4.942	4.914	4.868	3.586	2.963
25%	8.672	8.261	8.226	8.052	7.902	7.515

, illustrating the performance of Model [14], VRC-Net, and our model on the building test dataset. The d_CD score is multiplied by 10³, and lower d_CD scores indicate better performance.

In Figure 12, the visualization shows the variation of d_CD values for three models: VRC-Net, Model [14] and Our.

Even as the number of input points gradually decreased from 10% to 25% compared to the true point cloud data, our model remained ahead in terms of building point cloud completion compared to Model [14] and VRC-Net. Nevertheless, when the missing rate of building data surpassed 20%, the substantial loss of building point cloud data resulted in a notable absence of contour structure information for the building. This makes it difficult for us to construct contour constrained polygons that fit the original building, leading to a noticeable decrease in the point cloud completion performance of our model for buildings. Nonetheless, even in the case of high data loss, our approach can effectively fill in the missing parts of the point cloud data. Compared to other models, it still demonstrates notably impressive building point cloud completion capabilities.

4.6. Ablation Experiment

To evaluate the influence of the shape latent code with incorporated contour constraints in our approach on point cloud completion, we performed ablation experiments using the BuildingNet dataset. The errors reported in the table are mainly divided into two parts: G→R and R→G, with d_CD scores multiplied by 10³. Smaller d_CD scores indicate better performance.

The quantitative results are shown in

Table 4. Comparative results of our approach to ablation experiments d_CD on the BuildingNet dataset. For each model, the paired numerical values represent d_CD for G→R and R→G. Lower values indicate better performance.

	Contour-Constrained Shape Latent Code		dCD (↓)
Type	$\mathit{C}$	$\mathit{M}$	G→R	R→G
Baseline Model	×	×	1.711	1.508
Model 1	✔	×	1.556	0.923
Model 2	×	✔	1.231	1.126
Our	✔	✔	0.928	0.751

. When using the point cloud diffusion probability model alone to extract features, the highest error values were obtained, with d_CD values of 1.711/1.508. When adding the contour-constrained shape latent code $C$, the results were better than the Baseline Model, with a d_CD value of 1.556/0.923. Similarly, when adding the contour-constrained shape latent code $M$, the results were also better than the Baseline Model, with a d_CD value of 1.231/1.126. The contour-constrained shape latent code $C$ introduces outward redundancy, which results in a larger difference between the generated d_CD and the ground truth, leading to relatively higher d_CD values for G→R. However, this redundancy increases the coverage of the ground truth surface by the predicted point cloud, and hence the d_CD value for R→G is relatively lower. On the other hand, the contour-constrained shape latent code $M$ causes inward contraction, leading to a relatively smaller difference between the generated d_CD and the ground truth. However, this contraction reduces the coverage of the ground truth surface by the predicted point cloud, resulting in relatively lower d_CD values for G→R and relatively higher d_CD values for R→G. Our model combines contour-constrained shape latent codes $C$ and $M$, leveraging their advantages to minimize the difference between the generated point cloud and the real point cloud while achieving a higher coverage of the ground truth surface by the predicted point cloud. The d_CD values were 0.928/0.751, which was better than all the models mentioned above. This suggests that the inclusion of building contour constraints enhances the performance of point cloud completion.

The visualization results, as shown in Figure 13, demonstrate that the point cloud diffusion probability model based on building contour constraints produces the best results. As shown in Figure 13, the Baseline Model can extract the complete shape of building point clouds. However, in comparison to the real building model, there is some redundancy at the edges of the completion area. This suggests that the point cloud diffusion probability can capture global building features. The model that only integrates the contour-constrained shape latent code $C$ shows slight redundancy at the edges of the completion area. Compared to the Baseline Model, the completed point cloud deviates less, aligning better with the real building point cloud shape. The model that only integrates the contour-constrained shape latent code $M$ exhibits some minor point cloud omissions at the edges of the completion area compared to real buildings. However, the completed point cloud deviates less overall. Our model combines both contour-constrained shape latent codes $C$ and $M$. In the completion area, the point cloud at the roof’s edge is nearly identical to the real building point cloud. This indicates that the building contour constraints effectively suppress the generation of point clouds outside the building’s geometric space, significantly improving the geometric accuracy of point cloud completion and reducing the generation of non-realistic point clouds.

5. Discussion

In the digital realm, 3D architectural surfaces are typically composed of multiple planar polygons, which contain crucial information regarding the geometric shape of the building. Hence, we can employ geometric methods to construct high-precision constraint outlines for missing regions, thereby rectifying incomplete architectural point cloud 3D models. However, while the overall shape of the building point cloud encoded by the encoder $\phi$ remains intact, the encoded shape in the missing regions often deviates from the original geometric positions. Consequently, we amalgamated these two characteristics by merging the building-outline-constrained shape latent code with the building’s original point cloud shape latent code. This yields a shape latent code that more closely conforms to the architectural geometric shape, thus more effectively constraining the point cloud completion model to generate point clouds that correctly fill the missing regions.

In this undertaking, the construction of precise constraint outlines plays a pivotal role in enhancing the accuracy of point cloud completion. Achieving high geometric accuracy in constraint outlines primarily focuses on two critical aspects: ① Improving the geometric shape accuracy of the planar polygons extracted from the building surface point cloud; and ② rationalizing the grouping of building surface polygons.

Therefore, future work will emphasize the following two aspects: firstly, extracting planar polygons with high geometric accuracy, ensuring they accurately reflect the geometric shape of the building surface, and secondly, optimizing the grouping method of polygons to eliminate interference from irrelevant polygons and generate more rational constraint outlines within blocks that have missing regions.

6. Conclusions

This paper presents a 3D point cloud completion method based on a building-contour-constrained diffusion probability model. It constructs contour shape latent codes using building contour information, which consists of both walls and roofs, to constrain point cloud generation using the diffusion probability model. This ensures that the generated point cloud has the overall shape with geometric details of building edges. Our method effectively addresses the issue of incorrect geometric distribution in completing point clouds for missing areas in buildings. We tested our method’s performance in building point cloud completion on the BuildingNet dataset and reached the following conclusions:

(1)

By grouping building surface polygons, we simplified the process of constructing building-contour-constrained polygons. This eliminates the need for training complex point cloud semantic segmentation models to obtain the required building-contour-constrained regions.

(2)

In the comparison experiments presented in Table 1 and Table 2, our method demonstrates superior building point cloud completion accuracy compared to other point cloud completion methods, such as PCN, PF-Net, and VRC-Net. This is because we incorporated the geometric contour information of the target building point cloud during the building point cloud completion process, resulting in a point cloud distribution that closely matches the geometric shape of the target building.