3D City Reconstruction: A Novel Method for Semantic Segmentation and Building Monomer Construction Using Oblique Photography

Abstract

Existing 3D city reconstruction via oblique photography can only produce surface models, lacking semantic information about the urban environment and the ability to incorporate all individual buildings. Here, we propose a method for the semantic segmentation of 3D model data from oblique photography and for building monomer construction and implementation. Mesh data were converted into and mapped as point sets clustered to form superpoint sets via rough geometric segmentation, facilitating subsequent feature extractions. In the local neighborhood computation of semantic segmentation, a neighborhood search method based on geodesic distances, improved the rationality of the neighborhood. In addition, feature information was retained via the superpoint sets. Considering the practical requirements of large-scale 3D datasets, this study offers a robust and efficient segmentation method that combines traditional random forest and Markov random field models to segment 3D scene semantics. To address the need for modeling individual and unique buildings, our methodology utilized 3D mesh data of buildings as a data source for specific contour extraction. Model monomer construction and building contour extractions were based on mesh model slices and assessments of geometric similarity, which allowed the simultaneous and automatic achievement of these two processes.

1. Introduction

Three-dimensional (3D) city reconstruction is a popular research topic in virtual geographical environments, photogrammetry, and computer vision. As an effective means for the multidimensional presentation of urban elements, 3D city reconstruction has broad application prospects in fields such as natural resource supervision, urban planning, engineering simulations for construction purposes, and the scheduling of emergency protocols. With the recent development of unmanned aerial vehicle platforms, remote sensors, graphic image processing, 3D visual [1], and other technologies, oblique photography has gradually become the mainstream method for 3D modeling and is widely used in large-scale 3D city reconstructions [2,3].

Oblique photography 3D reconstruction has the advantages of fast data acquisition, low production costs, a high degree of automation, and a rich expression of results; however, a common disadvantage is the lack of semantic information in such models. Computers cannot automatically identify and distinguish between the ground, trees, buildings, roads, and other objects in a scene, i.e., it is difficult to perform monomer management of ground object elements. Therefore, a 3D model of a city as reconstructed via oblique photography can traditionally be used only for simple displays, and a connection between this virtual scene and the real-world parameters cannot be established through semantic information [4]. Increasing attention is being paid to the semantic classification and segmentation of 3D scenes and models to remedy this situation. Proposed methods include semantic classification based on manual annotation [5,6], a 3D point cloud classification based on deep learning [7,8,9,10,11], semantic segmentation of imagery based on dimension reduction [12,13], and a semantic classification based on 3D voxels [14,15]. However, mature and effective theoretical methods, technical systems, and software platforms have not yet been developed for semantic segmentation of urban 3D scenes acquired via oblique photography.

Due to its modeling principles and technical limitations, a model constructed using oblique photography presents a continuous whole, such as a triangular mesh. This is commonly known as a “surface” model, which cannot distinguish between individual buildings in geometric space and does not allow the objectified management of such models. To solve this challenge, it is often necessary to conduct monomer processing in subsequent stages [16,17] via physical cutting [18], dynamic labeling based on 2D vector graphics, manual ID labeling, and semiautomatic model replacement. However, in practice, the effect of these methods is not ideal, and they require considerable manual intervention or rely on third-party data sources (such as planning, or the construction management data related to building outlines). That is, it is difficult to satisfy the requirements of objectification without building outline data and a high degree of automation. Developing effective automated monomer processing in 3D models established via oblique photography is critical for achieving these goals.

This study explored methods for semantic classification, automatic individualization, and contour extraction using 3D model data from oblique photography. It proposes corresponding implementation paths and specific algorithm procedures and conducts experimental analysis. This research aimed to explore a beneficial method for processing and applications in urban 3D reconstruction using oblique photography. This study reflects innovation in the point set processing of oblique photography 3D data, superpoint construction, and feature fusion. A unique method for elevation plane cutting and polygon similarity determination is proposed for building model monomers, which can simultaneously achieve model monomer and contour extraction without relying on any third-party data.

2. Materials and Methods

2.1. Oblique Photography 3D Semantic Segmentation

Implementation Pathway for the Semantic Classification of 3D Mesh Data

The basic principles and objectives of semantic segmentation of 3D data are consistent with those valid for 2D image data [19,20,21,22,23,24,25]. However, unlike raster 2D images, urban 3D models are characterized by spatial dispersion, rich information, and large data scales. It is necessary to consider the characteristics of urban data and research the corresponding solutions in data organization, feature selection, and spatial retrieval for the semantic segmentation of 3D model mesh data from oblique photography. This study considered the following three key problems:

• The organization and formatting of 3D data. Each minimum semantic unit needs to be marked and classified. For the 3D modeling results of oblique photography, the data units that can be used for semantic segmentation comprise triangular vertices and facets. Data can also be reorganized to meet the needs of the semantic segmentation of large-scale 3D urban scenes;
• The classification and calculation of features. Feature selection is critical and normally evaluated manually; however, fusion can also compute features. Overall, the features of 3D data can be categorized into textural, color, spatial geometric, and fusion features. Another key issue in feature calculation is identifying and selecting the feature neighborhood space;
• Classifier training and classification optimization. The last step of semantic segmentation is classification calculations. Existing machine learning and deep learning algorithms train the classifier using sample data and then classify the test data [26]. An effective classifier must be selected to optimize classification by considering the global continuity of geospatial data.

Based on the three challenges above, this study proposes the following path for 3D semantic segmentation of oblique photographs (Figure 1).

The main processes are as follows:

• Step 1: Mapping an oblique photography 3D mesh model to point cloud data. To facilitate machine learning calculations, we performed point aggregation mapping of a 3D mesh model of oblique photography to transform the 3D mesh model into a point cloud model. In point cluster mapping, more abundant spatial location data are obtained while retaining topological relationships between points in the mesh. As a result, this type of point set provides richer semantic information than conventional point cloud data;
• Step 2: Rough geometric segmentation of point cloud data based on spatial clustering. The 3D point dataset obtained in Step 1 was mapped and divided into different geometric regions. Each region is called a superpoint, and all the points that constitute the superpoint possess the same semantic attributes. A spatial clustering method based on regional growth was adopted to conduct the subsequent semantic segmentation based on superpoints, reducing the overall computational complexity;
• Step 3: Point cloud spatial neighborhood search and geometric feature calculation. Feature extractions were performed for the adjacent 3D point regions for which the neighborhood search methodology is key. This study proposes using a geodesic distance calculation based on a surface model that considers the topological constraints of the grid (rendering it a more reasonable method than one that employs Euclidean distances);
• Step 4: Clustering center feature solving. Feature extraction and computation are prerequisites for semantic segmentation. In this study, the geometric features of individual points were first obtained, and then feature fusion was performed on superpoints based on the features of individual points. Compared to individual points, the feature dimension of superpoints was increased.
• Step 5: Semantic segmentation calculation. Semantic segmentation depends on the characteristics of the classifier. To conduct feature extraction, the 3D data need to define the characteristics of each object with a segmentation probability. This study proposed dividing the 3D scene into four categories: “ground”, “building”, “trees”, and “other”. We employed Markov random field and energy minimization to optimize the classification globally to sharpen category boundaries and render the local neighborhood classification as smoothly as possible.

2.2. Point Set Mapping and Point Cloud Geometric Segmentation

2.2.1. Mapping the Mesh Model to a Point Set

The spatial positions of the vertices of a triangular mesh representing a continuous 3D space will determine the geometric information of the triangular surface, including the location, area, normal vector, and texture information. Solving the problem of each triangular surface relies on the information from its three vertices when extracting spatial features. In geometric space, the processing of points is easier and more direct than that of surfaces in terms of computational complexity. To facilitate calculations, this study selected the vertices of triangular slices as computational units (Figure 2). The vertices of numerous triangular slices constitute a point aggregation model, thus transforming a triangular mesh’s semantic segmentation into a point cloud’s semantic segmentation. In contrast to a traditional LiDAR point cloud, the point cloud obtained through the transformation of mesh data possesses spatial geometry, texture, and color information while retaining a topological connection between points, which is of great value in describing local features.

Considering the textural information of the vertices from the oblique photography 3D model, the data model following transformation and reconstruction is described as follows: In the continuous 3D space, the point cloud of the triangle model is expressed as S = {F, P, T}, where F is the collection of facets that constitute the triangulation network, P is the collection of vertices that constitute the triangulation network, and T is the topological relationship between the triangles. This can be expressed as F = {F_i|i = 1, 2, 3,…, n} and F_i = {P_i1, P_i2, P_i3}, where P_i1, P_i2, and P_i3 are the three vertices that constitute the triangular sheet F_i. The vertex feature f(P_i) includes coordinate and texture information: f(P_i) = {$x_{P_i}, y_{P_i},z_{P_i},r_{P_i},g_{P_i},b_{P_i},N_{P_i}$}, where $x_{P_i}, y_{P_i},z_{P_i}$ are the coordinate values of the vertex, $r_{P_i},g_{P_i},b_{P_i}$ are the texture color values of the vertex, and $N_{P_i}$ is the set of points connected to the vertex P. The subsequent rough data segmentation utilizes the topological relationship (T) between triangular vertices.

2.2.2. Geometric Segmentation of the Point Cloud via Spatial Clustering

The 3D mesh data from oblique photography were converted to a point cloud model (Section 2.2.1). To reduce the computational time during subsequent machine learning and improve the model’s scalability, the point cloud was divided into different subsets via clustering, with points in the same regional subset possessing homogeneous semantics. Following the terminology of Landrieu [27,28], each subset was labeled a “superpoint”. Compared to the unprocessed point cloud data, the data after clustering contained more information, such as cluster size and the relationship between classes. Considering the large number of planar regions in a 3D urban landscape, this study adopted a plane detection method based on region growth to conduct a coarse segmentation clustering of point clouds, in which each point set is regarded as belonging to a specific plane as a cluster. This method assumes that points in the same plane region of clustering have similar normal vectors [29]. The following iterative approach was adopted:

• Step 1: An available seed point is selected from the dataset, and its neighborhood space is determined;
• Step 2: The neighborhood that meets the regional plane detection rules is included in the region;
• Step 3: Steps 1 and 2 are repeated for all neighborhoods;
• Step 4: A new region is created until no neighborhoods meet the 3D rules of regional plane detection.

Our plane detection based on 3D points adopted a plane-fitting method based on principal component analysis. When the normal vector of a point is less than the threshold value, and the distance from the plane is less than a certain value, that point is included in the regional plane. Each planar region is represented by a superpoint element.

2.3. Neighborhood Search and Feature Extraction

2.3.1. K-Neighborhood Search Based on Geodesic Distances

An effective neighborhood search is the key to feature extraction from a continuous space. For a given 3D point set, defining neighborhoods typically includes the following three processes [30]: the spherical [31], cylindrical [18], and k-neighborhood methods [32]. The selection of suitable k values for spherical and cylindrical neighborhoods depends on the empirical judgment of different datasets. Neighborhood k exhibits better flexibility when the point cloud density changes and is more suitable for the 3D urban environment, in which point cloud density varies significantly. As a result, we utilized the k-neighborhood method to determine the vertex neighborhood of each triangular facet.

The basic principle of this method is to calculate the Euclidean distances between a given point and other points, then sort them in ascending order. The so-called “k points” located closest to the examined point are the k-neighborhood points. However, such a direct use of Euclidean distances would abandon the plane topology constraint, and it is more reasonable to identify the similarity of samples in a manifold space. Therefore, we used the geodesic distances based on the surface model to quantify distances in the k-neighborhood.

The key to calculating the geodesic distance between two points on the surface of a triangular grid model is to identify the geodesic. In this study, the geodesic refers to the shortest path between two points on the surface of a 3D model and is, therefore, limited to its surface. Geodesic distance calculations can be divided into accurate and approximate. The precise method poses high requirements regarding model topology, which often requires many calculations and can be extremely time-consuming. In this study, we selected the Dijkstra shortest path geodesic distance algorithm to calculate the surface distance between two vertices, considering the time cost, computational accuracy, and model topology involved. The vertices of the triangles represent the points of the Dijkstra diagram, and the lengths of the edges connecting the points are the weights of the edges of the Dijkstra diagram, as illustrated by the numbers in Figure 3. The shortest route between the source point S and target point E was the line segment set {SP₂, P₂P₆, P₆E}, as shown by the blue lines in Figure 3, and the shortest path was 15.

2.3.2. Geometric Feature Extraction of 3D Points

Each 3D point in this study contained both geometric and textural features. The four geometric features selected for extraction were the local planarity (f_p), local horizontality (f_v), local sphericity (f_s), and relative elevation (f_z). Using these values, a 4-dimensional vector space can be constructed for each point using V($p_i$) = {$f_p\left(p_i\right)$, $f_v\left(p_i\right)$, $f_s\left(p_i\right)$, $f_z\left(p_i\right)$}, constituting the feature space of the point. The calculation of each variable is described below:

• Local planarity (f_p). Local planarity can be expressed as the flatness of the neighborhood space and is an important feature of many 3D urban objects, such as building façades, pitched roofs, and flat ground, all of which exhibit good local planarity. In the k-neighborhood space, local flatness can be calculated as the sum of the squares of the distances from all points in the neighborhood space for the optimal fitting plane of the local neighborhood. Therefore, the key is determining the neighborhood’s optimal fitting plane. This problem can be solved by establishing a neighborhood covariance matrix and estimating the local surface normal vectors based on calculating eigenvectors and eigenvalues [33,34]. Given a 3D space point p_i, with its k-neighborhood space being N (p_i) and the centroid of its neighborhood space being $\overline{p}$, the 3 × 3 covariance matrix (C) is calculated as follows:

  $$C=\frac{1}{k}{\sum }_{i\in N}{\left(p_i-\overline{p}\right)}^T\left(p_i-\overline{p}\right).$$

  (1)
  Following Equation (1), C is a symmetric semi-positive definite matrix. It has three orthogonal eigenvectors, 𝑣₀, 𝑣₁, and 𝑣₂, which correspond with three non-negative eigenvalues, λ₀, λ₁, and λ₂, respectively. If λ₂ ≥ λ₁ ≥ λ₀ ≥ 0 is assumed, the minimum eigenvalue λ₀ corresponds with the eigenvector 𝑣₀, which approximately represents the normal vector of the optimal fitting plane in neighborhood space [35]. As such, λ₀ represents the deviation from the local plane along the surface normal vector 𝑣₀. Therefore, we can define the flatness metric at p_i according to Equation (2) [36]:

  $$f_p\left(p_i\right)=1-\frac{3{\lambda }_0}{{\lambda }_0+{\lambda }_1+{\lambda }_2};$$

  (2)
• Local horizontality (f_v). Horizontal features are more evident on flat roofs and urban roads. Horizontality can be measured (Equation (3)) using the normal vector Pi’s absolute value and the vertical z-axis’s cosine. A cosine value close to zero indicates high horizontality, whereas a value close to 1 indicates a vertical plane.

  $$f_v\left(p_i\right)=\left|v_0.n_z\right|, n_z=<0,0.1>;$$

  (3)
• Local sphericity (f_s). Sphericity determines whether geometric shapes within a local area resemble a ball shape, typical for tree crowns. As previously defined, spherical properties can be calculated based on the difference between three characteristic values: λ₂ ≥ λ₁ ≥ λ₀ ≥ 0. The larger the value obtained with the following equation, the better the spherical properties:

  $$f_s\left(p_i\right)=\frac{{\lambda }_0}{{\lambda }_2};$$

  (4)
• Relative elevation (f_s). Different ground object types are linked to certain height classifications in urban environments. Owing to topographic fluctuations, absolute elevation information lacks a unified differential expression. Therefore, a relative elevation was defined in this study to describe the elevation information of each object, calculated as follows:

  $$f_z\left(p_i\right)=\frac{z_i-z_{\mathrm{m}\mathrm{i}\mathrm{n}}}{z_{\mathrm{m}\mathrm{a}\mathrm{x}}-z_{\mathrm{m}\mathrm{i}\mathrm{n}}},$$

  (5)

  where z_i is point p_i’s elevation; Zmax and Zmin are the maximum and minimum elevation values of all points in the neighborhood space.

2.3.3. Feature Fusion and Extraction of Superpoints

Feature fusion and extraction were conducted on each superpoint for which the fusion features were mainly the average of all 3D point features contained in the superpoint cluster, i.e., the average local planarity $f_{\overline{p}}\left(s_j\right)$, average local horizontality $f_{\overline{v}}\left(s_j\right)$, average local sphericity $f_{\overline{s}}\left(s_j\right)$, and average relative elevation $f_{\overline{z}}\left(s_j\right)$. The extracted features were obtained by calculating the statistical features of the 3D points within the superpoint. All points within a superpoint are considered local neighborhood space, and the corresponding eigenvector values of the local neighborhood covariance matrix were calculated as λ_s2 ≥ λ_s1 ≥ λ_s0 ≥ 0. Three geometric features were defined for each superpoint: the length $f_l\left(s_j\right)$, area $f_a\left(s_j\right)$, and volume $f_v\left(s_j\right)$. These values were calculated using: $f_l\left(s_j\right)$ = λ_s2, $f_a\left(s_j\right)$= λ_s1λ_s2 and $f_v\left(s_j\right)$= λ_s1λ_s2λ_s0. Thus, each superpoint was ultimately characterized with a 7-dimensional vector: V($S_j$) = {$f_{\overline{p}}\left(s_j\right)$, $f_{\overline{v}}\left(s_j\right)$, $f_{\overline{s}}\left(s_j\right)$, $f_{\overline{z}}\left(s_j\right)$, $f_l\left(s_j\right), f_a\left(s_j\right)$, $f_v\left(s_j\right)$}.

2.4. Random Forest Semantic Segmentation

The basic idea of 3D semantic segmentation is to calculate the probability of an object belonging to each classification label and then attribute the label attaining the highest probability value. This can be conducted via a supervised classification based on machine learning concepts, and specific training can be understood as a classifier. For a given feature vector x, the prediction belongs to the category label y with conditional probability p (y|x). Therefore, our primary objective was to select an appropriate classifier. We opted to use a random forest as a classifier for 3D urban landscapes with large amounts of data, which has the advantages of parallel computation [37], support for high-dimensional features, and strong generalizability.

A prior probability distribution of the observational data was created using a random forest. However, random forest classification only provides the category probability of a single discrete object without considering the correlation between spatial objects in its neighborhood. Consequently, some point clouds can be isolated into incorrect categories, and their classification labels are often not optimal solutions. Simultaneously, there are significant contextual prior knowledge constraints in 3D classifications (such as the elevation of the ground being lower than that of the building and roof, the building being adjacent to the ground, and the absence of trees on a roof). Classification results should discern between the boundaries of different categories as clearly as possible and obtain the smoothest possible classification in the local neighborhood. To this end, we employed the Markov random field model [38] to determine optimal classification labels.

Let P = {P1, P2,…, Pn} be a set of points, L = {L1, L2,…, Lm} comprise a set of classification labels, and m is the number of classification labels. X = {X1, X2,…, Xn} is a set of random variables in the set P, and each 3D point is associated with a random variable. According to Bayes’ rule:

$$P\left(s|d\right)=\frac{P\left(d|s\right)P\left(s\right)}{P\left(d\right)},$$

(6)

where $P\left(d|s\right)$ is the conditional probability distribution of the observed value d (also known as the likelihood function); $P\left(d\right)$ is a fixed value; and $P\left(s\right)$ is the prior probability of the initial value generated via random forest classification. Assuming that the optimal state is $\mathrm{s}*$, the solution to the problem can be expressed as

$$\mathrm{s}*=\mathrm{m}\mathrm{a}\mathrm{x}P\left(s\right)P\left(d|s\right),s\in S.$$

(7)

According to the Hammersley–Clifford theorem, the probability distribution of the Markov random field undirected graph must be expressed as the product of non-negative functions on the largest clique that satisfies the Gibbs distribution [39,40]. According to Gibbs measures of the Potts model [41], the solution of Equation (7) can be transformed into the minimum value of the solution of Equation (8):

$$E\left(s\right)={\sum }_{i=1..N}{D}_i\left(s_i\right)+\gamma {\sum }_{i,j\in N\left(p_i\right)}{V}_{i,j}\left(s_{i,}{s}_j\right),$$

(8)

where the semantic meaning of $D_i\left(s_i\right)$ can be understood as the loss of the current object, designated under a different classification tag. When this object is labeled with tag X_i, the consistent difference between its observed and expected features is greater; the greater this value, the smaller the matching degree. Furthermore, $V_{i,j}\left(s_{i,}{s}_j\right)$ describes the degree of proximity between adjacent objects, where i and j represent two adjacent points in the neighborhood space; the larger this value, the larger the range of smoothness and the smaller the overall smoothness. $\sum _{i=1..N}{D}_i\left(s_i\right)$ of the equation sign in Equation (8) expresses the global optimality of the classification results, and $\gamma \sum _{i,j\in N\left(p_i\right)}{V}_{i,j}\left(s_{i,}{s}_j\right)$ expresses the local smoothness of the classification results. In the classification problem based on the Markov model, Equation (8) was solved for the minimum value. The α-expansion algorithm [42] was used to solve the smallest energy.

2.5. Monomer and Contour Extraction

Thus far, architectural model objects have been extracted from a photograph using a semantic segmentation method. Based on an analysis of existing methods related to buildings, we believe that the following aspects should be considered in establishing 3D building monomers:

• The monomer modeling process should avoid using third-party data;
• Irreversible damage to the original data should be avoided;
• The degree of monomer automation must be maximized;
• Contour extraction should be optimally integrated with monomer processing to avoid multiple traversal data calculations.

Considering the characteristics inherent to the 3D data of oblique photography and the need for calculation efficiency, this study utilizes a 3D mesh as a data source. The proposed monomer and contour extraction method is based on a hierarchical “slicing” of oblique 3D models to determine the geometric similarity of polygons.

2.5.1. Contour Set Generation Based on Horizontal Slicing of Grid Model Data

Our basic principle behind building contour extraction was to consider a building contour collection using sections from different elevations; geometric similarity of the contour line is determined to extract the optimal contour line and perform logical monomer processing (Figure 4).

In the oblique photographic 3D model, the horizontal plane where the highest point of a building is located is taken as the starting level. The model is “sliced” horizontally at certain intervals between the highest and lowest points of the building (Figure 5). Note that no physical or geometric cuts of the model are involved; purely the projection of all grid mesh data above the cutting plane elevation is obtained for the horizontal cutting plane at each elevation interval to facilitate subsequent contour extraction.

The triangular plane of the model was projected onto the elevation plane and rasterized to allow the extraction of a binarized raster orthographic projection image on each plane. The size of grid cells selected for rasterization was 0.1 m × 0.1 m, balancing extraction efficiency and accuracy (for large cell sizes, the extraction accuracy is low, but the calculation efficiency is high). The process of projecting the triangle onto the elevation plane and performing binarization is summarized in Figure 6.

A binarized raster image of building contours was obtained for each elevation plane. Examples of binarized raster images are shown in Figure 7.

The raster image data of the projected building surfaces were obtained through binarization. To obtain building contours, building edges were extracted from the raster data. This involves the vectorization of raster data using relatively mature algorithms. However, in contrast to the classification of alternative image data, ground objects are complex and include many different linear and areal entities. The binary raster data processed in this study contained simple information and a large image range; the vector objects generated were all polygons. Considering these characteristics, we used a method based on image edge detection [43] and boundary tracking [44] to extract polygon contours from the binary imagery.

2.5.2. Monomer and Contour Selection Based on Geometric Similarity

Using the methods described in Section 3.1, multiple contour lines corresponding to different high-level horizontal sections were obtained, forming a contour line set. To extract a unique optimal contour for each building, two tasks must be accomplished: (1) collating the set of contour lines belonging to the same monomer and (2) selecting one contour line (among those cut from the same monomer) as the building contour line.

In this study, the monomer of the building adopted the logical monomer method, which does not cut the model geometrically or physically but generates a bounding box of the model and attaches the model ID and attributes. Multiple and unique contour line sets were obtained for separate high-rise buildings without logical connections. Determining whether the contour sets belong to the same building is the key to determining the logical monomer. Our monomer determination was based on the following principles:

• The contour of a single building “sliced” at different elevations is completely closed;
• For a contour set of the same building, two geometrically similar contour lines must exist at different elevations;
• The ideal situation for a regularly shaped building is one in which the contour polygons of different elevations overlap completely when projected onto the same XOY plane. The vector polygons obtained by “slicing” the inclined model blocks at different elevation planes were compared; corresponding inclined model blocks of the vector polygons, for which the position and shape deviation was less than the set threshold, were determined to represent the same inclined building monomers.

2.5.3. Geometric Similarity Indicators of Contours

The following three indicators were proposed to measure the spatial geometric similarity between two different contours, acting as quantified indicators for the subsequent monomer determination and contour selection:

• Positional deviations of contour planes. This refers to the spatial distance between contour polygons in the XOY plane, as measured by the distance between the geometric centers of two contour polygons. Contour lines of the same building should exhibit relatively small plane space deviations at separate elevations;
• Area deviations of contour polygons. This refers to the difference in surface areas between two contour polygons. The more the areas of two different contour lines resemble each another, the higher their geometric similarity;
• Geometric deviations of contours. This reflects the global similarity between two contours, mainly characterized by contour shape and size.

To increase the computational efficiency and facilitate the description of the above indicators, an outer bounding box of a building contour was first established when determining the geometric shape of the plane space position and contour. The plane space position of the contour is replaced by the space position of the outer bounding box, and the geometric shape of the contour is determined by the length and width of the outer bounding box. As shown in Figure 8, a series of contour sets were obtained for each building via horizontal “slicing” at different elevation planes. Subsequent monomer and contour selections were calculated and solved based on the geometric information of this contour series.

Table 1. Calculation of contour geometric similarity indicators.

Geometric Similarity Index	Deviation Formula	Source Contour of the Computed Object	Target Contour of the Computed Object
Positional deviations	ΔP = |P1 − P2|
Geometric deviations	ΔL = |L1 − L2|
	ΔW = |W1 − W2|
Contour area deviations	ΔS =|S1 − S2|

lists the formulas used to calculate each of the three indicators. First, the outer bounding boxes were solved for each contour. P₁ and P₂ are the geometric centers of the outer bounding boxes of source contour C₁ and target contour C₂, respectively. S₁ and S₂ represent the areas of these two contours. L₁ and L₂ are the heights, and W₁ and W₂ are the widths of the outer bounding boxes of these contours, respectively.

Using the above three indicators, conditional strategies can be applied to determine whether two contours are similar. The indicators had to meet predetermined thresholds simultaneously before two contours could be labeled “similar”. These conditions were as follows:

• The deviation of the plane position is not more than 0.1 m, the threshold value is empirical, and the position similarity (SimP) is calculated as follows:

$$\mathrm{S}\mathrm{i}\mathrm{m}\mathrm{P}=\left\{\begin{array}{c} True, \Delta \mathrm{D} \le 0.1\mathrm{m}\\ False, \Delta \mathrm{D}>0.1\mathrm{m}\end{array}\right.;$$

(9)

2.

The difference between the length and width of the outer bounding box adopts a relative threshold that does not exceed 1/20 of the sum of the compared quantities when geometric similarity (including length and width similarity) is calculated as

$$\mathrm{S}\mathrm{i}\mathrm{m}\mathrm{L}=\left\{\begin{array}{c} True, \Delta \mathrm{L} \le (L_1+L_2)/20\\ False, \Delta \mathrm{L}>(L_1+L_2)/20\end{array}\right.,$$

(10)

$$\mathrm{S}\mathrm{i}\mathrm{m}\mathrm{W}=\left\{\begin{array}{c} True, \Delta \mathrm{W} \le (W_1+W_2)/20\\ False, \Delta \mathrm{W}>(W_1+W_2)/20\end{array}\right.;$$

(11)

3.

The contour area deviation adopts the relative threshold of no more than 1/20 of the sum of the comparison contests. The similarity in the contour areas (SimS) is determined as follows:

$$\mathrm{S}\mathrm{i}\mathrm{m}\mathrm{S}=\left\{\begin{array}{c} True, \Delta \mathrm{S} \le (S_1+S_2)/20\\ False, \Delta \mathrm{S}>(S_1+S_2)/20\end{array}\right.;$$

(12)

4.

When the values obtained for SimP, SimL, SimW, and SimS are all “True”, the two contour profiles were considered similar. This can be expressed as follows:

Sim(C₁, C₂) = SimP & SimL & SimW & SimS.

(13)

2.5.4. Monomer Judgment and Contour Selection

Based on the assumptions listed in Section 2.5.3, building monomers were determined as follows: Contour polygons of differently inclined models were compared; when all geometric similarity indicators met the predetermined requirements, the two contour polygons were considered to correspond to the same building monomer. The maximum value of the corresponding elevation plane slice and its generated vector polygon (meeting the similarity conditions) represented the contour line of the building. During calculations, small or large contours were removed based on the given area range, decreasing the interference of noisy data. Such area ranges can be customized by users according to the types of buildings present. The stepwise monomer determination and contour selection process is as follows:

• Step 1: Beginning with the highest elevation slice, assign one of the vertical surfaces of the contour polygon as F_S, the source of the polygon outline for the current elevation (elevation = H). At subsequent (lower) vertical “slices” (elevation = H − ΔH), find the outline polygon for a target contour (Fo) that renders Sim (F_S, Fo) = True; in this case, F_S and Fo belong to the same building monomer. F_S is accepted as the outline of the current building, incorporated into the contour set F, and the polygon Fo is determined to be extracted;
• Step 2. If no Lo can be found in the elevation plane H − ΔH and Sim (F_S, Fo) = True, F_S does not represent the contour line of a single building. The next polygon is selected from the elevation plane H as the source polygon, and Step 1 is repeated;
• Step 3. After all contour polygons in elevation plane H have been used as source contours, the H − ΔH elevation plane is taken as the object and the contour polygons in this elevation are selected as source contour polygons. The source polygons are then searched for the next elevation at plane H − 2ΔH, and contours are added to set F. The processes outlined in Step 1 and Step 2 are repeated; (In this step, extracted contour polygons are no longer included in the calculation.)
• Step 4. Traverse all “sliced” surfaces of different elevations in descending order from high to low, repeating Step 1 to Step 3 until all elevation cut surfaces are traversed, and the selected contour set is F.

In addition to buildings, there may be objects such as trees and lampposts. The outline area of these features is small, and the distribution is scattered. To minimize the influence of these ground objects on the accuracy of building monomer extraction, we established a filtering area threshold value, S_t, and eliminated contour values whose area is smaller than from the final contour set F.

3. Results

3.1. Semantic Segmentation of an Oblique Photography 3D Model

The data used for the 3D semantic segmentation in this study comprised three sets: a training set, a test set, and experimental segmentation data. Training set data were used to train the classification model, and test set data were used to test the classification results and evaluate the model’s classification accuracy. Our experimental segmentation data were used to perform a 3D segmentation experiment according to a semantic segmentation model. Due to a lack of publicly available oblique photography datasets, we independently annotated the training and test data. Their data were divided into packets for small scenes of 140 m × 140 m to facilitate data annotation and training tests. The training set included 3D data for 10 scenes, including 3,325,855 triangular mesh surfaces. The test set included the 3D data for two scenes, including 944,858 triangular mesh plates. The experimental segmentation data included the 3D data for two scenes obtained via oblique photography, covering approximately 3 km² and 0.1 km², respectively. This experimental data were selected from an oblique photography 3D model of Changsha City, China, and the data objects covered buildings, ground, trees, and other elements with strong representation, as shown in Figure 9.

3.1.1. Data Format Conversion and Point Cloud Mapping

According to this study’s point set mapping method, all grid data were converted and mapped to 3D point cloud data. Figure 10 illustrates the process of point set mapping; Figure 10A is the original oblique photography model, Figure 10B is its corresponding grid model, Figure 10C is an enlarged local grid model, and Figure 10D shows the data mapped as a point cloud.

3.1.2. Data Classification Annotation

Currently, there are no accessible training or test datasets for the 3D semantic segmentation of oblique photography. Semantic marking tools were used to annotate point cloud data after 3D tilted mesh data mapping (as shown in Figure 11) to provide training data for our classification model and conduct tests.

Figure 12A depicts the training set data for 10 selected scenes, including buildings, the ground, trees, and other objects. This study classified data into four categories: ground, buildings, trees, and others. Figure 12B shows the product data after converting the original grid data from the training set (OBJ files) to point cloud data containing texture information. Following semantic annotation, the model file contained all the classification annotation information.

3.1.3. Model Training

The process of 3D semantic segmentation involves several key parameters, including the k-neighborhood number and energy function coefficient γ. During our model training, we adjusted training accuracy by dividing the relative influences of these two parameters as follows:

• Influence of the k value of k-neighborhoods on classification accuracy. With γ set to a fixed value of 0.2, the empirical results of training with different pairs of k values are shown in

  Table 2. Results of classification accuracy and F1 scores under different neighborhood k values.

  K values	4	6	8	12	16	20	30
  Classification accuracy	0.9553	0.9561	0.9583	0.9565	0.9519	0.9460	0.9460
  Mean F1 score	0.8796	0.8774	0.8875	0.8912	0.8354	0.8737	0.8737

  . The effects on the classification accuracy and mean F1 score were also investigated. According to a comprehensive analysis, when the k value was between 6 and 12, the training classification accuracy and F1 score obtained good results (Figure 13). As a result, the empirical value for k in this study was set to 10.
• The influence of the energy functions coefficient γ on classification accuracy. With k set to a fixed value of 10,

  Table 3. Results of classification accuracy and F1 scores under different energy function coefficients (γ values).

  Gamma (γ) values	0.01	0.1	0.4	0.6	1.0	2	6
  Classification accuracy	0.9535	0.9553	0.9566	0.9544	0.9471	0.9351	0.8844
  Mean F1 score	0.8700	0.8556	0.8907	0.8873	0.8278	0	0

  presents the training results using different pairs of γ values. Their effects on classification accuracy and mean F1 scores were investigated. According to a comprehensive analysis, a γ value between 0.4 and 0.6 delivered both training classification accuracy and an F1 score that obtained good results (Figure 14). As a result, the empirical value for γ in this study was set to 0.5.

3.1.4. Semantic Segmentation

Three scenes were selected for the segmentation test, as shown in Figure 15 and Figure 16.

The precision, recall and interchange ratio (IoU) were used to evaluate the classification results of multiple categories: a binary confusion matrix was adopted, the positive case was assumed to belong to the current class, and the negative case was assumed to belong to the non-current class. This study first used the training set data to train the model and then applied the test set data to the trained model. The results are summarized in

Table 4. Test results based on the training model.

Category	Precision	Recall	IoU
Ground	0.795524	0.796856	0.661390
Building	0.872807	0.822344	0.797025
Trees	0.766479	0.895214	0.703374
Other	0.748356	0.823761	0.723279

. In the test dataset, the classification accuracy was high, particularly when the building accuracy reached 0.87. For 323,081 points in the test scenario data, the running time of semantic segmentation was 98 s.

3.2. Building Monomer and Contour Extraction

A building model with an area of approximately 1 km² was selected and sectioned at different elevation planes. The projection and binarization results of grid model slices are shown in Figure 16, where h represents the elevation of the different planes.

After obtaining the binarized raster image of elevation planes as shown in Figure 17, filtering and edge detection technologies were used to extract the initial edge contours of the building, as shown in Figure 18.

The practical application of this method primarily focuses on two indexes: one is the monomer extraction accuracy, and the second is the recall rate of monomeric buildings. To evaluate the accuracy and recall ratio, this study calibrated the true value data according to the vector data of urban architecture status. According to the proposed algorithm, the building monomer extraction’s accuracy and recall ratio is primarily affected by the elevation difference ΔH used in ”cutting” and the area threshold S_t used in the filtering of building contour.

Table 5. Accuracy and recall under different elevation differences, ΔH.

ΔH (m)	Accuracy	Recall
0.5	0.812	0.922
0.8	0.821	0.905
1.0	0.833	0.893
1.5	0.834	0.894
2.0	0.843	0.885
2.5	0.854	0.871
3.0	0.877	0.797
5.0	0.885	0.786
6.0	0.901	0.732

and

Table 6. Accuracy and recall under different area thresholds, S_t.

St (m2)	Accuracy	Recall
3.0	0.732	0.903
5.0	0.781	0.894
8.0	0.792	0.886
10.0	0.866	0.879
20.0	0.893	0.885
50.0	0.905	0.864
80.0	0.914	0.822
100.0	0.923	0.782
150.0	0.945	0.742

compare the accuracy and recall under varying elevation differences and area thresholds, respectively. When evaluating the influence of the above two factors on the monomer effect, the other factor must be assigned a constant value. In Table 5, the area threshold is 200 m; in Table 6, the elevation difference is 2 m.

Figure 18 and Figure 19 show the corresponding change curves in Table 5 and Table 6.

As shown in Figure 19, as the elevation difference ΔH increases, the accuracy of the monomer increases, whereas the recall rate decreases. This is consistent with the intuitive judgment based on experience. When the elevation difference increases, some low trees and non-building features are filtered out to improve the accuracy. However, extremely large elevation differences may result in the omission of some low buildings during extraction, thereby reducing the recall rate.

As shown in Figure 20, the accuracy of the building monomer extraction and the recall rate increase with the increase in the area threshold. This is also consistent with the empirical judgment. When the size threshold increases, smaller trees and non-building features are eliminated; however, smaller house structures may also be eliminated.

Figure 21 depicts the monomer rendering results according to the building contours and height information extracted via monomer segmentation. The monomer segmentation and extraction of an oblique photography 3D model with a range of 5 km² took approximately 456 s. Through manual interpretation and verification, the accuracy of this monomer extraction was approximately 90%.

4. Discussion

This study addressed the lack of semantic information in 3D models of oblique photography. A semantic segmentation method was proposed based on 3D mesh data obtained from oblique photography. A point set mapping method was adopted to convert the mesh data into point cloud data, and rough geometric segmentation was performed simultaneously. We adopted a neighborhood search method based on geodesic distances in the local neighborhood computation of semantic segmentation. During feature selection and calculation, superpoint theory was used to fuse local features, realize the feature extraction of the segmented object, and reduce the computational load of machine learning. Finally, the oblique photographic model was partitioned into ground, trees, buildings, and others to segment the scene visually. Our proposed technique can advance the 3D semantic segmentation of oblique photography based on the successful implementation results. The semantic segmentation method proposed in this study for oblique photography 3D data was validated in Section 3.1. From the implementation results, this method can segment the continuously surfaced 3D mesh model into different land object types, which is beneficial for the subsequent processing and application of oblique photography 3D data.

To acquire 3D building monomers and conduct contour extraction using semantically segmented oblique photography, our technique included hierarchical “slicing” of a 3D model followed by similarity assessments of polygon geometry. This method used a 3D mesh model as the object, projected the triangular meshes of buildings onto different elevation planes, and then processed mesh binarization and mesh vector conversion to extract initial contour sets for buildings. Finally, a polygon space similarity discrimination method calculated a single body and extracted the adopted building contour from the contour set data. The proposed method for individualizing building models based on oblique photography 3D data was validated in Section 3.2. The implementation results show that this method can accurately extract individual components from oblique photography 3D models. This is beneficial for the subsequent object-oriented management of oblique photography 3D models.

This study also analyzed and evaluated the time efficiency of the proposed method. In terms of 3D semantic segmentation, the computation time of the model presented in this paper is approximately 98 s/km². For building individualization, the computation time of the method proposed in this paper is approximately 90 s/km². Based on an area of approximately 200 km² for a medium-sized city, it is estimated that the model in this paper will take around 10.4 h to complete 3D semantic segmentation and individualization. This processing time is acceptable in practical applications.

Compared to existing methods for semantic segmentation and individual extraction of urban 3D models, this study exhibits several characteristics and innovations, including the following:

(1) Regarding 3D semantic segmentation, this study utilized oblique photography-based 3D models as the data source and conducted point cloud processing, forming superpoints through clustering to reduce computational complexity. A search method based on surface model geodesic distance was proposed for feature neighborhood searches. It differs from traditional Euclidean distance clustering methods for discrete point cloud data by leveraging spatial topological information of mesh data and providing a more rational approach to spatial distance measurement;

(2) In terms of monomer and contour extraction of oblique photography-based 3D architectural models, this study innovatively proposes an approach based on high-level plane segmentation and geometric similarity analysis. Compared to physical or interactive individualization methods, our method has the advantages of not relying on third-party data, possessing a high degree of automation, and integrating monomer and contour extraction;

However, this study has some limitations. Three aspects of developing 3D reconstructions using oblique photography require further research: First, a more refined semantic segmentation is needed. Microscopic and fine semantic recognition should include the classification of components such as windows, balconies, and roof structures. Second, in terms of 3D semantic segmentation, more advanced deep learning methods can be employed to further improve segmentation accuracy. Third, the extracted single building contours should be regularized to reduce jagged edges to meet application needs.

5. Conclusions

Oblique photography 3D reconstruction is an important simulation modeling method with broad applications. To meet the needs of semantic extraction and building monomers from such photography, this study used 3D mesh data as the research object and proposed a holistic solution. Our technique enriched the post-processing system for the 3D reconstruction of oblique photography, exploring several problems and offering a feasible solution.

First, a machine learning-based semantic segmentation method is proposed for oblique photography 3D model data. According to the characteristics of the data to be semantically segmented, in the specific solving process, various methods, such as mesh data point mapping, superpoint construction, and spatial measurement based on geodesic distance, are comprehensively used for data processing and neighborhood retrieval. This allows for feature fusion and extraction of segmentation objects, followed by the use of traditional machine learning methods to accomplish semantic segmentation.

Second, relying solely on the oblique photography model as the data source, a monomer and contour extraction method is proposed based on model elevation plane “cutting” and polygon geometric similarity judgment. This method can simultaneously realize model individualization and outline extraction for buildings.

The experimental results support the effectiveness and applicability of the proposed method. However, there are some limitations in this study. In the future, further research is warranted regarding semantic segmentation granularity, the introduction of deep learning methods, and the regularization of building outlines.