A Method for 3D Reconstruction of the Ming and Qing Official-Style Roof Using a Decorative Components Template Library

Abstract

The ancient roof decorative components of the official-style architectures from the Ming and Qing dynasties in China hold both physical and symbolic significance. These roof structures are the essential objects in three-dimensional (3D) modeling of ancient architectures for traditional Chinese cultural preservation. Although ancient architectures can be surveyed by a 3D laser scanner, the complex geometry and diverse pattern of their roof decorative components make the 3D point cloud reconstruction challenging, or at some points, nearly impossible in a fully automated manner. In this paper, we propose a method to ensure that the 3D shape of each roof decorative component is accurately modeled. First, we establish a decorative components template library (or “template library” in short hereafter), which is the first of its kind for the roofs of Ming and Qing official-style architectures. The process of establishing the decorative components template library begins with a remote collection of survey data using a terrestrial laser scanner and digital camera. The next stage involves the design and construction of different 3D decorative components in the template library with reference to the manuscripts written in the Ming and Qing dynasties’ architectural pattern books. With the point cloud data collected on any Ming and Qing official-style architecture, we further propose a geo-registration mechanism to search for an optimal fitting of the decorative components from the template library on the collected point cloud automatically. Based on the experimental results, the accuracy of point cloud registration yields less than 0.02 m, which meets the accuracy of the 3D model at LoD 300 level. Time consumption is less than 5s and stable, for large volume computing capacity has good robustness. The proposed strategy provides a new way for the 3D modeling of large and clustered historical architectures, particularly with complex structures.

1. Introduction

The Ming and Qing dynasties (1368–1912) were the last feudal period in China. In this period, the hierarchy was strictly divided, and the construction method of royal residence was unified, which created a unique official-style architecture system. The Ming and Qing official-style architectures, which are considered as the last peak of Chinese architectural history, are an important carrier of traditional Chinese culture [1]. The protection of these ancient architectures is imminent due to the occurrence of natural disasters and man-made destruction. Traditional documentation methods of the Ming and Qing official-style architectures mainly rely on the use of paper documents (including paintings and tables), two-dimensional (2D) digital images, and so on. A paper document tends to be abstract and is unable to reflect the detail geometry of the architecture. Additionally, a large batch of storage results in inconvenient information retrieval. Although 2D digital images have the privilege of intuitive display, it is mostly impossible to perform geometric measurements or retrieve the dimension of the objects. Therefore, all these approaches are unable to provide reliable data support for reconstruction and restoration, and it is not conducive to serve the purpose of accurate preservation and scientific research. With the development of digital technology, especially the emergence of the three-dimensional (3D) laser scanners (or LiDAR), digital preservation has become viable for the Ming and Qing official-style architectures [2,3,4]. Nevertheless, the unstructured and bulky 3D point clouds require intensive post processing. It thus gives rise to certain challenges—for example, transforming the unstructured 3D point clouds into a digital 3D model for subsequent analysis and cultural heritage management.

The use of point clouds for 3D reconstruction can be commonly divided into three categories: data-driven methods [5,6,7], model-driven methods [8,9,10] and knowledge-driven methods [11,12,13]. Studies in [14,15] provide comprehensive reviews regarding the results of 3D building reconstruction using point clouds. Comparing to modern buildings, the structure of historic architecture is rather complex and heterogeneous. Therefore, it is mostly impossible to adopt any one of the aforementioned methods to generate an accurate 3D model for historic architecture. Thus, integrated methods are proposed in order to overcome the limitations [16,17,18]. In [19], two types of Milan Cathedral’s main spire structure were individually reconstructed using different methods. The first type are mostly regular shapes, they were reconstructed by a model-driven method. The second type of structure consisted of flower decorations or artistic components, which cannot be modeled using parameters. Thus, these components were represented by 3D mesh models based on the data-driven method. The final digital 3D model of Milan Cathedral’s main spire was generated by merging these two types of components. In [20], a similar method is also applied to the loggia of Castel Masegra (Sondrio, Italy). For regular and irregular shapes, existing libraries and Non-Uniform Rational B-Splines (NURBS) are used for 3D reconstruction, respectively. Although such an integrated method can also be adopted for the reconstruction of the Ming and Qing official-style architectures, this approach heavily relies on manual intervention and is time consuming. Hence, an automatic 3D reconstruction approach for the Ming and Qing official-style architectures is still desired.

As shown in Figure 1, the Ming and Qing official-style architectures mainly consist of four parts: (1) platform or stylobate, (2) column or wall, (3) Dou-gong and (4) roof [21]. The components of each part can be further divided into two categories: regular components and irregular components. Generally speaking, the regular components have well-defined characteristics and follow the potential geometric relationship [22,23]. Taking the roof structure as an example, it is mainly composed of ridge, tile and decorative components. The section shape of the semicircular tile is a half-round ridge, where the section shape of the board tile is one-eighth round [24]. The geometric features, such as straight-line segments, curve and plane, can facilitate automatic 3D model construction from the point cloud and/or image with the aid of this prior information. The irregular components mainly contain the decorative components, especially the roof decorative components, which are hard to model based on the geometric features. Hence, the mesh model is still needed in the process of 3D modeling. Since the data points corresponding to the decorative components cannot be completely segmented, this makes automatic reconstruction more challenging. To overcome this problem, it thus inspires us to create a new template library for each decorative component. After the decorative components are detected in the collected point clouds of any Ming and Qing official-style architectures, the 3D model of the detected decorative component from the pre-defined template library can be directly applied after the rigid transformation. This not only reduces the 3D modeling work of repetitive complex type components, save computational resources, but also better ensures the accuracy of the constructed model. There is no denying that establish a template library requires a great deal of preparation, including the inevitable manual modeling effort, and that getting the template library to cover the full range of components will be a challenging task. However, since the decorative components on the roof of the Ming and Qing, official-style architectures are highly similar and with a limited number of types, the constructed template library can be universally adopted on any Ming and Qing official-style architectures.

A similar approach for constructing a template library can be found in [25,26,27,28] (see

Table 1. A comparison of existing template library used in 3D reconstruction.

Study	Object Type	Basis for Library	Software/Tools	Method of Application	Analysis
Zdeněk Poloprutský [24]	Coaching inn in Kostelec nad Vltavou (Písek District, Czech Republic)	Surveying data (Photo documentation, point cloud and drawings)	Autodesk Revit software	/	This modeling method can design a library usable for detailed BIM models.
Murphy [25]	European classical architecture	Vitruvius to the 17th and 18th century Architectural Pattern Books	GDL embedded in the Graphisoft ArchiCAD. Decoration and non-uniform shapes based on NURBS, meshing and Boolean operations	The library is designed as a plug-in for existing software platforms for mapping these objects onto point clouds and image surveys.	Placing a construction element or GDL object, the default parameters need to be manually edited in order to correspond with the survey data.
A. Baik [26]	Old Jeddah historical buildings	Laser scanning point clouds and photogrammetric survey	Modeling library elements in 3D modeling software such as Autodesk Revit and Rhinoceros	The library is used as plug-in of existing BIM software platforms such as Autodesk Revit.	Determining the accurate position of the object’s elements requires preprocessing.
C Dore [27]	Building facades	Laser scanning, photogrammetric data, and references to Vitruvius to Palladio to the architectural pattern books of the eighteenth century	Parametric objects were built using GDL embedded in the ArchiCAD BIM software	Automatically position library components based on architectural rules and proportions, but manually resize and plot the components.	Users have a full control over the global and local facade for simple structure components.

). Most of the existing template libraries do not bundle with an automatic mechanism of identifying components. Instead, the majority of them rely on manual operation to identify a specific component type, and subsequently let the operator perform “drag and drop” in order to align the component into the corresponding location. Although this method ensures the accuracy of the reconstructed 3D model, its efficiency is limited. Therefore, having an automatic 3D reconstruction with reference to the decorative component template library is sound, particularly for the large number of existing and complex structure of the Ming and Qing official-style architectures.

2. Method

In this study, the contributions are twofold. First, we establish a decorative component template library, which is the first of its kind for the roofs of Ming and Qing official-style architectures. The library development is based on several studies that constructed decorative components and template libraries for different architectures [29,30,31]. Second, this template library can be used to automatically reconstruct 3D roof decorative components of any Ming and Qing official-style architecture based on a geo-registration mechanism.

After the template library was constructed, a geo-registration mechanism based on object identification and point cloud matching is proposed for 3D reconstruction. In this process, each decorative component can be positioned, located and scaled automatically in accordance with the architectural rules and proportions, and thus the process is instructive for other similar work. The automatic geo-registration mechanism is divided into four steps (see Figure 2). First, the corresponding decorative component type is detected based on the “You Only Look Once v3” (YOLOv3) algorithm [32]. Second, the corresponding decorative component model from the template library is transformed into a 3D point cloud. Third, a coarse registration based on the Sample Consensus Initial Alignment (SAC-IA) algorithm [33] is performed. Fourth, a fine registration based on the Scaling Iterative Closest Point (SICP) algorithm [34] is performed. The YOLOv3 detection results are further filtered by the following criteria: (1) if the confidence coefficient of the YOLOv3 detection result is greater than the set threshold, then the subsequent point cloud registration will be carried out directly; (2) if the confidence coefficient of the YOLOv3 detection results is less than the set threshold, then the first three results with the highest confidence coefficient are selected for point cloud registration. The object whose registration result reaches the preset accuracy value and performs best is regarded as successful; (3) if it does not conform to 1 and 2, it will be considered as a failure. The operation of this component will be skipped directly.

3. A Template Library of Roof Decorative Components

The understanding of the roof decorative components is an important prerequisite for the establishment of the decorative component template library which include their category, shape, location, function, rule, and cultural connotation. These are required to reconstruct the roof according to the architectural pattern books and manuscripts (see Section 3.1. for more details). In this template library, the coarse 3D model of decorative components is reconstructed based on point cloud data with the aid of photogrammetric computer vision or laser scanning technology. The collected point cloud can be classified into two main categories: dense image matching (DIM) point cloud and laser scanning point cloud. Subsequently, the roof decorative components in the 3D model are refined with reference to the design and detail listed in the architectural pattern books and manuscripts using 3D modeling software. These roof decorative components are then stored in the template library following a pre-defined coding schema. In general, this strategy is not bound by historic architecture. It also applies to any type of building with specific design rules so that a corresponding template library can be established accordingly.

3.1. An Overview of Roof Decorative Components in the Template Library

The “Qing Structural Regulations”, compiled by SiCheng Liang in 1934, lists the details of the Ming and Qing official-style architectures, including the construction method, layout, shape and structure of the buildings, as well as the name, dimension, topology and function of each component [35]. As shown in Figure 3, the roof can be classified into five main categories according to the structure, which are hip roof, gable-and-hip roof, pyramidal roof, overhanging gable roof, and flush gable roof (Figure 3a–e). The hip roof, gable-and-hip roof, and pyramidal roof can be further divided into double-eave and single-eave (Figure 3f–h). Moreover, the gable-and-hip roof, gable roof, and overhanging gable roof can derive another type of round ridge roof (Figure 3i). These roofs not only inherit the style of the Ming and Qing official-style architectures but also represent the social hierarchy of the inhabitants. From a high to low rank, the roof type is double-eave hip roof, double-eave gable-and-hip roof, double-eave pyramidal roof, hip roof, gable-and-hip roof, pyramidal roof, overhanging gable roof, round ridge roof, and flush gable roof [21].

With reference to the template library, the procedure of 3D modeling the roofs of Ming and Qing official-style architectures follows certain rules and schemes [36]. First, the number of roof decorative components located on different types of roof is specified. For instance, the Hall of Supreme Harmony in Forbidden City (Beijing, China) is the highest-grade Ming and Qing official-style architecture, and the roof type is double-eave hip. The number of Zou shou, which is a divine creature on the roof ridge that is said to ward off evil, is ten. Since the grade of the other Ming and Qing official-style architectures is lower, the number of Zou shou, which can only be nine or less, has to be an odd number [37]. In addition, some roof decorative components can only be found on a specified type of roof. We thus divide the roof decorative components into five categories: Chui shou, Tao shou, Wen shou, Zou shou and other types of decorative components (see Figure 2) according to historical records, especially the records from the Qing Dynastic Ministry of Works [38]. These specific rules and proportions outlined in the manuscripts related to the official-style architectures can provide an initial estimate of the position and size of the roof decorative components. This prior knowledge certainly helps reduce the amount of further editing when performing a digital 3D modeling of the official-style architectures from the collected data and facilitates the possibility to use the template library automatically.

3.2. Construction of the Template Library

The Ming and Qing official-style architectures are composed by different complicated and irregular components. The constructed 3D model in the proposed template library should possess different levels of detail (LoDs) that can be used to serve different purposes. For roof decorative components, they need to be accurately and adequately represented so that the geometry, attribute, and parameter proportion are preserved. Therefore, the LoDs and the desired accuracy of these components should be determined before creating the template library.

In terms of modeling the buildings, previous studies [39] mostly followed the standardized protocol of LoDs in Building Information Modeling (BIM) defined by the American Institute of Architects to specify the LoD of 3D architectural models [40], although some studies followed the level of geometry [41,42] and the grades of generation [43]. Similarly, we also classify the 3D model into five LoDs (see

Table 2. The description and accuracy of the level of details.

Level of Details	Accuracy of Planar/Elevation	Scale
LoD 100	>LoD200	/
LoD 200	0.05 m/0.05 m	1:500
LoD 300	0.02 m/0.02 m	1:200
LoD 400	0.01 m/0.01 m	1:100
LoD 500	0.005 m/0.005 m	1:50

). LoD 100 models the decorative components using 2D historical manuscripts and pattern books without accurate parameter information. LoD 200 describes the outline of decorative components by prismatic extrusion solid with certain parameter information. LoD 300 represents the character of decorative components by the collected data (point cloud or digital image), and the outline is represented by either a Delaunay triangular mesh or raster data. At LoD 400, the external geometric form is represented by detailed surfaces and shapes, allowing the proper orientation of Heritage Building Information Models (HBIMs) for in-depth BIM-based analyses. LoD 500 provides the geometric details of external and internal forms and new levels of information sharing for different purposes.

Brumana et al. [44] discussed the HBIM level of detail-geometry-accuracy. In their work, they adopted a scale ranging from 1:1000 to 1:10. As a result, the models of decorative components at LoD 100 to LoD 200 do not have shapes in detail. Hence, the accuracy of the decorative components in the template library should not be less than that of LoD 300 (≤0.02 m), which corresponds to the scale of 1:200. This thus meets the need for 3D reconstruction of the roof of the Ming and Qing official-style architectures.

After the desired accuracy is defined, the following method was applied to construct each decorative component in the template library. The library components were first generated from the DIM point clouds that represent the geometry and fabric of the roof decorative components, and the Delaunay triangulation was implemented to automatically reconstruct the 3D mesh models. Three-dimensional mesh can be used to reconstruct standard shapes (lines, circles, parabolas, etc.) or free-form profiles. However, such an automatic reconstruction is unlikely to meet the expected accuracy requirements. In addition, the details of decorative components cannot be revealed. Therefore, we imported the 3D mesh models into the Autodesk 3Ds Max software and performed manual editing on the model with reference to their materials, historical research, existing drawings and pictures.

An example of the reconstruction of decorative components is illustrated in Figure 4. This complex component represents the Wen shou, which is usually located on either side of the main ridge. Ribs have a circular structure that can be reconstructed with manual measurements on the point cloud. This allows a geometric reconstruction that preserves the uniqueness in terms of both component style and structure. The proposed approach can be summarized as follows:

• Acquisition and registration of the decorative component’s point cloud (Figure 4a);
• Surface generation using the point cloud and the mesh network (Figure 4b);
• Construction of a 3D model by editing the mesh model with a manual, semiautomated or automated approach (Figure 4c).

At present, this template library includes 105 models of decorative components, which are located in different regions of China and built by following the construction style of the Ming and Qing dynasties. Among them, the official-style decorative components are divided into five most representative types—Chui shou (16), Tao shou (12), Wen shou (18), Zou shou (32) and other types of decorative components (27), as described in

Table 3. The main roof decorative components of the Ming and Qing official-style architectures.

Categories	Shape	Location	Function	Rule	Cultural Connotation
Chui shou		The Chui shou is behind the Zou shou.	The Chui shou is also used to cover and decorate the iron nails that prevent the vertical ridge tiles from slipping and reinforce the roof ridges	Both the levels of Zou shou and Chui shou are the same. The height of the Chui shou is 1.5 times to the height of the forehead height, and the width to height ratio is 1~1.2 to 1.	/
Tao shou		The Tao shou is located at the top of the wing corner beam or corner beam.	The Tao shou is responsible for waterproofing and decoration.	The Tao shou has a height-to-length ratio of 1 to 1.4, and its thickness is consistent with height. The shape usually is a lion or the head of a dragon.	According to ancient Chinese legend, Tao shou can protect homes from fire.
Wen shou		The Wen shou is located above the midline of the ridge, and at the ends of the main ridge	Seal the gap of the tile ridge to avoid water seepage, enhance the stability of the main ridge, and promoted the Tenon-and-mortise structure to be tightly joined.	The levels of the wen shou and the main ridge are the same. When the architecture has double eaves, the level of the wen shou can be one level higher than the main ridge. The width to height ratio of the Wen shou is 7 to 10.	The moral of the Wen shou is to protect the family rich in food and clothes and has a large population, and Wen shou can also keep their homes safe in legend.
Zou shou		Zou shou is distributed in the lower half of the vertical ridge. Celestial being is usually located above the hooks at the corners of the roof, with Zou shou behind it.	The tiles of the vertical ridge are fixed with iron nails to prevent slipping, and the Zou shou is used to cover and decorate the iron nails.	The width to height ratio of the Zou shou is about 4 to 10 more than 6 to 10, and the ratio of thickness to eyebrow height is 3 to 10. According to the social hierarchy of the occupants to determine the number of placements. The total number is usually singular (not containing the Celestial man). The order of arrangement is Dragon, Phoenix, Lion, Tian ma, Hai ma, Suan ni, Xia yu, Xie zhi, Dou niu, and Xingshi.	Dragon, a legendary creature that can thrive in clouds and rain, and is a symbol of imperial power; Phoenix, a metaphor for a person with virtue, is also the name of a queen; Lion, the guardian of Buddhism, a symbol of bravery and majesty. Tian Ma, a loyal beast, chasing the wind; Hai Ma, the brave beast, enters the sea and enters the abyss. Suan Ni, playing the role of the mighty beast; Xia Yu, strange beast in the sea, making clouds rain, firefighting, and disaster prevention; Xie Zhi, brave and fair, implying justice and fairness; Dou Niu, a mascot for disaster elimination; Xing Shi, a bi-winged monkey statue, magic reduction, and lightning protection, is the last beast.
Others		Wa dang, Xuan yu and tiles.	It not only reinforces the roof structure; they also deemed to be an important carrier of Chinese folklore.	/	People place their hopes on decorative components, looking forward to every family’s peace, and prosperity, health, and happiness.

.

3.3. Template Library Coding Schema

A hierarchical coding schema is introduced in the template library in order to efficiently manage the components according to the rules of the Ming and Qing official-style architectures. As shown in Figure 5, the code of the decorative components consists of three blocks: the rigid code, flexible code and flow code. The rigid code defines the Ming and Qing official-style architectural style by the type of roof, as listed in

Table 4. Design of rigid code.

Type of Roof	Code	Type of Roof	Code
double-eave hip roof	DH	pyramidal roof	SP
double-eave gable-and-hip roof	DG	overhanging gable roof	SO
double-eave pyramidal roof	DP	gable roof	SG
hip roof	SH	flush gable roof	SF
gable-and-hip roof	SA

. The flexible code is divided into three levels, as shown in

Table 5. Design of flexible code.

First-Level Code		Second Level-Code		Third-Level Code
Platform or Stylobate	P	Intermediate pier	IP	…
		Step	ST
		Curb	CU
		Corner pier	CP
Column or Wall	C	Corner column	CC	…
		general bay column	GB
		Base	BA
		Sill	SI
		Wall	WA
		Window	WI
Dou-gong	D	Corner set	CS	…
		Lintel	LI
		Intermediate set	IS
Roof	R	Ridge	RI	…
		Beam	BE
		Tile	TI
		Decorative element	DE	Wen shou	WS
				Zou shou	ZS
				Chui shou	CS
				Tao shou	TS
				Others	DO

Note: this table is only an example of the coding design of flexible code segment and does not include the complete structural type of the Ming and Qing official-style architectures and does not represent the coding design of all Ming and Qing buildings. This paper focuses on the decorative components of the roof, the rest parts are simplified expression.

. The first-level code is named by a capital English letter, indicating the reconstruction object that belongs to a certain part of the Ming and Qing official-style architectures. The second-level code is named by two capital English letters, indicating the specific structure, and the third-level code refers to the name of the decorative components. The flow code can be used to represent variations of the same category of decorative components.

4. Geo-Registration Mechanism for Template Library

After the template library is established, the next task is to perform a geo-registration of the corresponding model of the roof decorative components in the library with the point cloud collected for any Ming and Qing official-style architecture. The process first detects the decorative components on the collected digital images based on the You Only Look Once v3 (YOLOv3) algorithm and determines the corresponding model in the template library according to the coding schema. Subsequently, the chosen 3D model in the template library is placed on the correct location of the roof using a point cloud matching algorithm; i.e., the Iterative Closest Point (ICP) algorithm [45]. Although the ICP point cloud registration method is proven accurate, it is also computationally demanding. Therefore, it is not conducive to process a large amount of data and is contrary to our intention to improve the degree of automation. It is worth noting that the registration results are better when the spatial location of the two sets of point clouds are close to each other. In this case, the target point cloud is likely to converge to a local minimum, which inspires us to propose a two-step mechanism with a coarse registration first and then a fine registration. This geo-registration mechanism not only overcomes the drawbacks of existing approaches relying on the manual operation of the template library, but also optimizes the model fitting results.

4.1. Automatic Detection of Roof Decorative Components

Prior to the geo-registration, the roof decorative components should be automatically detected on the collected images of any Ming and Qing official-style roof architecture. In this study, we adopt a deep learning algorithm for roof object detection that includes two major steps. First, because the Ming and Qing official-style architectures are famous and widely distributed in China and are very large in number, using the web crawler to obtain abundant Ming and Qing architectural roof decorative component images from the open network platforms is a good way to acquire this information. At the same time, a lightweight neural network was built to assist rapid image cleaning. After this, a dataset of roof decorative components can be successfully constructed through a few artificial discrimination and marks. Then, the YOLOv3 algorithm was applied to extract multi-level features of images and further integrated them into high-level semantic features of images to realize the detection of decorative components. The backbone of YOLOv3 is the Darknet-53 feature extractor. It has a stronger ability to extract image depth features and performs better on ImageNet datasets than Darknet-19, Resnet-101 and Resnet-152 [46,47]. Moreover, it achieves the highest floating-point calculation per second and higher efficiency [48]. Figure 6 shows the detection results of Chui shou and Zou shou on the roof of the Hall of Supreme Harmony (Beijing, China).

4.2. Transformation of 3D Template Roof Model into 3D Point Cloud

Once the roof object is successfully detected, the corresponding roof decorative components in the template library can be used to perform geo-registration. However, the 3D model cannot be used directly to match with the collected point cloud data. Therefore, it is necessary to transform the 3D model in the template library into a 3D point cloud. There may be a difference in terms of the point cloud density and the scale between the collected point cloud $P$ and the point cloud transformed from the template library $Q$; therefore, we intend to adjust the selected decorative component in the template library to be consistent with the collected point cloud. We reduce the density of the 3D point cloud of the decorative components to a specific density level that matches with the point cloud of the collected architecture prior to geo-registration. The ratio of the scales $S$ between two sets of point cloud data is shown in Equation (1).

$$\frac{S_P}{S_Q}=\frac{D_P}{D_Q}$$

(1)

In this equation, the $D_P$ is equal to the average distance from each point from collected point cloud $P$ to its centroid, and the $D_Q$ is equal to the average distance from each point from transformed point cloud $Q$ to the centroid.

4.3. Coarse Registration Based on SAC-IA

The coarse registration aims to determine an initial relative relationship between the two sets of point clouds based on a classical method named SAC-IA. SAC-IA was selected because this algorithm can ensure that the “match point cloud” and “target point cloud” roughly have the same spatial position. Because this algorithm uses Fast Point Feature Histogram (FPFH) features for point registration, compared with other algorithms (such as 3D scale-invariant feature transform (SIFT) feature matching algorithm), the computational efficiency is higher. Moreover, this algorithm adopts the matching strategy of random sampling, which can also realize registration quickly. Prior to the coarse registration, the FPFH features of each point were calculated, which can make the operation more efficient [33]. The SAC-IA can be divided into three steps.

(1)

Select $n$ points from the collected point cloud $P$ on the unknown Ming and Qing official-style architecture. In order to ensure that the selected points have different FPFH features, the distance between each selected point should be greater than a pre-defined minimum distance threshold $d$.

(2)

In the transformed point cloud $Q$ derived from the template library, the SAC-IA first searches the points which have similar FPFH features of the selected points from collected point cloud $P$. Then, these points found within the searched points are selected as the corresponding points.

(3)

Calculate the transformation matrix between the corresponding points, and then evaluate the performance of the coordinate transformation based on the sum of the distance error after using the corresponding points for transformation. In this paper, the sum function of the distance errors can be expressed as the Huber penalty measure $H$ as follows:

$$f={\displaystyle \sum }_{i=1}^{n}H\left(l_i\right)$$

(2)

where,

$$H\left(l_i\right)=\left\{\begin{array}{c}\frac{1}{2}{l}_i^2,\hspace{1em}\mathrm{if}\Vert l_i\Vert \le ml\\ \frac{1}{2}ml\left(2\Vert l_i\Vert -ml\right),\hspace{1em}\mathrm{if}\Vert l_i\Vert >ml\end{array}\right.$$

(3)

where $ml$ is a pre-defined threshold and $l_i$ is the difference of distance between the corresponding points. Figure 7c shows the initial registration result.

4.4. Fine Registration Based on SICP

The scale of the two sets of point clouds has been preliminarily unified after the coarse registration. However, their spatial positions are still not uniform and so this algorithm requires a further refinement. In this section, we adopt the improved SICP algorithm to register the transformed point cloud $Q$ to the collected point cloud $P$. SICP was selected because the algorithm is not limited to local matching but considers the overall scaling of two sets matching point cloud data, which made the result more accurate [34]. This algorithm can be expressed as Equation (4).

$$\underset{S, R, t, j \in \left\{1, 2,\dots ,N_q\right\}}{argmin}\left({\displaystyle \sum }_{i=1}^{N_p}\Vert \left(RS{p}_i+t\right)-q_j\Vert \begin{array}{c}2\\ 2\end{array}\right)$$

(4)

In Equation (4), $S=diag\left(s_1,s_2,\dots ,s_i,\dots ,s_m\right)$, $s_i\in \left[a_i,b_j\right]$ is a non-zero scale matrix, $R$ is an orthogonal matrix, and $t$ is a translation vector. The implementation of this algorithm is an iterative process and is described as follows:

• Step 1: the correlation is established by the current transformation $\left(S_k,R_k,t_k\right)$. The calculation can be expressed as Equation (5).

  $$c_{k+1}\left(i\right)=\underset{j\in \left\{1,2,\dots ,N_q\right\}}{argmin}\left(\Vert \left(R_k{S}_k{p}_i+t_k\right)-q_j\Vert \begin{array}{c}2\\ 2\end{array}\right)$$

  (5)
• Step 2: make $S=diag\left(s_1,s_2,\dots ,s_i,\dots ,s_m\right)$, and calculate the new transformation $\left(S_{k+1},R_{k+1},t_{k+1}\right)$. The calculation can be expressed as Equation (6).

  $$\left(S_{k+1},R_{k+1},t_{k+1}\right)=\underset{s_j\in \left[a_j,b_j\right],R,t}{argmin}\left({\displaystyle \sum }_{i=1}^{N_p}\Vert RS\left(R_k{S}_k{p}_i+t_k\right)+t-q_{c_{k+1}\left(i\right)}\Vert \begin{array}{c}2\\ 2\end{array}\right)$$

  (6)
• Step 3: repeat step 1 and step 2 until the changed amount $\Delta S$ = $\left|S_{\left(k+1\right)}-S_k\right|$ is less than the threshold or the iteration number is more than the maximum iteration number. Figure 7d shows the result of fine registration.

After the fine registration, the location information is calculated. The corresponding template 3D model from the template library can be retrieved and placed on the roof directly.

5. A Case Study: A Classic Official-Style Architecture

In this section, we use the constructed decorative components template library for the 3D reconstruction of the Ming and Qing official-style architecture roof without oversimplifying the details of the irregularly shaped decorative components. Here, we aim to demonstrate and verify the feasibility of our template library and the geo-registration mechanism by using a real modeling case. A classic Ming and Qing official-style architecture, which resides in the Forbidden City, was selected in the case study.

5.1. Historical Background

The Gate of Supreme Harmony was built in the 18th year of Yongle of the Ming Dynasty (1420), which is located south of the Forbidden City (see Figure 8). It is the largest palace gate in the Forbidden City and the main gate of the imperial palace of the Outer Court. The Gate of Supreme Harmony has nine rooms in width and four rooms in depth, with a construction area of 1300.00 m². The roof is a double-eave gable-and-hip type and occupies more than half of the main building; the beams and ridges are covered with exquisite dragon patterns, in addition to the Chui shou, Tao shou and Wen shou being fixed on the ridges, and there are seven Zou shou and a fairy man on each hip ridge. In the Ming Dynasty, the Gate of Supreme Harmony was the place where the emperor received worship and reports from his subordinates, issued imperial edicts, and handled political affairs.

5.2. Data Collection

In order to construct a 3D model, a complete 3D point cloud of the Gate of Supreme Harmony was collected. The survey was mainly carried out based on an image-based approach. A complete photogrammetric survey was conducted using an Unmanned Aerial Vehicle (UAV) to capture oblique images and a digital camera to collect nadir images. To improve the LoDs for a better documentation of the bottom part (including the walls and platform), the UAV data were integrated with the close-range digital images.

Regarding the UAV equipment, we selected the DJI Phantom4 quadcopter. The design of flight line is shown in Figure 9. The environment and weather will undoubtedly directly affect the collected data’s quality. Therefore, it is necessary to select cloudless and sunny conditions to carry out the data collection work. Using a photogrammetric five-direction flight function, the angle of inclination is about 45–48 degrees. The down-looking images had a forward overlap and side overlap of approximately 60% and 30%, respectively. The flight plan was implemented at an altitude of approximately 60 meters above the ground, and the camera equipped with a lens of focal length 8.8 mm. Finally, a total of 225 oblique images were collected with a 2cm ground resolution of pixels. DJI Phantom 4 is a professional photogrammetric system. Since this study uses small-area data collection, the POS system of this device can fully support the demand for photogrammetric accuracy, so no checkpoint is set up. The terrestrial acquisitions were recorded using a Nikon D750 full-frame digital camera equipped with a lens of focal length 17 mm, and the ground resolution of the terrestrial photo acquisition pixel is 2 cm. The shooting distance was controlled within 5 m, and a total of 474 digital images were captured. Finally, the image acquired by the UAV and the terrestrial image were processed jointly by the bundle adjustment method.

The image data were registered and transformed into DIM point cloud data using the Context Capture Center software [50], and then the DIM point cloud data underwent the process of coloring, filtering, registration, and geo-referencing, etc. Through the point cloud accuracy evaluation method described in the paper [51], the platform and walls of The Gate of Supreme Harmony were used to evaluate the accuracy of the overall point cloud data. Figure 10 shows the result of the point cloud with a precision close to ±3 mm.

5.3. From the Point Cloud to the 3D Model

Once the data collection was accomplished, the digital photos and generated point cloud of the Gate of Supreme Harmony underwent the proposed geo-registration process in order to generate a 3D model. The entire process was implemented based on the integrated development environment, Microsoft Visual Studio 2015, running on a DELL Precision 5510 with a CPU of Intel Core i7-2760QM 2.4 GHz, memory 2.98 GB, and a Windows 10 operating system. Autodesk 3Ds Max was chosen as the 3D modeling environment.

Components, which follow simple design rules, such as ridges and tiles, can be directly modeled with the tools provided in the software. The accuracy of the modeled components can certainly meet the requirement of LoD 300. Regarding the decorative components, the performance of the YOLOv3 algorithm (i.e., YOLOv3, see Section 4.1. for more details) was tested by datasets obtained by web crawlers. A total of 572 samples were selected and tested, which contain 2809 roof decorative components of five categories of Ming and Qing official-style architecture. The overall Average Precision (AP) value of this algorithm reached 0.74. Under the conditions of the confidence threshold of 0.5 and the Intersection over Union (IOU) threshold of 0.6, the test accuracy was 82.2% and the average single-image reasoning speed was 66ms per frame. Both accuracy and speed can meet the application requirements of the geo-registration mechanism proposed in this paper, and then the corresponding component model was directly retrieved from the template library according to the coding schema. The result in this case is accurate compared with the actual situation of the roof (see Figure 11).

Among all the detected roof decorative components, we selected the following representative components—Chui shou (DHRDECS01), Tao shou (DHRDETS01), Wen shou (DHRDEWS01), and Zou shou (DHRDEZS01) from the roof of the Gate of Supreme Harmony for further 3D modeling, as illustrated in Figure 12. In the YOLOv3 detection, the confidence values were 0.86, 0.71, 0.89, and 0.88, satisfying the threshold setting.

The corresponding 3D model detected in the template library was automatically placed on the roof of the Gate of Supreme Harmony according to the geo-registration mechanism. Since the point cloud density is an important parameter that affects the final registration result, in this experiment we randomly diluted the 3D model of the template library into a set of point clouds by 10%, 25%, 50% (the total number of point clouds is 10,000), and evaluated their registration results. This approach also simulated point clouds collected from different data acquisition devices with different LoDs. Figure 13 shows the best registration results with 10,000 data points. The accuracy and time consumption of this process are further analyzed below.

The experimental results of 3D reconstruction were assessed based on computing the Root Mean Square (RMS) error. As shown in Figure 14, the average value of RMS was less than 0.02 m most of the time for the four decorative components. Thus, the registration results certainly fulfilled the construction requirement of modeling the precision of LoD 300. When the number of points was diluted into 1000 points from the roof decorative component 3D model retrieved from the template library, the registration result was the worst with an average RMS error of 0.019 m. When the number of points was diluted to 10,000, the registration result yielded to the best with an average RMS error of less than 0.01m. Although the RMS error of the registration slightly reduced with the increased number of points, the reduction was not significant after multiple experiments, meaning that the point density has no influential impact on the results. It thus proves that the higher the point density of the registered object is, the better the registration accuracy that can be achieved. Among the four experimental roof decorative components, the registration result of Chui shou was the worst. Comparing to the other components, Chui shou has the most complex structure. Therefore, it can be inferred that the complexity of components also affects the registration result. When the number of points yielded to 10,000, the RMS of Zou shou dramatically increased. We suspect that the registration is misleading due to the fixed tile found in the Zou shou. Therefore, when establishing the template library, we should fully consider the actual situation and enrich its content to cater different decorative components. Moreover, the processing time was stable or had a slight increase with an increase in the number of points. The experimental results show that not only the accuracy of the registration results can meet the requirements of reconstruction, but also the working efficiency can be guaranteed under the condition of stable computational time consumption.

The final 3D roof model of the gate of Supreme Harmony was made up of the following components: 2 ceilings, 1 beam, 12 ridges, 6326 tiles and 58 decorative components, as shown in Figure 15. The rest of the building structures were reconstructed according to the pattern books manually with reference modeling tools, which offers a good representation of the proportion and spatial relationship between these structures of the Gate of Supreme Harmony.

6. Conclusions

In this study, we establish a roof decorative components template library for the Ming and Qing official-style architectures and propose a 3D reconstruction method based on an automatic geo-registration mechanism combined with the template library. The contribution of this work is twofold. First, based on the proposed geo-registration, the roof decorative components can be positioned, located, and scaled correctly on the roof of any surveyed Ming and Qing official-style architecture through matching a corresponding component from the template library. Second, our template library is of high universality and can facilitate the 3D reconstruction of most of the Ming and Qing official-style architectures’ roofs. The core idea of this work is to ensure the details of each roof decorative component can be accurately represented under automatic 3D reconstruction. In addition, this method allows an accurate and efficient 3D modeling of any historic architecture collected by laser scanning or photogrammetric image data. The constructed 3D model can be considered as a reasonable representation and interpretation of the architecture with different levels of complexity (e.g., ridges, simple structures, decorative components, complex structures, etc.).

The proposed strategy not only provides an effective method for the 3D reconstruction of the Ming and Qing official-style architectures’ roof, but can also be applied to other building components or parts, and other similar architectures which follow certain design rules, for instance, the Dou-gong of the Song dynasty architectures, the dome of Islamic architectures, the statues of European churches, and so on. However, it is worth noting the topological relationships between the “component to component” and “component to architecture”, which were not addressed in this paper. The topology is especially important for creating physical copies of the scanned components (e.g., using 3D printing), which will be a particular focus of further study.

The content of the template library is currently under expansion in order to cover more roof decorative components, as well as other architectural components of Ming and Qing official architectures. In future, it is intended to build an interoperable platform with the template library for supporting fully- or semiautomated process of detection/matching and geo-registration, by following related standards. Through information sharing and exchange among different HBIM systems, the template library can be further expanded and enriched, and thus provides an effective way for its further development and benefits from a wider application. As such, the template library can be extended to not only contain roof decorative components, but also cover various types of roof components (roof ridges, tiles, etc.), and even all parts of the whole building. The variety of component forms and rich datasets also provide better test conditions for the success/failure rates detection of the geo-registration mechanism. Further, such a collaborative platform can certainly improve the feasibility for the long-term management of the corresponding ancient architectural heritage and HBIM.