A Post-Scan Point Cloud Colorization Method for Cultural Heritage Documentation

Abstract

The 3D laser scanning technique is important for cultural heritage documentation. The laser itself normally does not carry any color information, so it usually requires an embedded camera system to colorize the point cloud. However, when the embedded camera system fails to perform properly under some external interferences, a post-scan colorization method is always desired to improve the point cloud visuality. This paper presents a simple but efficient point cloud colorization method based on a point-to-pixel orthogonal projection under an assumption that the orthogonal and perspective projections can produce similar effects for a planar feature as long as the target-to-camera distance is relatively short (within several meters). This assumption was verified by a simulation experiment, and the results show that only approximately 5% of colorization error was found at a target-to-camera distance of 3 m. The method was further verified with two real datasets collected for the cultural heritage documentation. The results showed that the visuality of the point clouds for two giant historical buildings had been greatly improved after applying the proposed method.

1. Introduction

The three-dimensional (3D) laser scanning, also commonly known as the terrestrial light detection and ranging (LiDAR), is nowadays an important technology for cultural heritage documentation [1]. A laser scanner can collect millions of points in a few minutes to digitalize outlines of most antique structures. The scanner comprises a laser component that emits and receives laser signals, along with measurements collected from digital angle decoders, to generate a cloud of 3D coordinates (point cloud) of a target. The mode of the laser emission and reception can be broadly classified into pulse-based and phase-based. For tasks that require a higher accuracy in terms of 3D reconstruction for the cultural heritage (usually for international standards), a phase-based laser (scanner) is preferred. Nevertheless, neither mode of the laser loads a color-related information or color model (e.g., the red, green and blue model) but only a returned-laser intensity. Therefore, many terrestrial scanners are embedded with a digital camera, which provides colors to every single point in the point clouds. This is one of the most common colorization methods adopted by the current commercialized laser scanners [2]. Accurate colorization of the point clouds is important as the color can greatly increase the information content of the point clouds, and therefore makes the point clouds become more effective in visual communication [3]. As colors are important components of culture and architecture, point cloud colorization is significant for cultural heritage documentation when laser scanners are used for such purposes [4,5].

The application of point cloud can often be extended when the color and other radiometric information are available. For example, Saglam et al. utilized the color information of the point clouds to enhance a point cloud segmentation algorithm [6]. Choi et al. incorporated the color of point clouds with their geometry to register point clouds collected from different stations [7]. Ling et al. considered distances, incident angles and target color for accurate evaluation of a scanner [8]. For a larger aspect, Chai et al. proposed a fusion-based building segmentation method that simultaneously considered the color of images and the geometry of point clouds to improve the segmentation accuracy [9]. Other than using the color information of a digital camera, the radiometric information provided by the multispectral camera can also be integrated with point cloud to enhance the applications. For example, Keskin et al. used the depth enhancement method to improve the radiometric resolution of airborne scanning data through co-registering color images to the point clouds [10]. Awrangjeb et al. proposed a roof extraction method using the point clouds and multispectral information [11].

The basic principle of the typical point cloud colorization method is that the transformation parameters (for six degrees of freedom, DoF) between the coordinate systems of the scanner and the camera are estimated via a calibration process [12]. A common target field should be set up to estimate the six transformation parameters based on the least-squares method, before the actual scanning target is performed. The point cloud will be transformed into the image plane to acquire the color using the parameters as illustrated by Figure 1. The transformation is governed by the collinearity equation (Equation (1)).

$$\left[\begin{array}{c}x-x_p\\ y-y_p\\ -c\end{array}\right]=\frac{1}{\lambda }\mathbf{M}\left[\begin{array}{c}X-X_c\\ Y-Y_c\\ Z-Z_c\end{array}\right]$$

(1)

where (X,Y,Z) is the coordinate of a point from the point cloud, (x,y) is the coordinate at the image space of the camera. c is the principal distance, λ is the scale factor depending on the object distance. For the transformation, M is the rotation matrix, and (X_c,Y_c,Z_c) is the camera position for the translation.

After the transformation parameters are estimated, Equation (1) can be reorganized as Equation (2) so that the scale factor can be eliminated [13]. Equation (2) is used to project every point in the point clouds onto the two-dimensional (2D) image plane of the camera, so each point falls onto a pixel with a color. In other words, every point will acquire a color after the projection (the perspective projection). The cameras will capture images immediately either before or after scanning for the scene. The image capturing and scanning processes are usually in sequence for a single scanning process so that the projection from point to image can be performed correctly. This process can be defined as an “in situ” point cloud colorization. The in situ colorization process can be interrupted by certain conditions such as limited lighting at the scenes. For example, limited lighting during scanning for the indoor structure of historical buildings (Figure 2) is not uncommon as the lighting systems are usually not well developed for such historical buildings as people usually do not work or live inside the buildings. Other color impairing conditions include shadowing, occlusion, overexposure, etc.

$$\{\begin{array}{c}x=x_p-c\frac{m_{11}\left(X-X_c\right)+m_{12}\left(Y-Y_c\right)+m_{13}\left(Z-Z_c\right)}{m_{31}\left(X-X_c\right)+m_{32}\left(Y-Y_c\right)+m_{33}\left(Z-Z_c\right)}\\ y=y_p-c\frac{m_{21}\left(X-X_c\right)+m_{22}\left(Y-Y_c\right)+m_{23}\left(Z-Z_c\right)}{m_{31}\left(X-X_c\right)+m_{32}\left(Y-Y_c\right)+m_{33}\left(Z-Z_c\right)}\end{array}$$

(2)

where m_ij is the element of row i and the column j of M, which can be expressed in terms of the camera orientation (Ω, Φ, and Κ) as follows.

$$\mathbf{M}={\mathbf{R}}_3\left({\rm K}\right){\mathbf{R}}_2\left(\Phi \right){\mathbf{R}}_1\left(\Omega \right)=\left(\begin{array}{ccc}{m}_{11}& m_{12}& m_{13}\\ m_{21}& m_{22}& m_{23}\\ m_{31}& m_{32}& m_{33}\end{array}\right)$$

(3)

where R₁, R₂, and R₃ are the 3D rotation matrices about the X, Y and Z axes, respectively.

As the in situ colorization process can be often hindered by certain conditions in a similar way that the photography can be, efficient post-scan colorization should be developed and applied to the point clouds. Nevertheless, there are still not many post-scan colorization-related works published in the literature. Crombez et al. used a visual servoing approach to register images with the point clouds. This method is accurate but requires interior and exterior orientation parameters, which might not always be available for users [14]. Pepe et al. installed target points onto the target for registration between a scanner and an external camera based on the bundle adjustment principle [15]. Even though the method produced high accuracy results, it required placement of target points, which would reduce the exposure of the original target to be surveyed.

Cao and Nagao developed a point cloud colorization method based on the generative adversarial networks (GANs) to colorize a set of 10,000 point clouds within 16 categories [16]. This method has a very high accuracy but relies on intensive training of the network using many sample point clouds. It is not site-dependent, and therefore may not be suitable for cultural heritage documentation unless extensive modifications are incurred. Arshad and Beksi modified the traditional GANs with graph convolutions constructed by leaf output layers and branches [17]. This method also requires many training samples, and therefore may not be adopted to the point cloud of a unique historical site. Gaiani et al. made use of the principle of photogrammetry to correct the color of the point clouds obtained from laser scanning, including the automatic color balance, exposure equalization, etc. [18]. Julin et al. proposed a new colorization evaluation method to quantify the color sensing capability of four different brands of laser scanners [2]. Their work focused on a performance evaluation rather than development of a usable colorization algorithm.

The colorization of point clouds greatly supported the development of the virtual reality (VR) for historical sites. For example, Edler et al. reviewed the potential of immersive VR environments for a multifaceted and redeveloped area for several applications in post-industrial sites in Germany, with emphases on the usage of game engine-based VR technologies [19]. Buyuksalih et al. used the Riegl VZ-400 scanner to capture point clouds to input into a Unity game engine to create immersive and interactive visualization of a cave for a VR system in the Istanbul Catalca Incegiz caves in Turkey [20]. For potential development of a VR system, Lezzerini et al. mapped the facades of the medieval church of St. Nicholas with laser scanning and a geographical information system (GIS) in Pisa, Italy [21].

In this paper, we proposed a simple but efficient point cloud colorization method to perform post-scan colorization for planar features with some images acquired independently from the colorless scans obtained for dark rooms or rooms with vast occlusions inside historical buildings. For the proposed method, a single image is enough to perform the colorization for a single planar feature as it is assumed that the perspective projection produces a similar effect as the orthogonal projection for a planar feature when the image capturing distance is relatively short (within several meters). The proposed method aimed at providing a relatively coarse colorization as the calibration process is not involved. Nevertheless, as demonstrated by the results, our method can deliver visualization results well and is easy to be implemented as the user does not require any information about the camera (e.g., the interior orientation parameters) but only a single image for colorization of a single planar feature.

2. Method

The proposed method is a semi-automatic approach for post-scan point cloud colorization for large planar features such as internal walls in a historical site. The workflow of the proposed method is shown in Figure 3. The method first invokes the random sample consensus (RANSAC) algorithm to separate individual planar features [22]. This process involves some visual inspection and manual editing to ensure that the planar features are segmented correctly. Then, the normal vectors and areas are estimated for each planar feature for identification of their attributes (walls, floors, ceilings, doors, windows or others). When the planar features are identified, images collected at the same place under improved conditions for photography or at different places with similar planar features (e.g., another similar wall with better lighting condition) are used to colorize the point cloud by using an orthogonal point-to-pixel projection under an assumption that the effects of orthogonal and perspective projections are similar for planar features when the target-to-camera distance is relatively short (within only several meters). The distribution of normal vectors of some selected planar features (or parts of them such as a windowsill) are considered to enhance the intensity variation of the assigned colors via a color intensity adjustment process.

2.1. Planar Feature Extraction

The planar feature extraction consists of three main point cloud processing steps: (1) ground filtering; (2) noise-removal; (3) RANSAC plane fitting. Firstly, the ground, which usually has the highest (or nearly the highest) point density, can be first separated from the entire point cloud via the clothe simulation filtering (CSF) method [23]. The CSF is almost parameter-free, so it can be readily applied. Secondly, the noise-removal method adopted is a simple radius-based method based on the k-nearest neighbor [24]. A point will be considered as “noisy” and be removed if that point does not possess a certain number of neighboring points within a spherical space defined by a radius and a center point (the point’s own position). Finally, the non-ground point clouds can be input to a RANSAC algorithm to separate individual planar features (Figure 4), then the normal vector can be estimated using the Principal Component Analysis (PCA) for each feature to define its orientations and positions [25]. Visual inspection and manual editing are required to guarantee the overall quality of the separated planes.

2.2. Point-to-Pixel Projection

The extracted individual planar feature will be used to generate a minimum boundary rectangle so that the size can be estimated to further compute a scaling factor [26], s, which is used to scale up the image coordinates to object coordinates (Equation (4)) associated with a pixel. The normal vector of the planar feature is computed for alignment of the central axis orthogonal to the image plane. Then, each point from the point cloud can be projected orthogonally onto a pixel with an object coordinate close to the point (searched by the k-nearest neighbor algorithm) to acquire a color (RGB value), as illustrated in Figure 5.

$$\left[\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\\ Z^{\prime }\end{array}\right]=s\left[\begin{array}{c}i\\ j\\ 0\end{array}\right]$$

(4)

where (i,j) is the image coordinate, and (X′,Y′,Z′) is the object coordinate associated with a pixel.

2.3. Color Intensity Adjustment (CIA)

Some walls inevitably contain windowsills or some other similar structures. For image capturing, the color intensity is usually weaker at the sill-like structure as less amount of light can reach there. As a result, it is needed to adjust the color intensity of the points lying on the sill-like structure during the point cloud colorization. As illustrated in Figure 6, points at the sills possess normal vectors (blue and orange), and these normal vectors are orthogonal (or approximately orthogonal) to normal vectors of the main surface of walls (red). The reduction of color intensity is governed by the intensity adjustment factor (c), which is a function of the angle between the two sets of abovementioned normal vectors:

$$c=\left(1-\frac{{⟨}\overrightarrow{a},\overrightarrow{b}{⟩}}{\alpha }\right)$$

(5)

where $\overrightarrow{a}$ is the normal vector of the main surface of walls; $\overrightarrow{b}$ is the normal vector of the sill-like structure. α is the angular factor (in radian) that determines the extent of the RGB intensity reduction. In practice, it can be set as 3.49 (~200°), based on empirical experience.

3. Experiment

3.1. Simulated Dataset

A point cloud of a planar feature (a facia board of the Sun Yat-sen University) captured by a Trimble SX 10 scanner with color observed by its embedded camera (Figure 7a) was used for a simulation experiment. The point cloud is first translated by its centroid and rotated, so its principal direction is parallel to the X-axis. The point cloud is assumed to be a real object, and an image of point cloud was simulated using the collinearity equation augmented with some realistic parameters listed in

Table 1. Parameters and their values used for the simulation of images.

Param.	μ (m)	xp (m)	yp (m)	c (m)	Xc (m)	Yc (m)	Zc (m)	Ω (°)	Φ (°)	Κ (°)
Value	1.37 × 10−5	10−5	10−5	4 × 10−3	0	1,3,5,7,9,11,13,15	0	90	0	0

. The IOP values are obtained from the default setting from a photogrammetric software (e.g., ContextCapture). Then, the image is used to colorize the point cloud (with the new color overlaying the old color) using the proposed method for the verification of the practicality. The root-mean-square error (RMSEcolor) of the colorization can be computed as:

$$RMS{E}_{color}=\frac{{{\sum }}_{i=1}^n{\left(\delta RG{B}_i\right)}^2}{n}$$

(6)

where, for each point, the RMSE of the red, green and blue is

$$\delta RG{B}_i=\sqrt{\frac{{\left(\delta R\right)}^2+{\left(\delta G\right)}^2+{\left(\delta B\right)}^2}{3}}$$

(7)

$\delta R$, $\delta G$, and $\delta B$ are the differences of the red, green, and blue intensity (0–255) between the color estimated by the proposed method and the original color of the point cloud, respectively. The RMSE and the standard deviation are used because they can quantify the average colorization errors. The Y position of the camera is being changed from 1 to 15 m with an interval of 2 to simulate the shift of camera position along the Y-axis to examine the impact to the results with the increasing image capturing distances.

3.2. Real Dataset

The proposed method was applied to two real datasets collected by using a Trimble SX 10 scanner and a Nikon D5600 camera (Figure 7b). The first dataset was captured on 7 November 2020, for the Zou Lu House (Figure 8a), which was the home of a late Chinese educator, Prof. Lu Zou (1885–1954), the first principle of the Sun Yat-sen University. The Zou Lu House is located in Chayang, Mei Zhou, Guangdong Province of China. The house was built in the early Qing dynasty (1644–1911), with the Hakka style. It was listed as one of the conserved cultural heritage sites by the Guangdong Province of China in 2005. The second dataset was captured on 10 December 2020, using the same set of equipment for the Wesley House (Figure 8b) located at the Sun Yat-sen University’s south campus in the Haizhu district, Guangzhou, China. The Wesley House was built in 1924 based on the eclecticism, supported by the Wesleyan Methodist Missionary Society of England. Similar to the Zou Lu House, the Wesley House was officially listed as a conserved cultural heritage site by the Guangdong Province of China. They are selected as the study cases in this work as some colors of their point clouds are deteriorated (appeared almost all black) due to missing light sources in rooms without windows or serious occlusions. The details of the datasets are tabulated in

Table 2. Details of the two real datasets captured for the experiment.

Site	Area (m2)	Number of Scans	Number of Images
Zou Lu House	1352	21	389
Wesley House	406	32	582

.

The variation of color across a point cloud can be quantified by computing the standard deviation, σ, of the color of the point cloud for the planar feature, j:

$${\sigma }_j=\sqrt{\frac{{{\sum }}_{i=1}^n{\left(f_i-u\right)}^2}{n}}$$

(8)

where f is the color of a point, u is the mean color, and n is the number of points on the point cloud. The higher the contrast of the color, the higher σ is.

4. Results

4.1. Results for the Simulated Data

The RMSE of the colorization (RMSE_color) is plotted versus the image capturing (T-C, target-to-camera) distance in Figure 9. It can be seen that the RMSE_color increases as the T-C distance increases. For a T-C distance less than 3, the RMSE_color is only 15, which means the intensity of the RGB color will have an RMSE of approximately 5.86% of the total colorization error. This is acceptable for many applications as changes of 15 out of 256 intensities will not cause a significant difference in visual perception.

On the other hand, Figure 10 shows the comparison of the colors between the original point cloud and the colorized point cloud of the facia board at different T-C distances. It can be seen that the colorized point clouds appear almost the same as the original point cloud (object) at the T-C distance of 3 m (comparing Figure 10a,b). This is consistent with the result of the RMSE_color of ~15 (low) shown in Figure 9, suggesting that the proposed method is practically correct under the assumption that the effects of orthogonal and perspective projections are similar for a planar feature as long as the T-C distance is within several meters. It can be seen in Figure 10 that the colorized point clouds are erroneous at T-C distance of 9 and 15 m, while the RMSE_color for both simulations is higher than 45, which is approximately 18% of the total colorization errors.

4.2. Results for the Real Data

Figure 11 shows the original registered point cloud of the Zou Lu House, and the same point cloud after the proposed colorization method is applied. The original point cloud has approximately 30% of the planar features scanned under darkness thanks to the underdeveloped lighting system, so the point clouds of those planar features are colorless (or black). Images acquired from daytime are used to colorize those planar features using the proposed method, it can be seen that the visuality of the point cloud has been greatly improved. Two walls are shown in Figure 12 and Figure 13. It can be seen that the wall, windows and door can be colorized properly using the proposed method.

The Wesley House is currently being used as an office for a research center (China Regional Coordinated Development and Rural Construction Institute of the Sun Yat-sen University), it is occupied by more than 3 faculty members and 20 graduate students. Therefore, the Wesley House was busy while the scans were collected. Many obstacles, shadows, and noises were found as seen in Figure 14a. After applying the proposed colorization method, the abovementioned issues were mostly remedied as can be seen from Figure 14b. Figure 15 and Figure 16 show two zoomed details of the Wesley House, the colorized point clouds produce apparently higher visual perspective effects. In Figure 15, there are some artifacts caused by opening windows (the right window, the glass caused the deflection of the laser, and some mossy portions). After applying the proposed method, the artifacts were removed as images of brick walls are used to colorize the point clouds. The images were taken by a brick wall nearby. In Figure 16, it can be seen that the contrast of the floor color was significantly improved after applying the proposed method.

To quantify the change in color of the point cloud after applying the proposed method, the histogram analysis was performed and the standard deviation of the color of the point cloud was computed using Equation (8). The standard deviation of the color of the point cloud of four selected planar features from the two datasets was tabulated in

Table 3. Standard deviation of the color of the point cloud.

Datasets	Planar Feature	Original Point Cloud (Unitless)	Colorized Point Cloud without CIA (Unitless)	Colorized Point Cloud without CIA (Unitless)
Zou Lu House	1	235	371	626
	2	293	407	583
Wesley House	3	1034	1201	1395
	4	311	427	548

. A larger standard deviation implies a higher variety of the color of the point cloud. The point clouds are usually more visually realistic, in normal circumstances, when the color contrast is higher. As a result, the proposed CIA algorithm can further increase the standard deviations of the color so that the variation of color across the features are increased.

It can be seen from Figure 17 and Figure 18 that the gray values of the point clouds shift toward the white region, and the contrasts increase after the proposed colorization method is applied. As a result, the CIA is shown to be an essential process to colorize the sill-like component of walls to gain higher color contrast. On the one hand, it can be seen from Figure 17 that the distribution of the gray value becomes closer to the normal distribution. The visuality of the outcome becomes more realistic. It is much better to assign a homogenous color (e.g., the white color) to the wall, as illustrated in Figure 17. On the other hand, it can be seen from Figure 18 that the distribution of the gray value also becomes closer to the normal distribution. The fluctuation of the gray value is significantly reduced after applying the proposed method, as shown in Figure 18. The range of the gray values span more widely in the histogram after applying the proposed method. These significantly enhance the visuality of the wall.

For the point clouds of the sill-like component, they often suffer from the edge effect (the severity depends on the beamwidth of the laser) [27] and the field of view issues so that the quality of the color of the point cloud has also deteriorated. This is the reason for why the intensity of the color should be adjusted to deliver more realistic results. It is worth noting that the proposed CLA algorithm works in a way that the color of image is transferred to point clouds with fairly high accuracy to enhance the visuality and comprehensiveness of the point clouds. Therefore, the final outcomes actually depend on the quality of the input images. The input images can be captured at different times so that the flexibility of usage of the method is high. As long as the images have high contrast and visuality, the images can be used to colorize the point cloud.

5. Discussion

The proposed method consists of a straightforward approach to project the color of images (captured independently) onto the point clouds. As shown in Section 4, the proposed could complete the colorization, turning colorless or color-faded point clouds into colorful and realistic ones. Compared to other colorization method based on machine learning [16,17], the proposed method does not require collection of training datasets. It only requires a single image captured under a better condition (e.g., with extra lighting) or for a similar object/scene. Based on our results obtained from simulation with a projection of an actual image onto a point cloud of an object, it is confirmed that the prospective nature of the imaging caused only slight displacement of color (with only 5% colorization error at a camera-to-target distance of 3 m). Normally, the image is captured within 2 m away from the target, so the colorization error can be further reduced. In addition, the proposed method was applied to two historical point clouds. It is shown that the promising visual effects were achieved.

6. Conclusions

In this paper, we proposed a simple but efficient point cloud colorization method to remedy destroyed or lost color of the point clouds of planar features after scans are collected and registered, as long as some images of the features can be collected independently. The method does not require any calibration (between the scanner and the camera) or any information about the camera (e.g., the interior and exterior orientation parameters) to colorize large planar features such as walls for buildings. To conclude, the method is based on a point-to-pixel orthogonal projection, which assumes the effects of orthogonal and perspective projections are close to each other for planar features when the target-to-camera distance is relatively short (within several meters). This assumption is verified by a simulation experiment, and the result suggested that the proposed method normally will cause approximately 5% of colorization error at the target-to-camera distance of 3 m, so the point clouds colorized by the proposed method appear almost the same as the point cloud with the actual color. The method was also verified with two real datasets collected in Guangzhou, China for cultural heritage purposes. Our results indicate that the visuality of the point clouds for two giant historical buildings greatly improved after the proposed method was applied.