Digital Recording of Historical Defensive Structures in Mountainous Areas Using Drones: Considerations and Comparisons

Abstract

Digital recording of historic buildings and sites in mountainous areas could be challenging. The paper considers and discusses the case of historical defensive structures in the Italian Alps, designed and built to be not accessible. Drone images and photogrammetric techniques for 3D modeling play a fundamental role in the digital documentation of fortified constructions with non-contact techniques. This manuscript describes the use of drones for reconstructing the external surfaces of some fortified structures using traditional photogrammetric/SfM solutions and novel methods based on NeRFs. The case of direct orientation based on PPK and traditional GCPs placed on the ground is also discussed, considering the difficulties in placing and measuring control points in such environments.

1. Introduction

Nowadays, 3D modeling from drone images has reached significant maturity and a high automation level. Digital photogrammetry coupled with computer vision methods (e.g., Structure from Motion/SfM) can automate most phases of the reconstruction workflow [1]. In addition, flight planning software also allow users to automate the data acquisition phase, which can be remotely planned before reaching the site [2].

This paper aims to show some of the latest advancements in drone image processing for 3D modeling applications, focusing on using Post-Processed Kinematic (PPK) for direct georeferencing [3], reducing the need for ground control points (GCPs) measured on-site. The manuscript then discusses Neural Radiance Fields (NeRF) [4] compared with the more traditional extraction of point clouds and meshes with dense image-matching techniques.

The particular case of fortified heritage constructions in mountainous areas is discussed. Such defensive structures were designed to be not easily accessible [5], making drones a powerful solution to capture their irregular geometry. Here, digital recording for restoration and conservation can pose several challenges, including vegetation on ruins and around the construction. Drones can become an essential tool for the opportunity to reach parts that are not accessible, providing images and videos not only limited to visual documentation.

After orienting images with photogrammetric techniques, SfM and NeRFs were used to reconstruct the geometry of three fortified constructions in the Italian Alps. A preview of the achieved results with NeRFs is shown in Figure 1, which depicts some frames extracted from a rendered video. Input data for NeRFs training are photogrammetrically oriented images acquired with a Mavic 3E drone supported by PPK for georeferencing. A simple qualitative inspection shows the excellent quality of the vegetated background achieved with NeRFs.

Overall, the main topics considered in this manuscript can be summarized as follows:

• The use of drones with PPK for digital recording heritage ruins with limited accessibility and comparison with more traditional surveying strategy (laser scanning and photogrammetry with GCPs);
• Comparison between traditional photogrammetry/SfM and new methods based on NeRFs generated with Nerfstudio [6].

1.1. Digital Recording of Ruins for Restoration and Conservation

The versatility of advanced digital recording tools proves to be invaluable in the field of conservation. Restoring and conserving historical buildings necessitates a precise understanding of their geometrical features. Digital recording techniques, such as laser scanning and digital photogrammetry, offer a very high level of detail, providing geometrically accurate information. This data can be utilized to analyze the historic building based on its geometrical proportions and to facilitate further investigations, including assessing the material composition and state of conservation.

The successful preparation of various conservation actions relies on creating a comprehensive recording dossier, instrumental for analyzing the proportions and architectural heritage. This includes verticality deviations or other deformations, understanding the construction logic, identifying materials, and assessing their conservation conditions.

Digital recording techniques offer the advantage of flexible 2D and 3D representations utilized throughout the conservation process, from preliminary studies and intervention identification to devising enhancement strategies to ensure ongoing use and accessibility of the buildings.

This study conducted three digital recording campaigns using drones on architectural ruins, namely, Castel Grumello and Castel Mancapane in Valtellina, Italy, and the Rocca di Vogogna in Val d’Ossola, Italy.

Following the guidelines of the Italian Code of Cultural Heritage and upholding a long-standing tradition in preservation, conducting meticulous surveys is crucial for the correct development of conservation design. The three cases discussed here involve archaeological remains—medieval fortifications still preserving some volumes of their original configuration, including square towers and defensive circuits. These tangible physical remnants bear testimony to values that must be preserved.

Upon initial observation, these defensive buildings conjure a distinctive spatial organization of the territory, known as the fortifications period, bridging the transition from the administrative structure of the Roman Empire to the fragmented power landscape of the Middle Ages. Furthermore, the material remnants preserve the vestiges of ancient construction traditions, employing stone components and organizing massive masonry structures.

Unraveling a comprehensive understanding of all the elements encompassing these archaeological sites demands a meticulous analysis of the buildings, ranging from their geometrical proportions to the characteristics of their structural elements and components. Advanced surveying techniques provide the crucial information necessary for further conservation strategy analyses. This includes studying the building’s various phases of transformation, mapping materials, and their deterioration, evaluating mechanical damages, and more.

Digital techniques able to record complex heritage sites are a valid support for preserving ruined buildings, which are particularly fragile in terms of materials. By creating digital records, experts can document the condition of the existing structures and identify deteriorated areas. Digital tools also facilitate advanced data analysis, such as studying material composition, structural integrity, and environmental factors, aiding in interpreting the transformations of the building over time and, hence, the potential risks they could face in the future. Moreover, they assist in assessing historical buildings’ structural stability.

Structural modeling software can simulate different conditions, helping to identify potential vulnerabilities and decisions on conservation and safety measures. Moreover, digital tools eventually simulate various restoration scenarios, interpreting the previously defined conservation strategies and allowing experts to assess their potential impact (e.g., the perception by the public) before implementing physical interventions.

1.2. Challenges in Heritage Recording in Mountainous Areas

Creating a detailed and accurate digital record of historical fortifications requires a combination of digital recording methods and tools. Photogrammetry and laser scanning are helpful solutions that capture the surfaces without requiring direct contact with the object. However, this paper does not explicitly focus on using different digital tools and their integrated use for surveying complex geometry. The reader is referred to several technical papers describing real case studies [7,8,9,10,11,12,13,14,15,16,17,18].

As mentioned in the previous section, this paper aims to evaluate and discuss the use of drones (and photogrammetry with images captured by drones) for the particular case of fortified constructions in mountainous areas. Photogrammetric processing using SfM methods is highly automated today thanks to different commercial and open-source solutions that can generate accurate point clouds and textured meshes. Fortified heritage often includes complex structures (castles, towers, forts, among others) with irregular geometry. An accurate and comprehensive digital record requires techniques capable of revealing the actual shape with all irregularities. Point clouds produced by drone images captured with suitable GSD (ground sampling distance) can show small details and geometric anomalies. Planning a complete survey requires knowledge of project requirements and images with a sufficiently short distance to reach the desired GSD while keeping a suitable geometry of the image block for photogrammetric processing.

In addition, drones equipped with GNSS receivers can also provide direct georeferencing using RTK or PPK, reaching centimeter-level precision. For fortifications built on mountains, placing and measuring control points with traditional surveying equipment (e.g., a GNSS antenna on top of a pole) could be challenging. Points with a homogeneous distribution on the site are also necessary. The opportunity to measure camera poses with centimeter-level precision coupled with a photogrammetric block with suitable geometry and sufficient overlap can be exploited to georeference the project without using external control points.

Drones are particularly suitable for capturing elevations and producing orthophotos, which are used for condition mapping (materials, construction technologies, degradation) and designing interventions. External elevations could also be rather challenging to reach with traditional terrestrial methods. At the same time, accurate and detailed site plans can be created using high-resolution orthophotos produced in the cartographic plane or a local Cartesian system. The user must be aware that products in a cartographic system could be affected by cartographic effects, making angles, distances, and areas different from the “real” values, which can be captured with laser scanning.

Although photogrammetry with drones can be helpful in these types of surveying activities, several limitations exist. For instance, parts covered with thick vegetation cannot be reconstructed. Vegetation in abandoned ruins is often present. A reconstruction from images would provide a model of the vegetated areas, which is also often incomplete for the intrinsic difficulties in modeling such irregular areas.

The case studies in this work represent three medieval fortified structures, partially collapsed, realized in two different contexts in the Lombardy and Piedmont regions in Italy. They were built on mountain cliffs and have an irregular shape in the plan’s organization for adapting the different parts of the buildings to the local topography.

2. Case Study Description

Three case studies are considered in this paper:

• Castel Grumello;
• Castel Mancapane;
• Rocca di Vogogna.

Detailed information about the case studies (Figure 2) are presented in the following paragraphs and summarized in

Table 1. Summary of presented cases study.

Case Study	Location	Approximate Elevation [m]
Castel Grumello	Montagna di Valtellina (Sondrio, Italy)	490
Castel Mancapane	Montagna di Valtellina (Italy)	890
Rocca di Vogogna	Vogogna (Verbania, Italy)	320

.

2.1. Castel Grumello

Castel Grumello is a medieval fortress in Montagna di Valtellina, Italy. The ruins of the castle, which are surrounded by the famous vineyards of Valtellina, strongly characterize the site where built heritage and landscape are merged into a fascinating combination of architecture and nature.

The fortress is a rare example of a twin castle used for residential and military purposes (Figure 3). It was part of a more complex system of fortifications that controlled Valtellina in the Middle Age. Together with Mancapane Castle (see the following sections) and other fortresses along the valley, it symbolized the economic and political powers of the families that controlled the territory.

The fortress’s history shows how the destiny of medieval castles is strictly related to the landlords’ fortune (or misfortune). The castle is mentioned in a document currently preserved in the Como national archive, written between 1329 and 1333. Historians agree on the possible construction of the castle in the 14th century by the De Piro, a powerful feudatory family in conflict with the Capitanei. The contrasts between the two families (reflecting the conflict between the Guelfi and the Ghibellini at the local level) are at the base of the ruin of Castel Grumello, which was then dismantled in 1526 by Grigioni, who conquered the valley. After a period of abandonment, the destiny of the castle was connected to the productive history of the Valtellina vineyards (probably cultivated since the 16th century), as it was used as a quarry to reorganize the cultivation of the vineyards through an enlargement of the terraces. Only in 1990, the FAI (Fondo per l’Ambiente Italiano) bought the ruins, and the fortress was restored thanks to a vast conservation program designed by the architect, S. Tirinzoni [19].

The castle is characterized by two main constructions along the crest of the hill. The west building was probably used for residential purposes, while the east was for military purposes. The different uses were mainly documented during the archaeological excavations at the beginning of this century. The restoration by Tirinzoni aimed at preserving the authenticity of the ruins. For this reason, the interventions fully respected the original materials and preserved the transformations over time that document the site’s complex history.

The masonry underwent a cleaning process that did not remove the patina chromatic variations, and oxidation happened over time. The cornices were carefully dismantled stone by stone and reassembled with the same blocks and new lime mortar. Reconstruction was carried out only if strictly necessary for consolidation and preservation. Mortar joint repointing was conducted using selected and differentiated lime and aggregates tailored to match the existing chromatic variations and textures in different areas.

The main goal of the restoration was to preserve the perception of the site by the local community, who traditionally use the site in their spare time to relax and enjoy the fantastic view from the ruins. For this reason, the interventions were aimed at improving the accessibility of the site by adding stairways, services, and barriers. Thanks to the conservation strategy and the beautiful landscape with vineyards, local communities and other tourists continuously visit the castle.

2.2. Castel Mancapane

In the mountainous context surrounding Sondrio, not far from Castel Grumello, another fortified structure with a square tower can be reached following a path on the mountains (Figure 1). Because of a lack of documentation, the origin of the fortress is unknown, but thanks to an accurate evaluation of the masonry pattern, the building was dated back to the 13th century [20]. It became the property of the De’ Capitanei family, who tried to establish their hegemony in the valley since the 14th century.

The structure comprises irregular stones with roughly rectangular blocks at the corners. As observed in the case of Castel Grumello, also this castle was abandoned during the period of domination by Grigioni, when the fortress system of the valley was dismantled. The ruins of the complex are composed of a simple rectangular wall, realized along the slope of the mountain, with a square tower set in the middle of the north wall.

The complex is more isolated than the Grumello Castle. It is surrounded by dense vegetation. Stefano Tirinzoni, who designed the conservation plan also for this archaeological site, removed some plants from the internal courtyard and the outer sides of the walls to guarantee the accessibility of the castle [21].

2.3. Rocca di Vogogna

The Rocca di Vogogna (Figure 4) was a defensive structure of a comprehensive military system composed of strongholds diffused along the Ossola Valley.

The origin of the building is commonly posted in the medieval period. Still, a systematic studio [22] of the fortress and the stratigraphy of its structures show that the archaeological remains can be divided into different phases, revealing that several parts were built after the 15th century.

Nowadays, the ruins of the fortress are surrounded by the vegetation of the Val Grande natural park, a protected area formed by the forests on the mountains in a large natural basin. The building dominates the historical center of Vogogna from the south side of Orsetto Mountain, facing the Simplon road. The ruins appear behind the medieval buildings characterizing the town, an important administrative center that, in 1348, became the capital of the lower Ossola Valley. The municipal palace was realized by Giovanni Visconti, the future Duke of Milan, who reinforced the fortifications of the town, connecting the city walls to a castle (known as the Castello Visconteo of Vogogna) located on the top of the city center and facing the Rocca of Vogogna on the mountain.

The relationship between the Castle of Vogogna, the medieval city center, the valley’s natural environment with the river Toce, the mountain hamlet, and the archaeological remains of the fortress required a complementary analysis carried out by using historical research and the recording of the complex. The southwest side of the building is composed of different bodies realized along the slope of the mountain, creating a irregular hypogeal distribution of the spaces. An existing record of the plan of the various levels was not available.

According to Babbini [22], who provides a stratigraphic analysis of the complex starting from a detailed survey of the main masonry patterns characterizing various areas of the fortress, the building can be divided into an ancient core, conserving the remains of a central square tower (which dates back to the 12th century), connected to a yard enclosed into defensive walls and sub-circular minor towers at the edges. This part of the fortress is connected to other spaces displayed on different levels. They appear as rooms with large openings facing the valley.

The complex results from several additions, including the works in the 15th century when Vitaliano Borromeo bought the Feud of Vogogna and introduced significant renewals to the town’s fortifications [23]. The fortress was deeply damaged after an attack in 1514 organized by Domodossola [23], the capital of the north district of Ossola Valley. The complex was therefore abandoned.

With the rearrangement of the territorial administration after the unification of Italy, Vogogna lost its administrative role and became an ordinary town. Its defensive structures were transformed, changing the military use into rural ones. The fortress was misused with the changing economic assets, moving from rural to industrial. The surviving structures, with the buttresses, the patrol walkways, and the bombards, are proof of the military origin of the building. In the same way, the ruins showing the addition of new spaces used as storage, and the subdivision of pre-existing large rooms into minor ones, require additional studies for interpreting the complex stratification of uses characterizing the complex and its transformations over time.

The difficult accessibility of the external walls, directly placed on the sides of the mountain, and the dangerous paths connecting the different levels of the structures require a detailed survey of these spaces, aimed at overcoming the lack of graphic documentation about the fortress, actually documented by two tables published by Airoldi [23].

3. Image Acquisition and Orientation

Acquisition and processing of the images for the proposed case studies were carried out using different approaches depending on the characteristics of the buildings and the availability of specific drones. The three approaches (

Table 2. Acquisition an orientation strategy for the presented case studies.

Case Study	Image Acquisition	Drone Used	Orientation Method	Expected Deliverables	Software Used	NeRF Test
Castel Grumello	Automated flight plan (normal and oblique images)	Mavic 3 Enterprise	Bundle adjustment constrained by camera positions (PPK)	3D model-site orthophoto-GIS	REDtoolbox-Agisoft Metashape	No
Castel Mancapane	Manual flight (normal and 360° around the castle)	Mavic 3 Enterprise	Bundle adjustment constrained by camera positions (PPK)	3D model	REDtoolbox-Agisoft Metashape	Yes
Rocca di Vogogna	Manual flight (normal and oblique images)	DJI Air 2S	Bundle adjustment constrained GCPs	3D model-site orthophoto-elevation orthophotos	Agisoft Metashape	Yes

) can be summarized as follows:

• Castel Grumello: complete direct orientation combining photogrammetric processing and PPK-automatic flight;
• Castel Mancapane: partial direct orientation integrating PPK in photogrammetric processing, only for a specific set of images for the presence of thick vegetation and lack of fixed-flight conducted manually;
• Rocca di Vogogna: georeferencing using a set of GCPs measured with RTK-manual flight (the used drone did not support RTK or PPK).

More details are provided in the following subsections.

3.1. Castel Grumello

Image acquisition was carried out with a Mavic 3 Enterprise, using an automated flight plan with normal and oblique images (40°). The acquisition took about 30 min, obtaining 1224 images at an average elevation (above ground) of 30 m organized in different strips.

A reference GNSS antenna (master) was also placed on-site, recording raw data in static mode for about 45 min (RINEX). The coordinates of the master were then calculated in static post-processing using 1-second RINEX from a permanent GNSS station and Leica Geo Office. Data were provided by the SPIN3 GNSS service (https://www.spingnss.it/ (accessed on 1 August 2023), a network of 33 permanent GNSS receivers covering three Italian regions (Piedmont, Lombardy, Valle D’Aosta).

PPK was used to calculate camera poses using the raw data acquired with the drone. The software used was REDtoolbox (https://www.redcatch.at/redtoolbox (accessed on 1 August 2023)), obtaining a fixed solution for all images. The flight was conducted with complete sky visibility, without occlusions, and in a fully automatic way.

Images were imported in Agisoft Metashape. The calibration parameters were loaded from a previous (specific) calibration project and were assumed to have constant values during bundle adjustment. Georeferencing was obtained with a rigid seven-parameter transformation (scale, rotation, translation), obtaining RMSE values on camera poses of 16 mm for East, 30 mm for North, and 22 mm for altitude. Figure 5 shows the results after image orientation. More details about generating additional deliverables are provided in the next section.

3.2. Castel Mancapane

The acquisition was conducted with images acquired with a Mavic 3 Enterprise with an RTK module. Rinex data were stored and post-processed using PPK. The master GNSS station is a part of the permanent network SPIN3 GNSS (https://www.spingnss.it/ (accessed on 1 August 2023)). The closest permanent receiver is located in Sondrio, a few kilometers from Castel Mancapane.

As the castle is surrounded by vegetation, a manual flight composed of several vertical strips was carried out. A total number of 549 images was acquired in about 30 min, also featuring a convergent geometry around the castle. Before running photogrammetric processing, PPK was used to calculate precise camera poses, which were used to georeference the project. However, a precise (fix) camera pose was achieved only for 347 images (63.2%), whereas 202 images (36.8%) had a float solution. Such a result was expected because vegetation strongly reduced sky visibility, especially for images near the ground. A trigger map and a height map after PPK processing are shown in Figure 6. As can be seen, the fix was lost and obtained again after a few vertical strips. The software used was still REDtoolbox.

Image orientation was carried out using only camera poses with a fixed solution. Images with a float solution were added to the project and included in image matching and bundle adjustment so that they were oriented together with the other images in a combined orientation procedure. In other words, coordinates from the float solution were excluded from processing, and georeferencing was carried out with a seven-parameter transformation calculated only using images with a fixed solution.

The software used for processing was Agisoft Metashape. The camera was also calibrated beforehand, and the calibration parameters (interior orientation and distortion coefficients) were imported and assumed as fixed values in processing. The RMSE values on camera poses after matching and bundle adjustment were about 9 mm for East, 7 mm for North, and 12 mm for altitude. An image showing camera poses around the castle is shown in Figure 7.

The achieved metric accuracy using PPK is sufficient for producing deliverables at a metric scale of 1:50. From this point of view, the results are satisfying, considering the complexity of accessing the place, especially with light equipment: only the drone was taken to the castle. More information about this point is also compared with a traditional laser scanning digital survey.

3.3. Rocca di Vogogna

Images of the Rocca di Vogogna case study were acquired using a DJI Air 2S. Image acquisition was carried out using a manual strategy. The choice of a manual flight was due to the atmospheric conditions. A strong wind characterized the area during the acquisition, and the drone pilot manually operated it to maintain control of drone movement during wind gusts. Apart from the manual flight, the acquisition was carried out using a schema combining normal images (aimed at creating an orthophoto of the fortress and nearby structures) and a set of oblique (45°) and convergent images to reconstruct the vertical surfaces.

Indeed, the reconstruction of vertical structures and their texture, like the remains of the fortress tower and the external fortification walls, are of fundamental importance for any stratigraphic analysis. However, this is not possible starting from ground acquisition due to the hill’s steep slope on which the fortification walls are built. In more detail, 373 normal images were acquired at a flight altitude of approximately 25 m above the ground organized in different strips, and 390 oblique and convergent images were captured covering the entire fortress area. A total of 763 images were acquired in approximately 1 h.

The project’s georeferencing was carried out using a set of GCPs placed on the ground and measured with RTK. Indeed, as previously mentioned, the drone used in this project, the DJI Air 2S, does not support RTK or PPK. In particular, 16 points have been materialized by using checkerboard targets (dimension 30 × 30 cm) in the area of the fortress. The measurement of targets was performed using a receiver Emlid Reach RS2 receiving corrections from service provided by the network of permanent stations developed by SPIN3 GNSS. The precision of the measured target positions is about ±2 cm.

The images were oriented using the software Agisoft Metashape (Figure 8). Due to the good geometry of the image acquisition, no fixed calibration was imported, and the calibration parameters were estimated during bundle adjustment. For the georeferencing of the project, 10 points were defined as GCPs and 6 were used as Check Points (CPs) for the evaluation of the orientation (Figure 9).

The RMSE on the GCPs target position after bundle adjustment is $1.0$ cm for East, $1.1$ cm for North, and $0.8$ cm for altitude. Similar results can be observed with CPs. RMSE is $0.9$ cm for East, $1.4$ cm for North, and $0.8$ cm for altitude.

4. Production of Deliverables with Digital Photogrammetry: Some Considerations

Different deliverables can be produced after orienting the images with different approaches (described in the previous section). We discuss the case of a traditional acquisition with laser scanning (often integrated with terrestrial and drone images) compared to the approach proposed in this paper, i.e., only drone images.

4.1. Castel Grumello

A textured 3D model was generated from the set of oriented images supported by PPK. A digital orthophoto (top view in the UTM cartographic plane) was then produced and imported into an existing GIS system developed to host different datasets related to Castel Grumello. Some images of the obtained textured 3D model are shown in Figure 10.

The GIS was created using ArcGIS Pro and ArcGIS Online, starting from several photogrammetric and laser scanning datasets acquired in previous years. Creating a complete GIS required a combined approach for the survey, which was georeferenced using some ground control points placed on the ground and measured with a GNSS in RTK. The new acquisition presented in this paper is instead directly integrated into the GIS thanks to the camera poses calculated using PPK.

A drone orthophoto was also available in the GIS. The previous digital recording was carried out with another drone (Mavic Air 2S) about two years before. As mentioned, some GCPs were necessary for the lack of PPK for the drone previously used. Another difference between the two drone surveys is the different geometry for acquisition. The old capture had only normal images, and vertical surfaces were integrated with laser scanning and terrestrial photogrammetry. The new drone flight also has oblique images, which provide a good reconstruction of walls, although without the same level of detail as a terrestrial capture. Of course, internal spaces could not be captured only with the drone.

An image of the GIS of Castel Grumello is shown in Figure 11. Creating vector files and the association of information requires (long) manual work. Digital orthophotos and georeferenced point clouds are valid geometric support for producing this kind of deliverables, in which automation in the restitution phase is still minimal and require manual work conducted by specialists.

4.2. Castel Mancapane

The set of oriented images with the (partial) direct orientation provided by PPK was used to generate a textured mesh of the castle. The software is Agisoft Metashape, which produced a textured mesh of about 5 million faces. As can be seen, the software was able to provide also a good model of the vegetation around the castle. A large bounding box, including trees around the castle, was necessary for the texture-mapping phase. The extraction of a mesh limited to the volume occupied by the castle resulted in a wrongly textured model, in which the vegetation was also projected on the external walls. This is not only a visual artifact; it could lead to a wrong interpretation of the conditions of the external surfaces, which would present biological colonization during the condition mapping phase, an essential step of the restoration process.

Some pictures of the final 3D model are shown in Figure 12. The reconstruction seems complete and includes the defensive wall (both external and internal surfaces) and the external surfaces of the tower, including the top level.

As mentioned in the previous section, comparing the results of such an acquisition with a laser scanning survey carried out in 2020 using a Faro Focus HDR is also interesting. Although the time required to capture the castle was less than a day, it was necessary to bring a total station (and a tripod) and a GNSS antenna (on a 2 m pole) to georeference the laser scanning survey. Static GNSS was used because of the dense vegetation in the area.

Using a drone with PPK (as in the latest acquisition) is a more comfortable choice, notwithstanding the quality of the laser scanning points is higher (in terms of data density and metric accuracy). In addition, the laser scanner also allowed for capturing the tower’s interior, whereas the latest drone acquisition could only capture the external surfaces.

Using both techniques is still the best approach to obtain the complete acquisition. Still, the use of a laser scanner can also be limited to internal parts and those external elements in which a higher level of detail can be necessary to generate an accurate 3D model (for instance, the staircases added during the restoration intervention). At the same time, the laser scanning acquisition for the walls close to vegetation could only capture the lower parts of the walls.

Another consideration is related to the reconstruction of small details and the need for specialists involved in the conservation process. The staircase added during the restoration could be an example (Figure 13). An automatic reconstruction obtained from images does not separate the different elements, like walls, the metal arch supporting the ruins, and the metal-wooden staircase. Drone images also have a GSD insufficient to provide a sufficiently good geometry.

The laser scanning point clouds have a better resolution instead. They were used in manual modeling enormity to create a 3D model which separates the different constructive elements. Point clouds and textured meshes greatly support heritage documentation, restoration, and conservation. However, other deliverables are often required when other specialists need to design interventions, and direct outputs from photogrammetry and laser scanning should be considered as tools for producing additional deliverables.

4.3. Rocca di Vogogna

The orientation results generated a textured 3D model of the Rocca di Vogogna. Bentley ContextCapture was used for this project. The choice of Bentley ContextCapture instead of Agistoft Metashape relates to the completeness of the mesh obtainable with the software. External and internal orientation results obtained with Agistoft Metashape were exported and used as input in Bentley ContextCapture. The obtained textured model is presented in Figure 14.

Once the textured 3D model is obtained, two main deliverables and results are expected:

• Creation of an overall orthophoto of the Rocca di Vogoga site;
• Creation of orthophotos of the different elevations facing steep slope hills for stratigraphic analysis.

The 3D model generated in Bentley ContextCapture was imported back into Agistoft Metashape to generate those two typologies of deliverables. Indeed, Agistoft Metashape allows for a more straightforward definition of the projection planes for orthophoto definition and has some editing functions.

The Rocca di Vogogna’s overall orthophoto was generated using only normal images. This choice is motivated by the need for a uniform ground sampling distance of $1.0$ cm and the reduction of illumination and shadow differences among the images used to generate the deliverable. The final orthophoto is shown in Figure 15.

The second deliverable typology consists of orthophotos of the fortress’s different elevations (Figure 16). As previously mentioned, the fortress is built on the top of a steep slope hill, preventing an effective acquisition of ground data for stratigraphic analysis of the different masonry structures constituting the walls. For this task, a drone survey was fundamental to reconstruct the different fortress units’ textures properly. Orthophotos were defined for the different elevations of the structure: external main fortifications walls and remains of the fortress tower. Only oblique and convergent images were utilized to generate the elevation orthophotos. The final orthophotos have a GSD of $0.5$ cm.

5. NeRFs for Processing Drone Images

5.1. Preliminary Considerations

Reconstruction of 3D scenes starting from images is a research topic in which scholars have been developing new methods since the beginning of the 20th century. The approach receiving more attention so far is the one based on the geometric consideration of image geometry and image-matching technique for radiometric consistency [1,24,25,26]. New approaches are emerging in the field based on implicit representations. In those methods, a reconstruction model can be trained from a set of 2D images, and the trained model can learn implicit 3D shapes and textures [27]. In particular, Neural Radiance Field (NeRF) models are gaining popularity recently. NeRF models use volume rendering with implicit neural scene representation by using Multi-Layer Perceptrons (MLPs) to learn the geometry and lighting of a 3D scene starting from 2D images. The first paper introducing the concept of NeRF was published by Mildenhall et al. [4] in 2020, and to date (June 2023) it has received more than 3000 citations (source: Google Scholar), proving the increasing interest in this topic. Providing a literature review of this topic is out of the scope of this paper, and the reader is referred to recent review papers [28,29,30]. In this paper, we introduce the central concepts of NeRF, as well as the required input data, and we propose some considerations on NeRF in general and their application to Cultural Heritage applications. NeRF models are gaining attention due to their advantages compared to other similar approaches:

• NeRF models can be trained using only multi-view images of a scene. Unlike other neural representations, NeRF models require only images and poses to learn a scene;
• NeRF models are photo-realistic;
• NeRF models require low memory and perform computations faster.

In its basic form, a NeRF model represents three-dimensional scenes as a radiance field approximated by a neural network. The radiance field describes color and volume density for every point and every viewing direction in the scene. NeRF models are self-supervised. This means that they do not need extra data (like 3D or depth information) for the 3D reconstruction: the input of a NeRF neural network are camera poses (i.e., spatial location and viewing directions) of a set of 2D images of a scene. The output is the volume density of the scene and the view emitted radiance (RGB color) in any direction and for each scene location. The function mapping camera poses to a radiance field (color and volume density) are approximated by one or more Multi-Layer Perceptrons (MLPs). The prediction of the volume density (i.e., the scene’s content) is constrained as independent concerning the viewing direction, while the color is set as dependent on both view direction and scene position. In its standard formulation, NeRF models are designed in two stages. The first stage takes as input the camera poses, passes them to a Multi-Layer Perceptron, and outputs volume density and a high-dimensional feature vector. In the second stage, the feature vector is concatenated with the viewing direction, using them as input for a further MLP, which outputs colors. Then, starting from a trained NeRF, new synthetic views can be created using this workflow:

• For each pixel to be created in the new synthetic image, camera rays and some sampling points are generated into the scene;
• Local color and density are computed for each sampling point by using the view direction, the sampling location, and the trained NeRF;
• The computed colors and densities are the input to produce the synthetic image using volume rendering.

NeRFs are an emerging technology that may provide novel outcomes in the heritage documentation field. A review of possible applications in digital recording and a comparison between NeRFs and traditional photogrammetric approaches was recently proposed in [31,32,33,34]. Previous work was also carried out by [35]. The author recognized the advantages in the case of complex objects with occlusions or irregular shapes, such as vegetated areas (one of the main problems faced in this work). The results also seem promising for homogenous surfaces or transparent/shining objects, which are difficult to model with photogrammetric methods. The metric accuracy of the achieved results is still questionable, especially when compared to photogrammetric or laser scanning point clouds. However, recent neural surface reconstruction methods seem very powerful [36].

Images and videos acquired from drones can also be processed using NeRFs, and previous work was carried out by [37]. The growing interest in such novel technology is demonstrated by the number of examples generated with Luma AI, accessible at https://lumalabs.ai/featured (accessed on 1 August 2023). We tried to process the high-resolution datasets oriented with photogrammetric methods using Nerfstudio, although advancements and updates in NeRFs are expected in the short term.

5.2. Castel Mancapane

5.2.1. Processing with Nerfstudio

The first test with NeRFs was carried out with the images oriented in Metashape and Nerfstudio (https://docs.nerf.studio/en/latest/ (accessed on 1 August 2023)). Exterior, interior, and calibration parameters were exported from the software and processed using Nerfacto. Processing was very rapid (a few minutes, faster than the generation of the point clouds and mesh with Metashape), even considering the significant number of images in the project (549). Default parameters were used in the processing. An image of the viewer after training is shown in Figure 17, whereas some frames of the rendered video were illustrated at the beginning of the manuscript (Figure 1).

Rendering a video is very straightforward, and the quality of the results was excellent. In this sense, NeRFs provided excellent results for Castel Mancapane. Creating a camera path is also relatively simple, making NeRFs an alternative to traditional approaches for rendering based on point clouds or meshes.

Nerfstudio also allows for exporting the geometry. The point cloud was then exported, setting the number of points to 20 million and using the default setting provided by Nerfstudio. A visual comparison with the point cloud generated by Metashape is shown in Figure 18, which comprises 90 million points. The results achieved with Metashape are more complete and less noisy. However, processing was carried out using the default set of options, and the authors know that results could be improved. NeRFs are also relatively new compared to traditional photogrammetric processing, and improvements are indeed expected with future research work.

5.2.2. A Few Considerations on (Some) Other NeRFs Implementations

The same dataset was processed using Luma AI (https://lumalabs.ai/ (accessed on 1 August 2023) and Instant-ngp (https://github.com/NVlabs/instant-ngp (accessed on 1 August 2023)). Luma AI provides a fully automated workflow through a website in which images can be loaded as a single zip file. As the maximum size is 5 gigabytes, the images were resized by 50%. Instant-ngp can be downloaded from the provided website, and we used COLMAP for image orientation. Additionally, in this case, compressed images were used in the workflow, knowing that image resizing affects the final results.

Rendering a video was straightforward with both solutions. The visual quality was excellent, especially for areas with vegetation, which is always difficult to model from a set of images and photogrammetric techniques. The background was also visible in the videos, including elements far from the castle. This interesting result could significantly improve traditional rendering from point clouds or meshes. A couple of sample frames from both videos are shown in Figure 19.

5.3. Rocca di Vogogna

A second test with NeRF was carried out for the Rocca di Vogogna dataset. This test aimed to compare results obtained with traditional image-matching techniques, those obtained with NeRF, and a common benchmark represented by a terrestrial laser scanning point cloud. A complete survey of the area for the Rocca di Vogogna is available. It was carried out using a Faro Focus X130 and a Leica BLK360.

For this case study, the comparison is between Metashape and Nerfstudio. The same images were used for both reconstructions: 390 oblique and convergent images. The set of normal images was discarded since it determined some issues with the NeRF training. Default parameters were used in the processing workflows with both software. Once the training in Nerfstudio was completed, the point cloud was exported, setting the target points to 20 million. The point cloud obtained with Metashape has about 35 million points and was then subsampled to 20 million for further comparison.

As previously noticed, also in the case of the Rocca di Vogogna dataset, a visual comparison shows a NeRF point cloud that is more noisy and less detailed than the one obtained with Metashape (Figure 20).

During the comparison with the terrestrial laser scanning data, the absolute distance between the point cloud is considered the test metric, and the laser scanning point cloud is considered the reference one. The computation of the distance between point clouds was performed by using the software CloudCompare (https://www.danielgm.net/cc/ (accessed on 1 August 2023)). Results of the comparison for image matching and NeRF are reported in Figure 21. The color scale is the same in both figures.

No systematic error or discrepancy can be visualized in comparing both point clouds. However, some more significant problems with the NeRFs solution can be observed in correspondence of vegetated areas, which are not detectable visually.

As shown in Figure 21, the discrepancies between image matching and laser scanning data are in the order of 2–3 cm. In particular, 50% of the discrepancies are below $3.6$ cm.

On the other hand (Figure 21-right), discrepancies for the NeRF reconstruction are in the order of magnitude of 4–5 cm, and 50% of the discrepancies are below $4.9$ cm. In addition, the distribution of discrepancies for the image matching shows a lower dispersion of this dataset than the NeRF results, as it can be observed both in the density and the cumulative distribution of discrepancies for the two datasets (Figure 22). This is probably related to the higher noise in the NeRF solution.

6. Discussion

We discussed a workflow using PPK for direct georeferencing after bundle block adjustment for two case studies (Castel Grumello and Castel Mancapane). The advantage of PPK compared to other solutions based on Ground Control Points for the specific case of castles and fortresses is related to the location of such monuments. Indeed, these structures are frequently built on the tops of steep hills, often surrounded by wooded areas. This circumstance makes it challenging, and, at times, not feasible, to identify natural points and set targets to be used as Ground Control Points.

A solution based on the measurement of camera position can be the only alternative. Both RTK and PPK can be used for camera pose measurement. The RTK connection allows for better waypoint navigation in the case of autonomous flight, which can be useful for challenging environments. However, RTK requires an on-field connection between the master and the rover (in this case, the drone) that may be difficult to obtain in mountain areas. In the PPK model, an on-field connection is unnecessary, and data can be post-processed. The baseline between the master and drone should be carefully planned to avoid a long baseline which may end up in low-accuracy corrections and/or may determine the impossibility of determining a fixed solution. For the same reasons, it is important to set up the cut-off angles and the noise-to-signal ratio in the master antenna. In addition, as presented for the Castel Mancapane case study, the solution may remain a float solution in areas with high vegetation, making those positions unusable for georeferencing after bundle block adjustment.

A second topic explored in this paper is the possibility of using NeRFs (specifically the Nerfstudio implementation) as an alternative to traditional image-matching algorithms for geometry reconstruction. While NeRF approaches present a new paradigm in computer graphics, their utilization for the metric reconstruction of large and complex sites is not fully explored. In addition, a reduction of image size (with a factor of 2, 4, and 8) may reduce the final metric quality of the results, especially for the definition of details. In this paper, a first test was conducted comparing the metric accuracy of NeRFs and a laser scanning dataset considered ground truth.

The metric adopted for the comparison is the unsigned distance between the laser scanning and the NeRF point cloud. This metric can provide some insights into the overall quality of the NeRF solution. The aim was to understand if such results can be used for further processing (e.g., production of architectural drawings) needed to define a conservation strategy. The NeRF point cloud has a good overall consistency with the laser scanning point cloud but also shows large noise and metric discrepancies compared to traditional photogrammetric approaches. However, in the future, significant improvements in NeRF models are expected.

Metrics other than unsigned distanced are also worth testing and evaluating. For example, the definition of geometrical features like point normal and eigenvalues are of primary importance for automated classification algorithms. Their consistency in NeRF-based point clouds is fundamental for evaluating their applicability in conservation projects requiring accurate metric documentation. This topic will be studied in future works.

7. Conclusions

The paper presented some projects using recording techniques based on digital photogrammetry with drones. PPK was a powerful solution to overcome traditional limitations for heritage ruins in mountainous areas. Camera poses provided by PPK allow for a reduction (or a total substitution) of GCPs on the ground, although having some GCPs is always helpful, as well as some check points to verify metric accuracy.

One of the main advantages of drone-based photogrammetry is the opportunity to extract dense point clouds and meshes of parts that are difficult to reach with terrestrial surveying methods. The case of heritage ruins considered in this paper (especially fortified constructions on mountains designed and built to be not accessible) is a clear example where drones help capture external elevations, the remains of tall structures, and the local topography.

Advanced digital surveying techniques can drive the definition of more reliable 3D geometrical models of the built heritage, which is a fundamental step for elaborating the analyses characterizing the conservation process. The digital model provides the common 2D graphic layout for describing the ruined building from the geometrical, technological, and conservation points of view. Moreover, according to the level of complexity of the site, the management of the conservation plan can be structured through HBIM models, providing an interoperable tool for those subjects involved in the preservation process, such as archaeologists, architects, engineers, and stakeholders.

Creating a detailed digital record cannot be carried out only using drones. Laser scanning, terrestrial photogrammetry, and their integrated use with GNSS and total stations are often necessary for the complete acquisition of the ruins. In addition, point clouds and meshes can rarely be considered the final deliverables for restoration and conservation, which usually require additional outputs such as orthophotos, technical drawings (CAD with plans, sections, elevations), GIS, or 3D models from modeling environments allowing further processing for designing interventions. Most of such deliverables still need manual work by specialists, which can use the point clouds/meshes as a reference and produce additional deliverables. Interpreting the different constructive elements (and their separation in the modeling phase), materials, and condition assessment is still conducted manually for complex projects.

The manuscript also considered the case of novel approaches based on NeRFs. Two datasets were also processed with Nerfstudio starting from the initial orientation results achieved with Metashape. Rendering provided very satisfactory results and excellent outputs for complex scenes with vegetation. The extraction of 3D geometry (point clouds in the proposed examples) did not reach the same quality (in terms of metric accuracy and point density) as traditional photogrammetric outputs. However, the rapid development of this relatively new technology is surely expected, and the application of NeRFs in heritage documentation could provide novel alternative processing methods.