Next Article in Journal
A Vectorial Current Density Imaging Method Based on Magnetic Gradient Tensor
Next Article in Special Issue
Traffic Stream Analysis by Radar Sensors on Two-Lane Roads for Free-Moving and Constrained Vehicles Identification
Previous Article in Journal
Temporal Evaluation of the Surface Area of Treated Skin Ulcers Caused by Cutaneous Leishmaniasis and Relation with Optical Parameters in an Animal Model: A Proof of Concept
Previous Article in Special Issue
Pavement Roughness Grade Recognition Based on One-dimensional Residual Convolutional Neural Network
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

A Robotized Raspberry-Based System for Pothole 3D Reconstruction and Mapping

1
Department of Civil, Constructional and Environmental Engineering, Sapienza University of Rome, Via Eudossiana 18, 00184 Rome, Italy
2
AIPSS Associazione Italiana Professionisti Sicurezza Stradale, Piazza del Teatro di Pompeo 2, 00186 Rome, Italy
*
Author to whom correspondence should be addressed.
Sensors 2023, 23(13), 5860; https://doi.org/10.3390/s23135860
Submission received: 18 May 2023 / Revised: 21 June 2023 / Accepted: 21 June 2023 / Published: 24 June 2023
(This article belongs to the Special Issue Low-Cost Sensors for Road Condition Monitoring)

Abstract

:
Repairing potholes is a task for municipalities to prevent serious road user injuries and vehicle damage. This study presents a low-cost, high-performance pothole monitoring system to maintain urban roads. The authors developed a methodology based on photogrammetry techniques to predict the pothole’s shape and volume. A collection of overlapping 2D images shot by a Raspberry Pi Camera Module 3 connected to a Raspberry Pi 4 Model B has been used to create a pothole 3D model. The Raspberry-based configuration has been mounted on an autonomous and remote-controlled robot (developed in the InfraROB European project) to reduce workers’ exposure to live traffic in survey activities and automate the process. The outputs of photogrammetry processing software have been validated through laboratory tests set as ground truth; the trial has been conducted on a tile made of asphalt mixture, reproducing a real pothole. Global Positioning System (GPS) and Geographical Information System (GIS) technologies allowed visualising potholes on a map with information about their centre, volume, backfill material, and an associated image. Ten on-site tests validated that the system works in an uncontrolled environment and not only in the laboratory. The results showed that the system is a valuable tool for monitoring road potholes taking into account construction workers’ and road users’ health and safety.

1. Introduction

The detection and analysis of road surface defects impact the definition of the most appropriate maintenance strategy to ensure traffic safety and ride comfort [1]. Effective maintenance planning also reduces maintenance costs for road managers who face increasing budgetary constraints [2]. Pavement Management Systems (PMSs) are a systematic methodology to assist decision-makers in identifying the best strategy [3]. In this regard, pavement data collection plays a pivotal role and is the most expensive and time-consuming PMS component [4,5].
The detection and assessment of road surface distress are grouped for flexible pavements into cracking, distortion, disintegration, and skid hazards [6]. Disintegration includes potholes which are local bowl-shaped depressions (usually less than 750 mm diameter) classified into three severity levels—Low (L), Medium (M), and High (H)—based on the pothole diameter and depth according to Table 1 [7].
The progressive pavement disintegration into small parts lost with time and under traffic causes potholes. Freeze-thaw cycles are a common factor in pothole forming, where water seeps into pavement cracks and then freezes, causing the pavement to crack and break apart [8]. The repeated passage of traffic-induced loads, poor construction, and the characteristics of the subgrade soil can contribute to pothole occurrence [9]. In the case of Rome, where the climate is relatively mild, the primary causes of potholes may differ from areas with harsher climates: fatigue and pumping from rain can be significant contributors to pothole formation in areas with heavy traffic and poor drainage. It can cause the road surface to weaken, crack, and eventually break apart, forming potholes. Potholes are frequent in urban areas and imply a significant danger to road users, specifically those with two-wheeled vehicles, and potential damage to vehicles [10]. In 2018 potholes and other pavement obstacles on Italian roads caused the death of 75 motorcyclists and injured another 1773. These numbers represent 8.9% and 3.9% of deaths and injuries among two-wheelers, respectively [11].
Hence, pothole detection during routine inspections and its repair [12] are a crucial challenge for road managers because they limit maintenance costs and extend the pavement service life [13]. On the other hand, promptly repairing potholes can return a quality similar to the original pavement, although patching is considered a pavement defect [14].
The traditional manual visual inspections have been the most commonly used road pothole detection method, but it is time-consuming and expensive, exposing workers to traffic hazards too [15]. Visual inspections of potholes are part of a comprehensive pavement assessment to calculate the Pavement Condition Index (PCI) by counting the number of potholes that are low-, medium-, and high-severity [7]. In recent years automated pothole-detection and measurement methods are becoming increasingly popular as they offer a faster and more efficient way to detect and measure potholes [16]. Based on the latest literature, commonly these methods are classified according to [17]:
  • Vision-based methods are fed input visuals (image or video) and include image processing and deep learning algorithms to determine the presence of potholes [18,19,20]. These methods are limited in measuring potholes’ geometric properties, such as depth and volume, because they use two-dimensional (2D) information. In other words, a 2D image or video can only provide us with a two-dimensional representation of a pothole, which makes it difficult to determine its shape and size;
  • Vibration-based methods use data collected from inertial sensors inside test vehicles and assume potholes as a significant acceleration impulse [21,22]. However, these methods cannot detect road surfaces (only wheel paths) [23] and the proper identification of abnormal vibration causes (e.g., distinguishing between potholes and pavement bumps) [24];
  • Three-dimensional (3D) reconstruction-based methods generate pothole 3D representations, from multiple images of the same scene, in the form of point cloud models, mesh models, and geometric models. By using stereovision-based methods—including photogrammetry and Structure-from-Motion (SfM) techniques—it is feasible to assess the pothole size, shape, and depth and measure its volume accurately. Some commonly used 3D pothole reconstruction-based methods include LiDAR (Light Detection and Ranging) scanning [25], stereo vision [26], and photogrammetry [27]. Various techniques are employed to extract depth and geometric information from images; by identifying corresponding points or features in multiple images, the camersas’ position and orientation is determined, and the 3D structure of the scene can be reconstructed [28]. The quality and accuracy of the 3D model depend on factors such as the number and distribution of images, image resolution, camera calibration, and the effectiveness of feature-matching algorithms [29].
  • The photogrammetry approach has been successfully tested by Tion et al. [30] to classify pothole samples in terms of severity levels. In recent years, Atencio et al. [31] developed a method for pothole measurement from 3D models generated from photographs shot by unmanned aerial vehicles (UAVs), determining the optimal flight parameters.
Integrating automated pothole-detection and measurement methods with a Geographic Information System (GIS) can further enhance the efficiency and effectiveness of pavement maintenance and repair efforts by enabling real-time monitoring and analysis of pothole data and facilitating targeted repair and maintenance activities based on accurate and up-to-date information. Once road distresses are detected, their location and characteristics can be captured and georeferenced using GIS, creating a detailed map of pavement failure distribution. Fendi et al. [32] proposed an easy-to-use and cost-effective system for creating a GIS database of geo-tagged photos to help local authorities automate the process of recording and reporting pavement distresses. Nautiyal and Sharma [33] used GIS to store, display, and analyse relevant road data. They developed a tool within GIS to automatically select the most appropriate maintenance technique for each road based on the information stored in GIS. Obaidat et al. [34] used GIS software to analyse data from state-of-the-art technology in pavement management based on the Pavement Condition Index (PCI).
This study aims to overcome the weakness of the pothole 3D reconstruction methodology, which has been revealed in the literature to be the most expensive among the automated methods [35]. The Raspberry Pi technology has been exploited to develop a low-cost prototype for photogrammetric data acquisition. A camera module has been connected to a Raspberry Pi board to capture and save digital images of each pothole from different points of view lined up along predefined circular paths. The latest skill has been made possible by mounting the Raspberry-based prototype on an autonomous and remote-controlled robot to reduce workers’ exposure to live traffic. During field surveys, it works without traffic, protected by a service vehicle. Finally, collected 2D digital photographs have been post-processed using photogrammetry software commonly used in engineering to generate 3D models of objects [36,37]. Significant attention has been paid to the computation of the pothole volume. Data processing is not performed in real-time because the challenge and feature of the system is to be part of a PMS. Indeed, data post-processing should identify the best intervention strategy, and it can require the expertise of multiple technical personnel before filling potholes or scheduling other works. This specific point of the research is a task of the ongoing European research project (InfraROB) focused on a 3D printer able to extrude a bitumen-based mixture for filling in potholes. The pothole volume computation aims to evaluate the amount of material needed to fill in the pothole and to be uploaded on the small tank of the 3D printer. The photogrammetry processing software outputs have been checked by comparison with laboratory tests on a tile made of asphalt mixture, reproducing a real pothole. Integrating a Global Positioning System (GPS) module and GIS technologies allowed the identification of potholes on a map. Each pothole is represented by a point on the map, with information about its centre, volume, backfill material, and an associated image.
On-site tests have been performed considering 10 potholes in a private parking area to demonstrate the system works in an uncontrolled environment.

The InfraROB Project

InfraROB is a project funded by the EU research program Horizon 2020 with the Grant Agreement N. 955337 to maintain road infrastructure integrity, performance, and safety through autonomous robotised solutions and modularisation. It has to be noticed that the European Union’s Horizon 2020 program is funding also the HERON project to develop an integrated automated system and an autonomous ground robotic vehicle for the maintenance of road infrastructures [38].
A specific task of the InfraROB project focuses on a 3D printer to extrude a mixture to repair potholes. A cold asphalt mixture composed of 100% RAP and an innovative bitumen rejuvenator has been designed in InfraROB to repair potholes [39]. The printer is on a small autonomous carrier (TinyMobileRobots [40], TMR) provided by a project partner. The integrated system allows a preventative approach to road maintenance focused on the early stages of road degradation when the potholes are still small. The machine can autonomously reach the position of the pothole to repair while the operator remains parked in a safe roadside area monitoring remotely the robot only, thus reducing workers’ exposure to live traffic. Due to the reduced dimensions of the autonomous carrier and the limited space for the printer allocation, the choice of the mixture had to comply with different constraints. The authors studied a material was studied that did not need compaction to avoid adding a piece of compaction equipment and lighten the carrier. Therefore, the study about the suitable mixture focused on a material that, immediately after the laying, can be compacted by the wheels of the repairing vehicle itself and later by the traffic vehicles. The mixture was studied to achieve and balance conflicting performances of consistency, flowability homogeneity, and internal structure and to be used to repair potholes without preliminary operations (e.g., cutting and cleaning of the asphalt). The final choice was a mixture composed of 100% Reclaimed Asphalt Pavement (RAP) mixed with a rejuvenator [39]: it provides a “green solution” using 100% RAP, that is, without the need to use natural resources.
The dimension of the equipment also limits the maximum weight the small robot can carry. Indeed, the fully loaded weight of the autonomous carrier is 50 kg, and the maximum weight of the mixture that can be loaded is 5 kg since the 3D printer alone weighs 20 kg. Therefore, it could fill in only small potholes: the maximum pothole dimensions to fill-in in a single step are 20 cm-diameter and 2 cm-depth. With these pothole dimensions, the robot can fill 3 potholes per mission. After filling, the autonomous system can return to its base (a van parked in a safe zone) to be re-filled in.

2. The Proposed Integrated System

In this section, the authors describe the developed pothole monitoring system. A novel Raspberry-based system is designed to automatically collect a series of 2D images of the potholes, reproduce precise 3D reality models, and measure the properties of the potholes through digital photogrammetry processing software. The proposed monitoring system can survey up to 75 cm-diameter potholes.
The prototype is integrated into a remote-controlled robotised carrier to capture photos by following a circular path around the potholes, ensuring that each photo overlaps the previous one. A GPS module connected to the Raspberry Pi acquires location data that, combined with GIS technologies, makes it possible to display each pothole on a map as a point, with details about the centre’s location, volume, amount of filling material, and a photo. Figure 1 summarises the overall procedure adopted in this research to reconstruct and map potholes.
The proposed system is currently under development, but it cannot be used in adverse weather conditions both for reasons related to the performance of the fill material (presence of water on the pavement surface) and to avoid potholes during the survey being filled by water or hail that causes variations in the volume calculation. In addition, lighting conditions affect and, in some cases, limit the operation of the system: at the moment, night-time surveys are not possible although they are suitable for safety operational reasons.

2.1. Hardware Platform

The Raspberry Pi technology has become popular due to its affordability, versatility, and easy-to-operate [41]. Several applications in road engineering are available in the scientific literature. Some researchers indirectly monitor the road conditions inside vehicles by using a low-cost inertial measurement unit (IMU) and a GPS module connected to a Raspberry Pi Zero W [42,43,44]. Ambrož et al. developed and tested a low-cost device based on the Raspberry Pi single-board computer for assessing road surface friction [45]. Kulkarni and Baligar propose a system that uses a camera module connected to a Raspberry Pi to detect, track, and count vehicles in real-time [46]. Wu et al. introduced a method that combines UAVs, high-resolution cameras, infrared thermography, Raspberry Pi computers, and deep neural networks to provide a cost-effective and automated solution for infrastructure condition assessment [47].
In this study, the hardware platform assembled and set up consists of consumer-grade components integrated around a Raspberry Pi 4 Model B 8 GB RAM [48] (Rpi 4B, Figure 2), for a final cost of about €250.00 also with the opportunity to purchase on the most popular e-commerce platforms. The hardware does not have enough computing resources to calculate potholes’ volume using photogrammetric techniques. On the other hand, Raspberry Pi technologies have been exploited to acquire data cost-effectively to support road managers in monitoring activities.
The dimensions of this computer (88 × 58 × 19.5 mm) are like those of a credit card: its compact size makes the device portable. A 3A at 5V power bank powers the Rpi 4B. The Pi 4 has dual-band 802.11ac wireless networking, Bluetooth 5.0, Gigabit Ethernet, 2 USB 3.0 and 2 USB 2.0 ports. It also has dual micro-HDMI ports. The operating system installed on the prototype developed in this study is Raspberry Pi OS, based on the Debian Linux distribution. One feature of the Raspberry Pi 4 board is the General-Purpose Input/Output (GPIO) header with 40 pins to connect different sensors. Additionally, the camera port between the HDMI and Ethernet port allows easy connection of official Raspberry Pi cameras.

2.1.1. GPS Module

A U-blox NEO-6M GPS module (NEO-6M GPS Module, Figure 3) [49] has been connected to Rpi 4B (Table 2) to add GPS tags to images. The goal is to upload captured images of potholes into a GIS platform to enable analysis and mapping of potholes in the urban road network: this can assist the road manager in planning and prioritising road maintenance activities based on the severity and location of potholes. This procedure has been studied to define the coordinates of the potholes to repair, which are the necessary input for the TMR to reach and fix the identified potholes.
More in detail, the U-blox mini GPS module NEO-6M is a receiver that uses single-point positioning with C/A Code of L1 frequency from the GPS constellation and receives augmented data from satellite-based augmentation systems. In standard scenarios, the single-point positioning technique with single L1 frequency and civilian code has a typical horizontal position error of 13 m at a 95% probability level. Satellite-based augmentation systems (SBAS) can reduce errors and improve data integrity, availability, and continuity. The GPS module’s update rate was 1 Hz. The Python library GPSD [50] allows the acquisition of PVT data on the Raspbian environment through the NMEA protocol. The obtained GPS data includes geographic coordinates, geometric height, UTC time, and velocity.

2.1.2. Camera Module

A Raspberry Pi Camera Module v3 [51] (Rpi Camera Module v3, Figure 4) allowed the collection of overlapping 2D digital photos of potholes to generate the 3D pothole model.
The Rpi Camera Module v3 features a Sony IMX708 image sensor with a high signal-to-noise ratio (SNR) and built-in 2D Dynamic Defect Pixel Correction (DPC). It also features Phase Detection Autofocus (PDAF) for rapid autofocus, supports HDR mode, and has the IR cut filter. The 25 × 24 × 11.5 mm camera has a resolution of 11.9 megapixels and a sensor size of 7.4 mm diagonally. The Camera Module v3 interfaces with the Raspberry Pi via the Camera Serial Interface (CSI) port, which enables high-speed data transfer between the camera module and the Rpi 4B.

2.2. Data Acquisition

The hardware platform was on the small autonomous carrier (Figure 5) whose cost is not herein disclosed due to agreements signed by the InfraROB partners. The Rpi 4B and its power supply (a power bank) have been fixed in the rear part of the robot. The GPS antenna was in the upper part of the single-board computer case. A camera clamp mounts the camera on the robot chassis, and a sponge material attenuates the vibrations transmitted to the module. Preliminary test shootings returned the best position of the Rpi Camera Module v3 about height (1 m from the road surface) and view angle (45° from the vertical).
The robot has a control app that provides a custom template for the potholes mapping task. As shown in Figure 6, considering the centre of the pothole as the centre of a circle, the robot can follow a circular path all around the pothole by defining the Radius value as input. Several circles with different radii in the range of 1.0–1.4 m around the pothole need to take enough photos for a complete and detailed pothole 3D reconstruction.
The circular path has been discretised in 30 points, describing circumferential arcs of equal length. The robot velocity driving to waypoints in automatic mode was 0.4 m/s. By pausing for 6 s at each point, the robot allows the camera capturing at least one picture of the pothole in static conditions for each point as the camera mounted on the robot automatically takes a picture per 5 s.

Pothole Images Acquisition Using Python Script

A Python script has been implemented in Rpi 4B to capture geotagged images using Rpi Camera Module v3 and NEO-6M GPS module. The program has an infinite loop that captures an image and adds to it GPS metadata every 5 s. Within this loop, the program obtains the GPS coordinates, captures an image using the Picamera library [52], saves it to a JPEG file format, and then adds GPS metadata to the image file using the piexif library [53]. The Picamera library is a Python interface providing easy access to the camera module of the Raspberry Pi. The library supports several features, including capturing images and video in various formats and resolutions and configuring camera settings such as brightness and exposure. The piexif library is another Python library that simplifies reading and writing metadata (EXIF, IPTC, XMP) in JPEG image files, including camera settings and geolocation data. The library also supports new image files with metadata, allowing you to add custom metadata to images.
At boot time, Rpi 4B has been connected automatically to the internet network to realise remote access and launch the command routine without being in situ. In addition, the storage has been set up automatically at the end of the survey phase in a cloud accessible from the PC equipped for post-processing.

2.3. Photogrammetric Data Processing

In this study, the authors exploited Bentley ContextCapture [54] to create 3D models of real-world scenes using multiple photos. However, it has to be noticed that other non-commercial and commercial specialized software (e.g., Pixpro [55], Agisoft Metashape [56], MicMac [57], Meshroom [58], VisualSFM [59]) are available to generate 3D models and shall be investigated for further comparative analyses, that are beyond the scope of this study.
Photographs around the pothole need to provide information about the geometry and appearance of the scene from different viewpoints. It is necessary to have enough overlap between the photos—the overlap between consecutive photographs should typically exceed two-thirds—so that the software can detect common features and accurately match them.
ContextCapture uses photogrammetry techniques to analyse the photos and reconstruct the 3D geometry with aerotriangulation. In this study, user tie points gave scaling constraints during aerotriangulation. These user-tie points have been identified by selecting at least two photos with overlapping views and specifying the distance between them. After the aerotriangulation step is completed, a 3D mesh model of the scene composed of interconnected polygons represents the surveyed surface. The resulting mesh can be exported in various formats as STL: the 3D printer designed as part of InfraROB requires this format file as input. Tools are also available to perform measurements such as area and volume.

2.4. Computation of Pothole Fill Material

Given the pothole’s volume (V) and the mixture characteristics, Equation (1) gives the mixture weight to be extruded (W). Specifically, the bulk density of the mixture (ρ) obtained in a previous study is 2.12 g/cm3 [39].
W = ρ × V

2.5. GIS Technologies to Map Potholes

The position of the centre of the pothole where the robot must initially go and stand is obtained by reading the GPS data acquired with the NEO-6M GPS module during the survey. Before starting, the GPS antenna coincides with the centre acquired for defining the path around the pothole. The filter on the data to identify the position when the robot stands in the centre of the pothole is possible by comparing acquisition times of GPS data and images captured by the camera, already activated during the approach to the pothole. The open-source software QGIS [60] was used to visualise the volume, the amount of fill material, and the pothole’s position as a point on the map, with an attached image of the pothole. A Python script automated this operation. The code loads the list of pothole coordinates, volumes, and backfill material from a CSV file, creates a new point layer “POTHOLES” in QGIS, and adds the referenced points, the attributes “Volume” and “Material to Fill Potholes” to the layer. It also allows the user to select the photo for each point by providing a photo path as input.

3. Results

3.1. Laboratory Tests

Laboratory tests were conducted on an asphalt tile simulating a real pothole (Figure 7). A 37 × 32 × 8 cm wooden formwork was filled with cold-mix asphalt and compacted until the upper surface was horizontal. A bowl shape has been opened near the tile’s centre to simulate a pothole.
The robot collected the centre of the asphalt tile by including its coordinates from the NEO-6M GPS module. During its riding, the Rpi Camera Module v3 shot 90 static pictures (image dimensions: 4608 × 2592 pixels) within minutes. Three circular trajectories with 1.0, 1.2, and 1.4 m radii were set. The photos were transferred to a computer as dataset input to reproduce a detailed 3D model of the pothole with Bentley ContextCapture software. The desktop Lenovo ThinkStation P620 with Windows 11 Pro 64-bit, an AMD Ryzen Threadripper PRO 3955WX 16-Cores processor, 32.0 GB of RAM, and an NVIDIA Quadro RTX 5000 have been used for the photogrammetric processing. In this study, the processing time (queue) was about 25 minutes, and the quality report generated by the software revealed an average ground resolution equal to 0.40267 mm/pixel.
A scale constraint during aerotriangulation was entered by selecting in 2 different photos user tie points A and B corresponding to the ends of one of the sides of the wooden formwork of known dimensions (31.5 cm) and then specifying the distance between them. Aerotriangulation main settings are described in Table 3.
After the aerotriangulation step, the 3D model was created (software’s default processing settings) and then questioned to evaluate the volume of the pothole (Figure 8). More in detail, this software task includes manually drawing the boundaries of the pothole as a Region of Interest (ROI). Once the ROI is defined, Bentley ContextCapture calculates the volume by comparing the model surface with a reference plane (i.e., the top surface defined by its mean plane determined as a function of the pothole edge). This approach involves dividing the 3D model into a grid of prisms, which represent an approximate discretization of the volume occupied by the model.
The actual volume was then measured in the laboratory by filling the pothole with water to verify the output from the software. For this purpose, the pothole was waterproofed by applying a synthetic thermoplastic material film; after which the distilled water contained in a graduated cylinder (±5 mL) was poured in (Figure 9).
The water volume at which water leaked out of the pothole was 485 mL: it confirmed that the proposed methodology is effective as the water volume returned by the software is very close (a relative error of 1%).
The weight of the material needed to fill the pothole was then determined (Equation (1)). In the experimental set-up, it was 1.02 kg.
Finally, all the information was uploaded and mapped in QGIS (Figure 10).

3.2. On-Site Tests

Ten on-site tests were carried out in a private parking area to validate that the system works in an uncontrolled environment and not only in the laboratory. Figure 11a–e shows how the main phases of the methodology have been applied.
Table 4 shows the computed pothole volume and the actual amount of filling material (Equation (1)) used to fix potholes after compaction.
The results confirmed that the laboratory set-up did not differ from the actual in situ conditions, as the extruded loose material (after compaction) effectively fixed the potholes by restoring surface evenness.

4. Conclusions

The most frequent distress on urban asphalt pavements are cracking, deformation, bleeding, and potholes. Potholes on the wearing course and the consequent material loss significantly impact road safety since they can damage vehicles and cause unforeseen accidents for road users. The essential needs of maintaining the integrity, performance, and safety of road infrastructures have been tackled in InfraROB, an ongoing research project funded by the EU research programme Horizon 2020 that aims to promote autonomous robotized solutions and modularization. A specific project task focuses on a 3D printer able to extrude a cold asphalt mixture composed of 100% RAP and fix small potholes.
This study presents a low-cost, high-performance pothole monitoring system. A Raspberry-based prototype has been developed and mounted on an autonomous and remote-controlled robot to avoid safety hazards to workers during field surveys. The proposed integrated system involves a camera and GPS modules managed by a mini-computer Rpi 4B. The robot can describe a circular path around the pothole at 0.4 m/s. Meanwhile, the prototype collects the coordinates of the pothole centre and shoots overlapped 2D images in static conditions and from different view angles. Laboratory tests on an asphalt tile simulating a real pothole validated the proposed methodology. The collected images (i.e., 90) are then processed through photogrammetric techniques using Bentley ContextCapture software to create the pothole 3D model and calculate the volume to fill. The processing time has been fast because the entire queue last about 25 min. Nevertheless, any photogrammetry software that returns data in the format required by the 3D printer in the filling phase could be used.
The bulk density of the mixture designed (i.e., 2.12 g/cm3) in this InfraROB task combined with the computed volume returns the weight to be extruded. Finally, for managing the available information as the coordinates of the pothole centre, the calculated volume and backfill material, and one photo are uploaded and mapped on a GIS platform.
The proposed approach is a valuable tool for road agencies to monitor road potholes and implement preventive maintenance strategies reducing risks for road workers and staying within budget constraints. On-site tests demonstrated that the adopted procedures can be implemented without modification in an uncontrolled environment (e.g., open-to-traffic pavements) because the laboratory set does not differ from in-situ conditions.
Further developments include integrating Building Information Modelling (BIM) and GIS can be leveraged to support the development of sustainable built environments and provide an effective PMS. This approach is becoming important according to European legislation [61] that encourages the adoption of BIM in public works and infrastructure projects. As for the on-site activities, the analysis will focus on a 3D printer to extrude the mixture and repair potholes. Finally, field tests will be performed on different damaged sections in an urban road network to monitor the behaviour of the new equipment. Future studies also include adapting the system to different distress (e.g., all type of cracks).

Author Contributions

Conceptualization, S.B.; Data curation, S.B. and L.M.; Formal analysis, G.L.; Funding acquisition, C.P.; Investigation, S.B.; Methodology, S.B., P.D.M. and N.F.; Project administration, C.P.; Resources, C.P. and A.D.; Software, S.B., G.L. and L.M.; Supervision, A.D. and L.M.; Validation, G.C., N.F. and A.D.; Visualization, S.B., N.F. and L.M.; Writing—original draft, S.B. and P.D.M.; Writing—review & editing, G.L. and G.C. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded within the research project InfraROB, funded by the European Commission’s research programme Horizon 2020 with the Grant Agreement N. 955337.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The data presented in this study are available on request from the corresponding author. The data are not publicly available due to confidentiality reasons.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Bruno, S.; Colonnese, S.; Scarano, G.; Del Serrone, G.; Loprencipe, G. Pavement Distress Estimation via Signal on Graph Processing. Sensors 2022, 22, 9183. [Google Scholar] [CrossRef] [PubMed]
  2. Basri, E.I.; Razak, I.H.A.; Ab-Samat, H.; Kamaruddin, S. Preventive Maintenance (PM) Planning: A Review. J. Qual. Maint. Eng. 2017, 23, 114–143. [Google Scholar] [CrossRef]
  3. Fengier, J.; Słowik, M.; Pożarycki, A. Contactless Approach to Determine Pavement Skid Resistance for Pavement Management System. In MATEC Web of Conferences; EDP Sciences: Les Ulis, France, 2018; Volume 222, p. 01003. [Google Scholar]
  4. Cafiso, S.; Di Graziano, A.; Goulias, D.G.; D’Agostino, C. Distress and Profile Data Analysis for Condition Assessment in Pavement Management Systems. Int. J. Pavement Res. Technol. 2019, 12, 527–536. [Google Scholar] [CrossRef]
  5. Loprencipe, G.; Cantisani, G.; Di Mascio, P. Global Assessment Method of Road Distresses. In Life-Cycle Design, Assessment, and Maintenance of Structures and Infrastructure Systems, Proceedings of the International Symposium Life-Cycle Civil Engineering, Lake Como, Italy, 11–14 June 2015; Furuta, H., Frangopol, D.M., Akiyama, M., Eds.; CRC Press: Boca Raton, FL, USA, 2015; pp. 1113–1120. [Google Scholar]
  6. Mouaket, I.M.; Al-Mansour, A.; Sinha, K.C. Evaluation of the Cost-Effectiveness of Pavement Surface Maintenance Activities; Indiana Department of Transportation: West Lafayette, IN, USA, 1990; p. 303. [Google Scholar]
  7. ASTM D6433-20; Standard Practice for Roads and Parking Lots Pavement Condition Index Surveys. ASTM: West Conshohocken, PA, USA, 2020.
  8. Biswas, S.; Hashemian, L.; Bayat, A. Investigation of Pothole Severity and Maintenance Methods in Canada through Questionnaire Survey. J. Cold Reg. Eng. 2018, 32, 04018002. [Google Scholar] [CrossRef]
  9. Kanoungo, A.; Sharma, U.; Goyal, A.; Kanoungo, S.; Singh, S. Assessment of Causes of Pothole Development on Chandigarh Roads. J. Inst. Eng. India Ser. A 2021, 102, 411–419. [Google Scholar] [CrossRef]
  10. Bosurgi, G.; Modica, M.; Pellegrino, O.; Sollazzo, G. An Automatic Pothole Detection Algorithm Using Pavement 3D Data. Int. J. Pavement Eng. 2022, 1–15. [Google Scholar] [CrossRef]
  11. Incidenti Stradali: Ancma, Un Decesso Su 10 Dovuto a Buche e Ostacoli in Strada. Available online: https://www.ansa.it/canale_motori/notizie/sicurezza/2019/07/28/incidenti-stradali-ancma-un-decesso-su-10-dovuto-a-buche-e-ostacoli-in-strada_fa845c55-9280-4172-b323-b5105aca96a3.html (accessed on 16 March 2023).
  12. Biswas, S.; Hashemian, L.; Hasanuzzaman, M.; Bayat, A. A Study on Pothole Repair in Canada through Questionnaire Survey and Laboratory Evaluation of Patching Materials. Can. J. Civ. Eng. 2016, 43, 443–450. [Google Scholar] [CrossRef]
  13. Pożarycki, A.; Górnaś, P.; Fengier, J. Mechanistic Approach and LCCA for Pavement Rehabilitation in the Absence of Ability to Change Its Vertical Alignment. Arch. Inst. Inż. Lądowej 2015, 63–76. [Google Scholar]
  14. Hadjidemetriou, G.; Serrano, M.; Vela, P.; Christodoulou, S. Patch Defects in Images Using Support Vector Machines. Civ. Comp Proc. 2015, 108. [Google Scholar]
  15. Feng, Z.; El Issaoui, A.; Lehtomäki, M.; Ingman, M.; Kaartinen, H.; Kukko, A.; Savela, J.; Hyyppä, H.; Hyyppä, J. Pavement Distress Detection Using Terrestrial Laser Scanning Point Clouds–Accuracy Evaluation and Algorithm Comparison. ISPRS Open J. Photogramm. Remote Sens. 2022, 3, 100010. [Google Scholar] [CrossRef]
  16. Garbowski, T.; Gajewski, T. Semi-Automatic Inspection Tool of Pavement Condition from Three-Dimensional Profile Scans. Procedia Eng. 2017, 172, 310–318. [Google Scholar] [CrossRef]
  17. Kim, T.; Ryu, S.-K. Review and Analysis of Pothole Detection Methods. J. Emerg. Trends Comput. Inf. Sci. 2014, 5, 603–608. [Google Scholar]
  18. Ibragimov, E.; Lee, H.-J.; Lee, J.-J.; Kim, N. Automated Pavement Distress Detection Using Region Based Convolutional Neural Networks. Int. J. Pavement Eng. 2022, 23, 1981–1992. [Google Scholar] [CrossRef]
  19. Gajjar, K.; Niekerk, T.; van Wilm, T.; Mercorelli, P. Vision-Based Deep Learning Algorithm for Detecting Potholes. J. Phys. Conf. Ser. 2022, 2162, 012019. [Google Scholar] [CrossRef]
  20. Tedeschi, A.; Benedetto, F. A Real-Time Automatic Pavement Crack and Pothole Recognition System for Mobile Android-Based Devices. Adv. Eng. Inform. 2017, 32, 11–25. [Google Scholar] [CrossRef]
  21. Mednis, A.; Strazdins, G.; Zviedris, R.; Kanonirs, G.; Selavo, L. Real Time Pothole Detection Using Android Smartphones with Accelerometers; IEEE: Piscataway, NJ, USA, 2011; pp. 1–6. [Google Scholar]
  22. Du, R.; Qiu, G.; Gao, K.; Hu, L.; Liu, L. Abnormal Road Surface Recognition Based on Smartphone Acceleration Sensor. Sensors 2020, 20, 451. [Google Scholar] [CrossRef] [Green Version]
  23. Jo, Y.; Ryu, S. Pothole Detection System Using a Black-Box Camera. Sensors 2015, 15, 29316–29331. [Google Scholar] [CrossRef]
  24. Kyriakou, C.; Christodoulou, S.E.; Dimitriou, L. Smartphone-Based Pothole Detection Utilizing Artificial Neural Networks. J. Infrastruct. Syst. 2019, 25, 04019019. [Google Scholar] [CrossRef]
  25. Ravi, R.; Habib, A.; Bullock, D. Pothole Mapping and Patching Quantity Estimates Using LiDAR-Based Mobile Mapping Systems. Transp. Res. Rec. 2020, 2674, 124–134. [Google Scholar] [CrossRef]
  26. Li, Y.; Papachristou, C.; Weyer, D. Road Pothole Detection System Based on Stereo Vision; IEEE: Piscataway, NJ, USA, 2018; pp. 292–297. [Google Scholar]
  27. Tan, Y.; Li, Y. UAV Photogrammetry-Based 3D Road Distress Detection. ISPRS Int. J. Geo-Inf. 2019, 8, 409. [Google Scholar] [CrossRef] [Green Version]
  28. Remondino, F.; El-Hakim, S.F.; Gruen, A.; Zhang, L. Turning Images into 3-D Models. IEEE Signal Process. Mag. 2008, 25, 55–65. [Google Scholar] [CrossRef]
  29. Remondino, F.; Guarnieri, A.; Vettore, A. 3D Modeling of Close-Range Objects: Photogrammetry or Laser Scanning? Beraldin, J.-A., El-Hakim, S.F., Gruen, A., Walton, J.S., Eds.; SPIE Library: San Jose, CA, USA, 2005; p. 216. [Google Scholar]
  30. Tiong, P.L.Y.; Mustaffar, M.; Hainin, M.R. Road Surface Assessment of Pothole Severity by Close Range Digital Photogrammetry Method. World Appl. Sci. J. 2012, 19, 867–873. [Google Scholar]
  31. Atencio, E.; Plaza-Muñoz, F.; Muñoz-La Rivera, F.; Lozano-Galant, J.A. Calibration of UAV Flight Parameters for Pavement Pothole Detection Using Orthogonal Arrays. Autom. Constr. 2022, 143, 104545. [Google Scholar] [CrossRef]
  32. Fendi, K.G.; Adam, S.M.; Kokkas, N.; Smith, M. An Approach to Produce a GIS Database for Road Surface Monitoring. APCBEE Procedia 2014, 9, 235–240. [Google Scholar] [CrossRef]
  33. Nautiyal, A.; Sharma, S. Condition Based Maintenance Planning of Low Volume Rural Roads Using GIS. J. Clean. Prod. 2021, 312, 127649. [Google Scholar] [CrossRef]
  34. Obaidat, M.T.; Ghuzlan, K.A.; Bara’W, A.-M. Integration of Geographic Information System (GIS) and PAVER System toward Efficient Pavement Maintenance Management System (PMMS). Jordan J. Civ. Eng. 2018, 12, 449–460. [Google Scholar]
  35. Kim, Y.-M.; Kim, Y.-G.; Son, S.-Y.; Lim, S.-Y.; Choi, B.-Y.; Choi, D.-H. Review of Recent Automated Pothole-Detection Methods. Appl. Sci. 2022, 12, 5320. [Google Scholar] [CrossRef]
  36. Brzeziński, K.; Maślakowski, M.; Liszewski, P. Evaluation of the Volume Measurement Optical Method Suitability for Determining the Relative Compaction of Soils. Civ. Eng. J. 2018, 4, 2052. [Google Scholar] [CrossRef] [Green Version]
  37. de Oliveira, A.K.V.; Bracht, M.K.; Melo, A.P.; Lamberts, R.; Ruther, R. Evaluation of Faults in a Photovoltaic Power Plant Using Orthomosaics Based on Aerial Infrared Thermography. In Proceedings of the 2021 IEEE 48th Photovoltaic Specialists Conference (PVSC), Fort Lauderdale, FL, USA, 20 June 2021; pp. 2604–2610. [Google Scholar]
  38. Katsamenis, I.; Bimpas, M.; Protopapadakis, E.; Zafeiropoulos, C.; Kalogeras, D.; Doulamis, A.; Doulamis, N.; Martín-Portugués Montoliu, C.; Handanos, Y.; Schmidt, F. Robotic Maintenance of Road Infrastructures: The Heron Project. In Proceedings of the 15th International Conference on Pervasive Technologies Related to Assistive Environments, Corfu, Greece, 29 June 2022–1 July 2022; pp. 628–635. [Google Scholar]
  39. Cantisani, G.; D’Andrea, A.; Di Mascio, P.; Moretti, L.; Fiore, N.; Petrelli, M.; Polidori, C.; Venturini, L. Materials Study to Implement a 3D Printer System to Repair Road Pavement Potholes. In New Scenarios for Transport Infrastructure and Systems Transition to Inclusivity, Resilience and Sustainability, Proceedings of the AIIT 3rd International Conference, Rome, Italy, 2022, 15–16 September 2022; Giordano Editore: Rome, Italy, 2023; Volume 69, pp. 91–98. [Google Scholar] [CrossRef]
  40. TinyMobileRobots. Available online: https://tinymobilerobots.com/ (accessed on 18 April 2023).
  41. Ghael, H.D.; Solanki, L.; Sahu, G. A Review Paper on Raspberry Pi and Its Applications. Int. J. Adv. Eng. Manag. 2020, 2, 4. [Google Scholar]
  42. Loprencipe, G.; de Almeida Filho, F.G.V.; de Oliveira, R.H.; Bruno, S. Validation of a Low-Cost Pavement Monitoring Inertial-Based System for Urban Road Networks. Sensors 2021, 21, 3127. [Google Scholar] [CrossRef]
  43. Bruno, S.; Vita, L.; Loprencipe, G. Development of a GIS-Based Methodology for the Management of Stone Pavements Using Low-Cost Sensors. Sensors 2022, 22, 6560. [Google Scholar] [CrossRef]
  44. Bruno, S.; Loprencipe, G.; Marchetti, V. Proposal for a Low-Cost Monitoring System to Assess the Pavement Deterioration in Urban Roads. Eur. Transp. Eur. 2023, 91, 1–10. [Google Scholar] [CrossRef]
  45. Ambrož, M.; Hudomalj, U.; Marinšek, A.; Kamnik, R. Raspberry Pi-Based Low-Cost Connected Device for Assessing Road Surface Friction. Electronics 2019, 8, 341. [Google Scholar] [CrossRef] [Green Version]
  46. Kulkarni, R.B. Dynamic Decision Model for a Pavement Management System. Transp. Res. Rec. 1984, 8, 11–18. [Google Scholar]
  47. Wu, W.; Qurishee, M.A.; Owino, J.; Fomunung, I.; Onyango, M.; Atolagbe, B. Coupling Deep Learning and UAV for Infrastructure Condition Assessment Automation. In Proceedings of the 2018 IEEE International Smart Cities Conference (ISC2), Kansas, MI, USA, 16 September 2018; pp. 1–7. [Google Scholar]
  48. Raspberry Pi 4. Available online: https://www.raspberrypi.com/products/raspberry-pi-4-model-b/ (accessed on 8 February 2023).
  49. U-Blox NEO-6M GPS Module. Available online: https://www.u-blox.com/en/product/neo-6-series (accessed on 8 February 2023).
  50. GPSD Libray. Available online: https://pypi.org/project/gpsd-py3/ (accessed on 8 April 2023).
  51. Raspberry Pi Camera Module 3. Available online: https://www.raspberrypi.com/products/camera-module-3/ (accessed on 25 March 2023).
  52. Picamera Library. Available online: https://github.com/waveform80/picamera (accessed on 5 April 2023).
  53. Piexif Library. Available online: https://pypi.org/project/piexif/ (accessed on 5 April 2023).
  54. ContextCapture-Bentley Systems. Available online: https://www.bentley.com/software/contextcapture/ (accessed on 8 April 2023).
  55. Pixpro Photogrammetry Software. Available online: https://www.pix-pro.com/ (accessed on 17 April 2023).
  56. Agisoft Metashape. Available online: https://www.agisoft.com/ (accessed on 17 April 2023).
  57. MicMac. Available online: https://micmac.ensg.eu/index.php/Accueil (accessed on 17 April 2023).
  58. Meshroom. Available online: https://alicevision.org/#meshroom (accessed on 17 April 2023).
  59. VisualSFM. Available online: http://ccwu.me/vsfm/index.html (accessed on 17 April 2023).
  60. QGIS. Available online: https://docs.qgis.org/3.28/it/docs/user_manual/ (accessed on 23 April 2023).
  61. EUR-Lex Access to European Union Law (2014). Direttiva 2014/24/EU del Parlamento Europeo e del Consiglio, del 26 Febbraio 2014, Sugli Appalti Pubblici e che Abroga La Direttiva 2004/18/CE Testo Rilevante Ai Fini del SEE. Available online: https://eur-lex.europa.eu/legal-content/IT/TXT/?qid=1485620527972&uri=CELEX:32014L0024 (accessed on 17 April 2023).
Figure 1. The framework of the overall procedure.
Figure 1. The framework of the overall procedure.
Sensors 23 05860 g001
Figure 2. RPI 4B.
Figure 2. RPI 4B.
Sensors 23 05860 g002
Figure 3. A view of the GPS antenna and the U-blox GPS module connected to Rpi 4B.
Figure 3. A view of the GPS antenna and the U-blox GPS module connected to Rpi 4B.
Sensors 23 05860 g003
Figure 4. Rpi Camera Module v3 connected to Rpi 4B.
Figure 4. Rpi Camera Module v3 connected to Rpi 4B.
Sensors 23 05860 g004
Figure 5. The proposed integrated system.
Figure 5. The proposed integrated system.
Sensors 23 05860 g005
Figure 6. TMR path setting.
Figure 6. TMR path setting.
Sensors 23 05860 g006
Figure 7. Asphalt tile simulating a real pothole.
Figure 7. Asphalt tile simulating a real pothole.
Sensors 23 05860 g007
Figure 8. Determining pothole fill volume using photogrammetry software.
Figure 8. Determining pothole fill volume using photogrammetry software.
Sensors 23 05860 g008
Figure 9. Measuring pothole fill volume in lab tests.
Figure 9. Measuring pothole fill volume in lab tests.
Sensors 23 05860 g009
Figure 10. Pothole information in GIS. The red dot locates in the centre of the pothole.
Figure 10. Pothole information in GIS. The red dot locates in the centre of the pothole.
Sensors 23 05860 g010
Figure 11. Main phases of the presented methodology. (a) Photos of the pothole from different points of view; (b) 3D mesh model of the scene; (c) Volume computation using software tools; (d) On-site inspection by qualified personnel; (e) Amount of filling material.
Figure 11. Main phases of the presented methodology. (a) Photos of the pothole from different points of view; (b) 3D mesh model of the scene; (c) Volume computation using software tools; (d) On-site inspection by qualified personnel; (e) Amount of filling material.
Sensors 23 05860 g011
Table 1. Severity-based pothole classification.
Table 1. Severity-based pothole classification.
Maximum Depth [mm]Average Diameter [mm]
100–200200–450450–750
13–25LLM
25–50LMH
>50MMH
Table 2. Connections between the NEO-6M GPS module and the Rpi 4B.
Table 2. Connections between the NEO-6M GPS module and the Rpi 4B.
Raspberry PinSensor Pin
3V3VCC
GNDGND
GPIO 14RX
GPIO 15TX
Table 3. Aerotriangulation main settings in Bentley ContextCapture for the study processes.
Table 3. Aerotriangulation main settings in Bentley ContextCapture for the study processes.
ParametersSettings
Final rigid registrationRegistration constraints
Key point densityHigh
Pair selection modeDefault
Component construction modeOnePass
Blockwise colour equalizationEnabled
SplatsEnabled
Table 4. On-site test results.
Table 4. On-site test results.
PotholePothole Volume [cm3]Computed Filling Material [kg]
I3920.83
II10242.17
III9251.96
IV7451.58
V15193.22
VI4290.91
VII19254.08
VIII12592.67
IX20574.36
X4150.88
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Bruno, S.; Loprencipe, G.; Di Mascio, P.; Cantisani, G.; Fiore, N.; Polidori, C.; D’Andrea, A.; Moretti, L. A Robotized Raspberry-Based System for Pothole 3D Reconstruction and Mapping. Sensors 2023, 23, 5860. https://doi.org/10.3390/s23135860

AMA Style

Bruno S, Loprencipe G, Di Mascio P, Cantisani G, Fiore N, Polidori C, D’Andrea A, Moretti L. A Robotized Raspberry-Based System for Pothole 3D Reconstruction and Mapping. Sensors. 2023; 23(13):5860. https://doi.org/10.3390/s23135860

Chicago/Turabian Style

Bruno, Salvatore, Giuseppe Loprencipe, Paola Di Mascio, Giuseppe Cantisani, Nicola Fiore, Carlo Polidori, Antonio D’Andrea, and Laura Moretti. 2023. "A Robotized Raspberry-Based System for Pothole 3D Reconstruction and Mapping" Sensors 23, no. 13: 5860. https://doi.org/10.3390/s23135860

APA Style

Bruno, S., Loprencipe, G., Di Mascio, P., Cantisani, G., Fiore, N., Polidori, C., D’Andrea, A., & Moretti, L. (2023). A Robotized Raspberry-Based System for Pothole 3D Reconstruction and Mapping. Sensors, 23(13), 5860. https://doi.org/10.3390/s23135860

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop