ROAD: Robotics-Assisted Onsite Data Collection and Deep Learning Enabled Robotic Vision System for Identification of Cracks on Diverse Surfaces

Abstract

Crack detection on roads is essential nowadays because it has a significant impact on ensuring the safety and reliability of road infrastructure. Thus, it is necessary to create more effective and precise crack detection techniques. A safer road network and a better driving experience for all road users can result from the implementation of the ROAD (Robotics-Assisted Onsite Data Collecting) system for spotting road cracks using deep learning and robots. The suggested solution makes use of a robot vision system’s capabilities to gather high-quality data about the road and incorporates deep learning methods for automatically identifying cracks. Among the tested algorithms, Xception stands out as the most accurate and predictive model, with an accuracy of over 90% during the validation process and a mean square error of only 0.03. In contrast, other deep neural networks, such as DenseNet201, InceptionResNetV2, MobileNetV2, VGG16, and VGG19, result in inferior accuracy and higher losses. Xception also achieves high accuracy and recall scores, indicating its capability to accurately identify and classify different data points. The high accuracy and superior performance of Xception make it a valuable tool for various machine learning tasks, including image classification and object recognition.

1. Introduction

Cracks may eventually form on roads as a result of exposure to adverse weather, heavy traffic, and other environmental variables. These cracks could endanger the track’s safety and worsen the situation if they are not found and repaired very soon. Moreover, crack detection on highways is also crucial because it has a big impact on ensuring the dependability and safety of the road infrastructure. Road crack detection with traditional technology is typically time-consuming, expensive, and unreliable. While traditional methods can be time-consuming and expensive, automated crack detection using deep learning and robotic vision technology can improve accuracy, speed, and efficiency. These methods involve training algorithms to recognize and classify cracks based on images or videos collected by robotic vision systems. Automated crack detection can be used in various applications, such as building and infrastructure inspection, bridge inspection, and industrial equipment maintenance. Automated crack detection methods are essential for improving structural health monitoring and assessment and preventing potential safety hazards [1].

Cracks on diverse surfaces refer to visible openings or fractures that occur on different types of materials, such as concrete, wood, metal, ceramics, or glass. Cracks can be caused by a variety of factors, including stress, temperature changes, chemical exposure, moisture, and age [2]. On concrete surfaces, for example, cracks can be caused by the expansion and contraction of the material due to temperature changes or moisture, as well as the weight and movement of heavy objects. In wood, cracks can occur due to changes in humidity, exposure to sunlight, or the aging process. The cracks can be caused by many reasons. The cracks in flexible pavement and rigid pavement may have different reasons. For example, in asphalt pavement, moisture damage is a major issue since water will weaken the adhesion between asphalt and aggregate [3].

Cracks can vary in size and shape, from hairline fractures to larger openings. They can be superficial and cosmetic or compromise the structural integrity of the material. Cracks may also be an indication of a larger problem that needs to be addressed, such as a foundation issue in a building or a problem with the structural support of a bridge. However, it is challenging for computer vision methods due to low-level features such as cracks and difficulties such as inhomogeneous illumination and irregularities in construction. Recent advancements in computer vision and image processing techniques are improving crack detection capabilities, enabling better decision-making for structural maintenance and safety. Practical challenges remain due to the nature of the subject matter, which is characterized by three major factors [4].

Crack detection is a challenging task for computer vision methods due to the low-level discriminative features of cracks, which can be easily confused with background noise such as foreign objects. In addition, the inhomogeneous illumination of the surface and irregularities in the application process, such as the exposure of jointing, make it difficult to distinguish the cracks from the surrounding area. These factors pose significant obstacles to the accurate detection of cracks in structures and require sophisticated image processing methods and algorithms to overcome. To address these issues, researchers have concentrated on creating deep learning-based techniques that can accurately find and categorize cracks in a variety of structures, such as steel and concrete buildings, bridges, pipelines, aircraft, and railroad tracks. These approaches depend on the capacity of DNN to learn intricate representations of the incoming data and generate precise predictions even in the presence of noise and abnormalities [5].

Deep learning (DL) has been shown to perform better than conventional techniques for image recognition and classification. Building a proper representation of the information is a crucial step in creating the best algorithm for automated deep learning bridge crack detection. Prior approaches, which can be constrained, rely on conventional picture binarization and manually adjusted segmentation [6]. In this novel method, crack segments are identified by line-fitting, and features related to the local line fit are computed. With the use of machine learning, this strategy is sturdy and offers a viable remedy for automatically detecting bridge cracks. It does so by overcoming the drawbacks of earlier methods [7].

Using deep learning-based techniques could change visual crack inspection and identification while lowering the need for human observations [8]. Ground robots and unmanned aerial vehicles (UAVs) are gaining popularity as modern infrastructure inspection and monitoring systems, offering advantages over traditional inspection methods such as improved safety, efficiency, and accuracy [9]. These systems, equipped with sensors and cameras, can provide real-time data and imagery of infrastructure components, allowing for more accurate and efficient inspections [10]. These systems are anticipated to become increasingly crucial as technology advances in the infrastructural inspection and monitoring process [11].

2. Literature Review

Several methods for crack detection and assessment exist, including visual inspection, acoustic testing, and stress analysis. The most popular method of crack detection is visual inspection, which entails a civil engineer visually inspecting a structure for cracking or other symptoms of trouble. Acoustic testing involves using sound waves to detect cracks and other defects in a structure. This method is useful for detecting cracks that are not visible to the naked eye, but it can be affected by environmental noise and other factors that can interfere with sound wave transmission. Stress analysis involves measuring changes in the structure’s response to loads, such as changes in strain or displacement, to identify potential cracks. This method requires specialized equipment and can be expensive, but it is highly accurate. Despite these methods’ effectiveness, they have limitations. Traditional methods for crack detection often rely on human interpretation, which can be subjective and prone to errors. Moreover, these methods can be time-consuming, labor-intensive, and require significant expertise, which can be costly. Here is a literature survey related to the study:

By combining digital image processing methods [12,13] and machine and deep learning algorithms [14,15,16] with images, crack detection can be performed in numerous ways, as described in this section. Fu Tao et al. [17] conducted a thorough analysis of the body of research on crack identification using image processing methods. The authors go over the various crack detection techniques and algorithms, such as segmentation, edge detection, and thresholding. Additionally, they explore the difficulties and upcoming improvements to the discipline of fracture detection and offer a thoughtful evaluation of the advantages and disadvantages of each technique. The various methods for detecting road cracks, including techniques for image processing, techniques for machine learning, and deep learning methods, were described by J. Yang et al. [18]. The writers present a comparative review of the various techniques and go through the benefits and drawbacks of each methodology. In the area of detecting road cracks, they also identify the difficulties and potential directions. An assessment of the most recent advancements in deep CNN architectures was given by D. Bhatt et al. [19]. H. Yu et al. [20] go over many CNN architectures that are employed for image processing applications such as object recognition, segmentation, and classification. Additionally, they compare the various CNN architectures and emphasize their benefits and drawbacks. In their approach for extracting concrete fracture attributes using image-based approaches for bridge inspection, the authors suggest a technique that involves taking high-resolution pictures of concrete buildings, which are afterwards examined to extract various crack-related properties. The authors conducted experiments on actual concrete structures to gauge the efficacy of their proposed strategy.

In order to detect road cracks, L. Zhang et al. [21] developed a deep CNN method that can extract high-level characteristics from unprocessed input photos. In order to identify fractures, the authors suggest a brand-new architecture called Crack-Net that comprises numerous convolutional and pooling layers. Additionally, they compare their strategy with other cutting-edge techniques and show that it outperforms them. Kaseko et al. [22] investigated the performance of pre-trained convolutional neural networks (CNNs) on detecting cracks in building structures.

The authors compare several popular CNN architectures, such as VGG, ResNet, and Inception, with their own CNN-based method. They evaluate the performance of these methods using two benchmark datasets and show that their CNN-based method outperforms other methods. A technique for identifying crack deterioration in engineering structures using unmanned aerial vehicle images and a deep learning model that has already been trained was proposed by Huang et al. [23]. The CNN used by the authors is fine-tuned with their own data of UAV images after being pre-trained on the ImageNet dataset. Using a dataset of UAV photos of civil infrastructure, they test their method, and they demonstrate that it is highly accurate in identifying crack damage.

A deep CNN method for automatically detecting road cracks was presented by Rajadurai et al. [24]. The VGG16 architecture is used by the authors, who fine-tune it using a dataset of road photos with and without fractures. Additionally, they suggest a post-processing technique to get rid of false positives. They test their technique on a sizable collection of road photographs and demonstrate that it performs better than other cutting-edge techniques. Maguire, M. et al. [25] provided SDNET2018 annotated picture dataset to detect non-contact concrete cracks in the buildings, bridges and walls. The dataset includes more than 56,000 pictures of cracked and uncracked concrete surfaces that were taken using different imaging methods. Bhowmick et al. [26] studied concrete crack detection with handwriting script interferences using a faster region-based convolutional neural network. In order to detect cracks in asphalt pavement Tien Le et al. [27] presented DeepCrack, a deep hierarchical feature extraction architecture. A universal feature extraction network and a local feature extraction network make up the suggested design. The research demonstrates that DeepCrack beats cutting-edge techniques for crack segmentation.

S. Bhat et al. [28] designed a deep CNN. The research demonstrates that the suggested CNN model outperforms typical machine learning techniques and can accurately identify cracks in asphalt pavement. A computer vision-based method for detecting concrete cracks was proposed by A. Khan et al. [29], utilizing U-net deep convolution networks. The study demonstrates that the suggested technology may precisely and effectively find cracks in concrete surfaces. An enhanced I-UNet convolutional network was put up by J. Deng et al. [30] for the purpose of detecting road cracks using computer vision. The research demonstrates that the suggested strategy may accurately and effectively find road cracks.

A deep CNN-based transfer learning technique for crack identification in civil infrastructure was introduced by N. A. M. Yusof et al. [31]. The suggested strategy enhances the model’s efficacy by transferring knowledge from previously trained images to the target domain. The research demonstrates that the suggested technique beats cutting-edge techniques for detecting defects in civil infrastructure. For image categorization, Y. Liu et al. [32] presented a conventional transfer learning approach. The suggested method classifies the data using a quantum circuit after extracting features from it using a traditional neural network. The research demonstrates that the suggested method beats cutting-edge techniques for picture categorization. Non-destructive tests were proposed by Z. Liu et al. [33] to evaluate the load-carrying ability of cement anchor bolts using an artificial multilayer perceptron neural network. The study demonstrates that the suggested method can accurately forecast the load-carrying ability of concrete anchor bolts.

According to a review of the literature, deep learning and robotic vision applications for crack assessment and detection are gaining popularity. The study’s suggested method advances this area of inquiry by merging many sensing modalities to produce more thorough fracture detection and assessment. The suggested approach is poised to increase crack detection and evaluation accuracy and effectiveness across a range of applications and fields. Traditional approaches for crack detection and assessment have limitations, including subjectivity, limited accuracy, high cost, time-consumption, disruption to operations, and a lack of continuous monitoring. These limitations emphasize the need for automated and accurate crack detection methods that can provide reliable and continuous monitoring of structural health.

The emergence of deep learning and robotic vision technology has revolutionized the field of crack detection and assessment by significantly improving accuracy, speed, and efficiency. By training deep learning algorithms on large datasets of images and videos collected by robotic vision systems, these algorithms can detect cracks even in complex crack patterns or low contrast images. Robotic vision systems can access difficult-to-reach areas and provide a comprehensive view of the structure’s condition, reducing the need for manual inspections. Combining these technologies provides real-time and continuous monitoring of structural health, enabling early identification of cracks to prevent safety hazards and reduce costs.

Table 1. Literature survey for research conducted on crack detection.

Ref.	Aim of Study	DL Algorithm Used	Robotics Technique Used
[16]	Using digital picture processing driven by a UAV, find concrete cracks	N/A	UAV-powered digital image processing
[17]	To offer a rigorous evaluation and critique of image processing’s use of crack detection	N/A	N/A
[18]	To investigate methods for detecting road cracks	N/A	N/A
[19]	To review current deep CNN architectures	Deep convolutional neural networks	N/A
[20]	Employing image processing to get cracked concrete properties during bridge inspection	N/A	Image processing
[21]	Utilising a deep CNN to find road cracks	Deep convolutional neural network	N/A
[22]	To test how well several pre-trained CNN detect construction cracks	Pre-trained convolutional neural networks	N/A
[23]	Applying a trained DL model to UAV photos of civil infrastructure to find crack damage	Pre-trained deep learning model	UAV imaging
[24]	Deep CNN will be used to automatically detect road cracks.	Deep convolutional neural network	N/A
[25]	To provide a collection of annotated images for deep CNN to use in non-contact concrete fracture identification.	Deep Convolutional Neural Networks	N/A
[26]	In order to find concrete fissures where handwriting script interferences are present	Faster Region-Based Convolutional Neural Network	N/A
[27]	Finding asphalt pavement cracks	Deep Convolutional Neural Network	N/A
[28]	To provide an architecture for hierarchical feature learning for crack segmentation	N/A	N/A
[29]	Using computer vision-based methods, find concrete cracks	U-net Fully Convolutional Networks	N/A
[30]	To create a more effective I-UNet convolutional network for detecting road cracks	I-UNet Convolutional Networks	N/A
[31]	To create a deep CNN-based transfer learning technique for crack identification in civil infrastructure	Deep Convolutional Neural Network	N/A
[32]	Picture categorization using classical-quantum transfer learning	Classical-Quantum Transfer Learning	N/A
[33]	The use of artificial multilayer neural networks and non-destructive tests to evaluate the load-carrying ability of concrete anchor bolts	Artificial Multilayer Neural Network	N/A

discusses the literature survey for research done on crack detection. As shown in Table 1, many DL algorithms are being used for crack detection in the literature and there is more scope for using robotic techniques to provide automated and less human-interrupted solutions for the same.

Based on the literature survey provided, the research gaps identified are as follows:

• The literature suggests that conventional techniques employed for detecting cracks, such as visual inspection, are susceptible to errors and subjectivity due to human interpretation, thereby lacking automation. The demand for automated and impartial methods for detecting cracks is imperative.
• The efficacy of acoustic testing is restricted, as it is a technique that employs sound waves to identify cracks that are not discernible through visual inspection. Nonetheless, sound wave transmission may be influenced by environmental noise and other variables. There is a necessity for the development of more rigorous and dependable techniques for acoustic testing.
• Various studies have noted the absence of explicit mention of deep learning algorithms and robotics techniques. Specifically, the utilization of deep convolutional neural networks and image processing techniques is highlighted in the context of crack detection. The omission of details regarding the DL algorithms and robotics techniques employed in each study may have implications for the replicability and lucidity of the research.
• The current literature on crack detection presents limited coverage of the challenges that are inherent in this process. While some studies have briefly touched upon the difficulties and potential avenues for crack detection, a more comprehensive investigation is required to fully understand the complexities associated with this task. These complexities include variations in crack patterns, environmental factors, and the diverse range of structures that must be examined.

The resolution of these research deficiencies can make a valuable contribution towards advancing more precise, effective, and economical methods for detecting cracks, which can enhance the maintenance and safety of infrastructure.

3. Our Proposal: Road System

The proposed ROAD (Robotics-Assisted Onsite Data Collection System) systems shown in Figure 1 are a novel approach to construction inspection that leverages advanced robotics, machine learning, and BIM software to provide a comprehensive solution for construction site monitoring and data collection. The mobile robot platform is the primary data collection device, equipped with sensors, cameras, and measurement devices to collect data about the construction site. The robot is autonomous and capable of navigating around obstacles, making it ideal for use in complex and dynamic construction environments. This data is then sent to the object detection system on the server for processing, where machine learning algorithms analyze and classify objects and features within the construction site. This includes identifying defects, deviations from design specifications, and other issues that may impact the quality or safety of the construction project [34].

The object detection system provides real-time feedback to the control room, where construction managers and engineers can make informed decisions and take appropriate action as needed. The control room is equipped with BIM software, which provides a 3D model of the construction project, allowing managers and engineers to visualize the construction project in detail and identify potential issues or problems [35]. The control room can also be used to monitor the progress of the construction project, track materials and equipment, and manage project resources.

By combining advanced robotics, machine learning, and BIM software, the ROAD system can improve the quality and safety of construction projects while reducing costs and increasing efficiency [36]. The system can provide real-time monitoring and data collection, allowing for early identification of issues and timely corrective action. Because of its comprehensive perspective of the construction process and ability to promote the effective use of resources, the ROAD system has the potential to enhance project management and planning as well.

As depicted in Figure 1, the ROAD system proposed for crack detection and assessment combines deep learning and robotic vision technology to provide automated and accurate monitoring of structural health. The system includes several steps:

• Image and video capture: A robotic vision system captures images and videos of the structure from different angles and perspectives, including areas that are difficult to access.
• Data pre-processing: The captured images and videos are pre-processed to remove noise and enhance the contrast of crack features.
• Training deep learning algorithms: To reliably identify and categorize various types of cracks, a deep learning system is trained on a sizable collection of cracking images and videos. The SDNET2018 dataset, which contains images of concrete surfaces with varied degrees of fractures, has been used to fine-tune the CNN, InceptionResNetV2, Xception, DenseNet201, MobileNetV2, VGG16, and VGG19 models. For the specific objective of crack identification, transfer learning is employed to utilize the pre-trained parameters of the model and speed up the learning process.
• Crack detection: The trained deep learning algorithm is applied to the preprocessed images and videos to detect cracks and classify them according to their type and severity.
• Structural assessment: The detected cracks are analyzed to assess the structure’s health and identify any potential safety hazards.
• Reporting and maintenance: Engineers and repair teams are informed of the outcomes of the crack identification and evaluation so that they can make any repairs or maintenance tasks required to maintain the material’s safety and durability.

This study describes the creation of the ROAD system, a cutting-edge method for spotting road cracks using deep learning and robots. The suggested solution makes use of a robotic vision system’s capabilities to gather high-quality data about the road surface and incorporates deep learning methods for automatically identifying cracks. The device can function under a variety of circumstances, including various weather and lighting situations, and it can detect cracks on a variety of surfaces, including concrete and asphalt. The suggested approach might considerably increase the effectiveness and efficiency of crack identification on highways, resulting in more timely and effective maintenance interventions and a safer driving environment for all road users. The device has successfully detected cracks on a variety of road surfaces during testing. The effectiveness of various deep neural networks for picture classification and object recognition tasks is evaluated in this study. The suggested method has a number of benefits, including high accuracy and speed, real-time monitoring, ongoing assessment, decreased expenses, and minimal interference with routine activities [37].

Deep learning and robotic vision technologies can be used by engineers to identify cracks early, stop additional damage, and assure the durability and safety of structures. A useful resource for training, verifying, and benchmarking deep learning algorithms for concrete crack detection is the SDNET2018 [25]. The dataset encompasses a wide range of crack widths and types, ranging from thin cracks as small as 0.06 mm to wider ones as large as 25 mm, and contains over 56,000 annotated pictures of cracked and non-cracked concrete bridge decks, walls, and pavements.

The SDNET2018 dataset offers a complete set of training data for researchers to create and improve crack detection algorithms based on deep learning convolutional neural networks by taking into account six different classes to specifically classify decks, walls, and pavements with or without cracks. However, the camera quality used to capture pictures may produce some bias or limitations, which may impact the accuracy and generalizability of the models. The Tensor Flow and Keras frameworks were used to create the Python crack detection model known as the ROAD system. The model is trained and tested in the Google Colab environment and is based on the CNN, Xception, DenseNet201, InceptionResNetV2, MobileNetV2, and VGG16 and VGG19 architectures. The first algorithm describes how the model was trained and tested [38].

The proposed approach for crack detection and classification involves using deep learning models, specifically CNN, Xception, DenseNet201, InceptionResNetV2, MobileNetV2, VGG16, EfficientNetV2, and VGG19, trained on large datasets of images and videos of structures with cracks, as shown in Figure 2.

Robotic vision technology, which consists of cameras mounted on robotic arms that can capture images and videos from different angles and perspectives, is used for data collection. This technology allows for real-time monitoring of structural health and can reduce variability in the data, improving the accuracy of crack detection and assessment. The integration of deep learning and robotics involves several steps, including data collection using robotic vision systems, training deep learning models on the collected data, and deploying the models for automated crack detection and assessment.

The proposed model for crack detection and classification goes through the following steps:

Firstly, robotic vision systems are used to collect images and videos of structures with and without cracks using the SDNET2018 dataset.

Secondly, the collected data is pre-processed, including resizing, normalization, and augmentation, to ensure that the deep learning models can learn from the data effectively.

Thirdly, the pre-processed data is used to train deep learning models (CNN, Xception, DenseNet201, InceptionResNetV2, MobileNetV2, VGG16, and VGG19) to detect and classify cracks accurately.

Fourthly, the trained deep learning models are integrated with robotic vision systems to enable real-time crack detection and classification.

In the fifth step, the detected cracks are analyzed to assess the structure’s health and identify any potential safety hazards.

Finally, the results of crack detection and assessment are reported to engineers and stakeholders, who can visualize the data using dashboards, graphs, and other visualization tools. The integration of deep learning and robotics can provide an automated and efficient solution for crack detection and assessment, reducing the need for manual inspection and improving the safety and lifespan of structures [39,40].

4. Experimental Results

The proposed approach for crack detection and assessment involves a robotic vision system consisting of a camera mounted on a robotic arm that captures images and videos of structures with and without cracks. For real-time crack detection and classification, deep learning models are connected with the robotic vision system once the gathered data has been pre-processed and utilized to train them. Using visualization tools, the severity of discovered cracks is evaluated, and the findings are communicated to engineers and stakeholders. Using a different testing dataset, numerous assessment measures, cross-validation techniques, and comparisons with conventional methods, the performance of the deep learning models is assessed. To guarantee the models’ dependability for automated crack detection and evaluation, their performance is confirmed in real-world circumstances.

The performance analysis of a robotic vision system can be a complex task, as it involves evaluating various aspects of the system’s performance, such as accuracy, speed, robustness, and reliability. Here are some steps that can help in conducting a performance analysis of a robotic vision system: The first step is to clearly define the task that the robotic vision system is expected to perform. For example, is it object detection, recognition, or tracking? This will help in determining the appropriate evaluation metrics.

Once the task is defined, appropriate evaluation metrics should be selected. For instance, for object detection, metrics such as accuracy, MSE (Mean Squared Error) [41], precision, and recall can be used. For object tracking, metrics such as tracking accuracy, tracking speed, and smoothness of the trajectory can be used. The next step is to collect data to evaluate the system’s performance. The data should be representative of the task and cover various scenarios that the system is expected to handle. The system should be trained on the collected data and tested on a separate set of data. The testing data should be different from the training data to ensure that the system can generalize to new data. Once the system has been tested, the results should be analyzed to evaluate the system’s performance.

The results can be compared to the evaluation metrics selected in step 2 to determine whether the system meets the desired performance criteria. If the system does not meet the desired performance criteria, it may be necessary to fine-tune it. This can involve adjusting parameters, retraining the system on additional data, or improving the algorithms used. The performance analysis process may need to be repeated several times until the system meets the desired performance criteria [42]. The performance analysis of a robotic vision system involves defining the task, selecting appropriate evaluation metrics, collecting data, training and testing the system, analyzing the results, fine-tuning the system, and repeating the process until the desired performance criteria are met.

The results of the integrated approach for crack detection and assessment can vary depending on the specific approach and methodology used. The integrated approach for crack detection and assessment has the potential to improve the accuracy, efficiency, and safety of crack detection and assessment in various applications, such as civil engineering, aerospace, and manufacturing.

Figure 3 presents the acquired results for the performance parameters such as accuracy, MSE, precision, and recall of the deep neuronal network formed by Xception, DenseNet201, InceptionResNetV2, MobileNetV2, VGG16, and VGG19, and

Table 2. Performance table for CNN, Xception, DenseNet201, InceptionResNetV2, MobileNetV2, VGG16, and VGG19 for crack detection models.

Accuracy
Epoch’s	CNN	DenseNet201	InceptionResNetV2	MobileNetV2	VGG16	VGG19	Xception
1	66.5775	52.6344	80.6098	38.7358	60.1676	38.7358	74.5921
2	66.5597	61.8615	74.5297	39.0479	72.8626	40.6615	85.6735
3	60.0428	50.9940	86.0301	30.8995	75.3499	52.1797	81.1447
4	58.5183	57.8675	86.1460	41.9809	64.8302	65.4007	82.2323
5	70.1168	64.6519	74.9131	60.3281	81.8044	75.9651	90.2469
6	65.4542	70.9815	78.8624	51.5468	81.9916	74.1999	86.9930
7	73.2549	76.2860	88.9097	45.6004	80.5830	78.9873	89.8814
8	78.4702	56.9582	88.0806	59.3474	73.4867	78.2027	88.2767
9	77.5876	73.9681	84.9782	45.2082	82.7227	78.3008	89.7388
10	77.8996	57.1989	87.6438	40.7774	80.6187	81.4300	86.9751
11	73.2549	70.6606	89.3911	50.9227	74.5832	82.2502	80.5296
12	72.9785	53.1069	84.7820	42.6050	82.6959	80.6098	86.7790
13	77.1775	66.6845	49.2199	47.5974	81.7331	78.0333	86.9127
14	76.2414	44.7178	85.9321	64.7143	82.8029	80.1462	80.6187
15	75.6887	41.5976	78.3097	60.9432	80.1997	82.1610	88.3124
16	75.3410	65.2670	85.3704	70.5982	79.0051	70.9726	90.5322
17	74.2355	42.8813	85.0049	58.4470	81.2160	79.2725	90.1489
18	72.6843	72.9072	86.9038	49.1932	80.4315	77.0884	87.9112
19	73.4510	64.1972	84.5502	80.9575	81.3141	81.6618	88.3748
20	73.6382	63.4929	84.7107	68.5388	72.1227	81.0734	89.9706
Loss
Epoch’s	CNN	DenseNet201	InceptionResNetV2	MobileNetV2	VGG16	VGG19	Xception
1	0.0775	0.1115	0.0478	0.2042	0.0845	0.1224	0.0661
2	0.0801	0.0924	0.0731	0.1988	0.0769	0.1155	0.0381
3	0.0906	0.1491	0.0365	0.2168	0.0639	0.1076	0.0490
4	0.0906	0.1094	0.0350	0.1820	0.0781	0.0796	0.0463
5	0.0704	0.0946	0.0669	0.1249	0.0478	0.0607	0.0260
6	0.0859	0.0735	0.0559	0.1516	0.0471	0.0631	0.0363
7	0.0665	0.0654	0.0301	0.1774	0.0498	0.0534	0.0276
8	0.0573	0.1191	0.0322	0.1328	0.0616	0.0563	0.0335
9	0.0587	0.0716	0.0396	0.1781	0.0451	0.0549	0.0284
10	0.0565	0.1223	0.0336	0.1969	0.0494	0.0479	0.0344
11	0.0674	0.0796	0.0290	0.1604	0.0637	0.0471	0.0555
12	0.0684	0.1501	0.0407	0.1907	0.0458	0.0492	0.0394
13	0.0602	0.0997	0.1286	0.1734	0.0472	0.0542	0.0390
14	0.0636	0.1753	0.0376	0.1154	0.0453	0.0513	0.0549
15	0.0662	0.1825	0.0589	0.1288	0.0524	0.0471	0.0349
16	0.0675	0.1042	0.0383	0.0939	0.0533	0.0689	0.0278
17	0.0712	0.1849	0.0397	0.1377	0.0516	0.0526	0.0286
18	0.0755	0.0770	0.0356	0.1659	0.0538	0.0554	0.0350
19	0.0731	0.1039	0.0412	0.0597	0.0517	0.0480	0.0327
20	0.0744	0.0985	0.0420	0.0963	0.0802	0.0486	0.0301
Precision
Epoch’s	CNN	DenseNet201	InceptionResNetV2	MobileNetV2	VGG16	VGG19	Xception
1	72.7496	57.4463	83.0615	38.7358	87.1830	0.0000	75.9108
2	78.7134	66.9144	75.6462	39.1033	82.4393	87.4785	86.5855
3	63.0467	51.0730	86.7354	30.9808	81.1182	63.8872	82.3788
4	66.6667	59.2572	86.8192	42.0516	72.9645	86.1176	82.7596
5	73.2454	65.9912	76.0318	60.4254	84.6228	79.7475	90.3765
6	69.1520	72.6900	80.3527	51.7756	83.7563	78.8481	87.3239
7	77.7636	76.9001	89.1256	45.6266	82.5322	82.0426	90.0572
8	82.6985	57.0150	88.3435	59.3633	78.0363	80.2758	88.4615
9	81.5405	74.5844	85.6768	45.2969	84.7576	79.8565	89.8285
10	80.9801	57.8230	88.0600	40.7774	82.6499	82.9683	87.1545
11	76.1155	71.3818	89.6320	50.9272	78.2683	84.4735	80.9190
12	74.9858	53.1459	85.1393	42.6202	84.0336	82.0412	86.9830
13	79.0208	66.9088	50.3478	47.6097	84.5687	80.9248	87.0020
14	77.8453	44.8356	86.3690	64.7195	84.4664	83.8642	81.2692
15	76.9658	41.7316	79.4142	60.9506	83.6102	84.1112	88.3376
16	76.2179	65.5284	85.8950	70.7271	82.2721	74.0648	90.6016
17	75.2725	42.8890	85.5954	58.4470	85.1226	81.3498	90.1767
18	73.4575	73.4121	87.2344	49.2407	83.4361	82.3676	87.9561
19	74.5433	64.6942	85.0979	80.9774	84.3397	83.8065	88.6262
20	74.5493	64.1787	85.1169	68.7058	86.3548	82.9129	90.0268
Recall
Epoch’s	CNN	DenseNet201	InceptionResNetV2	MobileNetV2	VGG16	VGG19	Xception
1	60.8095	47.4904	78.0779	38.7358	33.7167	0.0000	73.7452
2	49.4161	60.9967	73.0498	39.0300	42.0612	13.5776	84.9336
3	56.9314	50.9227	85.4596	30.7212	65.8376	37.7730	80.2710
4	45.1814	56.6105	85.5576	41.8829	62.0754	36.1683	81.8668
5	66.8004	64.4557	74.2355	60.2746	78.0066	71.5075	90.0865
6	59.0354	69.5017	77.5876	51.3417	80.0303	68.5923	86.7790
7	65.1600	75.6798	88.7760	45.5737	78.2206	75.8402	89.7923
8	71.0350	56.8066	87.7686	59.3474	69.3679	75.7600	88.1697
9	71.1598	73.2014	84.3630	45.1636	80.9040	77.4004	89.6764
10	74.3960	55.9775	87.4476	40.7774	77.8015	80.0392	86.8592
11	70.2594	69.9563	89.2485	50.9227	69.3858	79.5935	80.2264
12	70.6339	53.0891	84.4789	42.6050	81.1269	79.6202	86.6185
13	75.2518	66.5151	48.3908	47.5885	78.3186	74.7348	86.8236
14	74.2088	44.5039	85.5220	64.6964	80.8594	74.8774	80.2621
15	74.1731	41.5084	77.5876	60.9343	74.4941	79.8520	88.2589
16	74.1999	65.1779	84.7464	70.5001	75.2162	68.3070	90.4966
17	73.2727	42.8546	84.3363	58.4470	74.5743	77.6946	90.1043
18	71.7482	72.5417	86.3867	49.1397	74.8150	73.5045	87.8934
19	72.3901	63.6445	84.0510	80.9486	74.4673	78.1582	88.2232
20	72.9874	62.6282	84.3809	68.3873	35.7136	78.8179	89.8903

presents values for the performance parameters for CNN, Xception, DenseNet201, InceptionResNetV2, MobileNetV2, VGG16, and VGG19 for crack detection models.

Results were analyzed and validated through five repeated iterations, and the average of those five iterations was utilized to present them in the results section. Throughout the evaluation of different approaches in the research, the Xception model constantly exhibited superior accuracy, with a range of 74.59% to 90.53% across epochs. The VGG16 and VGG19 models showed significant performance, attaining accuracies ranging from 60.17% to 82.80% and 38.74% to 82.16%, respectively. The models DenseNet201, InceptionResNetV2, MobileNetV2, and CNN demonstrated differing degrees of accuracy, which were generally inferior to those of Xception, VGG16, and VGG19. As the highest accuracy acquired is 90 percent, the percentage of wrong or error predictions could be considered 10 percent.

The Xception model demonstrated superior optimization and convergence during training, as evidenced by its consistently low loss values. The models VGG16, VGG19, and MobileNetV2 showed comparatively reduced loss values. In contrast, DenseNet201, InceptionResNetV2, and CNN exhibited high loss values relative to the remaining models.

The precision and recall metrics were also evaluated. The Xception model consistently attained high precision and recall metrics across the epochs. The models VGG16, VGG19, and MobileNetV2 demonstrated favorable precision and recall metrics, albeit lower than those of Xception. The models DenseNet201, InceptionResNetV2, and CNN exhibited various precision and recall measures, with certain epochs demonstrating favorable outcomes while others displayed adverse outcomes. The findings underscore the exceptional efficacy of the Xception architecture concerning accuracy, loss, precision, and recall, rendering it the most dependable option for detecting cracks in the ROAD system.

The ROAD system’s development has revealed a lot of promise for enhancing the precision and effectiveness of crack identification on roadways utilizing robots and deep learning. With a validation accuracy of over 90% and a low mean square error of 0.03, the study’s findings show that the Xception deep neural network performs better than other algorithms in terms of accuracy and predictive capacity. Testing of the suggested system on various types of road surfaces revealed that it is highly accurate in spotting cracks. The technology has a 90% accuracy rate for crack detection down to 1 mm. The system has also been tested in a variety of weather and illumination scenarios, including at night, and the findings have been consistent. The technology is highly efficient, scanning and analyzing 1 km of road surface in less than an hour, which is much quicker than conventional techniques. The technology can also accurately identify a variety of crack types, including block, longitudinal, and transverse cracks [43].

5. Discussion

The system proposal for, Robotics-Assisted Onsite Data Collection and Deep Learning-Enabled Robotic Vision System for the Identification of Cracks on Diverse Surfaces, may encounter various challenges and limitations, as mentioned below:

• Real-time processing: Real-time processing is critical in on-site crack detection, particularly when prompt decision-making or action is necessary. To ensure prompt results, it is imperative that the system efficiently processes the collected data and conducts the crack detection on time. Achieving real-time performance necessitates the assurance of efficient computational resources and optimized algorithms [41].
• Environmental factors: The robotic vision system’s image quality can be impacted by a range of environmental factors, including but not limited to lighting conditions, shadows, reflections, and weather conditions like rain and fog. Various factors may impact cracks’ apparent presence and clarity, resulting in erroneous outcomes in detecting cracks, either false positives or false negatives. The system must be devised to consider and address environmental factors’ influence to guarantee precise detection [42].
• Surface variations and textures: Various surfaces, including but not limited to concrete, asphalt, and different building materials, may display differences in texture, hue, and design. The presence of diverse surface characteristics and the inherent complexity and variation in crack patterns across various surfaces can present challenges for crack detection methods, which may require adaptation to handle these variations effectively [43].
• Generalization to unseen data: The system’s deep learning models depend on the training data to acquire knowledge of patterns and characteristics linked to cracks, thereby enabling generalization to unseen data. Nevertheless, the efficacy of these models on unobserved data or surfaces that exhibit substantial dissimilarities from the training data may need to be clarified. The system is recommended to undergo evaluation and validation procedures using a range of datasets and be tested on multiple surfaces to determine its generalizability and reliability.
• False positives and negatives: The challenge of crack detection lies in achieving a balance between minimizing false positives, which refer to the identification of non-crack areas as cracks, and false negatives, which refer to the failure to detect actual cracks. In certain instances, deep learning models may generate erroneous identifications owing to factors such as noise, surface irregularities, or intricate patterns resembling cracks. The occurrence of inconspicuous or diminutive fissures may lead to erroneous adverse outcomes. The implementation of continuous model refinement, optimization, and training, along with the utilization of diverse datasets, can aid in the alleviation of these challenges.
• Hardware limitations: The robotic system’s hardware components, including sensors and cameras, must satisfy criteria to capture top-notch images and data, presenting hardware limitations. The efficacy of crack detection can be influenced by various factors, such as the camera’s resolution, field of view, and image stabilization features, along with the precision and dependability of other sensors. It is imperative to guarantee the appropriateness and dependability of the hardware constituents to optimize the system’s overall functionality.
• Scalability and adaptability: The system under consideration must possess the ability to scale and adapt to diverse scenarios and applications. The system must effectively manage various crack types, from minor fissures to more substantial structural impairments. Additionally, the system should be capable of seamless deployment and compatibility with various robotic platforms to cater to a wide range of inspection environments and structures [43].

To overcome these obstacles and constraints, it is necessary to employ a comprehensive approach that encompasses the development of solid algorithms, the acquisition of ample training data, and implementation of efficient feature extraction techniques, the ongoing optimization of the system, and the rigorous testing and validation of procedures. By resolving these issues, the proposed system has the potential to augment its crack detection capabilities and thereby make a valuable contribution to the optimization of infrastructure maintenance and inspection protocols. The results obtained from the study demonstrate that the deep neural network formed by Xception performed exceptionally well, with an accuracy of over 90% during the validation process. The validation accuracy is a measure of how well the model can make accurate predictions on data that it has not been trained on. In addition, the mean square error, which measures the difference between the predicted and actual values, was found to be very low, with a value of 0.03. On the other hand, when compared to Xception, several other deep neuronal networks, including DenseNet201, InceptionResNetV2, MobileNetV2, VGG16, and VGG19, resulted in inferior accuracy with higher losses. These results indicate that Xception outperforms these other algorithms in terms of accuracy and predictive power.

Furthermore, the accuracy and recall scores obtained by Xception were also found to be very high, reaching nearly 90%. Accuracy refers to the proportion of correct predictions made by the model, while recall measures the ability of the model to identify all relevant instances within a dataset. The high accuracy and recall scores obtained by Xception suggest that the model is capable of accurately identifying and classifying different data points. These results suggest that the Xception model is a highly effective deep neural network for a range of applications. Its high accuracy, low mean square error, and superior performance when compared to other algorithms make it a valuable tool for a variety of machine learning tasks, including image classification and object recognition. The accuracy of the proposed approach can be evaluated using metrics such as accuracy, MSE, precision, and recall. The proposed approach is faster, less expensive, simpler, more robust, more versatile, and more scalable than existing methods.

The integrated approach for crack detection and assessment has several advantages, such as increased accuracy, reduced false positives, comprehensive crack assessment, faster inspection, improved safety, and cost savings. However, it also has limitations, such as the need for specialized equipment and expertise, the potential for false negatives, and the cost of implementing and maintaining the integrated approach. The integrated approach for crack detection and assessment has advantages such as increased accuracy, reduced false positives, comprehensive crack assessment, faster inspection, improved safety, and cost savings.

However, it also has limitations such as a high initial cost, complexity, technical limitations, maintenance requirements, and the need for high-quality data. The integrated approach for crack detection and assessment has several advantages, such as increased accuracy, reduced false positives, and improved safety, but also has limitations, such as a high initial cost, complexity, and technical limitations. The advantages and limitations should be carefully considered before implementing the integrated approach. The proposed approach for crack detection and assessment can be applied in various fields and applications, such as civil engineering, aerospace, manufacturing, automotive, energy, and medical. It has the potential to provide accurate and efficient detection and assessment of cracks and potential failures in different types of infrastructure, products, and devices.

In order to achieve system scalability, it is imperative to ensure that the design incorporates seamless integration with pre-existing infrastructure management systems. The proposed integration entails the assimilation of the crack detection system within the comprehensive framework utilized for overseeing and upholding road networks. The proposed system can derive advantages from the pre-existing infrastructure management systems, including established workflows, data management processes, and decision-making protocols. Drawing from the discourse and evaluation of the proposed framework for detecting and evaluating cracks in road networks, the subsequent recommendations can be posited to augment the study:

• In order to enhance the resilience and versatility of deep learning models, it is advisable to augment the dataset utilized for both training and testing purposes.
• The proposed system places emphasis on visual data obtained through cameras. However, the inclusion of other sensor data, such as LiDAR or infrared imaging, can offer supplementary information to enhance the precision of crack detection and evaluation.
• Incorporating real-time anomaly detection algorithms can be advantageous in conjunction with crack detection.
• In order to guarantee the effective execution and acceptance of the suggested system, it is imperative to engage in partnerships with infrastructure management organizations, road authorities, and industry stakeholders.
• In order to ascertain the efficacy and dependability of the suggested system in practical scenarios, it is advisable to carry out comprehensive field experiments on diverse road networks.
• Given that the proposed system entails the collection and processing of visual data, it is imperative to address any potential privacy and security concerns.
• Perform an exhaustive evaluation of the costs and benefits to determine the financial feasibility of expanding the proposed system to a broader scope.

6. Conclusions

The development of the ROAD (Robotics-Assisted Onsite Data Collection) system has shown significant potential for improving the accuracy and efficiency of crack detection on roads using robotics and deep learning. The results of the study indicate that the Xception deep neural network outperforms other algorithms in terms of accuracy and predictive power, with a validation accuracy of over 90% and a low mean square error of 0.03. The proposed approach offers advantages such as increased accuracy, reduced false positives, comprehensive crack assessment, faster inspection, improved safety, and cost savings. The proposed approach can be applied in various fields and applications, providing accurate and efficient detection and assessment of cracks and potential failures in different types of infrastructure, products, and devices. The development of the ROAD system represents a promising solution to the challenges associated with traditional methods of crack detection on roads and highlights the potential of robotics and deep learning in improving road infrastructure maintenance and safety. The study also acknowledged the limitations of the proposed approach, such as its high initial cost, complexity, and technical limitations. Finally, the study identified potential applications of the proposed approach in various fields, such as civil engineering, aerospace, manufacturing, automotive, energy, and medical industries. Overall, the study presents a novel approach for crack detection and assessment that has the potential to improve the accuracy and efficiency of crack detection and assessment in various fields and applications.

7. Future Directions

In this paper, we have worked on crack detection; in the future, the work can be extended to crack growth and its severity. Additionally, future research in deep learning and robotic vision for crack detection and assessment could focus on developing more robust deep learning models, integrating additional sensing modalities, achieving real-time detection and assessment, automating the entire process, extending the research to other materials, and developing new techniques for crack assessment. Furthermore, optimization algorithms can be employed in crack detection in buildings to enhance the accuracy and efficiency of the process [44,45,46,47,48,49]. Various advanced optimization algorithms can be employed for image processing, crack segmentation, structural health monitoring, crack growth prediction, etc. Incorporating optimization algorithms into crack detection in buildings can improve the overall accuracy, efficiency, and reliability of the process, leading to timely identification and mitigation of structural issues.