A Comprehensive Analysis of the Integration of Deep Learning Models in Concrete Research from a Structural Health Perspective

Abstract

Concrete stands as the most widely used construction material globally due to its versatility, encompassing applications ranging from pavement, multifloor structures, and bridges to dams. However, these concrete structures endure structural stress and require close monitoring to prevent accidents and ensure sustainability throughout their complete life cycle. In recent years, artificial intelligence (AI) and computer vision (CV) have demonstrated considerable potential in diverse applications within construction engineering, including structural health monitoring (SHM) and inspection processes such as crack and damage detection, as well as rebar exposure. While it is undeniable that CV and deep learning models are transforming the construction industry by offering robust solutions for complex scenarios, there remain numerous challenges pertinent to their applications that require attention. This paper aims to systematically and critically review the literature of the past decade on the application of deep learning models in the construction industry for SHM purposes in concrete structures. The review delves into proposed methodologies and technologies while identifying opportunities and challenges associated with these applications in practice. Additionally, the paper provides insights to bridge the gap between theory and application.

1. Introduction

Concrete is the most important and demanding construction material [1], and concrete structures have been influencing the construction industry for decades [2]. However, an ever-growing number of concrete structures worldwide are entering the aging phase [3]. Due to various factors, such as weather and environmental conditions, chemical reactions, and external and internal stresses, concrete structures are often subject to defects such as cracks, efflorescence, spalling, bar exposure, etc., and fail to meet the expected life cycle, aging earlier than expected [2].

The idea of structural health monitoring (SHM) first emerged in the early 2000s. Although initially, the sole focus of SHM was to monitor concrete bridges, in present times, it is defined as the method of continuously evaluating and assessing the condition and performance of any concrete structures, such as buildings, bridges, dams, pipelines, and other infrastructure, throughout their operational lifespan [4,5,6]. The objective of SHM is to detect any damage, deterioration, or changes in structural properties that could potentially compromise the safety, functionality, or longevity of the concrete structure and is crucial in maintaining structures in optimal condition [2]. The traditional methods for the SHM process primarily involve manual inspection, which is heavily dependent on the expertise of the inspector. However, these methods present various challenges, including time-consuming operation, varying subjectivity, or difficulties in inspecting components at elevated heights in tunnels/road pavement in busy traffic conditions [2,7]. Therefore, there is a pressing need for an innovative and precise inspection approach to effectively monitor the health condition of structures that can overcome the mentioned limitations.

The advancement of artificial intelligence (AI) and computer vision (CV) has enabled their various applications in the construction industry, ranging from automatic surface crack detection to tunnel lining defect detection [8,9]. In recent years, there has been a notable progression in the application of AI methodologies, particularly machine learning and deep learning techniques, for activities such as data diagnostics, data interpretation, and feature extraction in the realm of SHM for infrastructure systems. Recent advances in deep learning technologies have allowed for state-of-the-art performance in visual recognition problems such as image classification, object detection, and object segmentation. Thus, such technologies hold significant promise, with an extensive array of practical applications in the SHM area [10].

Boccagna et al. [10] stated that the primary challenges in the application of deep-learning-based technologies for automated SHM processes pertain to the establishment of a resilient framework that encompasses all the intervening stages, from data collection to result generation and analysis. Therefore, the objective of this study is to conduct a comparative analysis of existing and proposed deep-learning-based models and convolutional neural network (CNN) techniques in the field of SHM. This comparison considers various factors, including the type of concrete structure, the type of input data, the size of the data; the type of damage, and the accuracy achieved by each proposed model, as well as their limitations in certain applications and suggestions for future research. The aim is for this study to serve as a foundational reference for the selection of deep-learning-based approaches in future SHM problem-solving scenarios. Particularly, it can guide researchers and practitioners in choosing appropriate deep learning methodologies when encountering situations similar to those reported in prior studies and can assist in identifying systems that have demonstrated notable performance in structural defect detection in those cases.

To accomplish these goals, only studies published after 2017 were meticulously arranged to incorporate the most up-to-date developments and cutting-edge technologies. The selection process adhered to specific criteria established by prior review articles in the field of construction engineering research that employed CV and deep learning techniques [11]:

• The chosen studies were centered on deep-learning- and CV-based applications, irrespective of their outcomes, including aspects like visualization or quantification;
• The studies were focused on the SHM domain, spanning various construction stages and varying physical and environmental conditions.

Studies utilizing alternative methodologies like machine learning techniques such as artificial neural networks (ANNs), genetic algorithms (GAs), or support vector machines (SVMs) [12], as well as those emphasizing sensor-based approaches [13], were not included in the selection process. Figure 1 shows an outline of the research methodology. This study focuses solely on SHM application to concrete structures; hence, steel structures and other types are not considered. The chosen studies underwent a systematic analysis based on their methodologies and technologies, as well as their respective applications. Subsequently, a more in-depth examination identified those studies that warranted further exploration, guided by their methodological frameworks and specific areas of application.

2. Overview of Artificial Intelligence and Deep Learning

In 1955, one of the pioneers of AI, John McCarthy, defined AI as a means to develop machines that behave intelligently [14]. In present times, artificial intelligence is characterized as the “exploration and creation of intelligent agents” [1]. AI empowers computers to achieve human-like capabilities such as perception, knowledge representation, reasoning, problem solving, and planning. This enables them to address intricate and ambiguous problems in a purposeful, intelligent, and adaptable manner.

Investment in AI is experiencing swift expansion, with branches like machine learning and deep learning notably playing a significant role [15]. According to a study by Purdy and Daugherty [16], AI is in the process of reshaping all aspects of society, and this transformation might increase employee productivity by 40% and can increase annual economic growth rate by two times by the year 2035.

Although AI still lacks appropriate applications in the construction industry compared to other sectors [15], researchers have been attempting to implement different AI technologies in different areas of construction, such as cost prediction in the real estate business [17], SHM at construction sites [18], and supply chain management [19]. Chu et al. [20] carried out studies to investigate the applications of robotics in the construction industry. Chen et al. [21] proposed a fuzzy cognitive map model that can assess risk casualties and predict risk for construction projects. Zaira and Hadikusumo [22] adopted a knowledge-based AI approach and conducted structural equation modeling (SEM) to identify the factors with the most significant impact in terms of improving workers’ safety behavior. Artificial neural networks, and other machine learning approaches such as random forest, SVM, etc., are also popular AI techniques in the construction industry [23].

CV-based AI applications were initially constrained in the construction domain due to the complexity of data acquisition process and the requirement for computers with high levels of computing power. Nevertheless, as the prevalence of such devices and high-performance computing resources has grown, CV technologies have find applications in various construction domains, such as labor and asset monitoring, progress assessment, quality control and assurance, etc. [24]. Advancements in CNN architectures are currently revolutionizing the CV field. Convolutional neural networks (CNNs), which represent an essential part of deep learning techniques, are widely renowned for their excellence in top-tier CV projects.

History and Development of Deep Learning

Convolutional neural networks (CNNs) are advanced feedforward neural networks. A feedforward neural network operates by transmitting information from the input layer to the output layer in a unidirectional manner, without forming cyclic connections among its neurons [25,26]. When an image is introduced as input to the network, it traverses a sequence of convolutional and pooling layers before reaching the final stage of fully connected layers, usually concluding with a SoftMax layer, which yields the output [25] (Figure 1).

The concept of convolutional neural networks (CNNs) first came to light in the late 1990s, thanks to the work of LeCun et al. [27]. As noted by Sultana et al. [28], the success of LeNet-5 sparked researchers’ interest in exploring the vast potential of CNNs, particularly in tasks that involve pixel-level operations like detection and segmentation. Object classification by CNNs has found applications in categorizing various elements at construction sites, including workers, risk zones, and machinery, as well as safety measures or accident risk [29,30]. These classifications are typically conducted using images collected from sources like CCTV footage, unmanned aerial vehicles (UAVs), mobile robots, etc. [30,31,32]. Classifying workers with or without personal protective equipment is another popular application of CNNs in the construction engineering area. Such classification is achieved using histogram of oriented gradients (HOG) features extracted from hardhats, which efficiently describe detailed shape information [33].

CNNs excel over traditional machine learning techniques due to their adaptive nature, learning feature representations through end-to-end training, thereby eliminating the necessity for hand-engineered features and expert knowledge [34]. Additionally, deep learning models can fine tune themselves, making deep learning algorithms and CNNs popular in various sectors of computer vision, including image segmentation [35,36].

Image segmentation, a fundamental technique in computer vision, involves the partitioning a digital image into distinct sections based on various pixel characteristics [37]. Given the critical role of spatial information in semantically segmenting different regions of an image, image segmentation is typically considered a detail-level or pixel-level vision task, in contrast to tasks like classification and object recognition [28]. The primary objective of image segmentation is to extract specific information for further analysis. This process involves separating specific regions in an image in such a way that all pixels within each area share common attributes such as color, intensity, texture, etc. [38,39]. Chi and Caldas [40] were among the first to apply object segmentation in identifying equipment and workers at construction sites using a background subtraction algorithm. Recently, segmentation techniques were applied to the detection of workers around a fall pit in an attempt to reduce fall-induced hazards among workers, as well as to the detection of risk zones such as elevated edges or roofs [41,42].

In practice, there are three primary types of segmentation, as depicted in Figure 2:

• Semantic segmentation: This involves labeling image pixels with object categories [43];
• Instance segmentation: Here, the goal is to separate and identify all instances of objects within an image individually and independently [44,45];
• Panoptic segmentation (PS): Panoptic segmentation, as defined by Kirillov et al. [46], combines both semantic and instance segmentation, offering a comprehensive view of the scene by identifying object categories and individual instances.

The applications of convolutional neural networks (CNNs) in image segmentation are incredibly diverse. Farabet et al. [47] introduced a multiscale CNN designed for scene-labeling tasks. They trained and tested their model on three distinct datasets: Sift flow [48], the Barcelona dataset [49], and the Stanford background dataset [50]. Girshick et al. [51] proposed region-based CNN (R-CNN) in the PASCAL VOC semantic segmentation challenge, which was part of the Pascal Visual Object Classes Challenge in 2007. This approach employs region proposals in a CNN to localize and segment individual objects.

The concept of a fully convolutional network (FCN) was introduced by Long et al. [52] to address semantic segmentation tasks. For the generation of region proposals, Faster R-CNN utilizes a CNN-based region proposal network (RPN) that employs bounding boxes—a technique shared with instance segmentation models. Mask R-CNN, introduced by He et al. [53], closely resembles Faster R-CNN but includes a binary mask prediction branch to facilitate instance segmentation in addition to object detection. Huang et al. [54] presented Mask Scoring R-CNN, which enhances Mask R-CNN by incorporating a network block to improve the qualitative aspect of predicted masks. In later development, Kirillov et al. [46] introduced a cutting-edge instance segmentation model that utilizes point-based rendering within the framework of Mask R-CNN.

The application of such instance segmentation models has promising implications in construction science research; for example, Teizer and Vela [55] proposed a segmentation-based tracking system for workers at construction sites. Wang et al. [56] employed a two-level approach that combines Faster R-CNN and Mask R-CNN to identify the morphological aspects of damage. This includes recognizing features such as damage topology, area, and ratio.

3. SHM System Based on Deep Learning Models

Researchers in the construction engineering field have recognized the immense potential and innovative technological strides resulting from the utilization of deep learning methods [57,58]. Consequently, numerous initiatives have been undertaken to apply deep learning techniques to structural health monitoring (SHM) of concrete infrastructure [59]. In this section, we delve into deep-learning-based research in the SHM domain, with a specific focus on two facets: (1) damage identification and (2) concrete condition assessment.

3.1. Damage Identification

At the heart of any SHM system lies its capacity to conduct damage identification. Damage refers to alterations in a material’s physical characteristics caused by ongoing deterioration or a singular event affecting a structure. Such changes have the potential to compromise the performance and structural integrity of the concrete [60]. One limitation of applying deep learning techniques is that they require a large and annotated database, which is not always available, especially in the concrete research area. However, the application of transfer learning can eliminate this problem, allowing an existing deep learning model to be retrained with smaller amounts of new data [61]; accordingly, an increasing number of applications of deep learning models in concrete research and SHM have been reported.

For example, Gopalakrishnan et al. [62] applied transfer learning to a pretrained VGG-16 model for crack detection in hot-mix asphalt and Portland cement concrete-based pavement. Kolar et al. [63] also applied transfer learning to VGG-16 model to detect safety guardrails to promote on-site safety inspection.

Real-world scenarios often limit the applications of deep learning models at actual construction sites because of lighting and shadow issues. Cha et al. [64] trained a CNN with a large database of 40k images under various lighting conditions and achieved 98% accuracy in detecting concrete cracks. The authors later compared the performance of the proposed CNN using Canny and Sobel edge detection methods. Tong et al. [65] integrated three CNNs to perform recognition, localization, and feature extraction tasks, enabling the 3D reconstruction of hidden pavement cracks with images of cracks collected using ground-penetrating radar (GPR). Figure 3 demonstrates the proposed pipeline of 3D reconstruction pavement cracks with GPR data. Gibert et al. [66] combined multiple detectors for automatic inspection of railway tracks.

Assessment of post-disaster damage in concrete to provide valuable insights for necessary follow-up actions is another application of deep learning in the SHM area. Davoudi et al. [67] applied image segmentation to determine the state of the damage in reinforced concrete beams and slabs. Lattanzi et al. [68] also applied image segmentation via the MATLAB Image Processing Toolbox for the extraction of features from images of damaged reinforcement columns to estimate the maximum lateral displacement using a regression model. Spalling, a common type of damage in concrete structures, is another area of application of deep learning in SHM practice. For example, Dawood et al. [69] presented a hybrid model combining image processing and machine learning techniques to identify spalling distress in subway stations. Yeum et al. [70] applied AlexNet to both collapse classification and spalling detection in post-disaster analysis of concrete structures.

Kim and Cho [71] introduced a method utilizing unmanned aerial vehicles (UAVs) and R-CNN to detect cracks in old concrete bridges. They applied transfer learning to R-CNN, using crack images to enhance crack detection, and later, image processing was employed to quantify the identified cracks. Kang and Cha [72] also applied UAV-based damage detection with deep learning; however, they addressed one important issue, which is that UAVs often require a skilled pilot and autonomous flight with GPS in certain complex locations of structures, such as indoors or beneath bridges. The authors proposed an ultrasonic beacon for UAV navigation in GPS-incompatible environments.

Xue and Li [73] devised a three-tiered deep learning framework including an FCN, RPN, and position-sensitive region of interest pooling for identification of damage in tunnel linings. Hoang et al. [74] also compared the performance of a CNN with Sobel and Canny edge detection algorithms, as previously reported by Cha et al. [64], for a cyclic survey of pavement cracks. Similarly, Dorafshan et al. [75] compared the performance of four edge detection methods with CNNs in detail for crack detection in concrete. Four common edge detection methods in the spatial domain (Roberts, Prewitt, Sobel, and Laplacian of Gaussian) and two in the frequency domain (Butterworth and Gaussian), as well as the AlexNet model in three modes of training (trained, transfer learning, and without training), were compared, and the authors concluded that AlexNet showed superiority over other methods.

AlexNet was used by Wang et al. [76] as well. The authors utilized both AlexNet and GoogLeNet for the detection of various types of damage to masonry walls in historic structures, using sliding-window techniques to pinpoint concrete damage. Motivated by the ImageNet Challenge, Gao and Mosalam [77] proposed the concept of Structural ImageNet, with four intended tasks: component identification, spalling detection, damage condition evaluation, and damage type determination in concrete through the application of transfer learning in VGGNet (Visual Geometry Group).

Wu et al. [78] applied transfer learning to VGG16 and ResNet18 to detect two types of prevalent concrete surface defects, namely cracks and corrosion. Zhang et al. [79] proposed Faster R-CNN to determine the spatiotemporal information of the vehicles on bridges in order to determine the stress state and traffic densities. Wang and Cheng [80] proposed DilaSeg-CRF by integrating a CNN with a dense conditional random field (CRF) to improve the segmentation accuracy in sewer pipe defect detection, whereas Li et al. [81] addressed the issue of data imbalance in sewer damage detection by introducing a hierarchical classification approach to supervise the learning process at different levels. Zha et al. [82] applied transfer learning to ResNet (deep residual neural network) for eight types post-disaster concrete damage detection: scenario classification, damage detection, spalling detection, material identification, collapse detection, effected component identification, and damage level and type determination, which were categorized into binary or multiclasses according to the conditions. The authors used the 2018 PEER Hub ImageNet Challenge distributed by the Pacific Earthquake Engineering Research Center to evaluate the proposed methodology.

Jang et al. [83] used transfer learning in GoogLeNet with hybrid images, combining vision and infrared thermography images to enhance crack detection in concrete structures. The authors suggested the use of a UAV-mounted hybrid system comprising a vision camera, an infrared camera, and a continuous-wave line laser to capture images, particularly for large-scale structures, then used them for inspection of the respective structures. U-Net, which is famous for applications in biomedical image segmentation, was first applied by Liu et al. [84] to concrete crack detection and later compared with FCN using evaluation metrics such as precision and the size of the training set. The authors applied U-Net for localization of concrete cracks under various lighting and background conditions. Khani et al. [85] investigated the impact of preprocessing on a concrete crack detection pipeline based on a CNN trained with 700 labelled gas turbine images. The authors concluded that bilateral filtering improves the generalization ability of the suggested framework in cases with cracks on complex structures. Zhang et al. [86] argued that two-stage detectors such as Faster R-CNN and ResNet-101 have limited practical applications due to their slow speeds. The authors used a single-stage detector (SSD), YOLOv3 (You Only Look Once), to detect multiple types of concrete bridge damage, such as cracks, pop-outs, spalling, exposed rebar, etc.

Liu et al. [87] argued that the motion blur from excessive vibration in UAVs limits the accuracy of crack detection in high-rise buildings. The authors introduced a generative adversarial network (GAN) that incorporates the concept of localized skip connections that recognize the correlation between blurred and sharpened crack images. The proposed method was validated through experiments involving the investigation of skip connections in deblurring and compared with a state-of-the-art deblurring model. Kim et al. [88] applied transfer learning to Mask R-CNN for automatic concrete damage detection and localization in four classes—cracks, efflorescence, rebar exposure, and spalling—using an instance segmentation approach.

Mondol et al. [89] applied Faster R-CNN to detect post-disaster damage like surface cracks, exposed rebar, and buckled rebar using image data collected from concrete structures damaged during past earthquakes in Nepal (2015), Taiwan (2016), Ecuador (2016), Erzincan (1992), Duzce (1999), Bingol (2003), Peru (2007), Wenchuan (2008), and Haiti (2010). Deng et al. [90] introduced LinkASSPNet (LinkNet with atrous spatial pyramid pooling) and conducted a performance comparison with U-Net and LinkNet in the context of concrete bridge surface damage detection. Notably, this study stands out, as the models were trained on a relatively small dataset. It purports to address the challenge of variations in labeling areas among labelers in pixel-wise image segmentation tasks.

Zheng and Zhang [91] proposed a crack detection model for concrete based on image segmentation tasks and the FCN, R-CNN, and RFCN (Richer Fully Convolutional Networks) models. The training included a wide range of image data, including images of buildings, bridges, dams, roads, etc. Karaaslan et al. [92] proposed a combination of an SSD-based VGG-16 model and a modified SegNet, where the former detects regions of interest related to damage, such as cracks or spalling, upon verification by the respective inspector, and the latter then applies segmentation to the damage for further analysis. Miao et al. [93] proposed U-Net-based Damage-Net for semantic segmentation of seismic damage in reinforced concrete structures, where the authors adjusted the padding size and stride size to ensure that the input and output size were the same, which is usually not the case in U-Net. The proposed Damage-Net receives its encoder from the convolutional layers of VGG-16, allowing it to adapt transfer learning and to be trained on a comparatively smaller dataset. Based on this architecture, two individual models were proposed: Crack-Net for detecting cracks, and 4Category-Net for identifying four additional damage categories, namely concrete spalling and crushing, reinforcement exposure, buckling, and fracture.

Qiao et al. [94] proposed EMA-DenseNet, a combination of densely connected convolutional networks (DenseNet) integrated with an expected maximum attention (EMA) module in the last pooling layer for the detection of surface damage in concrete bridges in a set of images collected from multiple bridges located in Zhejiang (China). The authors claimed that the proposed model performs better than FCN, SegNet, DeepLab v3+, and SDDNet. Huang et al. [95] proposed a software system for damage detection in subway tunnels by integrating four separate functions: image fusion to splice the images acquired by different cameras, image preprocessing to remove background noise and other preprocessing tasks, damage identification performed by the R-CNN model and a data platform for evaluation by the respective personnel. Arya et al. [96] proposed a concrete pavement damage dataset consisting of 26,620 data point from multiple countries and investigated how the demographics of the damage data affect the model performance based on a YOLO-v5/YOLO-v4/cascade R-CNN-based ensemble model. Cui et al. [97] proposed an improved YOLO-v3 model for the detection of erosion damage that achieved up to a 75% mean average precision value.

Pozzer et al. [98] compared the performance of different models, i.e., VGG-16, ResNet-18, ResNet-50, MobileNet-V2, Xception, etc., in detecting concrete defects such as delamination, cracks, spalling, and patches in thermographic and regular images at varying distances and under varying conditions using semantic segmentation. Andrushia et al. [99] implied that most research on damage detection in concrete structures does not consider the complex background or environmental effects and therefore proposed a U-Net with an encoder–decoder framework for thermal damage detection in concrete structures in the event of fires.

Munawar et al. [100] introduced a cycle generative adversarial network (CycleGAN) with 16 convolution layers, providing additional support to refine predictions through guided filtering (GF) and conditional random fields (CRFs). The authors applied this model to inspect mid- to high-rise concrete structures constructed during the 2000s using segmentation techniques and drones. Zou et al. [101] proposed a YOLOv4-based approach to the detection of multiple types of damage, including both fine and wide cracks, spalling, exposed and bucking rebars, etc., that was integrated in a graphical user interface (GUI) to streamline the assessment of structural damage in reinforced concrete (RC) buildings following an earthquake. Han et al. [102] proposed the use of a transfer-learning-based AlexNet and threshold segmentation to precisely locate cracks in concrete structures.

Tanveer et al. [103] compared and analyzed the performance of five semantic segmentation models (ENet, CGNet, ESNet, DDRNet-Slim23, and DeepLabV3+ (ResNet-50)). These models were categorized as lightweight and heavyweight based on the parameter count. The evaluation focused on on-site damage detection in concrete structures using edge computing devices such as smartphones, tablets, etc. Bai et al. [104] proposed an EfficientNet-V2-based model for component damage recognition, serving both structural health monitoring (SHM) and post-disaster assessment purposes. They also investigated the relationship between damage type, component damage level, and the structural safety state. Crognale [105] compared four different image processing techniques, namely Otsu-method thresholding, Markov random field segmentation, the RGB color detection technique, and the K-means clustering algorithm, in corrosion and crack detection based on a case study. Chen et al. [106] proposed an AlexNet-based multiclass damage detection method for reinforced concrete bridges in high-speed rail systems.

Wan et al. [107] proposed a BR-DETR model, a concrete bridge damage detection model based on detection transformers (DETR), with deformable Conv2D in place of convolution, as well as with an additional convolutional project attention layer after the self-attention layer. Zhu and Tang [108] introduced a DeepLabV3+ network architecture with Xception as the backbone to automatically estimate detailed crack information in hydraulic concrete structures. Huang et al. [109] proposed a Faster R-CNN with Res-Net101 as the backbone for detection of damage like cracks, spalling, and precipitates in hydraulic concrete structures.

3.2. Damage Quantification

Damage quantification is the next step after damage identification. Concrete damage quantification aims to determine the extent, severity, and specific characteristics of damage, such as cracks, spalling, corrosion, or other forms of deterioration. Although using deep learning for concrete damage quantification is still a relatively new concept in SHM, researchers are continuously generating new ideas to automate the quantification process, given the inherent challenges associated with this topic.

Kim et al. [110] proposed a UAV-based digital image processing system integrated with imaging and distance-sensing technology to determine the width and length of the cracks in concrete surfaces. Tong et al. [111] proposed a CNN-based method to calculate the mean texture depth (MTD) of pavement surfaces from 3D scan data, which was tested on four different highways in Shanxi, China. Huang et al. [112] studied lining damage in tunnels with a rapid detection and assessment analysis system developed by Nanjing HuoYang Hou Mdt InfoTech Ltd. The system includes a multichannel array of high-speed CCD (charged couple device) cameras to obtaining image data, multiple sensors to mitigate the impact of vehicle vibration on the tunnel, a multilayer lighting system, multiple positioning technology (reference object positioning technology + image positioning technology + mileage positioning technology + infrared laser positioning technology), and a computer vision approach for damage identification and analysis. Tayo et al. [113] presented a device capable of portable crack width calculation in concrete road pavement using pattern recognition based on multiple image processing technologies, such as graying, enhancement, filtering and denoising, binarization, segmentation, etc. Kim and Cho [114] proposed Mask R-CNN+image processing techniques for successful detection and quantification of concrete cracks with widths of 0.3 mm or more. Wei et al. [115] applied the same approach to concrete surface bughole segmentation and diameter measurement.

Beckman [116] applied Faster R-CNN to automatically and simultaneously detect and quantify concrete spalling in multiple locations within the same surface. The authors used a depth camera to obtain the volume quantifications of the spalling damage. Park et al. [117] applied YOLO for both concrete crack detection and quantification (i.e., to determine the size of the cracks) in real time. The authors used laser beams with integrated distance sensors for accurate measurement of the crack size. Bhowmick et al. [118] applied U-Net-based segmentation for concrete crack localization and binarization to estimate quantitating properties of cracks, like length, width, area, orientation, etc., from video data collected by a camera mounted on a UAV. Flah et al. [119] applied a deep learning technique to identify both structural and durability-related damage in structural members and assess the condition in a short time span by combing a Keras classifier with Otsu image processing. The proposed method can classify cracks; quantify them in terms of length, width, and angular orientation; and evaluate the severity of the damage.

Yuan et al. [120] proposed an inspection robot that transforms the quantification of concrete damage from a 2D plane to 3D space with stereo vision and a Mask R-CNN approach. The robot is based on four different sensors, with a monocular camera as a visual sensor, a stereo camera with a sensor for inertial measurement (IMU) of six degrees of freedom that can be mapped for panoramic image stitching, and a LiDAR sensor to measure the distance between the RC structure and the camera. Miao et al. [121] proposed a GoogLeNet-based transfer learning approach incorporating a novel sliding technique known as neighborhood scanning. This method aims at the detection, segmentation, and quantification of concrete cracks, achieving an average relative error of 14.58% in crack calculation.

Song et al. [122] introduced a deep learning approach for crack segmentation and quantification utilizing MobileNetV1 and ResNet50, along with DeeplabV3+ and U-Net. MobileNetV1 and ResNet50 handle crack classification, while DeeplabV3+ and U-Net manage panoramic crack segmentation (Figure 4). The quantitative information of the crack was subsequently acquired by multiplying the actual physical size corresponding to the unit pixel, assuming the length of a single pixel as the unit length. Kumarapu et al. [123] introduced UAVIC, a system that integrates UAVs with an image processing technique, i.e., digital image correlation. This approach is employed for damage quantification on scaled bridge girders.

Bae et al. [124] proposed a computer-vision-based crack quantification algorithm using decision making based on statistical methods for accurate estimation and quantification of damage based on an image dataset of concrete building structures in South Korea. Li et al. [125] proposed a ResNet50-based improved You Only Look At CoefficienTs for Edge devices (YolactEdge) combined with digital image processing techniques for damage identification and quantification in hydraulic tunnels.

4. Strengths, Weaknesses, Opportunities, and Threats (SWOT) Analysis

SWOT analysis is a strategic tool utilized to assess the strengths, weaknesses, opportunities, and threats within a system or situation [126]. To facilitate future planning, decision making, and strategic development in construction engineering, a SWOT analysis was conducted to evaluate the deep-learning-based SHM systems discussed in this study.

• Strengths: The implementation of deep learning models can assist in improving the SHM process by systematically predicting patterns and anomalies in image data [127]. This results in more accurate damage identification in concrete and concrete structures. One advantage of deep learning is its integration capability with multiple other systems, such as sensors (LiDAR and IMU); UAVs or drones; and depth/stereo or infrared camera, which, in turn, aids in the quantification process. Combining deep learning with computer vision applications can not only eliminate the unsafe and lengthy manual inspection process but also enable the automation of the entire SHM system in real time [128].
• Weaknesses: The first weakness in the implementation of deep learning models is their requirement for large-scale annotated data. Obtaining high-quality labeled data, specifically related to concrete health conditions, remains a persistent challenge [61]. Another potential limitation is the requirement for extensive knowledge. Those aiming to implement a deep-learning-based SHM process must be familiar with both deep learning and structural engineering, posing an additional challenge.
• Opportunities: An SHM system based on deep learning models offers early damage identification and real-time continuous monitoring [2]. When integrated with an alarm system, it can promptly notify authorities during the early stages of damage. This enables rapid action, preventing the escalation of severity and reducing additional costs related to maintenance and damage repair. The field of construction engineering also often struggles with complexities in data, and deep learning has proven to be an effective solution to address these challenges [62].
• Threats: While deep learning applications can undoubtedly aid in the SHM process, the reliability of results becomes questionable without proper validation or practical testing of the trained models. Additionally, a deep-learning-based system should undergo regular updates with new data or guidelines to effectively tackle emerging challenges [61].

According to the SWOT analysis, the application of deep learning in SHM systems offers a vast array of opportunities; however, further research and studies are required to understand its limitations and threats. The next section describes some practical recommendations regarding training data requirements, model reliability, and other issues found in the studies published to date.

5. Discussion and Suggested Frameworks for the Future

The number of deep-learning-based applications in concrete research is rapidly growing, especially in the SHM area. Numerous applications have been reported with respect to both concrete damage identification and quantification. However, based on the abovementioned trends, the number of automatic concrete damage quantification studies in the SHM area was comparatively less before 2019. The application and integration of stereo cameras and sensors, such as LiDAR and laser sensors, have made deep learning applications for damage quantification. Many researchers have applied various image processing techniques rather than integration with depth cameras or sensors. However, concrete cracks are very fine, so whichever system is adopted must precisely quantify a particular property (either length, width, or diameter). AlexNet, GoogleNet, Faster R-CNN, Mask R-CNN, U-Net, VGG, and YOLO models seem to be popular choices for damage identification. However, compared to other industries, construction falls behind in terms of adopting digitalization; therefore the application of deep-learning- and vision-based systems to monitor concrete health in the SHM area is still not sufficient in real practice, mostly due to the following issues:

• Data shortage: Although transfer learning has made the adaptation of deep learning easier, there is still a lack of publicly available datasets in the construction domain. Raw data often need to go through many stages of post processing, which is very time-consuming and labor-intensive. Also, there is a need for annotated datasets, which are essential for any deep learning training [129]. Most studies have been conducted using private datasets; making such datasets public would open multiple doors for researchers in the SHM domain for multiple applications. Although data augmentation plays an important role in dataset incrementation, applying various transformations to existing data, such as rotating, scaling, flipping, or cropping images, is insufficient for research in the SHM area. An alternative method involves utilizing generative adversarial networks (GANs), where a deep learning model comprising two distinct networks (namely a generator and a discriminator) is employed to generate synthetic image data instead of relying on real-world camera inputs only, as reported in [87,100]. Deng et al. [130] implied that GANs trained on synthetic data often perform well in real-world scenarios.
• Impact of the training data on overfitting: Transfer learning has undeniably simplified the application of deep learning models in structural health monitoring (SHM). However, the persistent challenge of overfitting can arise, particularly in instances where there is a paucity of image data. Deep learning models characterized by multiple layers and millions of parameters demand extensive tuning, as illustrated, for example, by the necessity of adjusting at least 100 million parameters in VGG-16 for crack detection [61]. The insufficiency of training data, both in terms of quantity and quality, poses a significant obstacle, rendering a model incapable of performing effectively in real-world applications. It is imperative that the training data encompass diverse real-world scenarios, accounting for variations in background, lighting, and weather conditions, to ensure the model’s robustness and applicability.
• Requirement for high-performance computers: Many deep learning techniques necessitate several days for training due to the extensive calculations involved in computing related training parameters, such as loss functions. Adequate hardware, including high-capacity hard disks, multiple GPUs/CPUs, and substantial memory, is essential for storing these calculations. Researchers should prioritize discovering optimized model structures with fewer parameters, facilitating their seamless adaptation in structural health monitoring (SHM) applications. An attempt to address this concern was made by Zang et al. [86] with an SSD-based model.
• Dealing with background noise: On the other hand, in addressing various background noises in images, researchers have implemented different morphological changes in the CNN architecture [80,87,90,93,94,100,107] to increase the detection accuracy. However, the source code is typically not publicly available. Researchers should be encouraged to make their source code publicly accessible, enabling other researchers to enhance the architecture further and, consequently, increase its applicability in actual practice. Due to the image resizing requirement of deep learning models to be trained on computers with average computing capacities, generalization abilities are often lost. For example, stains are a common issue in concrete structures and often incorrectly identified as cracks. To solve this issue, stains and similar defects could be categorized as another class [131] to improve the generalization abilities.

Despite the challenges and limitations, the use of deep learning and computer vision technologies holds significant promise in structural health monitoring (SHM) and concrete research. Integrating deep learning applications into smartphones or tablets for on-site inspections, as well as utilizing UAV-based approaches for the inspection of high-rise buildings and long-span bridges, not only facilitates the whole inspections process but also saves time and effort while promoting workplace safety. Additionally, the integration of big data and data mining technologies with cloud computing can enhance data management. A collaborative effort from researchers, scholars, and engineers in the construction, computer science, and civil engineering domains can establish more effective deep -earning-based SHM inspection systems for both damage identification and quantification, regardless of severity, delicacy of the damage, and the influence of the surrounding environment.

6. Conclusions

The aim of this research was to conduct a systematic review of the utilization of deep learning in the identification and quantification of concrete damage for SHM purposes. This study delved into the concepts and historical development of artificial intelligence (AI), computer vision (CV), and deep learning. With the aim of including the latest advancements in concrete research, the analysis was focused on studies spanning from 2017 to 2023, particularly those addressing vision-based crack identification, categorization, and measurement analysis. Our comprehensive discussion of the applications, purposes, and limitations of deep-learning-based SHM research yielded the following key points:

• Although deep-learning-based damage identification is subject to multiple challenges regarding data acquisition, processing, training, and testing issues, it has demonstrated significant promise. The requirement for specific dataset preparation and strategic approaches during training could help the researchers overcome overfitting issues encountered as a result of limited resources.
• The integrations of deep learning in concrete damage quantification research is challenging due to the fact it can only provide pixel-based measurement, not an actual measurement. While unit conversion and image processing techniques can be applied to smaller cracks, large cracks may require depth or stereo cameras and remote sensing systems. However, most available depth cameras on the market have a short range (>10 m), so efforts should be made to develop longer-range depth cameras.

In this study, we addressed four critical issues related to the application of deep learning to concrete damage identification and quantification and suggested potential frameworks to deal with these issues. This research provides helpful insights that can aid in future applications and studies regarding deep learning in the SHM area.