Use of Parallel ResNet for High-Performance Pavement Crack Detection and Measurement

Abstract

In the process of road pavement health and safety assessment, crack detection plays a pivotal role in a preventive maintenance strategy. Recently, Convolutional Neural Networks (CNNs) have been applied to automatically identify the cracks on concrete pavements. The effectiveness of a CNN-based road crack detection and measurement method depends on several factors, including the image segmentation of cracks with complex topology, the inference of noises with similar texture to the distress, and the sensitivity to thin cracks. The presence of shadows, strong light reflections, and road markings can also severely affect the accuracy in detection and measurement. In this study, a review of the state-of-the-art CNN methods for crack identification is presented, paying attention to existing limitations. Then, a novel deep residual convolutional neural network (Parallel ResNet) is proposed with the aim of creating a high-performance pavement crack detection and measurement system. The challenge and special feature of Parallel ResNet is to remove the noise inference, identifying even thin and complex cracks correctly. The performance of Parallel ResNet has been investigated on two publicly available datasets (CrackTree200 and CFD), comparing it with that of competing methods suggested in the literature. Parallel ResNet reached the maximum scores in Precision (94.27%), Recall (92.52%), and F1 (93.08%) using the CrackTree200 dataset. Similarly, for the CFD dataset the novel method achieved high values in Precision (96.21%), Recall (95.12%), and F1 (95.63%). Based on the crack detection and image recognition results, mathematical morphology was then used to further minimize noise and accurately segment the road diseases, obtaining the outer contours of the connected domain in crack images. Therefore, crack skeletons have been extracted to measure the distress length, width, and area on images of rigid pavements. The experimental results show that Parallel ResNet can effectively minimize noise to obtain the geometry of cracks. The results of crack characteristic measurements are accurate and Parallel ResNet can be assumed as a reliable method in pavement crack image analysis, in order to plan the best road maintenance strategy.

1. Introduction

Monitoring, measuring, and evaluating pavement conditions are essential parts of road pavement maintenance activities, due to the perspective to plan corrective actions [1,2]. Cracking is one of the most common road diseases [3,4]. The identification of cracks is a highly attractive problem as it allows the planning of the most efficient preventive maintenance interventions; in fact, cracks activate the main degradation phenomena of the pavement [5,6]. To recognize the extent of the crack and the corresponding degree of severity found during the inspection allows the assessment of the level of pavement decay. As proof, the commonly used Pavement Condition Index (PCI) [7] provides a global assessment of flexible or rigid pavements by distinguishing the extent and severity of each defect. Traditional road crack detection methods are time consuming, wasteful, and subjective [8,9,10]. Recent advances in technology had a significant impact in the field of pavement distress evaluation and measurement, making it possible to analyze the damage more quickly and reliably than consolidated methods [11,12,13]. Hence, the road managers can prioritize and plan the maintenance of the road network, in order to keep the infrastructure in good condition and to extend their service life [14].

Crack identification systems based on digital image processing have already been used in the detection of transport infrastructures such as highways, tunnels, and bridges [15,16,17]. This approach requires image acquisition hardware and image detection software. Road surface images are typically collected by multi-functional road detection vehicles or surveyor robots [10,18]. Cracks in the images are distinguished into linear cracks and complex cracks (block or network cracks): linear cracks are evaluated by measuring the average width and length of cracks, while complex cracks are measured using the average crack width and crack area [7]. The original images are generally processed using the main algorithms of pre-processing, threshold segmentation, edge segmentation, and mathematical morphology [16,17,19].

In recent years, digital image detection based on machine learning algorithms or deep neural networks has been applied in many fields, such as equipment fault diagnosis, medical image classification, and facial recognition [20,21,22]. At the same time, it has also been used to automatically identify cracks on concrete pavements [23], with reliable detection results. However, some issues limit the automatic detection of road surface cracks based on digital image processing. Firstly, the problem of inaccurate crack detection with complex topology and noise inference should be solved [12,23,24,25]. Furthermore, it is very difficult to distinguish between certain noises in the image background (e.g., characterized by a dark and thin nature) and cracks. Then, the detection accuracy is low and the measurement error is very large on thin cracks or with complex topology [10,26,27].

In order to overcome the above-discussed problems of well-known road crack detection and measurement systems, in this study the Convolutional Neural Network (CNN) structure is improved by combining a fully convolutional layer network [28] and residual network [29]. Thus, in this study is proposed a novel parallel residual CNN, namely Parallel ResNet, with multiple branches of parallel structures, regularizing the loss function.

The mathematical morphology method to further filter noise, to eliminate blur, and to segment cracks is applied. Using the medial axis skeleton algorithm [30,31], the crack skeleton and the contour of the disease’s connected domain is extracted. The primary goals are the reduction in error when measuring the maximum and average crack width, distress length and area. Therefore, the proposed pavement crack detection and measurement method can improve the accuracy and reliability in road health assessment systems.

2. Related Works

This section briefly reviews the literature on automatic crack detection and measurement methods based on image processing applications.

2.1. Crack Detection

First studies on crack detection using image processing are related to the intensity-threshold approach, due to its simplicity and efficiency [32]. However, this method is generally suitable for image processing with consistent background grayscale, uniform illumination, and high contrast.

The gray level of the crack edge has an obvious step phenomenon and the background gray level changes slowly, so the crack target can also be segmented by detecting the edge using algorithms such as Roberts, Sobel, Canny, Prewit, or Log [33]. Qu et al. in [34] adopted basic mathematical morphology operations to complete the identification and connection of linear crack targets: too many threshold settings reduce the adoption of similar approaches. On the other hand, the segmentation algorithm of edge detection is mostly based on the local gray and gradient information to identify the crack edge, which is only suitable for crack maps with a strong edge label [35].

Kass et al. in [36] proposed the Minimal Path Method (MPM) to extract simple open curves from images, in order to avoid false detections and to measure the width of the cracks. Some researchers in [37] introduced an approach that selects endpoints globally, and then obtains a minimum path. Nguyen et al. in [38] applied Free-From Anisotropy to consider intensity and crack form features for detection on pavement surface images.

As the image data size increases, several studies have considered the use of image processing and machine learning approaches. Statistical analysis are the most popular and widely used methodological approaches [39,40]. Delagnes and Barba in [41] suggested the detection of poorly contrasted cracks in textured areas using a Markov random field model. Wang et al. in [20] employed a Support Vector Machine (SVM) to recognize aircraft skin cracks. Bray et al. in [42] used the neural networks classification approach to identify road defects. Yang et al. in [28] adopted artificial neural network models to separate crack pixels from the background by selecting appropriate thresholds. However, all these methods had disadvantages in detecting the complete crack structure over the entire image. Shi et al. in [24] proposed an automatic road crack detection method, i.e., CrackForest, based on random structured forests. The main purpose of this system is to characterize the distress and eliminate noise marked as cracks by learning the inherent structured information of road diseases. Low accuracy in complex crack detection is the biggest limitation of the CrackForest algorithm.

Deep learning is a subfield of machine learning. In recent years, Convolutional Neural Networks have not only performed better in multi-class classification tasks but have also made significant progress in segmentation and concrete cracks detection. Chambon and Moliard in [43] presented a comparative study of deep learning software frameworks, network architectures, selected hyper-parameters, and crack detection performances. Some researchers in [28,30] adopted the Fully Convolutional Network (FCN) to achieve automated road crack detection and measurement. Liu et al. in [44] applied U-net, an encoder–decoder network approach, to detect concrete cracks. In more detail, the encoder network is similar to a VGG-16 deep convolutional neural network, and it includes an up sampling layer and dense skip connection. The U-net-based method showed high effectiveness, good robustness, and better accuracy than previous FCNs. Liu et al. in [45] proposed a two-step pavement crack detection and segmentation method based on CNN. The method combines the detection and segmentation of cracks on the pavement together, so that the cracks found in the first step are segmented in the second step to improve accuracy. Lau et al. in [46] proposed a U-Net-based network architecture with an encoder of pre-trained ResNet-34. In this study, techniques such as freezing the layer groups, assigning different learning rates to each layer group, and incrementally increasing the image size were adopted. Song et al. in [47] proposed the Multi-scale Feature Attention Network as an innovative approach to automatically identify cracks. Some researchers in [23] involved the use of a cross-entropy loss function and the VGG16 network model for crack detection on concrete pavements. He et al. in [29] adopted the Residual Network (ResNet), which solved the network degradation problem with the residual network, a 152-layer successfully trained neural network. ResNet can obtain higher accuracy than VGGNet and GoogleNet, and higher computational efficiency than VGGNet. Shortcut connections in the ResNet realizes identity mapping, so the data stream can flow across layers, and a deeper network should improve image recognition. Kumar et al. in [21] proposed an ensemble of Fine-Tuned Convolutional Neural Networks for medical image classification. Veit Andreas et al. in [48] showed how residual networks work as relatively shallow networks.

2.2. Crack Measurement

Several researchers have focused on techniques to measure the width, length, and area of cracks: they are important factors in road safety diagnosis. Shan et al. in [49] pointed out that the type, number, width and length of road cracks represent the early levels of decay of concrete structures. Hyunwoo Cho et al. in [50] proposed a Crack Width Transform Method, that can be summarized in the following steps: (I) crack width transform, (II) aspect ratio filtering, (III) crack region search, (IV) hole filling, and (V) relative thresholding. The method has high accuracy and consistency in linear or simple crack width measurements.

Thinning algorithms for binary images can extract crack skeletons. Peleg et al. in [51] proposed an index table thinning algorithm, which can extract crack skeletons. However, the skeleton shape is easy to be deformed, and a spiculated margin is produced in the process of thinning. In this regard, the Hilditch thinning algorithm [52] reduces the distortion and the spiculated margin. However, due to too many conditions and the computational complexity, significant resources are required and the processing speed is slow. Zhang and Suen in [53] refined the algorithm: the processing speed is improved, but a single pixel of thinning cannot be guaranteed, the bifurcation point is difficult to find, and the thinning process has a spiculated margin. Some researchers in [27,54] pointed out that many methods were used for crack width measurement: commonly, the left and right boundary method, skeleton point oblique method and boundary point minimum distance method have been applied to crack width measurement. The crack length can be calculated from the crack skeleton, and the fracture area from the pixel statistical method. However, the skeleton extraction process of cracks with complex topology creates more spiculated margins.

3. Methodology

In this section, a novel method for crack detection and measurement based on raw input road surface images is presented. The overall procedure of the proposed approach is summarized in Figure 1.

A deep residual convolutional neural network, namely Parallel ResNet, is adopted to improve the crack detection performance of CNNs. The main features of the model include skip connection convolutional layer, parallel neural network structure, regularized loss function, and convolutional layer with a stride equal to 2. Supervised learning analyzes the crack image dataset (necessary for training) and yields the crack recognition as output.

The additional task of the novel approach is the crack measurement. The mathematical morphology method is employed to further sort out blur and noise and effectively segment the crack. Then, the crack skeleton is extracted by the medial axis skeleton algorithm from the crack recognition image. The crack length is calculated considering the total number of pixels of the crack skeletons with single pixel width. The outer contours of the connected domain are also extracted for the crack. Finally, the use of the normal vertical crack skeleton method allows the crack width and area of each connected domain to be measured and the crack characteristics to be counted, such as the crack width and area of the entire image.

3.1. Automatic Pavement Crack Detection Method

In this research, the innovative approach involves the multi-branch parallel combination of deep convolutional network structures, with the aim of providing a more accurate and efficient detection method than the single branch structure. This assumption is in accordance with real-life situations: the results obtained by merging the knowledge in crack identification from different people are more reliable than the ones obtained from a single person. The criterion is explained through the example in Figure 2.

The proposed methodology focuses on the implementation of residual convolutional neural networks. This classical CNN architecture adopts skip connection to return the feature map of a certain layer to the next one, or even deeper, avoiding the network model degradation due to gradient disappearance and explosion. Then, a parallel structure module of residual networks is introduced, which is based on the idea of paralleling multiple residual networks, replacing the traditional pooling layer with a convolutional layer characterized by a stride of 2, and regularizing the loss function.

The Parallel ResNet architecture, in the case of a number of branches equal to 3, is described in

Table 1. The architecture of Parallel ResNet (branches = 3).

Architecture of Parallel ResNet
Input (27 × 27 RGB image)
Conv16 [3 × 3, 1, 1]
Conv_A_1 16 [3 × 3, 1, 1]	Conv_A_2 16 [3 × 3, 1, 1]	Conv_A_3 16 [3 × 3, 1, 1]
Conv_B_1 16 [3 × 3, 1, 1]	Conv_B_2 16 [3 × 3, 1, 1]	Conv_B_3 16 [3 × 3, 1, 1]
$\oplus$Conv16	$\oplus$Conv16	$\oplus$Conv16
Conv_1 16 [3 × 3, 2, 1]	Conv_2 16 [3 × 3, 2, 1]	Conv_3 16 [3 × 3, 2, 1]
$\oplus$
Conv32 [3 × 3, 1, 1]
Conv_A_1 32 [3 × 3, 1, 1]	Conv_A_2 32 [3 × 3, 1, 1]	Conv_A_3 32 [3 × 3, 1, 1]
Conv_B_1 32 [3 × 3, 1, 1]	Conv_B_2 32 [3 × 3, 1, 1]	Conv_B_3 32 [3 × 3, 1, 1]
$\oplus$Conv32	$\oplus$Conv32	$\oplus$Conv32
Conv_1 32 [3 × 3, 2, 1]	Conv_2 32 [3 × 3, 2, 1]	Conv_3 32 [3 × 3, 2, 1]
$\oplus$
FC-64
FC-64
FC-25
Output (5 × 5 structured prediction)

.

The symbol $\oplus$ represents the tensor add operation, the terms “Conv*”, “Conv_*” and “Conv_*_*all” are convolutional layers: for example, in Conv16 [3 × 3, 1, 1] the depth is 16, kernel size is 3 × 3, stride is 1, and with zero padding. The term “FC-*” represents the full connection layer. The output patch 5 × 5 is a structured prediction center based on the input patch 27 × 27. In the final output, the activation function sigmoid is used for the binary classification task. The Rectified Linear Unit (ReLU) is employed to increase the non-linearity for hidden layers.

The Parallel ResNet architecture forward propagation diagram is described in Figure 3a: the size of the input patch is 27 × 27 with three channels; other cubes indicate the feature maps obtained from the convolutional layer (Conv), Parallel Residual Convolutional Neural Network Module (the Parallel ResNet Module), and Full Connection. An exploded view of the Parallel ResNet Module is shown in Figure 3b.

Additional information is provided for a better understanding of the discussed architecture. The convolutional layer (Conv) plays a key role in CNN. In the process of data transmission, sparse interaction and parameter sharing are realized, which reduce the number of parameters in the CNN model and allow the extraction of useful feature information. Turning to mathematics, Equation (1) represents the convolutional operation of the convolutional layer:

$$Conv{\left(x\right)}_{i,j}={\displaystyle \sum }_m{\displaystyle \sum }_n{X}_{i-m, j-n}{K}_{m,n}$$

(1)

where X is the two-dimensional input matrix; K is the convolutional kernel; and m and n are the row and column values of the convolutional kernel. In expanded form, Equation (1) is rewritten considering two different values of stride, for example, equal to 1 and 2 (Equations (2) and (3), respectively).

$$\left[\begin{array}{ccc}a& b& c\\ e& f& g\end{array} \begin{array}{c}d\\ h\end{array}\right]\otimes \left[\begin{array}{cc}i& j\\ k& l\end{array}\right]=\left[ai+bj+ek+fl bi+cj+fk+gl ci+dj+fk+hl\right]$$

(2)

$$\left[\begin{array}{ccc}a& b& c\\ e& f& g\end{array} \begin{array}{c}d\\ h\end{array}\right]\otimes \left[\begin{array}{cc}i& j\\ k& l\end{array}\right]=\left[ai+bj+ek+fl ci+dj+fk+hl\right]$$

(3)

where $\left[\begin{array}{ccc}a& b& c\\ e& f& g\end{array} \begin{array}{c}d\\ h\end{array}\right]$ is the input two-dimensional tensor; and $\left[\begin{array}{cc}i& j\\ k& l\end{array}\right]$ is a 2 × 2 convolutional kernel tensor.

Based on the convolutional layer parallel structure of the concept-v3 model [12,54], this paper proposes a network structure with a multiple skip connection convolutional layer (residual block) connected in parallel, i.e., the Parallel ResNet Module (Figure 4). For a better comprehension, it is distinguished in:

• Figure 4a, which represents two standard convolutional layers. The related function is defined by Equation (4):

$$H\left(x\right)=F\left(x\right)$$

(4)

• Figure 4b, which shows a residual module skipping two convolutional layers (L = 2). The mathematical expression is described by Equation (5):

$$H\left(x\right)=F\left(x\right)+x$$

(5)

• Figure 4c, a Parallel ResNet module with three branches in parallel. The structure of each branch in the parallel module is the same. For example, the first parallel branch is composed of a residual block skipping two convolutional layers (Conv_A_1 and Conv_B_1), and a convolutional layer (Conv_1) with a stride of 2. Finally, through the add operation, the module outputs $N\left(x\right)$

.

The Parallel ResNet module function is described by the following Equations (6)–(10):

$$H_i\left(x\right)=x^l+{\displaystyle \sum }_{k=l}^L{F}_i\left(x^k\right)$$

(6)

$$F_i\left(x^k\right)=f\left(C_0^k,C_1^k,\cdots ,C_c^k\right)$$

(7)

$$C_m^k={\displaystyle \sum }_{n\in x_m}{x}_i^{k-1}\ast w_{nm}^l+b_m^l$$

(8)

$$L_i\left(x\right)=H_i\left(x\right)\otimes W_i+t_i$$

(9)

$$N\left(x\right)={\displaystyle \sum }_{i=1}^n{L}_i\left(x\right)$$

(10)

where $H\left(x\right)$ is the output of the residual module; $L$ is the number of skip layers; $F\left(x^k\right)$ represents the output after $C^k$ is activated; $C_m^k$ is the m-th channel of the k-th layer’s characteristic graph; the function $f\left(\cdots \right)$ is the ReLU activation function; ${\chi }_m$ represents the subset of the input feature used to calculate $C_i^k$; $w_{nm}^l$ is the convolutional kernel matrix which size is $3\times 3$; $L_i\left(x\right)$ is the output tensor of the convolutional layer with a stride equal to 2; $\otimes$ denotes the convolution operation symbol; $W_i$ is the convolutional kernel matrix which size is also $3\times 3$; and n is the number of batch elements.

The Parallel ResNet module, compared with standard convolutional modules, has the following advantages:

• In the forward propagation of each branch, if the weights in the convolutional layer (Figure 2a) have not learned any information, it is equivalent to performing an identity transformation. If the convolutional layer has learned some useful knowledge, then it may have a better learning ability than the identity function;
• The skip connects change the output function of convolutional layer training from $H\left(x\right)$ to $H\left(x\right)=F\left(x\right)+x$. This ensures that the gradient calculated for $H\left(x\right)$ does not tend to zero in the process of back propagation. In addition, in back propagation, the residual block is more sensitive to the change of output, and can adjust the weight more finely than the standard convolutional layer;
• A parallel structure, through multiple branches in parallel, can learn more useful knowledge and can identify crack features in the training and learning phases.

Regularizing the loss function is another important issue to be considered in the developed methodology. Cross entropy is a common loss function, and it can be calculated as follows (Equation (11)):

$$Loss\left(x\right)=-{\displaystyle \sum }_{i=1}^N{p}_i\left(x\right)\log (q_i\left(x\right))$$

(11)

where $Loss\left(x\right)$ represents regularizing the loss function; N is the number of batch elements; $x$ is an input batch; and $p_i\left(x\right)$ and $q_i\left(x\right)$ are the actual probability and model prediction probability that the input batch belongs to the ith type.

After performing L2 regularization on the unit, the formula can be rewritten as follows (Equation (12)):

$$Loss\left(x\right)=-{\displaystyle \sum }_{i=1}^N{p}_i\left(x\right)\log (q_i\left(x\right))+\lambda {\displaystyle \sum }_{j=1}^M\Vert {\omega }_j^2\Vert$$

(12)

where M is the number of parameters in the regularization part; $\omega$ is the parameter item involved in the regularization; $\lambda$ is the regularization coefficient, which defines the degree of regularization; and $\Vert \cdots \Vert$ represents the L2 regularization method.

The regularization coefficient L2 limits the size of the parameter values and can prevent overfitting. During the tests, the regularization coefficient is set to 0.0005. In the crack detection method of Parallel ResNet, the convolutional kernel weight of a branch in the Parallel ResNet Module is randomly selected in the regularization process to participate in the L2 regularization calculation. In TensorFlow 2.0, the L2 regularization method is adopted ($R\left(\omega \right)=\Vert {\omega }_2^2\Vert =\frac{1}{2}\times {\displaystyle \sum }_j\left|{\omega }_j^2\right|$). Therefore, the derived expression (Equation (13)) can be written:

$$Loss\left(x\right)=-{\displaystyle \sum }_{i=1}^N{p}_i\left(x\right)\log (q_i\left(x\right))+\lambda \times \frac{1}{2}\times {\displaystyle \sum }_{j=1}^M\Vert {\omega }_{jk}^2\Vert$$

(13)

where $k=Rand\left(branchs\right)$; and $\lambda$ in this study is set to 0.0005.

A branch of the Parallel ResNet module is randomly selected for L2 regularization to provide a greater diversity of parameter learning and updating, which avoids over fitting and is useful in achieving the global optimal solution.

3.2. Crack Measurement Method

The automatic crack detection process does not completely eliminate inferences (e.g., irregular distributions, small areas, similar pixels and shapes to cracks) in images returned by Parallel ResNet. Hence, the mathematical morphology method has been used to further minimize noise and accurately segment the investigated road diseases. Mathematical morphology is based on set theory in order to describe different objects in an image, perform matching operations on the image by defining structural elements of specific sizes and shapes, and analyze and recognize the target area in the original image. Assuming that the pixel set of the image to be processed is I and the structural element is set to P, the mathematical morphology operation is the set operation of P and I. In this paper, the main operations of mathematical morphology (expansion, corrosion, opening and closing) are used to smooth the edges of the cracks, remove small holes, binary image deblurring and other actions, as summarized in

Table 2. Common operations in mathematical morphology.

Operation Name	Function	Action
Dilation	$I\oplus P=\left\{x|{\left(\widehat{P}\right)}_x\cap I\ne \phi \right\}$	Remove gaps and holes.
Erosion	$I\mathsf{\Theta }P=\left\{x|{\left(\widehat{P}\right)}_x\subseteq I\right\}$	Minimize the inference of noise points.
Opening	$I\circ P=\left(I\mathsf{\Theta }P\right)\oplus P$	The contour becomes smooth, the narrow discontinuity is broken, and the elongated crack is eliminated.
Closing	$I\cdot P=\left(I\oplus P\right)\mathsf{\Theta }P$	Smooth contour lines, remove smaller holes and fill up the breaks in contour lines

.

Then, the connected domain and the outer contour of the crack are extracted by mathematical morphology methods. The adopted criterion is to remove unnecessary contour points and keep only skeleton points; multiple iterations are performed to obtain the final result. In this work, the medial-axis skeleton algorithm is used to extract the crack skeleton. The purpose of extracting the crack skeleton in the pavement surface images is to measure the characteristic parameters of road diseases. As discussed above, cracks in digital images may be incoherent, thin and complex; topology cracks may also have multiple connected regions after being segmented, and can be divided into different connected domains. The overall procedure is presented in Figure 5.

Turning to mathematics, one by one statistical calculation, based on the connected domain of the divided cracks, is required to define the width, length, and area of the cracks. Based on the displacement of pixels for crack skeletons, the following Equation (14) can be used to calculate the crack length:

$$L_{crack}={\displaystyle \sum }_m{\displaystyle \sum }_{n}f\left(x,y\right)dl$$

(14)

where $f\left(x,y\right)$ are the calibrated pixel displacements in the coherent crack; $dl$ is the finite length of the crack skeleton elements; and m and n are the numbers of connected domains in the crack images and the total numbers of finite lengths of the crack skeleton elements in the coherent crack, respectively.

The crack skeleton normal line method measures the maximum crack width. The average crack width is estimated by dividing the crack area by the crack length, as written in Equation (15):

$$W_{avg}=\frac{A_{crack}}{L_{crack}}$$

(15)

Hence, the area of the crack ($A_{crack})$ is calculated by counting the total pixels of the connected domain. For the calculation of crack area with complex topology, not only does the area of the cracks themselves have to be counted, but also the minimum area of the external rectangle of the cracks. Finally, the process of crack characteristic measurement is shown in

Table 3. Crack measurement process.

Step Number	Crack Measurement Process
1	The cracks are segmented by mathematical morphology, and the contour of the cracks in the connected region is extracted.
2	Parallel thinning algorithm for crack thinning.
3	Crack skeleton extraction based on single pixel width.
4	The crack length is calculated according to Equation (14).
5	Calculation of maximum width of crack by vertical line method of crack skeleton.
6	Based on the pixel statistics, the area of the crack and circumscribed quadrilateral is calculated, and the average width of the crack is calculated according to Equation (15).

.

4. Experiments and Results

The proposed method, Parallel ResNet, is trained and tested on two different databases with RGB images: CrackTree200 [55] and CFD [24]. The CrackTree200 dataset contains 206 images (size 800 × 600 pixels) of cracks in concrete pavements, including noises such as shadows and uneven lighting. The CrackTree200 dataset contains cracks that are thinner in thickness and have a more complex topology. In this study, the CrackTree200 dataset was split into two subsets: 156 images for training and 40 for testing. The CFD dataset contains 118 RGB images with a resolution of 320 × 480 pixels; these data also have some inferences such as shadows, oil spots and water stains, and have non-uniform illumination. In this case, the number of samples was extended; the images were flipped up and down, left and right, and diagonally; and the expanded CFD dataset was then split into two subsets (288 images for training and 184 for test).

In the data pre-processing stage, a strategy with a change in the ratio of positive to negative training samples was proposed to solve the multi-label classification problem with imbalanced samples [56]. The batch size for each iteration was set to 256, and 10 epochs were conducted in the CrackTree200 and CFD datasets.

The computing platform was equipped with Intel Xeon CPU E3-1505M, GPU NVIDIA Quadro M1200 8G, Windows 10 System, and TensorFlow2.0.

In order to evaluate the performance of Parallel ResNet, three statistical parameters are considered: Precision, Recall and F1 scores. They are calculated as described in

Table 4. The evaluation parameters calculation.

Evaluation Parameter	Formula
Precision	$Pr=\frac{TP}{TP+FP}$
Recall	$Re=\frac{TP}{TP+FN}$
F1	$F1=\frac{2\times Pr\times Re}{Pr+Re}$

TP, true positive; FP, false positive; FN, false negative.

:

Considering that ground truth cracks are manually annotated and there are transitional areas between crack pixels and non-crack pixels in real images, it was proposed to tolerate a small distance between the detection and the reference segmentation for the calculation of the TP rate [57]. In this study, it was assumed that the detected pixels, which are no more than two pixels away from the manually labeled pixel, are also true positive pixels.

4.1. Multiple Branches of Parallel ResNet

By distinguishing the number of branches in the Parallel ResNet module, the performance of the method will be different. To analyze this assumption in a numerical way, several values (1, 2, 3, 4, and 5) were set as number of branches. Considering both available datasets (CrackTree200 and CFD), the Precision, Recall and F1 parameters were calculated (

Table 5. Statistical parameter evaluation varying the number of branches in Parallel ResNet.

Number of Branches	CrackTree200			CFD
	Precision	Recall	F1	Precision	Recall	F1
1	0.8936	0.9389	0.9126	0.8095	0.9743	0.8716
2	0.9033	0.9429	0.9189	0.9621	0.9512	0.9563
3	0.9427	0.9252	0.9308	0.9237	0.9723	0.9465
4	0.9476	0.9046	0.9235	0.9323	0.9581	0.9436
5	0.9021	0.9284	0.9112	0.9240	0.9524	0.9352

and Figure 6). Specifically, in the CrackTree200 database it was noted that F1 obtains the highest value for a number of branches equal to 3. Differently, for the CFD database, the best score for F1 is reached when the number of branches is equal to 2. Regarding the Recall parameter, the values obtained for different numbers of branches fall within a narrow range: for the CrackTree200 database they are between 0.9 and 0.943; and for the CFD database they are between 0.95 and 0.98. Comparing Figure 6a,c, the Precision and F1 parameters have a similar trend (a convex function) to the number of branches.

For automatic detection of crack digital images, the branch amount of Parallel ResNet was set according to the F1 score. Therefore, a number of branches equal to 3 was assumed for the CrackTree200 dataset, and equal to 2 for the CFD dataset. Deep learning optimization methods and hyper-parameters of Parallel ResNet model were selected by comparative experimental methods. A gradient-descent optimization algorithm, i.e., Adaptive Moment Estimation (Adam), was used in this study. The ReLU was employed to increase the non-linearity for hidden layers. Moreover, in the final output the sigmoid activation function was used for the binary classification task. It should also be noted that in the training phase, the regularization coefficient (λ) was set to 0.0005, and the learning rate was set to 0.001.

4.2. Experimental Results in Crack Detection

In order to validate Parallel ResNet as a reliable method in pavement crack image analysis, it was compared with six other competing methods defined in the literature, namely Canny, CrackForest, Modified VGG16, U-Net, Structured Prediction, and Ensemble Network. The Precision, Recall and F1 scores are reported in

Table 6. Detection performance of different methods for CFD and CrackTree200 databases.

Method	CrackTree200			CFD
	Precision	Recall	F1	Precision	Recall	F1
Canny	0.30	0.21	0.25	0.4377	0.7307	0.4570
CrackForest	0.7656	0.9133	0.8330	0.7466	0.9514	0.8318
Modified VGG16	0.912	0.891	0.901	0.889	0.903	0.896
U-Net	0.848	0.851	0.849	0.855	0.882	0.868
Structured Prediction	-	-	-	0.9119	0.9481	0.9244
Ensemble Network	0.8525	0.9091	0.8799	0.9552	0.9521	0.9533
Parallel ResNet	0.9427	0.9252	0.9308	0.9621	0.9512	0.9563

for the CrackTree200 and CFD datasets.

Thus, the novel deep residual convolutional neural network, as compared with well-known literature methods, achieved the following results:

• For the CrackTree200 dataset, the best scores in all performance metrics were found (Precision = 94.27%, Recall = 92.52%, F1 = 93.08%);
• For the CFD dataset, the best scores were in Precision (96.21%) and F1 (95.63%); while the obtained value in Recall (95.12%) was a little bit smaller than that achieved by applying Ensemble Network (but fairly similar);
• Problems also arose for Structured Prediction, which failed in cracks detection for the CrackTree200 database: the main reason may be that the crack width is very small, and the noise and crack pixels are very close in the raw image;
• At the same time, it can also be found that the crack detection ability based on the deep learning is much better than the traditional machine learning and the edge extraction algorithms, such as CrackForest, Canny, and Structured Prediction;
• The histogram in Figure 7 shows the results, in terms of evaluation parameters, limited to the deep learning-based methods considered in this paper.

4.2.1. Detection Analysis on the CrackTree200 Dataset

In Figure 8, four different methods (Canny, CrackForest, Ensemble Network, and Parallel ResNet) are compared on the CrackTree200 dataset. The traditional Canny edge detection algorithm was not found to be suitable for road crack detection due to its high sensitivity. CrackForest did not perform well on thin and complex topologies, as well as in the presence of crack images with noise inference. Regarding Ensemble Network, it detected most of the cracks, but there was still much noise left. The method proposed in this research showed promising results on the effective discernment of noise inference, especially in crack images with noises that have a similar texture to cracks. The example crack images in Figure 8 (second and fifth rows) include shadows very similar to cracks in pixels; the proposed method was able to minimize inference and identify cracks better than other crack detection methods.

4.2.2. Detection Analysis on the CFD Dataset

In Figure 9, five different methods (Canny, CrackForest, Structured prediction, Ensemble Network, Parallel ResNet) are compared on the CFD dataset. As can be intuitively observed, the method presented in this paper outperformed other algorithms, with the exception of Ensemble Network, which showed very similar detection results.

4.2.3. Performance Evaluation of Parallel ResNet under Cross-Dataset Scenarios

In this section, different cross-dataset scenarios are investigated in order to confirm the validation of the proposed model in terms of generalization. In particular, in this study the performance of Parallel ResNet merging the CFD and CrackTree200 datasets was further investigated in various ways:

• Training on CFD and testing on CrackTree200;
• Training on CrackTree200 and testing on CFD;
• Training and testing on a hybrid dataset from CFD and CrackTree200.

For a better comprehension, Figure 10 shows an application of the above-discussed approach.

Cross-dataset results using Parallel ReseNet are listed in

Table 7. Cross-dataset results.

	CrackTree200 Testing	CFD Testing
CrackTree200 (Training)	Pr = 0.9427	Pr = 0.9792
	Re = 0.9252	Re = 0.8992
	F1 = 0.9308	F1 = 0.9346
CFD (Training)	Pr = 0.4658	Pr = 0.9237
	Re = 0.6975	Re = 0.9723
	F1 = 0.5139	F1 = 0.9465
Hybrid dataset (Training)	Pr = 0.9335	Pr = 0.9288
	Re = 0.9006	Re = 0.9483
	F1 = 0.9131	F1 = 0.9369

:

Hence, the Parallel ResNet model trained on CrackTree200 and tested on CFD led to high precision and low recall, while the model trained on CFD and tested on CrackTree200 achieved low scores. In addition, training with hybrid data and subsequent tests with each of the publicly available datasets returned results slightly lower than the best performances.

4.3. Experimental Results in Crack Measurement

The crack detection algorithm cannot automatically remove noise from pavement images. Therefore, road surface crack measurement requires a mathematical morphology approach to minimize noise and blur in order to ensure accurate crack skeleton extraction and measurement of road distress characteristics. Figure 11 presents the main morphological steps in crack measurement: original image (Figure 11a), ground truth image (Figure 11b), recognition image of Parallel ResNet (Figure 11c), crack skeleton (Figure 11d), crack outer contour image (Figure 11e), crack circumscribed quadrilateral image (Figure 11f). In particular, the crack circumscribed quadrilateral image shows more clearly that the topologically complex crack is divided into multiple connected regions.

Applying the methodology just discussed, the measurement results are shown for the CrackTree200 and CFD databases in

Table 8. Measurement results for the CrackTree200 dataset.

Original Image (CrackTree200)
Morphological Features	Truth	Predicted	Truth	Predicted	Truth	Predicted	Truth	Predicted	Truth	Predicted
Length	1474	1265	1468	1045	4383	4241	2000	1810	3063	2600
Max Width	8.0	5.7	8.2	6.3	14	5.66	7.2	8.0	10.8	5.7
Mean Width	4.5	3.7	4.5	3.8	4.8	3.7	4.6	4.0	4.8	3.7
Area	6612	4490	6930	3917	22,334	15,568	9372	7310	15,127	9197

and

Table 9. Measurement results for the CFD dataset.

Original Image (CFD)
Morphological Features	Truth	Predicted	Truth	Predicted	Truth	Predicted	Truth	Predicted	Truth	Predicted
Length	3330	2833	540	539	697	689	523	535	451	455
Max Width	14.6	11.7	7.2	8.9	14.4	14.6	14.6	12.4	7.2	10
Mean Width	4.7	5.7	4.4	6.1	6.6	7.3	8.2	9.6	4.5	5.7
Area	15383	16,237	2403	3527	4894	5420	4325	5056	2068	2656

, respectively.

The numbers of truth and predicted data are in pixels.

The experimental results show that the measurement error of crack length, average width and area on crack images of the CrackTree200 dataset is larger than the CFD dataset.

As for the crack length and average width measurement errors, they are mainly caused by the failure of crack edge segmentation.

There are relatively large errors in the measurement of crack areas. One reason is that when there are multiple crack domains in a digital image with complex topology, the predicted crack area is larger than the true area due to overlap. Another possible reason is that some thin cracks are not detected, so the predicted crack area is underestimated.

Especially when the crack boundary is not smooth, the measurement result of the maximum crack width will be worse, and the main reason is that the crack calculation error of the crack crossing position is large. The operations of mathematical morphology, expansion and closing, have been applied to segment cracks. These methods can smooth the contour lines, eliminate the smaller holes, fill the breaks in contour lines and eliminate the isolated pixels, which can reduce the number of predicted crack pixels. At the same time, different combinations or sequences of morphological operations may affect the extraction of crack skeletons and the number of pixels in the crack.

5. Conclusions

In this paper, a novel automatic pavement crack detection and measurement method was proposed. The main goal is to provide an effective solution that overcomes the limitations of established crack detection approaches based on image processing. Currently, segmenting cracks with complex topology, reducing noises with similar texture to the cracks, or identifying thin cracks is highly challenging. In addition, a measurement technique was developed to accurately evaluate crack characteristics.

The Parallel ResNet Module was proposed in order to improve the deep neural network structure, effectively minimize noise, and automatically identify thin and complex crack topology in numerical pavement images. Publicly available datasets (CFD and CrackTree200) were used to validate the methodology. F1 scores reported in the experiments were analyzed to identify the optimal number of branches in the Parallel ResNet Module; a number of branches equal to three was assumed for the CrackTree200 dataset, and equal to two for the CFD dataset.

Experimentally comparing Parallel ResNet with other competing methods suggested in the literature, the novel methodology reached, in terms of detection performance:

• The maximum scores in Precision (94.27%), Recall (92.52%), and F1 (93.08%) using the CrackTree200 dataset;
• High values in Precision (96.21%), Recall (95.12%), and F1 (95.63%) using the CFD dataset.

Therefore, an improved crack measurement method is presented that employs mathematical morphology techniques to eliminate noise inferences and accurately segment the cracks. The extracted crack skeleton was used to measure the characteristic parameters of road diseases. The mathematical results show that the novel method, compared with several competing methods, performs well, although it can be improved in future works, especially in the estimation of the area and the maximum width of the crack.

It should also be pointed out that the method was only performed on accurate static images; video streaming was not considered. Therefore, tests on video databases are planned for future projects. Another important issue concerns the computational speed in crack detection: it will be necessary to focus on the possibility of increasing it. It should also be noted that the adopted algorithm requires manually labeling each crack pixel of ground truth images, which makes data acquisition very expensive: unsupervised and semi-supervised learning-based techniques can mitigate this problem. The determination of the hyper parameter of the final module also takes a lot of time, and further studies will focus on evolutionary computation to optimize the hyper parameters of the deep learning module. Finally, other recent machine learning techniques and optimization methods (e.g., [58,59]) will be investigated and incorporated into the proposed approaches in order to obtain even more accurate results.