Mixed Reality-Based Concrete Crack Detection and Skeleton Extraction Using Deep Learning and Image Processing

Abstract

Advancements in image processing and deep learning offer considerable opportunities for automated defect assessment in civil structures. However, these systems cannot work interactively with human inspectors. Mixed reality (MR) can be adopted to address this by involving inspectors in various stages of the assessment process. This paper integrates You Only Look Once (YOLO) v5n and YOLO v5m with the Canny algorithm for real-time concrete crack detection and skeleton extraction with a Microsoft HoloLens 2 MR device. The YOLO v5n demonstrates a superior mean average precision (mAP) 0.5 and speed, while YOLO v5m achieves the highest mAP 0.5 0.95 among the other YOLO v5 structures. The Canny algorithm also outperforms the Sobel and Prewitt edge detectors with the highest F1 score. The developed MR-based system could not only be employed for real-time defect assessment but also be utilized for the automatic recording of the location and other specifications of the cracks for further analysis and future re-inspections.

1. Introduction

Structures such as buildings, bridges, and pavements are subject to fatigue stresses, deterioration, and damage [1,2]. Cracks are among the major indices of the degradation and poor technical conditions of the structures, which can be formed and propagated in different types of construction materials such as concrete, brick, slab, asphalt, and beam [3,4]. Different factors such as aging, cyclic loading, poor workmanship and design, environmental parameters, and natural disasters can lead to the development of cracks [5,6,7]. Defect assessment and the maintenance of structures are extremely expensive exercises [6]. Accordingly, the early, fast, and efficient detection of cracks provides an opportunity to prevent further damage and minimize the risk of failure of building objects by taking necessary measures in a timely manner [4,5]. If cracks remain unknown, they can spread on the surface of structures and finally collapse them, which may cause various financial and safety problems [6].

Traditional approaches to defect assessment and crack detection are based on visual inspection. Such manual surveys, however, are associated with different challenges, including the following:

• A high dependency on the experience and skill set of individual inspectors;
• The subjective judgments of building surveyors;
• The need for excessive time, cost, and labor;
• Placing inspectors in dangerous situations [3,8,9].

To address these challenges, the literature emphasizes the increasing trend of the application of automatic procedures developed based on image processing (IP), computer vision, and Artificial Intelligence (AI) for the accurate and rapid classification, detection, and segmentation of cracks [3,10]. These cutting-edge technologies support the efficient evaluation and detection of surface defects to ensure the reliability of structure services, public safety, and financial stability [10]. Edge detection methods in IP, such as the Canny and Sobel operators, have been widely used for the automatic extraction of the skeleton of the cracks of concrete surfaces [11,12,13]. However, using IP methods alone may not lead to very high accuracies in crack detection due to factors such as a high sensitivity to the quality of the used images, noisy outputs, and a dependency on the appropriate selection of threshold parameters [14,15,16]. Accordingly, the application of Machine Learning (ML) and deep learning (DL) techniques has been considered to increase the accuracy of automatic crack detection practices [17,18]. A convolutional neural network (CNN) is the most developed and widely used DL method for various purposes such as classification, segmentation, recognition, detection, etc. [19]. In terms of defect assessment, a CNN is utilized for the extraction of deep convolutional features from crack and non-crack visual data and images through learning techniques [20]. Since the mid-2000s, deep CNNs (DCNNs) have been used to develop efficient and fast damage detection systems for civil structures [15]. In addition, more sophisticated CNN-based methods have been developed both for crack detection, e.g., the region-based CNN (RCNN) family, a Single Shot Detector (SSD), and the You Only Look Once (YOLO) family, and segmentation, e.g., U-Net, SegNet, and DenseNet [17,21,22].

For the development of state-of-the-art crack detection algorithms, the required visual input data can be provided by digital cameras, smartphones [23], Unmanned Aerial Vehicles (UAVs) [10], laser scanners [24], Ground Penetrating Radar (GPR) devices [25], etc. However, the developed techniques only provide input data for human inspectors. Despite the mentioned limitations regarding manual surveys, the inspectors’ expertise and experiences are critical [26]. Therefore, another solution could be the integration of human capabilities with IP and AI-based crack detection techniques in a systematic manner [27] to tackle the main challenges of manual inspections [28]. Accordingly, augmented reality (AR) and mixed reality (MR) can provide a great opportunity to capture, visualize, and process visual data while involving human expertise in different stages [29,30]. Among different devices that have been developed for AR and MR, Microsoft HoloLens has been introduced as a tool for the monitoring, maintenance, and management of civil structures. It has been equipped with different cameras and sensors to provide inputs and a powerful Central Processing Unit (CPU) and Graphics Processing Unit (GPU) for the automatic detection of cracks [26,30,31,32,33].

Considering the importance of developing advanced crack detection methods while benefiting from human proficiency, this paper aims to develop an integrated crack detection and skeleton extraction method based on IP and DL through MR capabilities. Previous studies have demonstrated the high accuracy of DL-based crack detection and segmentation models in structural images by using semantic segmentation networks and specialized convolutional models [34,35,36,37,38]. While these approaches achieve significant accuracy in controlled environments, they are computationally intensive, often requiring dedicated GPU resources and processing times that are impractical for real-time applications on MR devices such as the Microsoft HoloLens 2. In particular, end-to-end segmentation models like Mask RCNN and U-Net necessitate substantial processing power, which may reduce the frame rate and negatively impact the user experience in real time [39]. Therefore, this study adopts a hybrid approach, integrating CNN-based crack detection with IP-based edge detection for skeleton extraction. This combination leverages the strengths of DL for object detection and the efficiency of IP algorithms for edge detection, optimizing real-time performance within hardware constraints. Through this integrated approach, we develop an interactive HoloLens application that empowers human inspectors to facilitate automatic crack detection. The primary contributions of this paper are as follows:

• The introduction of an MR-based interactive system using Microsoft HoloLens 2. This system enables human inspectors to perform real-time concrete crack detection and skeleton extraction, enhancing the assessment process.
• The integration of DL and IP methods: combining these techniques facilitates the real-time assessment of concrete surfaces, balancing high accuracy and processing speed.
• Comprehensive performance analysis and model comparison. This paper evaluates different models to identify the most accurate and fastest solutions for integration into the MR system.

The remainder of this paper is organized as follows. In Section 2, related works are reviewed. Section 3 introduces the proposed method based on the integration of the YOLO v5 model for crack detection and different considered IP-based edge detectors. Section 4 discusses the transformation of the designed integrated defect assessment of concrete surfaces to the MR environment with Microsoft HoloLens 2. Section 5 provides the experimental settings and evaluation results of the considered methods, followed by related discussions. Finally, the conclusion and future work are presented in Section 6.

2. Related Works

This section reviews related works to crack detection using DL, crack edge detection using IP, and possible applications of AR and MR in the defect assessment of concrete structures. These three concepts are the main foundations of this paper, and they provide an integrated crack detection and skeleton extraction model through an MR-based system.

2.1. Crack Detection Using Deep Learning

Deep learning (DL), as a branch of Machine Learning (ML), uses multilayered neural networks (MNNs) for learning data features and feature extraction [17]. The term “deep” represents the large number of layers between the input and output layers [21]. DL methods have been used in different fields of science for classification, segmentation, recognition, detection, identification, sentiment analysis, prediction, social media services, medical services, robotics, machine translation, and surveillance with promising results [19]. With regard to crack detection, DL has the following capabilities:

• Classification: labelling the image patch to crack or non-crack;
• Detection: localization of the crack by drawing a bounding box around the crack region;
• Segmentation: dividing the image pixels into crack and non-crack pixels [16,21].

DL-based crack detection methods have less dependency on the quality of the input images, less sensitivity to noise, and a higher speed [16]. In DL, a CNN, introduced in 1980 [40], is the most established, innovative, and widely used neural network structure for image analysis [41].

For crack detection, three major deep learning architectures have been used for the localization of the cracks (i.e., drawing a bounding box around the crack region), including the RCNN family, SSD, and the YOLO family. RCNNs, Fast RCNNs, Faster RCNNs, and Mask RCNNs are members of the RCNN family [17,42]. An RCNN is a two-step detection process. In the first, the region proposal algorithm is utilized in order to generate the region proposals with the possibility of containing an object. Subsequently, the CNN features are extracted from the region proposals, and the objects are classified using the extracted features to obtain the final bounding boxes [43]. Kim et al. [44] used an RCNN for the crack detection of concrete bridges based on UAV images. Concrete road crack detection was also carried out using the Faster RCNN method by Hacıefendioğlu and Başağa [45]. Attard et al. [46] utilized a Mask RCNN to localize the cracks on concrete surfaces. In spite of the high accuracy of the Faster RCNN algorithm in crack detection, with a higher speed in comparison to other members of the RCNN family, it is still difficult to achieve real-time detection due to the large amount of time required to train the network through a two-step detection process [43].

The SSD and YOLO families are among other types of CNN-based detection algorithms that were introduced to perform all the computations in a single network instead of considering an extra module to discover the candidate regions of objects in the image space. This process is called “one-stage detection” [47,48]. For the crack detection aim, Yan and Zhang [49] used an SSD for asphalt highway pavement crack detection and highlighted the higher accuracy and speed of the one-stage detection method compared to two-stage object detection networks like the RCNN family.

2.1.1. YOLO for the Crack Detection of Civil Structures

After the first presentation of YOLO as a new approach for object detection in 2015 [50], many researchers have sought to investigate the application of YOLO for the detection of cracks in civil structures such as concrete walls [51], pavements [52,53], and bridges [43]. In addition to having a much higher speed in YOLO compared to a Faster RCNN, the accuracy of YOLO is significantly higher than a Faster RCNN, especially in some cases [54,55,56,57,58]. Furthermore, some authors pointed out the superiority of YOLO for gaining higher accuracies in civil crack localization in comparison to an SSD [54,58,59]. By a comparison of the ability of 11 well-known CNN models for concrete crack detection, Teng et al. [60] asserted that the YOLO method can lead to outstanding detection results if an appropriate feature extractor model and layer, training epoch, and testing image size are selected.

2.1.2. YOLO v5

YOLO v5 is a version of the YOLO architecture series with a very high detection accuracy, fast inference speed, and the fastest detection speed, which is up to 140 frames per second [61]. Some recent studies have attempted to investigate the capabilities of YOLO v5 for the detection of cracks in civil structures such as concrete structures and pavements.

Regarding the detection of cracks in asphalt pavements, Liu et al. [62] developed a transfer learning method called the Deep Domain Adaptation-based Crack Detection Network (DDACDN) based on YOLO v5 and highlighted its detection accuracy in comparison to other methods. Li, Gu, Xu, Xu, Zhang, Liu, and Dong [52] investigated the accuracy and speed of different versions of YOLO for the detection of concealed cracks in asphalt pavements using GPR data. Accordingly, they highlighted the highest accuracy for YOLO v5 and the highest speed for YOLO v4. In addition, they indicated that, among different series of YOLO v5, the small model called “YOLO v5s” has the highest speed. The higher speed of YOLO v5s for detecting cracks in pavements was also demonstrated in a study conducted by Hu et al. [63]. They compared different series of YOLO v5 and also asserted a higher accuracy for YOLO v5x.

In relation to defect assessment in concrete surfaces, Agyemang et al. [64] improved YOLO v5 using a non-local means (NLM) filter and feature refinement capabilities of a Convolutional Block Attention Module (CBAM) and highlighted the accuracy of the model for concrete crack detection. Zhao et al. [65] also detected the crack regions in images of concrete structures by YOLO v5 using bounding boxes and segmented the cracks by a Crack Feature Pyramid Network (FPN). Finally, an integrated approach for crack detection using YOLO v5 and crack quantification using IP led to a very high accuracy in a study conducted by Yu et al. [66].

2.2. Crack Edge Detection Using Image Processing

IP techniques detect cracks through the utilization of different filters, thresholds, morphological analysis, statistical methods, and percolation procedures [6]. The general structure of IP methods for crack detection includes image acquisition, image pre-processing, image processing, crack detection, and crack feature extraction [2]. Edge detection, thresholding and segmentation, region growing, and percolation-based techniques are different aspects of crack detection using IP [21].

For the skeleton extraction of cracks, IP-based edge detection can be used. Edge detectors are mathematical algorithms that are employed to identify the pixels in images at which the gray-level intensity varies dramatically. These pixels can be grouped into areas called edges. Since crack locations in concrete surfaces are highly associated with pixels having a discontinued gray-level intensity, edge detection methods can be significantly appropriate for identifying cracks in such surfaces [13].

IP techniques have been investigated in various studies to assess their capabilities for the recognition of the edges of cracks in civil structures, like pavements [67] and concrete surfaces [13], and have obtained promising results. These edge detection algorithms include the Canny [68], Sobel [69], Prewitt [13], Roberts [70], Fourier Transform [11], Wavelet Transform [71], Fast Haar Transform [11], Gaussian high-pass filter [70], bidimensional empirical mode decomposition (BEMD) [72], and Laplace edge detection [73] methods.

Many authors emphasize that crack detection using IP is a challenging process since cracks possess poor continuity and low contrast among neighboring pixels [19]. However, if the cracks are detected and the bounding boxes are drawn around them using DL technology, IP techniques are able to identify the skeleton of the cracks inside the bounding boxes with a high accuracy and a high speed [13,66].

2.3. AR and MR

During the last decades, state-of-the-art technologies such as AR and MR have promoted digitization and automation in the simulation and visualization of information [74]. AR superimposes virtual objects into the real world. In an AR system, the physical world is supplemented with virtual elements as computer-generated content in the same space [75,76,77]. MR, however, takes the process a step further and embeds the virtual content into the real world in such a way that users can interact with it [75]. Figure 1 illustrates the difference between AR and MR. In AR, digital information overlays the real world, enhancing the environment without enabling direct interaction between real and virtual elements. For instance, Pokémon GO exemplifies AR by layering digital objects on physical surroundings but without deeper interaction. MR, on the other hand, integrates real and virtual elements to create an interactive experience. This allows users to engage directly with both digital and physical elements, manipulating them in real time through advanced sensing and imaging technologies. MR thus enables a more immersive experience, where users can simultaneously be present in both real and virtual environments, as shown in the figure. This capability opens up new possibilities in areas like gaming, professional work, and, as in our study, defect inspection by merging human expertise with digital automation in real-world settings [78].

Milgram and Kishino [79] were the first to introduce and define MR as a particular subset of immersive technologies while incorporating the virtual world within real environments in a “reality–virtuality continuum” context. Since then, different research and varied relevant technologies have been developed, but there is no universally agreed definition for MR [80]. Microsoft HoloLens is the most widely used device for the development of MR-based applications. It is the first self-contained holographic computer device that provides users with the experience of astonishing holographic representations, engaging with high-fidelity holographic three-dimensional (3D) models, and interacting with a hybrid reality [74,81].

2.3.1. AR and MR in Civil Engineering

In the field of building industry and civil engineering, some researchers have attempted to investigate the capabilities of AR and MR for design, engineering, construction, and facility management. Kamat Vineet and El-Tawil [82] studied the practical application of AR for assessing earthquake-induced building damage. In the same year, the application of the Global Positioning System (GPS) and Three Degrees of Freedom (3DoF) angular tracking for addressing the registration problem during the interactive visualization of construction graphics in outdoor AR environments was investigated [83]. Bae et al. [84] presented a context-aware, vision-based mobile augmented reality system to enable field teams to query and access 3D information on-site. Later, a conceptual design was developed to improve structural inspections using the HoloLens [85]. Baek et al. [86] developed an AR-based system for facility management based on an image-based indoor localization method. The presented system has the ability to estimate the indoor position and orientation of the users by comparing their perspective to Building Information Modelling (BIM) using DL. Napolitano, Liu, Sun, and Glisic [29] presented a framework aiming at the documentation and visualization of the data inside and outside of the built environment based on the fusion of image-based documentation and AR.

2.3.2. AR and MR for Defect Detection in Civil Structures

Specifically for defect detection and maintenance operations using AR and MR, scholars have recently developed some frameworks and methodologies.

Table 1. Studies focusing on the application of AR and MR for the crack detection of civil structures.

Source	Aim	Tools and Models
[87]	Presenting a framework for proactive construction defect management.	Integration of AR, BIM, and ontology-based data collection template.
[26]	MR for smart assessment of bridge cracks.	HoloLens and SSD for crack detection and SegNet for crack segmentation.
[30]	Detection of cracks in concrete structures through MR.	HoloLens and computer vison (feature extraction of matching).
[27]	Real-time automated damage detection using AR smart glasses.	Hands-free Epson BT-300 smart glasses and SSD MobileNet clustering for crack detection and Mask RCNN for segmentation.
[88]	Presenting an innovative bridge maintenance system for storing, manipulating, and sharing inspection data and maintenance history.	HoloLens, BIM, and image processing.
[89]	Using AR for bridge condition assessment and the detection of bridge deck delamination, crack formation, and rebar corrosion.	GPR, Laser Distance Sensors (LDSs), and Infra-Red Thermography (IRT) cameras.

provides the aims, tools, and models in these studies. The point that should be highlighted here is that, despite emphasizing the importance and capabilities of MR-related devices, especially Microsoft HoloLens, the practical use of these technologies, while benefiting from advanced crack detection algorithms in DL and IP, is limited. The lack of a practical implementation and evaluation of the proposed models, the lack of providing information on the accuracy and speed of the developed models, a low accuracy, and the lack of the application of advanced crack detection methods based on IP and DL are among the different limitations of these relevant studies. Accordingly, further studies are required in the field to develop advanced crack detection methods using AR and MR capabilities to incorporate human expertise into the automated detection process with a high accuracy and speed.

3. An Integrated Method for Crack Detection and Skeleton Extraction

As previously mentioned, implementing a fully end-to-end DL approach for both crack detection and edge extraction can be computationally intensive and may reduce processing speed, especially on devices with limited hardware capacity, such as the Microsoft HoloLens 2. IP-based crack detection algorithms have been found to have lower accuracies in comparison to DL techniques. However, if the cracks are detected by advanced DL methods like YOLO, the crack edge detection methods in IP could be employed to extract the skeleton of the detected cracks by the DL techniques with an acceptable accuracy and a high speed [13]. Hence, developing an integrated DL and IP-based approach for crack detection and quantification can lead to promising results [66]. Accordingly, to achieve a balance between speed and accuracy for real-time inspection, this study attempts to develop an integrated methodology for crack detection and skeleton extraction (Figure 2) and subsequently implement the developed model in a Microsoft HoloLens.

In order to achieve this aim, different series of YOLO v5 are first adopted for real-time crack detection. YOLO v5 is selected as the primary DL model for crack detection due to its well-established balance of accuracy, processing speed, and computational efficiency [90], which are critical for real-time applications on resource-constrained MR devices like the Microsoft HoloLens 2. While newer versions of YOLO have been recently introduced, their capabilities in crack detection tasks have not yet been thoroughly investigated. YOLO v5’s extensive validation in recent crack detection studies [91,92,93,94] further supports its reliability and suitability for our application. By using YOLO v5, we achieve an effective trade-off between detection accuracy and real-time processing requirements, ensuring a high performance and responsiveness in the MR environment.

Subsequently, after crack detection, the performance of the three edge detection algorithms of Canny, Sobel, and Prewitt for the skeleton extraction of the detected cracks are compared to choose the most accurate edge detector. Indeed, after edge detection using YOLO v5 and drawing bounding boxes around the cracks, the selected edge detector is used for the extraction of the skeleton of the cracks inside the bounding boxes. The YOLO v5 and these edge detection methods will be further described in the following sub-sections.

3.1. YOLO v5 Algorithm Network

For the detection of objects and drawing a bounding box around them, the YOLO v5 utilizes four main stages in its architecture, including the input, backbone, neck, and head [65] (Figure 3). These stages will be further discussed below:

• In the input part, mosaic data augmentation, adaptive tracing frame calculation, and adaptive picture scaling are followed to enrich the surface dataset, calculate the optimal anchor point box, and avoid creating a lot of redundant information, respectively. These steps will improve training efficiency [63,95].
• The backbone module focuses on the feature extraction of the input image data. It is based on the focus structure, Cross Stage Partial Network (CSPNet), and Spatial Pyramid Pooling (SPP). The focus structure is mainly used for image slicing. The CSPNet is used to extract abundant information features from the input image. Finally, the SPP converts feature maps of all sizes into feature vectors with fixed sizes, thereby enhancing the acceptance field of the network and capturing functions of various sizes [61,95].
• The neck module of the YOLO v5 is responsible for performing the multi-scale feature fusion. It is established using FPN and Path Aggregation Network (PAN) structures that are mainly used to generate a feature pyramid. The FPN is a top–down structure that is used to fuse the feature information with the backbone network part to realize the communication of semantic features, whereas the PAN is a bottom–up structure that is used to realize the communication of strong localization feature information [63,66,96].
• The head module of the YOLO v5 is the final detection stage. Anchor boxes are used in this part to build final output vectors with class probabilities, objectness scores, and bounding boxes. Based on the Intersection Over Union (IOU) (Equation (1)), the generalized Intersection Over Union (GIOU) (Equation (2)) is employed as the loss function of the bounding box during the prediction step [95,96].

  $$IOU={\displaystyle \frac{\left|A\cap B\right|}{\left|A\cup B\right|}}$$

  (1)

  where $A$ is the target box and $B$ is the prediction box (Figure 4).

  $$GIOU=IOU-{\displaystyle \frac{\left|\frac{C}{A\cup B}\right|}{\left|C\right|}}$$

  (2)

  where $C$ is the minimum closed box that encloses both $A$ and $B$.

YOLO v5 is divided into different basic network structures based on network depth and width, including YOLO v5n (nano), YOLO v5s (small), YOLO v5m (medium), YOLO v5l (large), and YOLO v5x (extra-large). This categorization is based on the width and depth of the BottleneckCSP module, while the backbone, neck, and head are kept the same [61]. Out of these network structures, Zhao et al. [97] introduced YOLO v5s as the best architecture for real-time defect detection, considering that it met the real-time requirements of the production line. In this study, the possible application of different types of YOLO v5 for crack detection using MR is analyzed in terms of speed and accuracy to employ the best and most compatible model with the Microsoft HoloLens 2.

3.2. Image Processing-Based Crack Edge Detection

This study assesses the application of the Canny, Sobel, and Prewitt edge detection methods to select the best detector to be integrated with the YOLO v5 crack detection model. Short descriptions of these edge detection algorithms are presented in this section.

3.2.1. Sobel

The Sobel edge detector [98] recognizes edges by using two 3 × 3 convolution kernels of $G_x$ (Equation (3)) and $G_y$ (Equation (4)) to compute the derivatives in both the x and y directions, respectively.

$$G_x=\left[\begin{array}{ccc}1& 0& -1\\ 2& 0& -2\\ 1& 0& -1\end{array}\right]$$

(3)

$$G_y=\left[\begin{array}{ccc}1& 2& 1\\ 0& 0& 0\\ -1& -2& -1\end{array}\right]$$

(4)

The final outputs of the gradient approximations are combined to provide the gradient magnitude using Equation (5).

$$G_s=\sqrt{{(G_x·N_{IS})}^2+{(G_y·N_{IS})}^2}$$

(5)

where $N_{IS}$ denotes an image neighborhood with a size of 3 × 3 pixels.

Furthermore, a threshold value TS identifies the presentation of the output image with all edges being displayed. If the Sobel gradient values of pixels are less than the threshold value TS, they are substituted by the threshold value [13].

3.2.2. Canny

The Canny edge detector [99] employs a multi-step algorithm in order to detect a wide range of edges and concurrently suppress noises in the images. At the first step of the edge detection, a Gaussian filter (Equation (6)) is applied for image smoothing and removing the noise and redundant details or textures.

$$g\left(m,n\right)=G_{\sigma }\left(m,n\right)\times f(m,n)$$

(6)

where $G_{\sigma }$ is a Gaussian function with the variance of ${\sigma }^2$ as Equation (7), $m$ and $n$ are the indices used to specify the location of a pixel within an image, and $f(m,n)$ represents an image neighborhood at the pixel coordinate of $(m,n)$.

$$G_{\sigma }={\displaystyle \frac{1}{\sqrt{{2\pi \sigma }^2}}}\mathrm{e}\mathrm{x}\mathrm{p}(-{\displaystyle \frac{m^2+n^2}{{2\sigma }^2}})$$

(7)

In the next step, the gradient of $g\left(m,n\right)$, using a certain gradient operator (e.g., Sobel), is computed as Equation (8).

$$g_{m,n}(m,n)=\sqrt{g_m^2(m,n)+g_n^2(m,n)}$$

(8)

Subsequently, a non-maximum suppression is performed to convert the blurred edges in the image of the gradient magnitudes to sharp ones. After the suppression step, it is still possible that some pixels that are noise or color variations have been incorrectly specified as edges. Hence, a double thresholding strategy is considered. Accordingly, the edge pixels that are stronger than the upper threshold T2 are marked as strong edges. Meanwhile, those edge pixels that are weaker than the lower threshold T1 are suppressed. Furthermore, the edge pixels that range between T1 and T2 are identified as weak edges. Finally, the Canny algorithm suppresses all the edge pixels not connected to the strong edge pixels, which belong to the weak edge group [100].

3.2.3. Prewitt

The Prewitt algorithm was first developed by Prewitt [101]. It is based on two 3 × 3 filters, which facilitate the estimation of the derivatives of each location within an image: one for horizontal (Equation (9)) and one for vertical changes (Equation (10)).

$$P_x=\left[\begin{array}{ccc}-1& -1& -1\\ 0& 0& 0\\ 1& 1& 1\end{array}\right]$$

(9)

$$P_y=\left[\begin{array}{ccc}-1& 0& 1\\ -1& 0& 1\\ -1& 0& 1\end{array}\right]$$

(10)

These two horizontal and vertical filters are combined to calculate the gradient magnitude using Equation (11).

$$P_p=\sqrt{{(P_x·N_{IP})}^2+{(P_y·N_{IP})}^2}$$

(11)

where $N_{IP}$ denotes the source image [13].

4. Transformation of the Model to the HoloLens

The experiments were conducted using the Microsoft HoloLens 2, an advanced MR headset equipped with a Qualcomm Snapdragon 850 Compute Platform and a custom Holographic Processing Unit. It features 4 GB of DRAM and 64 GB of UFS 2.1 storage, designed to meet the extensive computational demands of spatial computing and interactive applications. The visual output is facilitated by a high-resolution 2k 3:2 light engine, which delivers a resolution of 1440 × 936 pixels per eye. The field of view provided by the device is 43 degrees horizontally, enhancing the immersive experience necessary for detailed visual analysis. The HoloLens 2 also includes an RGB camera with a resolution of 2272 × 1278, utilized for capturing the video essential for concrete crack detection. This setup is critical for the accurate operation of deep learning models and image processing algorithms in our research.

For the implementation of the integrated crack detection and skeleton extraction model in the Microsoft HoloLens 2, first, the YOLO v5 detection models were trained through the PyTorch 1.11.0 framework [102]. Subsequently, the trained PyTorch models were transformed into the Open Neural Network Exchange (ONNX) format, which is a lightweight model format compatible with the HoloLens and Unity real-time development platform [103]. To test the YOLO v5 models, a C# 9.0 with .NET 5.0 application was implemented with the post-processes, including parsing the detection result and applying a non-maximum suppression.

Since the nature, length, thickness, and other characteristics of the cracks could be different, the MR-based system was developed in such a way that, if necessary, a combination of different real-time crack detection and skeleton extraction models could be used. To be more specific, the accuracy and speed of different types of YOLO v5 methods for crack detection and different considered IP-based edge detection algorithms were assessed. The real-time system was subsequently developed in such a way to provide the opportunity for users to change the models and compare the results on the fly. It meant users could swap between the models in HoloLens. The MR application was developed in Unity, which has capabilities for integrating various applications in HoloLens.

5. Experimental Analysis, Results, and Discussion

This section analyzes and evaluates the performance of the proposed MR-based crack edge detection and skeleton extraction algorithm.

5.1. Dataset Preparation

Two datasets were used in this study: one dataset for the training, validation, and testing of the YOLO v5 crack detection algorithm and one dataset for the evaluation of the edge detection methods. These two datasets are further discussed in the following sub-sections.

5.1.1. Dataset for Crack Detection

For the acquisition of the images required for crack detection using DL, the initial image dataset, including 410 images at a resolution of 256 × 256 pixels, was obtained through Bianchi and Hebdon [104] and Maguire et al. [105]. The input images were then manually annotated. Larger input images can improve the training speed and accuracy of the model [54]. Hence, in order to expand the dataset, some augmentation methods were automatically applied to the initial images as advised by different authors to enhance the performance of the models, including the following:

• Flip: horizontal and vertical [106];
• Grayscale [107];
• Brightness [108]: between −25% and +25%;
• Exposure [109]: between −15% and +15%;
• Blur [110]: up to 0.5 px.

These augmentation processes not only increased the number of input images to 996 but also provided images in the dataset with different lighting conditions. The latter helped to maximize the training accuracy of the detection algorithm under various complex lighting conditions and to avoid over-fitting [111].

After the finalization of the image dataset for crack detection using YOLO v5, 78% of the images in the dataset were randomly selected for the training of the model, 8% of the images were selected for the validation of the model, and the remaining images were used for the testing. Figure 5 shows some of the images used for crack detection using YOLO v5 and their annotated bounding boxes with different conditions.

5.1.2. Dataset for Crack Skeleton Extraction

For the evaluation of the edge detection algorithms, another dataset was utilized, including 444 images, which were downloaded from [112]. The dataset includes 4032 × 3024 high-resolution images together with their alpha maps indicating the crack presence. Figure 6 shows a sample of the used images with their relevant ground truths.

5.2. Assessment Metrics

An accuracy assessment of both the crack detection (bounding box identification) and crack skeleton extraction procedures can be carried out using the classical evaluation metrics of precision (P) (Equation (12)), recall (R) (Equation (13)), and F1 score (Equation (14)).

$$P={\displaystyle \frac{TP}{TP+FP}}$$

(12)

$$R={\displaystyle \frac{TP}{TP+FN}}$$

(13)

$$F1={\displaystyle \frac{2\times P\times R}{P+R}}$$

(14)

where $TP$ (true positive) is the number of the target boxes that are correctly predicted as defects, $FP$ (false positive) is the number of the boxes that are incorrectly predicted as defects, and $FN$ (false negative) is the number of the target boxes that are incorrectly predicted as non-defects [54,63]. Typically, an IOU ≥ 0.5 is referred to as a positive detection (TP), whereas an IOU < 0.5 means a false detection (FP) [16]. $P$ identifies the number of images that were correctly labelled among all the images labelled positive (including TP and FP) by the classifier, while recall specifies the number of all the positive images in the dataset that were correctly labelled as positive. In addition, the F1 score expresses the performance of the detection algorithm by calculating the harmonic mean of the P and R rates, with the best value equal to 1 and the worst value equal to 0 [113].

Average precision (AP) is also the accuracy of all the categories, standing for the area enclosed by the precision–recall (PR) curve and the coordinate axes. Indeed, the PR curve is a graph with P values on the y-axis and R values on the x-axis. It is calculated as the weighted mean of the precisions achieved at each threshold, with the increase in R from the previous threshold used as the weight. The mean AP (mAP) is subsequently computed as the mean of the AP of all damage modes [54]. The mAP 0.5 is calculated when the IOU ≥ 0.5 and is referred to as a TP. Furthermore, the mAP 0.5 0.95 corresponds to the average AP for the IOU from 0.5 to 0.95 with a step size of 0.05. Both the mAP 0.5 and mAP 0.5 0.95 are considered for the evaluation of the different types of YOLO v5 crack detection methods.

5.3. Model Implementation and Testing

5.3.1. Crack Detection

Considering the aim of this study being real-time crack detection and the limitations of implementing large networks in Microsoft HoloLens 2, three smaller structures and faster types of YOLO v5, including YOLO v5n, YOLO v5s, and YOLO v5m, were selected to be run.

In the first step, the models were pre-trained, and it was found that their loss functions stopped decreasing significantly in about 180 epochs. This is roughly the number of times that the YOLO v5 models explore the whole training dataset. It means they stop learning from the data. Since too many training epochs can lead to over-fitting issues, the models were trained with 150 epochs. Figure 7, Figure 8 and Figure 9 present the changes in the mAP 0.5 and mAP 0.5 0.95 of the training process after the increase in the number of epochs for the three structures of nano, small, and middle, respectively.

After the training and validation of the models, they were tested, and the results are provided in

Table 2. The results of comparing the YOLO v5n, YOLO v5s, and YOLO v5m crack detection methods.

Method	mAP 0.5	mAP 0.5 0.95	Speed (FPS)
YOLO v5n	0.927	0.701	4.5
YOLO v5s	0.895	0.702	3
YOLO v5m	0.890	0.728	1

. Accordingly, YOLO v5n was associated with the highest mAP 0.5 and the highest speed. However, in terms of the mAP 0.5 0.95, YOLO v5m showed the highest accuracy. Indeed, YOLO v5n was the fastest model and the best model in terms of the ability to find cracks, while YOLO v5m was the most accurate model for crack detection. Hence, it was decided to implement both of these models and provide the possibility to the user to try both models in the field. Figure 10 provides an example of the bounding boxes identified around the detected cracks in different images by Yolo v5n (orange), Yolo v5s (blue), and Yolo v5m (green) models.

Our system has been optimized to detect cracks with widths as narrow as 0.06 mm within the Australian standard range, which classifies fine cracks in concrete structures up to 1 mm in width [114]. Additionally, the system operates effectively within the recommended viewing range for the Microsoft HoloLens 2, specifically between 1.25 and 5 m, with an optimal focal distance of approximately 2 m [114]. This range adheres to the HoloLens 2 specifications, maintaining visual clarity and user comfort while enabling accurate and responsive crack detection, as shown in Table 2 and Figure 10.

5.3.2. Crack Edge Extraction

All three considered edge detectors were applied on the dataset prepared, and their P, R, and F1 scores were computed (

Table 3. The results of comparing the Sobel, Canny, and Prewitt crack edge detectors.

Algorithm	Precision (P)	Recall (R)	F1 Score
Sobel	0.89	0.70	0.77
Canny	0.93	0.87	0.88
Prewitt	0.94	0.55	0.67

). Accordingly, the Canny algorithm led to the highest F1 score. According to Figure 11, which provides the analysis of the P, R, and F1 scores of the three edge detectors, the boxplot comparison of the F1 score and R illustrates that the Inter-Quartile Range (IQR) of the Canny algorithm was shorter. This indicates that the dispersion of the F1 score and p value of the Canny algorithm was smaller than the other two algorithms. This demonstrates a more stable performance of the Canny algorithm. Although the Prewitt algorithm had a better result in the precision box plot, its R and, accordingly, F1 score was very low. The Canny algorithm’s P was also close to the P of Prewitt. Hence, the Canny algorithm was selected to be implemented in the HoloLens 2 for the extraction of the skeleton of the cracks identified by the selected detection methods of YOLO v5n and YOLO v5m.

Figure 12 also provides an example of the outputs of the Sobel, Canny, and Prewitt algorithms for the extraction of the skeleton of the cracks in an image and inside the bounding box based on the output of the applied DL-based crack detection. This figure represents the efficiency of Canny in comparison to the other two methods, since their detections have a high level of noise. On the contrary, the Canny filter leads to less noise, especially inside the bounding box specified around the crack by YOLO v5. Accordingly, the Canny algorithm was selected as the edge detector technique for the extraction of the detected cracks. This indicates the reliability of the developed integrated method in this paper. Indeed, by edge detection using YOLO v5, the cracks are detected with a high accuracy and bounding boxes are drawn around the detected cracks. Subsequently, the skeletons of the cracks are extracted using the Canny algorithm inside the bounding boxes to avoid noise, which is the main error associated with IP-based edge detectors.

5.3.3. Integrated Model in the HoloLens

The MR application was developed in Unity by integrating the YOLO v5n and YOLO v5m crack detection methods and the Canny edge detector with the capability for users to change the crack detection models and compare the detection results of the two models of nano and middle in the field. Accordingly, by wearing the HoloLens 2, the inspector would be able to observe the real world through special glasses, while, simultaneously, with the help of the model developed in the machine, scanning the concrete surfaces and detecting possible cracks and their edges. Therefore, the use of this system can detect and analyze cracks in an MR environment in a real-time mode and provide insight to users for further assessments.

Figure 13 provides some examples of testing the implemented model in the HoloLens 2. In this figure, some cracks in concrete walls and the corresponding detection of those cracks using the HoloLens are presented. For this aim, the real-time experiment of defect assessment in concrete surfaces using the developed MR system was carried out in a building. The videos were recorded through the HoloLens and the corresponding photos were taken with a mobile phone for comparison and visual evaluation. Figure 14 also provides an example of the detection of a crack using both the YOLO v5n and YOLO v5m methods and edge detection using the Canny algorithm in the HoloLens 2.

Despite the acceptable accuracy of the system in detecting cracks in concrete surfaces during practical testing, some limitations were also observed. First, the accuracy of the Canny algorithm varies when the user’s distance and angle to the surface change. Indeed, the best edge detection occurs when the individual looks directly at the crack at a distance of about one meter. In addition, factors such as noise, a lack of sufficient light, and the width of the crack can decrease the accuracy of the Canny algorithm in extracting the edges of the cracks. In the photo on the right in Figure 13, it is observed how shadow and a lack of light can negatively affect the crack edge detection process.

Device movement and stability are critical factors in the performance of MR applications. Our results, detailed in Table 2, show that the YOLO v5n with Canny achieved 4.5 fps, resulting in a new inference every 222 ms approximately, while the YOLO v5s with Canny achieved 3 fps, with a new inference every 333 ms approximately. The YOLO v5n with Canny, at a lower rate of 1 fps, presented a new inference every 1000 ms. Despite the variations in frame rates, the high fps configurations (v5n and v5s) meant that typical device movements, such as head shaking, did not significantly affect the detection results. However, in the v5s configuration, rapid movements of the head and device led to some degradation in detection accuracy, indicating that more intense or quicker movements can still pose challenges to detection stability in the HoloLens 2.

Generally, in the real experiment, both the YOLO v5 nano and middle methods showed a very high level of accuracy. However, the edge detection accuracy of the Canny algorithm was not as high as expected based on the results of the accuracy assessment using the P, R, and F1 score metrics. An alternative solution could be to use DL for both crack detection and segmentation. Nevertheless, the YOLO v5n model, as the smallest and fastest type of YOLO v5 requires 0.23 s to detect cracks in one frame. We estimate that an AI-based segmentation model with a similar performance would require an even longer time to identify the skeleton of the cracks. Therefore, by the integration of two AI-based models, one for detection and one for segmentation, a software cannot run better than 2 FPS in the HoloLens 2 considering the configuration of it based on the Qualcomm Snapdragon 850 Compute Platform SOC [115]. Accordingly, such an application on the HoloLens 2 cannot guarantee an acceptable user experience. That is the main reason for using IP-based crack edge detection algorithms in addition to AI-based crack detection methods in this study. Improvements in the capabilities of the HoloLens in future versions in terms of the CPU and GPU may provide a better opportunity to develop CNN-based models for both crack detection and segmentation in an MR system.

By using YOLO v5, we achieved an effective trade-off between detection accuracy and real-time processing requirements, ensuring a high performance and responsiveness in the MR environment. The promising performance of the developed system, as demonstrated in this study, highlights the suitability of YOLO v5 for real-time crack detection in resource-constrained MR applications. Specifically, the balance between accuracy and processing speed achieved in our results validates YOLO v5 as the optimal choice for delivering both reliable detection and a responsive user experience. These findings support the use of YOLO v5 in MR applications where computational efficiency is essential, indicating that this model can provide a solid foundation for future advancements in real-time structural inspection systems.

6. Conclusions

The automatic assessment of defects in civil structures using advancements in IP and DL technologies while benefiting from the experiences and expertise of inspectors is of major importance. Hence, this study developed an integrated crack detection and skeleton extraction method in the Microsoft HoloLens 2 MR device for real-time crack monitoring. The main novelty of this research is the development of both crack detection and skeleton extraction using the YOLO v5 method and Canny edge detector in an MR environment along with a comprehensive accuracy and speed assessment of different structures of YOLO v5, as well as IP-based edge detectors.

This study’s hybrid approach, combining YOLO v5 for crack detection with the Canny edge detection algorithm, has proven effective for real-time MR-based crack assessments on the Microsoft HoloLens 2. This approach successfully balances processing speed and detection accuracy, optimizing performance within the hardware constraints of current MR devices. Future improvements in the HoloLens hardware may open opportunities to implement fully DL-based detection and segmentation models, further enhancing the accuracy and automation of defect assessment. Continued development in MR technology, coupled with enhancements in model efficiency, will likely enable more sophisticated systems for structural monitoring and maintenance in the years to come.

There are also some other issues that should be addressed in future relevant studies by improving the training process and the dataset used, including the impact of distance, angle, light, shadow, and crack width. Furthermore, newer versions of YOLO have been recently released. Hence, they could also be employed to evaluate the possible increases in the accuracy and speed of crack detection. Finally, further research should be followed to use the existing standards for crack classification, which could be embedded in the MR system. Indeed, after crack detection and skeleton extraction, the system could be advanced in such a way as to help inspectors during their defect assessment practices. The developed MR-based defect assessment model has various advantages for the application of automatic state-of-the-art methods for the management and maintenance of structures. Among them, the possibility of recording the location of the crack (position), name of the building, floor, and other descriptions for future maintenance activities, as well as future investigations, can be highlighted. To be more specific, first, the inspector can quickly scan a concrete surface to detect possible cracks in different parts of the surface. This speeds up evaluation and reduces human error. At the same time, by automatically extracting the crack skeleton, one can use one’s experience and expertise and available instructions to determine feasible actions based on the crack conditions. Second, some cracks may need further assessment to determine possible future changes in their size and width. Recording the specifications and location of such cracks is complex for inspectors. However, the designed MR-based system can help them to automatically save the relevant properties and make future evaluations much faster and more cost-effective.