TMCrack-Net: A U-Shaped Network with a Feature Pyramid and Transformer for Mural Crack Segmentation

Abstract

The detection of crack information is very important in mural conservation. In practice, the number of ancient murals is scarce, and the difficulty of collecting digital information about murals leads to minimal data being collected. Crack information appears in pictures of paintings, which resembles painting traces and is easy to misidentify. However, the current mainstream semantic segmentation networks directly use the features of the backbone network for prediction, which do not fully use the features at different scales and ignore the differences between the decoder and encoder features. This paper proposes a new U-shaped convolutional neural network with feature pyramids and a transformer called TMCrack-net. Instead of U-Net’s jump-join, an AG-BiFPN network is used, which consists of two modules: a channel cross-fusion (CCT) module with a transformer and a bidirectional feature pyramid network. While fully using the information in different network dimensions, the channel cross-fusion module optimizes the final features of each layer to reduce the confounding effect caused by the fusion of features. For the fusion of multi-scale channel information with decoder features, we designed a fusion module based on a channel attention (called FCA) to guide the fusion of enhanced encoder features with decoder features and reduce the ambiguity between the two feature sets. To demonstrate the effectiveness and generalization of the model, TMCrack-Net was evaluated on the Tang Dynasty tomb chamber mural dataset and Crack500. MIou values of 0.7731 and 0.7944 were achieved, respectively, which are better than those of other advanced crack detection methods. The method yields accurate segmentation performance and is advantageous for mural painting crack segmentation tasks.

1. Introduction

Ancient murals are regarded as valuable cultural heritage, reflecting the social landscape of the times they are from. They provide a valuable basis for studying ancient cultures because of their scientific, historical, and humanistic value. However, under the influence of natural and artificial factors, many murals [1] have suffered from problems such as cracks, puckering, and scratches, which weaken the information expression of frescoes and are not conducive to preserving history and culture, or the transmission of the art.

Screen cleaning, reinforcement, paper, cloth stickers, disease analysis, expert evaluation, and disease labeling [2] are all involved in the manual disease labeling of ancient murals. A manual method will not only cause secondary damage to the murals but will also be very inefficient. Therefore, image processing technology must be introduced to help heritage conservationists protect murals.

Accurate segmentation of cracks in mural images is challenging, and mural paintings have complex background structures. The painting of the polo picture in the tomb of Prince Zhang Huai in Tang Dynasty is rich in content; the painting has different backgrounds of people, horses, trees among mountains and rocks; and the complex image content increases the difficulty of crack recognition. Cracked images are shown in Figure 1.

Among the crack detection algorithms of traditional image processing, Hou proposed a K-means Sobel algorithm to extract the disease edges of murals. Two evaluation [3] criteria were given for the specificity of murals: disease recognition rate and edge continuity. Subsequently, his team [4] used geographic information system technology (GIS) to produce high-precision digital orthophoto maps (DOM) of murals, establish a spatial database of diseases, obtain disease maps and trend maps, obtain the locations and severity of diseases through spatial analysis and spatial statistics, and use decision trees to classify disease classes. Gancarczyk [5] applied data mining techniques to image segmentation techniques to extract crack information with unsatisfactory results. Cornelis [6] proposed a multi-scale high-hat transform and K-SVD method and a post-processing method based on semi-supervised clustering to eliminate mislabeling, and weighted fusion of the three methods was performed to calibrate crack information of different sizes and brightnesses. Two types of methods are described below: (1) manual depiction using GIS software; (2) image processing using edge detection and multivariate filtering. Though both methods involve non-contact extraction, they are mostly human–computer interactions and semi-automatic methods requiring improved accuracy.

With the development of computational vision and deep learning, different neural networks are used for mural painting disease detection. Lin [7] used hyperspectral images for disease region identification, using minimum noise fraction (MNF) to focus on different bands and reduce the effect of noise on the data. The mural images were classified into several types of damaged areas and normal areas by back propagation (BP) neural network. Yu [8] used a U-Net network with multi-scale details to detect paint peeling disease in mural paintings of Fengguo Temple of Liao Dynasty in Yi County, Jinzhou City, China. The shallow features of the wall paintings were injected into different stages of the encoder to obtain detailed information about the paint peeling disease, and attention mechanisms were added to suppress invalid features in the encoding stage. Wu [9] proposed the Ghost-C3SE YOLOv5 network to detect the damage to cave murals. The YOLOv5 network structure was adjusted to reduce the dimensionality of the convolutional layers. An attention mechanism was added to the backbone network to adjust the importance of different feature channels.

This paper proposes the TMCrack-Net model using the ConvNext network as the encoder skeleton. Bottom-up and top-down fusion of the extracted features take place using feature fusion networks with attention. Circulation of semantic and detailed information through the feature network. Each layer’s final features are optimized using multi-headed self-attention to reduce the confounding effect of feature fusion. The enhanced features are guided to fuse with the decoder’s features using the FCA module to reduce the ambiguity between the two features:

1.

We created a Tang Dynasty tomb chamber mural dataset, the TMCrack database, to implement the segmentation model, which includes images of the real tomb chamber mural and the corresponding crack label images.

2.

Based on the U-Net algorithm, we proposed TMCRack-Net with BiFPNet and a transformer for fresco crack segmentation. Instead of jumping joins in U-Net, a feature fusion network is used to fully use features at different scales to enhance the crack detail information. An attention mechanism was added after the feature fusion network to enhance the weights of useful features and reduce the influence of irrelevant features.

3.

To better fuse encoder and decoder features, we designed a fusion module based on channel attention (called FCA) to calculate the similarity matrix between encoder and decoder features, enhance each feature with a strong correlation with at least one other, suppress the features with weak correlations, and reduce the ambiguity between the two features.

2. Related Work

Different CNN models are used in crack segmentation methods. The effectiveness of CNNs in crack segmentation has been demonstrated in a large number of experiments.

2.1. Feature Pyramid

Multi-scale images are one of the most challenging aspects of computer vision, and most networks directly use the features extracted from the backbone network to predict the target. This pioneering work [10] proposed a top-down approach to multi-scale features using the feature pyramid network (FPN). Following this idea, PANet [11] adds a bottom-up path aggregation network on top of FPN to further exploit the different information between the bottom and top-level features. NAS-FPN [12] uses neural architecture searching to design feature network topologies automatically but requires thousands of hours of GPU time and makes it difficult to interpret the network structure. BiFPN [13] adds a bi-directional scale join based on removing nodes with only one input edge and adding a residual structure to join the original junction and the output node. Each of the four network structures is shown in Figure 2.

The fusion of features at different scales achieved good results using all of the methods above, but feature fusion leads to confounding effects [14], along with bias towards local interactions. By combining the CCT module, we optimize features across multiple scales after feature fusion to reduce confounding effects and establish long-range dependencies.

2.2. Vision Transformer

The transformer [15] program has achieved great success in the natural language processing domain, and many researchers have applied the program to tasks related to computer vision. A pure transformer vision model (Vision Transformer, ViT) was proposed by Dosovitskiy [16], which divides, convolves, and spreads images into two-dimensional vectors to solve the transformer’s input problem in the image domain. Liu [17] proposed the Swin-Transformer, which uses a hierarchical construction method similar to that used in convolutional neural networks: chunking the feature maps to reduce computational effort and using spatially shifted bit windows to facilitate information exchange between different windows, thus improving the model’s ability to extract local information. The state-of-the-art level of ImageNet classification is achieved. Inspired by Swin-Transformer, Cao [18] proposed Swin-Unet, the first purely transformer-based U-shaped structure, which uses Swin-Transformer to replace the convolution module in U-Net. Wang [19] proposed UCTransNet, using a CTrans (channel transformer) module instead of skip connections in U-Net. The CTrans module consists of multi-scale channel cross-fusion (called CCT) and channel cross-attention (called CCA). The CCT module performs cross-fusion of multi-scale features through the transformer. The CCA module directs the fusion of the fused multi-scale features with the decoded features through attention.

The Swin-Unet algorithm requires a large amount of data to obtain better performance. It lags behind the convolutional network when small amounts of data are available. UCTransNet directly uses the features of the backbone network as input to the transformer. It does not take full advantage of the underlying detailed information. This paper uses the pure convolutional network ConvNext as the backbone to extract features that better fit small datasets. The information at different scales is better utilized using a feature fusion network.

3. Proposed Methods

3.1. Model Architecture

TMCrack-Net consists of three main modules: (1) Top-down extraction of crack features using the ConvNext network as the backbone of the U-net network. (2) The AG-BiFPN module includes BiFPN and CCT, a feature fusion network, to exploit the multi-scale features of the image better. (3) An attention fusion module to better enhance useful features and suppress invalid ones. Figure 3 shows the TMCrack-Net with BiFPNet, Transforme, and FCA.

In this paper, we consider the crack segmentation task of mural images as a pixel-level semantic segmentation problem, where 0 represents “background pixels” and 1 represents “crack pixels”. Compared with natural images, the number of mural images is small, and the background information is complex, so the feature extraction ability of the backbone network directly affects the segmentation effect. Our backbone is the ConvNext [20] network proposed by Facebook AI Research and the University of California, Berkeley, which optimizes ResNet to reach the performance limits of pure convolutional models. As a pure convolutional network, it achieves 87.8% TOP-1 accuracy on the ImageNet dataset while maintaining the simplicity and efficiency of standard convolutional networks.

ConvNext consists of downsample blocks and ConvNext blocks, repeatedly stacked. A ConvNext block consists of depthwise convolution, and ordinary Conv is combined, using the GELU function as the activation function. A downsample block consists of layer norm and 2 × 2 convolution. Figure 4 shows the structures of downsample and ConvNeXt blocks.

With the small number of mural images and complex background information, low-level information and high-level semantic information are particularly important for crack segmentation. Using only the backbone network to extract features from the images does not make good use of the channel information at different scales, and a feature fusion network needs to be added to recycle the multi-scale features, but feature fusion confuse the localization of cracks in the images. Attention mechanisms need to be used to optimize the feature pyramid, and using attention computation at each layer of the pyramid is costly. Therefore, we introduced the CCT [19] module. By analyzing the information provided by each element layer, CCT can automatically determine the importance of different channels, increase the weights of channels that contribute to segmentation, and suppress channels that do not. Due to the ambiguity between the features of the codecs, in this paper, we design an FCA module based on a self-attentive model to filter the channel information and better fuse the features between them.

3.2. AG-BiFPN Module

UCTransNet demonstrates that not all jump-joins are beneficial for segmentation, and simple replication of features at the encoder stage can be detrimental to feature fusion, so more suitable feature fusion methods are needed to join the encoder and decoder.

In order to better utilize the features extracted from the backbone network, we invoke a combination of BiFPN and CCT modules instead of the original hop-join in the U-net. By fusing feature information at different scales through the BiFPN network, the input image retains a richer level of detail contained in the shallow features, better utilizes local information, and focuses on extracting crack details.

BiFPN was proposed by Google Research, based on the bi-directional scale join. It removes the nodes with only one input edge, which do not perform feature fusion and contribute little to the network, and add the residual structure to join the original junction with the output node, which can fuse more features. The BiFPN infrastructure is shown in Figure 2d.

The single-use feature fusion network with only a convolutional structure is biased toward local interaction and brings some confounding effect, which interferes with recognizing cracks. By adding an attention mechanism after BiFPN, as shown in Figure 5, the CCT module can better emphasize the global information and alleviate the confounding effect at each layer of the pyramid.

CCT computes the weights of different feature layers through a multi-headed self-attentive mechanism and sums the weights of each layer with the corresponding feature layer to establish global relationships and suppress invalid features. The implementation of CCT can be expressed as follows:

1.

Five features ${\mathrm{E}}_{i_{-}\mathrm{out}},(i=1,2,3,4,5)$ are obtained by feature fusion. Firstly, the features are divided into a series of patches, and the number of feature channels is kept constant. The patches are flattened and given location information to obtain tokens of different scales as $T_i(1,2,3,4,5)$, and the four tokens are combined to obtain $T_C=Concat(T_1,T_2,T_3{T}_4,T_5)$.

2.

Tokens are fed into the multi-headed self-attentive mechanism, with five $T_i$ as the query and $T_C$ as the key and value:

$$Q_{\mathrm{i}}=T_{\mathrm{i}}{M}_{Q_i},K=T_c{M}_K,V=T_c{M}_V$$

(1)

where $M_Q$, $M_K$, and $M_V$ are the weights of the different input features, and the weights $W_{vi}$ for each layer of features are obtained by $Q_i$ and K. The formula for $W_{vi}$ is:

$$\begin{array}{cc}\hfill W_{vi}& =softmax\left[\sigma \left(\frac{Q_{\tilde{i}}^{T}K}{\sqrt{C_{\mathrm{\Sigma }}}}\right)\right]\hfill \\ \hfill & =softmax\left[\sigma \left(\frac{{\left(T_{\mathrm{i}}{M}_Q\right)}^T{T}_C{M}_K}{\sqrt{C_{\mathrm{\Sigma }}}}\right)\right]\hfill \end{array}$$

(2)

where $\sigma$ represents the instance normalization [21], $C_{\mathrm{\Sigma }}={\sum }_{i=1}^N{C}_i,(i=1,2,3,4,5)$ denotes the sum of the number of channels of the input features. In our implementation, $C_1=96$, $C_2=192$, $C_3=384$, $C_4=768$, and $C_5=768$, values weighted by $W_{vi}$:

$$A{t}_i=W_{vi}{V}^T$$

(3)

In the case of N head attention, the output after multi-head cross-attention is calculated as follows:

$$MA{t}_i=\left(A{t}_i^1+A{t}_i^2+A{t}_i^3+,\dots ,A{t}_i^N\right)/N$$

(4)

where N is the number of heads. A series of experiments on layers 1, 2, 3, 4, and 5 of CCT in the implementation found that the best performance can be achieved on the tomb mural dataset with one layer and six heads of CCT. The following output is obtained by using MLP and residual join:

$$O_i=MA{t}_i+MLP\left(Q_i+MA{t}_i\right)$$

(5)

Finally, the five outputs $O_1$, $O_2$, $O_3$, $O_4$, $O_5$ are reconstructed by up-sampling through convolution, and the corresponding decoder features are concatenated.

3.3. Decoder with FCA Module

We designed a feature fusion module with attention guidance to better fuse the two features. Figure 6 illustrates the FCA module. This module does not perform dimensionality reduction operations. ECA-Net [22] found that dimensionality reduction adversely affects the attention paid to the learning channel.

After encoder feature enhancement, the FCA module first compresses the encoded features $O_i$ and decoded features $D_i$ using the global average pooling layer to obtain global spatial information. Secondly, global attention is obtained by the spreading operation with a linear layer. Again, the encoder $O_i^{\prime }$ and decoder $D_i^{\prime }$ features with attention are obtained using the spreading operation. Finally, $O_i^{\prime }$ and $D_i^{\prime }$ are multiplied to get the similarity matrix $S_i^{\prime }$, which is normalized by the softmax function to get the weights of different features, enhance the features with strong correlations, suppress the features with weak correlations, and eliminate the ambiguity existing between the encoder and decoder to some extent.

As shown in Figure 6, in FCA, we obtain the global spatial information by spatially compressing the features $O_i$ and $D_i$. Space compression is performed by space compression by the global average pool (GAP) layer:

$$G\left(O_i\right)=\frac{1}{H\times W}\sum _{i=1}^H\sum _{j=1}^W{O}_i^k\left(i,j\right)$$

(6)

$$G\left(D_i\right)=\frac{1}{H\times W}\sum _{i=1}^H\sum _{j=1}^W{D}_i^k\left(i,j\right)$$

(7)

where k is the table k-th layer channel, and i is the i-th layer. Then, to improve the model fitting ability, the features are linearly varied to obtain trainable vectors $l\left(o_i\right)$ and $l\left(d_i\right)$. The linear variation is implemented through the linear layer:

$$L\left(G\left(O_i\right)\right)=G\left(O_i\right)A_1^T+B_1$$

(8)

$$L\left(G\left(D_i\right)\right)=G\left(D_i\right)A_2^T+B_2$$

(9)

After restoring the original dimensions of $l\left(o_i\right)$ and $l\left(d_i\right)$, the two dimensions of H and W are flattened to obtain $O_i^{\prime }$ and $D_i^{\prime }$, and the similarity between them is calculated to obtain the weight matrix $S_i^{\prime }$. The specific operation is:

$$O_i^{\prime }=Flatten\left(\mathrm{unsqueeze}\left(l_{o_i}\right)\right)$$

(10)

$$D_i^{\prime }=Flatten\left(\mathrm{unsqueeze}\left(l_{d_i}\right)\right)$$

(11)

$$S_i^{\prime }=O_i^{\prime }·D_i^{\prime T}$$

(12)

The attention mask is generated by normalizing these weights using the softmax function for the similarity matrix $S_i^{\prime }$. After decompressing $S_i$, it is multiplied by $O_i$ and ${\widehat{O}}_i$ is obtained by the RELU activation function as follows:

$$S_i=\phi \left(\mathrm{unsqueeze}\left(S_i^{\prime }\right)\right)$$

(13)

$${\widehat{O}}_i=\psi \left(O_i·S_i\right)$$

(14)

where $\phi (·)$ represents the softmax function and $\psi (·)$ represents the Relu function. Finally, merge $F\widehat{{}_i}$ with the corresponding upsampling features.

4. Experiments and Results

4.1. Implementations Details

We implemented our model using PyTorch on an NVIDIA GeForce RTX 3090 GPU card with 24 GB of RAM. To avoid overfitting, we performed data enhancement, including horizontal flipping, vertical flipping, and random rotation. Pre-trained weights trained on ImageNet using ConvNext-T were used to initialize the backbone of the model. The input resolution of the dataset was set to 224 × 224. In this study, we used an Adam optimizer to train our model with batch size set to 8, initial learning rate set to 1 × 10${}^{-4}$, and cross-entropy loss as our loss function to train our network.

4.2. Evaluation Criteria

Six evaluation metrics were used to verify the performance of the model, including precision rate (P), regression rate (R), F-score ($F_1$), mean intersection over union ($MIou$), DiceScore ($Dice$), and Jaccard Distance ($Jaccard$). Equations are as follows:

$$P=\frac{TP}{TP+FP}$$

(15)

$$R=\frac{TP}{TP+FN}$$

(16)

$$F_1=\frac{2PR}{P+R}$$

(17)

$$MIou=\frac{1}{k+1}\sum _{i=0}^k\frac{TP}{FN+FP+TP}$$

(18)

$$Dice=\frac{2TP}{FP+2TP+FN}$$

(19)

$$Jaccard=\frac{TP}{TP+FP+FN}$$

(20)

where K represents the number of categories, $TP$ represents positive samples which have been correctly classified as positive, and $FP$ represents false positive samples which have been incorrectly classified as positive samples. $FN$ represents false negative samples which have been incorrectly classified as negative samples. Precision rate (P) is the probability that all predicted positive samples are positive samples. Recall that (R) is the probability that the original sample is predicted to be a positive sample among positive samples. F-Score is a balanced value when both accuracy and recall are considered simultaneously, and the highest value is taken when both are reached simultaneously. In the problem of semantic segmentation, the intersection and merge ratio is the ratio of the intersection and merging of the true labels and predicted values of the class. The dice coefficient and Jaccard distance represent the similarity between the true and predicted samples.

4.3. Dataset

Mural data for the Tang Zhang Huai Prince tomb polo: the whole picture is sparse, staggered, and very successful at presenting harmonious rhythmic beauty. The detailed portrayal of people and horses and the rude outline of ancient trees in the mountains and rocks in the picture can give people a sense of ancient and elegant beauty. This picture truly reflects an actual polo match in the Tang Dynasty and provides informative physical data for the study of polo in the Tang Dynasty, which is of great historical value. The polo picture from the tomb of Prince Zhang Huai of the Tang Dynasty was selected for crack disease extraction experiments to provide a reference for the conservation and restoration of the mural.

The field data were acquired using a Swiss Sinar P2 large format technology camera and a Sinar 75LV digital back, with a color gamut model of Adobe RGB. Contact type split shot parallel acquisition. The frescoes were divided into fresco chunks with a pixel size of $224\times 224$ format, and 700 fresco samples were selected and named TMCrack. See Figure 7.

The images were labeled using the LabelMe tool, and the corresponding image label mapping was obtained after subsequent conversion processing. The images in the dataset were randomly segmented, including 60% as the training set, 10% as the validation set, and 30% as the test set.

Table 1. Dataset settings.

Dataset	Dataset Settings
TMCrack	Train set	Validation Set	Test Set
	420	70	210
Crack500	Train set	Validation Set	Test Set
	250	50	200

shows the settings of the dataset.

To verify the generalization of the model, we used Crack500 [23] for training and testing. Crack500 was taken by Yang et al. using a cell phone at the main campus of Temple University, USA. It has a total of 500 crack images, and it includes labeled images corresponding to semantic segmentation. The dataset is divided into training images, validation images, and test images, and the ratio is 5:1:4.

4.4. Performance Comparison of the ConvNext with Different Version

ConvNeXt was designed in different variants, ConvNeXtT/S/B/L. The variants differ only in the number of channels and the number of block repetitions in each stage. To better select the backbone network, we used different variants as the backbone of the u-net network and tested the data in the same configuration. The results are shown in

Table 2. Performances of different versions of ConvNext.

Vesrison	Channel	Stage	MIou%	Parame (M)
ConvNeXt-T	[96 192 384 768]	[3 9 3 3]	74.22	29
ConvNeXt-S	[96 192 384 768]	[3 3 27 3]	73.18	50
ConvNeXt-B	[128 256 512 1024]	[3 3 27 3]	74.02	87
ConvNeXt-L	[192 384 768 1536]	[3 3 27 3]	74.10	198

.

The channel represents the number of input channels in the four stages, and the stage represents the number of times each stage is repeatedly stacked with blocks.

4.5. Performance Comparison of the CCT with Different Numbers of Attention Heads and Layers

A comparative experiment was conducted with the other unchanged modules to determine the number of attention heads in the CCT module. As shown in

Table 3. Performances with different numbers of attention heads.

Number of Head	${\mathit{F}}_1$%	MIou%	Dice%
2	85.53	76.67	73.19
3	85.67	76.87	73.46
4	85.78	77.03	73.67
5	85.92	77.19	73.91
6	86.04	77.31	73.40
7	85.77	76.92	73.60
8	85.53	76.78	73.18
9	85.57	76.61	73.20

, as the number of attention heads increased, the MIou value increased, and as the number of attention heads increased to six, the performance did not increase and appeared to decrease, and as it continued to increase, the performance then decreased.

Keeping the other modules and parameters constant, the effect of the number of CCT modules in the model was tested, as shown in

Table 4. Performance with different numbers of layers.

Layers of CCT	${\mathit{F}}_1$%	MIou%	Dice%
1	86.04	77.31	74.11
2	85.80	77.05	73.71
3	85.82	76.95	73.67
4	85.79	76.88	73.59
5	85.70	76.77	73.42

, and degradation in performance occurred when two more layers were added. MIou decreased as the number of layers increased. The final choice was to use one layer in the network.

4.6. Other Methods

1.

U-Net: U-Net uses an encoding–decoding structure to build a U-shaped network. It uses convolutional and pooling layers to extract features and contextual information and can maintain a certain performance level on a small amount of data by linking contextual information through hopping.

2.

PSPnet: PSPnet is designed with a pyramid pooling module (PPM) to obtain global information at different scales by using global pooling at different scales and fusing the obtained global features with the original input features. The feature map carries both local and global contextual information.

3.

DeepLab v3+ obtains feature information of different perceptual fields by atrous convolutions, uses spatial pyramid pooling (SPP) to fuse feature information under different perceptual fields, and uses an encoder–decoder structure to recover spatial information to obtain target boundaries.

4.

UNet++ [24]: U-Net++ is an improved jump-join method for U-Net networks. It uses nested and dense jump-joins to flexibly fuse features at different scales, compensate for semantic loss between the encoder and decoder, and enhance segmentation capabilities.

5.

Swin-Unet: Swin-Unet is the first pure transformer-based u-architecture; both encoder and decoder are built based on the Swin-transformer block. The extracted contextual features are upsampled by the decoder with the patch extension layer, and the spatial resolution of the feature map is recovered for segmentation prediction by fusing with the multi-scale features of the encoder through jump connections.

6.

UCTransNet: UCTransNet proves that not all jump-joins are powerful for the segmentation task and enhances the effective feature layer by computing the importance of features at different scales via the transformer. An attention mechanism is used to guide the feature fusion between encoder and decoder, reducing the semantic gap between them.

4.7. Experimental Results

Results on TMCrack: We compare the proposed models with the models in Section 4.6 using MPA, recall, $F_1$, MIou, dice score, and Jaccard on input images of size 224 × 224. All models were trained and tested on the TMCrack dataset, and the dataset settings were kept consistent. The results are listed in

Table 5. Performances of different methods on TMCrack.

Method	P%	R%	${\mathit{F}}_1$%	MIou%	Dice%	Jaccard%
U-Net	85.08	82.02	83.52	74.22	69.24	52.96
PSPnet	85.18	77.14	80.96	70.86	65.39	46.61
DeepLab V3+	85.98	80.06	82.91	73.33	67.72	51.19
UNet++	85.26	85.03	85.14	76.28	72.47	56.82
Swin-Unet	82.53	76.97	79.65	69.59	61.61	44.52
UCTransNet	82.79	83.48	83.13	73.87	68.93	52.69
TMCrack-Net	86.49	85.26	85.87	77.31	73.76	58.42

.

Table 5 shows that our network achieved an accuracy of 0.8649, a recall of 0.8526, an $F_1$ value of 0.8587, an MIou of 0.7731, a dice value of 0.7376, and a Jaccard value of 0.5842 on the TMCrack test set. All six of these metrics are better than those of the other methods.

Figure 8 shows the crack prediction results of different methods on the TMCrack test dataset. We tested the segmentation performance of various networks on simple and complex crack images. For cracks with simple or obvious precise shapes, most of them were detected in almost all tested models. For images with complex shapes and obscure cracks, TMCRack-Net had better performance. The PR curves of TMCrack-Net and other methods are shown in Figure 9a, which were used to evaluate the generalization level and implementation of each model from a qualitative point of view, and it can be seen from the figure that TMCrack-Net outperformed the other methods, segmenting more cracks and maintaining the accuracy of the locations.

Results on Crack500: To verify the robustness of the model and the effectiveness of the improved method, the Crack500 dataset, widely used for crack detection, was selected. The improved model was retrained on Crack500 and compared with the model in Section 4.6. The results are shown in

Table 6. Performances of different methods on Crack500.

Method	P%	R%	${\mathit{F}}_1$%	MIou%	Dice%	Jaccard%
U-Net	85.40	88.90	87.11	78.86	75.82	61.06
UNet++	88.04	85.28	86.38	78.28	74.73	59.65
PSPnet	87.37	77.78	82.30	72.43	65.43	48.62
DeepLabV3+	88.03	84.43	86.20	77.70	73.86	58.55
Swin-Unet	86.56	82.59	84.52	75.57	70.66	54.63
UCTransNet	88.42	83.49	85.88	77.23	73.13	57.64
TMCrack-Net	88.09	86.85	87.47	79.44	76.41	61.82

.

Figure 9b shows that TMCrack-Net achieved the highest precision–recall curve, and had better generalization level and performance, as shown in Table 5. Its precision rate was 0.8809, recall rate was 0.8685, $F_1$ value was 0.8747, MIou was 0.7944, and Dice value was 0.7641 on the Crack500 test dataset.

Figure 10 illustrates the crack image prediction results from the test set. Figure 10 shows the images where the crack shapes vary from simple to complex to reflect the effect of crack shape on the detection of different models. It can be seen from the results that TMCRack-Ne has a better global grasp of the segmented cracks due to having more complete shapes compared to other segmentation networks.

4.8. Model Complexity

We used the number of floating point operations (FLOPs) and frames per second (FPS) to measure the time complexity of the different models; the smaller the FLOPS, the smaller the computational demand of the model, and the FPS indicates the speed of detecting cracked images per second. Params are used to measure the spatial complexity of different models, and the parameter size indicates the memory size occupied by the model, and the results are shown in

Table 7. The FLOPs, Params, and FPS of all methods on the TMCrack dataset.

Method	Params (M)	Flops (G)	FPS
U-Net	43.93	17.62	46
PSPnet	49.07	11.82	47
DeepLab V3+	5.81	5.05	50
UNet++	47.2	153.25	20
Swin-Unet	154.24	119.37	17
UCTransNet	66.43	32.94	18
TMCrack-Net	89.27	19.15	25

.

Table 7 presents the FLOPS, FPS, and PARAMS of all methods based on the TMCracks dataset. The detection speed of the pure convolutional model was better than that of the model with a transformer, and DeepLab V3+ had the fastest detection speed among all methods, at 50 FPS. The detection speed of this model was 25 FPS, meaning it can process 25 crack images per second, and it has a smaller computational volume with less memory usage than the model with a pure transformer. TMCrack-Net has more parameters than most of these methods, so the model proposed in this paper also has some limitations.

4.9. Ablation Experiments

To investigate the effectiveness of BiFPN, CCT, and FCA in TMCRack-Net networks, we assumed that the U-net with ConvNeXt as the backbone was the baseline, and all parameters were the same except for the combination of each module and ConvNeXt.

As shown in the

Table 8. Ablation study on TMCrack for the each module.

Method	P%	R%	${\mathit{F}}_1$%	MIou%	Dice%	Jaccard%
U-Net	85.08	82.02	83.52	74.22	69.24	52.96
U-Net+CCT	87.03	83.55	85.25	76.37	72.44	56.79
U-Net+BiFPN	85.22	84.16	84.69	75.70	71.57	55.73
U-Net+FCA	86.58	84.80	85.68	76.97	73.38	57.95
U-Net+CCT +BiFPN	85.08	84.63	84.58	75.91	71.91	51.64
U-Net+CCT+FCA	85.58	85.64	85.60	76.88	73.34	57.90
TMCrack-Net	86.49	85.26	85.87	77.31	73.76	58.42

, the best performance of our approach was achieved with the addition of the three modules, which indicates that the improvements are cumulative.

By explaining which regions the model focuses on and visualizing them through Grad-CAM [25], Figure 11 shows the changes in the model’s focus regions after the ablation experiment, from which it can be seen that the TMCrack-Net network with the addition of BiFPN, CCT, and FCA obtained the best detection results.

Figure 11 shows the effect of different modules in the model. The simple U-Net network did not sufficiently focus on the area where the cracks were located for complex and inconspicuous crack images, and the addition of CCT and FCA modules increased the effective features.

5. Conclusions

In this work, we compared the capability of different semantic segmentation models in mural crack segmentation. To improve the model’s performance for mural crack segmentation, we proposed TMCrack-Net with feature fusion and a transformer. In addition, we built a mural crack library with 700 mural images annotated as a dataset. Our results showed that our model has some advantages for recognizing mural cracks. We use BiFPN to fuse the detailed information and semantic information of cracks, the CCT module to optimize the feature fusion to enhance the effective feature layer, and the FCA module to guide the fusion of encoded features and decoded features through an attention mechanism to reduce the ambiguity between them and segment more cracks. The main work and conclusions of this study are as follows:

•

We constructed a dataset of Tang Dynasty tomb chamber murals. The murals genuinely reflect the actual polo matches in the Tang Dynasty, providing informative physical data for the study of polo in the Tang Dynasty, which is of great historical value. The mural background is complex, having people, horses, and ancient mountains and rocks. Complex backgrounds are suitable for reflecting the model’s applicability in complex backgrounds.

•

Different layers of convolutional layers have different image features. In the shallow network, the features contain detailed information about the cracks, and in the higher layers, the features are represented as the semantic features of the cracks. The fusion pre-screening of information at different scales facilitates the extraction of crack information.

•

In order to obtain the number of CCT modules and self-attentive heads, the experimental comparison results in Section 4.8 show that the results are better when the number of transformer modules is one and the number of self-attentive heads is six. P, F₁, MIou, dice score, and Jaccard improved by 0.0181, 0.0056, 0.0067, 0.0087, and 0.0106 compared to the model using only convolution.

•

The FCA module is designed to integrate the encoding features with the decoding features. In Section 4.8, the results showed that by adding the FCA module, the MIou value of the model was improved by 0.0275, the p value by 0.015, recall by 0.0278, $F_1$-score by 0.0216, dice score by 0.0414, and Jaccard by 0.0499.

•

The proposed model TMCrack-Net was compared with semantic segmentation networks of different structures. The evaluation metrics and visualization results in Section 4.7 show that TMCrack-Net achieved the highest values of P, R, $F_1$, MIou, dice score, and Jaccard with 0.8649, 0.8526, 0.8587, 0.7731, 0.7376, and 0.5842, respectively.

In the future, we will further address the limitations of our model. We will improve the model’s accuracy and enhance the model for segmenting cracks in more complex mural images. We will investigate better fusion methods with attention mechanisms for improvement. As the number of tomb murals is scarce and precious, we will continue to collect more murals images to build a larger dataset so that the model can better learn the crack features.