Transfer Learning for Improving Seismic Building Damage Assessment

Abstract

The rapid assessment of building damage in earthquake-stricken areas is of paramount importance for emergency response. The development of remote sensing technology has aided in deriving reliable and precise building damage assessments of extensive areas following disasters. It is well documented that convolutional neural network methods have superior performance in earthquake building damage assessment compared with traditional machine learning methods. However, deep learning models require a large number of samples, and sufficient numbers of samples are usually not available in the newly earthquake-stricken areas rapidly enough. At the same time, the historical samples inevitably differ from the new earthquake-affected areas due to the discrepancy of regional building characteristics. For this purpose, this study proposes a data transfer algorithm for evaluating the impact of a single historical training sample on the model performance. Then, beneficial samples are selected to transfer knowledge from the historical data for facilitating the calibration of the new model. Four models are designed with two earthquake damage building datasets and the performance of the models is compared and evaluated. The results show that the data transfer algorithm proposed in this work improves the reliability of the building damage assessment model significantly by filtering samples from the historical data that are suitable for the new task. The performance of the model built based on the data transfer method on the test set of new earthquakes task is approximately 8% higher in overall accuracy compared with the model trained directly with the new earthquake samples when the training data for the new task is only 10% of the historical data and is operating under the objective of four classes of building damage. The proposed data transfer algorithm has effectively enhanced the precision of the seismic building damage assessment in a data-limited context. Thus, it could be applicable to the building damage assessment of new disasters.

1. Introduction

Major earthquakes are one of the most destructive and devastating natural disasters, often triggering extensive secondary disasters that result in widespread building collapse and infrastructure damage [1,2,3,4,5]. In this context, the extensive building damage triggered by earthquake disasters is an overwhelming cause of human casualties and loss of assets [6,7]. Rapidly assessing the damage of the building during the emergency response period following the earthquake is an essential process, which would help to mitigate the loss of life and property [8,9,10]. Although traditional post-disaster field investigations are more accurate for damage mapping, it is time-consuming and less secure. In comparison, satellite or UAV-based imagery allows for faster, more precise, and more extensive coverage of the disaster-affected area, which could be analyzed for building damage assessment purposes, which could support the emergency response [11,12].

Earthquake building damage detection and assessment are performed based on pre-and post-earthquake images or a single post-earthquake image using various types of remote sensing data, including optical, LiDAR, and SAR [13,14]. In terms of detection and classification methods, visual interpretation [15], transformation-based methods [16], and post-classification comparison from multiple periods have been used [17,18]. In recent years, machine learning and deep learning methods have also been employed for earthquake building damage assessment [19,20,21]. Among these, the deep learning methods (Convolutional Neural Networks, CNNs) have been demonstrated to be more effective at assessing the seismic building damage classes, with their superior accuracy owing to their automatic feature extraction capability that allows them to extract the high-level features more effectively [22,23,24]. CNNs can not only automatically learn low-level and mid-level features from the raw images, but also automatically extract higher-level features with discriminating representation according to the problem [25]. The low-level CNNs could extract the basic features of buildings, such as the color, the edges, and the corners, and the deep-level CNNs derives high-level features, such as problem-specific features related to the new task [26]. Such automatic feature extraction abilities are an advantage compared to traditional machine learning models that require setting the features through expert experience for remote sensing image building detection [27,28]. The powerful learning capability of deep CNNs is mainly due to the use of multiple feature extraction stages, which can automatically learn representational features from the raw images according to the target task. Recent studies have demonstrated the impressive performance of deep CNNs for automatic feature extraction and the detection of buildings [29,30,31]. For instance, Vetrivel et al. (2016) compared the results of two supervised classification methods (expert knowledge extraction of image features and convolutional neural network automatic feature extraction) for collapsed building detection. The results showed that the convolutional neural network that automatically extracted features performed better. Moreover, the authors noted that with the increase in the remote sensing image spatial resolution, the performance of the convolutional neural network method for feature extraction will have more significant advantages [32]. Ci et al. (2019) built an earthquake building damage class assessment model based on the combination of a convolutional neural network and an ordered regression. The results demonstrated that the classification accuracy of the proposed model was significantly higher than that of traditional machine learning methods [33]. However, the established deep learning-based seismic building damage assessment models require a large amount of sample data for training, which is usually difficult to obtain during the emergency response period in a new earthquake-affected area. At the same time, these models were trained based on damaged building data within a single earthquake-stricken area, which affects the extrapolation capability of the trained models when they are applied to new earthquake scenarios that lack data. In addressing the issue of insufficient training data, transfer learning has been mostly used in the literature. For instance, Yu et al. (2019) found that the performance of the accuracy of image-based tumor diagnosis could effectively be improved by fine-tuning the pre-trained image deep learning model [34]. Qin et al. (2021) demonstrated that transfer learning outperforms traditional convolutional neural networks for landslide detection [35]. Few works in the literature use transfer learning to assess and recognize structural and building damage, but most of these works focus on feature-based and knowledge-based transfer learning [36,37]. Only a small amount of attention has been focused on the transfer of sample instance data. Among these, there are very limited seismic building damage studies based on data transfer.

To address this issue, this study proposes a data transfer algorithm based on instance evaluation to be able to maximize the use of historical heterogeneous data. Based on this knowledge, trained models, and historical data, the beneficial samples from historical data are effectively screened out to be used for improving the new task prediction. We attempt to solve the issue that only very limited labeled samples are available in the emergency response phase after the occurrence of a new earthquake.

2. Materials and Methods

2.1. Post-Earthquake High-Resolution Remote Sensing Images

In this study, high-resolution remote sensing images following two earthquakes were used to develop a transfer learning model for assessment of seismic building damage classes. These two earthquakes were the Yushu earthquake [38] in Qinghai Province in 2010 and the Ludian earthquake [39] in Yunnan Province in 2014. On 14 April 2010, an earthquake with a magnitude of 6.9 Mw occurred in the Yushu Tibetan autonomous prefecture, Qinghai Province, with an epicenter near 33.2°N and 96.6°E and a depth of 14 km. The Yushu earthquake destroyed a large number of buildings in Jiegu town and the surrounding areas in the Yushu Tibetan Autonomous Prefecture. The remote sensing images following the Yushu earthquake were 0.1 m spatial resolution aerial image data acquired on 16 April 2010, containing R, G, and B bands, and the collapsed buildings images near the Jiegu town are shown in Figure 1a. On 3 August 2014, an earthquake with a magnitude of 6.1 Mw occurred near 27.1°N and 103.3°E in the Ludian County, Zhaotong City, Yunnan Province, and the epicenter had a depth of 12 km. The remote sensing images following the Ludian earthquake were 0.2 m spatial resolution UAV image data acquired on 7 August and 14 August 2014, which contained R, G, and B bands, and the post-quake collapsed buildings near Longtoushan town are shown in Figure 1b.

2.2. Labeling of Damaged Buildings in Remote Sensing Images

The visual interpretation approach was used to vectorize the building outlines on the post-earthquake remote-sensing images. Based on the analysis of the length-to-width ratio of multiple vectorized buildings and the spatial resolution of remote sensing images, it was found that producing the building samples in a standard image of size 88×88 could better present the characteristics of various buildings. The building damage classes were set into four categories, including intact damaged (no observable damage), lightly damaged, heavily damaged, and collapsed, by examining the building features, including outline, geometry, texture, and the relationship with the surrounding area, on the remote sensing images and by incorporating the criteria for classifying earthquake-induced building damage from the literature. The damage classes of the building samples in the images were evaluated one by one through visual interpretation, with the field investigations after the 2014 Ludian earthquake as a reference. The numbers of building samples of each damage class for the Ludian and Yushu earthquakes are shown in

Table 1. The sample size of damaged buildings in the Ludian dataset and the Yushu dataset [33].

Damage Classes	Description	Ludian Dataset	Yushu Dataset
D0	No observable damage	2680	778
D1	Light damage	5013	918
D2	Heavy damage	2807	665
D3	Collapse	3280	1140
Total		13,780	3501

after visual interpretation, and the morphological samples of the damaged buildings in remote sensing images are shown in

Table 2. Sample of damaged buildings on remote sensing images in the Ludian and Yushu datasets.

Dataset	No Observable Damage	Light Damage	Heavy Damage	Collapse
Ludian dataset
Yushu dataset

.

The dataset obtained from the damaged buildings of the Ludian earthquake is referred to as the “Ludian dataset”. The dataset obtained from the damaged buildings of the Yushu earthquake is referred to as the “Yushu dataset”. The Ludian dataset contains a total of 2680 buildings with D0 damage level, 5013 buildings with D1 damage level, 2807 buildings with D2 damage level, and 3280 buildings with D3 damage level, for a total of 13,780 buildings. The Yushu dataset contains a total of 778 buildings with D0 damage, 918 buildings with D1 damage, 665 buildings with D2 damage, and 1140 buildings with D3 damage, for a total of 3501 buildings. In general, the sample size of the building samples of each damage level is relatively balanced within the two earthquake datasets, and the Ludian dataset is more abundant.

Although both the “Ludian dataset” and the “Yushu dataset” contain remote sensing image data acquired following the earthquake, the building damage classes and label evaluation criteria are identical in these two datasets. There are still differences between the two datasets due to various reasons, such as their geographical regions, imaging periods, aerial platforms, sensor parameters, local customs, etc., which are reflected by the different distribution of DN values of pixels in the remote sensing images. Such differences are tangible in the actual seismic building damage assessment. This study attempts to investigate the extrapolation of the model to different seismic scenarios, different regions, and the effect of the different amount of training data on the accuracy.

2.3. Transfer Learning

Transfer learning is a method of transferring the knowledge trained or learned from deep learning models to a new pending task. The advantage of transfer learning over traditional machine learning models is that the trained models and knowledge from historical data can be transferred to the new task instead of retraining a new model. This significantly reduces the number of training samples required for a new model, while at the same time it can improve the accuracy of the new model with the help of existing models, knowledge, and historical data [40].

Regarding the training for the Ludian dataset and the Yushu dataset, the Ludian dataset, with a more abundant sample size, is chosen as the historical earthquake scenario, and the Yushu dataset, with a relatively smaller sample size, is used as the new task scenario. Such a design is similar to the practical new earthquake scenario; i.e., there are more historical earthquake building damage samples, while new samples are usually relatively limited for the new earthquakes. In this case, the challenge for the building damage assessment of new earthquake scenarios is that there are already a small number of training samples and test samples with the same distribution. However, the number of these samples is too small, so it may not be possible to derive a reliable model by relying on these limited training and testing samples. It is also difficult to increase the number of training samples of new earthquakes, which would take more time and would not be sufficient for the rapid damage assessment for the emergency response. At the same time, there are a larger number of auxiliary samples (historical data). However, the distribution between these auxiliary samples and the training samples of the new earthquake scenario is not consistent.

The problem to be solved here is how to derive, with the help of these auxiliary samples, a new model that has the highest possible classification accuracy when applied to the new test samples. For this purpose, the following model framework is designed in this study (Figure 2).

Firstly, a convolutional neural network (CNN) combined with an ordered regression model is employed to build a seismic building damage class assessment model based on the Ludian dataset. This deep learning model architecture consists of a CNN feature extractor, i.e., the visual geometry group network (VGG), and an ordered regression classifier, which has been demonstrated to outperform other model architectures in seismic building damage class assessment. In this study, the VGG-OR network is employed to develop a VGG-OR(LD) model for the Ludian dataset. Then, four schemes are designed to evaluate the performance of different transfer learning schemes through comparative tests. The first scheme is to directly transfer the trained model, i.e., VGG-OR(LD), to the testing set of the “new” Yushu dataset for prediction. The second scheme is to retrain a new model based on the “new” Yushu dataset and the VGG-OR architecture, i.e., VGG-OR(YS). The third scheme is to utilize the training set of the “new” Yushu dataset to fine-tune the parameters of the VGG-OR(LD) model to derive a new model, i.e., VGG-OR(FT). The last scheme is to propose a new data transfer algorithm. The proposed data transfer algorithm is designed to identify potential samples of the historical data that are beneficial to the Yushu dataset, and these beneficial historical samples are used jointly with the training samples of the Yushu dataset to fine-tune the parameters of the VGG-OR(LD) model to derive a new model, i.e., VGG-OR(DT).

The basic idea of the data transfer algorithm proposed in this study is that although the historical samples and the target training samples are more or less different from each other, there should still be a portion of the historical sample that is suitable for assisting in training the model that could be adapted to the test sample of the new task. Therefore, the critical implementation of this data transfer algorithm is to build an evaluation model that retains the eligible historical data and removes the historical data that differs significantly from the target data. These retained historical samples would be used as added data to be used with the target training samples to improve the reliability of the model after filtering. The data transfer algorithm process proposed in this study is illustrated in

Table 3. Data transfer algorithm processes.

Input ${\mathrm{Z}}_{\mathrm{S}}$: Historical dataset (Ludian dataset) ${\mathrm{Z}}_{\mathrm{T},\mathrm{train}}$: Training set from the new task dataset (Yushu dataset) ${\mathrm{Z}}_{\mathrm{T},\mathrm{val}}$: Validation set from the new task dataset (Yushu dataset) i, j: subscript ${\mathrm{f}}_{\mathsf{\theta }}: \mathrm{f}$ refers to the CNN model and θ refers to the parameter of the CNN model ${\mathrm{I}}_{\mathrm{loss}}\left({\mathrm{z}}_{\mathrm{i}},{\mathrm{z}}_{\mathrm{j}}\right):$ loss function

Start ${\mathsf{\theta }}_{\mathrm{pre}}$: Pre-trained parameters obtained by training based on the historical dataset ${\mathrm{Z}}_{\mathrm{S}}$ ${\mathrm{f}}_{{\mathsf{\theta }}_0}\stackrel{{\mathrm{Z}}_{\mathrm{S}}}{\to }{\mathrm{f}}_{{\mathsf{\theta }}_{\mathrm{pre}}}$

Processes $1. \mathrm{For} \mathrm{each} {\mathrm{z}}_{\mathrm{i}}\in {\mathrm{Z}}_{\mathrm{S}},{ \mathrm{z}}_{\mathrm{j}}\in {\mathrm{Z}}_{\mathrm{T},\mathrm{val}}$, calculate the loss function on the validation set: $\mathrm{For} {\mathrm{z}}_{\mathrm{i}} \mathrm{in} {\mathrm{Z}}_{\mathrm{S}}$ $\mathrm{For} {\mathrm{z}}_{\mathrm{j}} \mathrm{in} {\mathrm{Z}}_{\mathrm{T},\mathrm{val}}$ $\mathrm{Calculating} {\mathrm{I}}_{\mathrm{loss}}\left({\mathrm{z}}_{\mathrm{i}},{\mathrm{z}}_{\mathrm{j}}\right)$ $\mathrm{If} {\displaystyle \sum }_{\mathrm{j}}{\mathrm{I}}_{\mathrm{loss}}\left({\mathrm{z}}_{\mathrm{i}},{\mathrm{z}}_{\mathrm{j}}\right)>0, \mathrm{then} \mathrm{remove} {\mathrm{z}}_{\mathrm{i}} \mathrm{from} {\mathrm{Z}}_{\mathrm{S}}$ $2. \mathrm{Beneficial} \mathrm{samples} {\mathrm{Z}}_{\mathrm{s}}^{\prime }$ $4. \mathrm{Combined} {\mathrm{Z}}_{\mathrm{T},\mathrm{train}}\cup {\mathrm{Z}}_{\mathrm{s}}^{\prime }$as the training set, Fine-tuning the pre-trained parameters ${\mathsf{\theta }}_{\mathrm{pre}} to {\mathsf{\theta }}_{\mathrm{T}}$ ${\mathrm{f}}_{{\mathsf{\theta }}_{\mathrm{pre}}}\stackrel{{\mathrm{Z}}_{\mathrm{T},\mathrm{train}}\cup {\mathrm{Z}}_{\mathrm{s}}^{\prime }}{\to }{\mathrm{f}}_{{\mathsf{\theta }}_{\mathrm{T}}}$

Output ${\mathrm{Z}}_{\mathrm{s}}^{\prime }$: Beneficial samples after selection ${\mathsf{\theta }}_{\mathrm{T}}$: Optimized model parameters

and Figure 3. The effect of the historical sample on the accuracy of the model in the test set of the new task is evaluated using the loss function proposed by Koh and Liang (2017), and the beneficial samples are selected for improving the model with the training set of the new task [41].

In the existing transfer learning studies, it is found that the best transfer learning effectiveness is achieved when the amount of data in the new task is close to 10% of the amount of the historical data. When the data size of the new task is more than 20% of the historical data size, the effectiveness of transfer learning decreases and may even be weaker than that of direct training based on only the new training data [42,43]. Therefore, this study limits the data size of the new task to approximately 10% of the historical data size when comparing the performance of the models, and also explores the effect of the training sample size for the new task on the performance of the different models.

2.4. Evaluation Methods

The confusion matrix, overall accuracy, kappa coefficient and mean squared error (MSE) were used to evaluate the performance of the four models. The training procedure of all the models in this study randomly divides the dataset into the training set, validation set, and test set. The three sets (training set, validation set and test set) are not intersected with each other. The training set is used to train the parameters of the deep learning model. The validation set is used for model selection, where the optimal model is selected based on the performance of the model with the validation set, or the models trained with different hyperparameters, including model structure, learning rate, number of iterations, etc., are selected based on the performance of these hyperparameters with the validation set. The test set is employed to examine the extrapolation performance of the final selected optimal model. Since the sample size of the Ludian dataset is comparatively sufficient for the training of the VGG-OR(LD) model, the Ludian dataset is randomly divided into three parts according to the ratio of 80%:10%:10%, mapping to the training set, the validation set, and the test set, respectively. For a more detailed explanation of the VGG-OR(LD) model, refer to the literature [33]. For the new earthquake scenario, i.e., the Yushu dataset, 60% of the samples are randomly divided into the training set, 20% into the validation set, and the remaining 20% into the test set, and the evaluation metrics are based on the test set. Thus, for the VGG-OR(LD) model, the test set of the Yushu dataset was directly employed to evaluate the performance of the model. For the VGG-OR(YS) model, the model is trained using 60% of the Yushu training dataset combined with the CNN-OR model structure, then 20% of the Yushu validation dataset is utilized to fine-tune the parameters of the VGG-OR(YS) model, and finally the performance of the VGG-OR(YS) model is evaluated using 20% of the Yushu test dataset. For the VGG-OR(FT) model, the training and validation sets of the Yushu dataset were employed to fine-tune the parameters of the VGG-OR(LD) model, and then the performance of the VGG-OR(FT) model was evaluated with the Yushu test dataset. For the VGG-OR(DT) model, the data transfer algorithm proposed in this work is used to identify the beneficial samples in the Ludian dataset, and these beneficial samples are used together with the training and validation sets of the Yushu dataset to fine-tune the VGG-OR(LD) model, and then the Yushu test dataset is employed to evaluate the performance of the VGG-OR(DT) model. The details of each evaluation method are described as follows.

The confusion matrix is widely used to evaluate the agreement of the observation and the model’s classification, which is one of the underlying methods for evaluating the performance of remote sensing image classification algorithms. This is achieved by counting the number of algorithmic classification result labels corresponding to the true category labels on a sample-by-sample basis. A confusion matrix is represented using C. Assuming that the samples have a total of K classes, the matrix C is a K-row and K-column matrix. C (i, j) represents the total number of samples whose observation category is i and predicts category j.

The overall accuracy is the rate of consistency between the results classified by a model and the observation. It can be derived using the following formula:

$$\mathrm{OA}=\frac{{{\displaystyle \sum }}_{\mathrm{i}}^{\mathrm{K}}\mathrm{C}\left(\mathrm{i},\mathrm{i}\right)}{{{\displaystyle \sum }}_{\mathrm{i}}^{\mathrm{K}}{{\displaystyle \sum }}_{\mathrm{j}}^{\mathrm{K}}\mathrm{C}\left(\mathrm{i},\mathrm{j}\right)}$$

(1)

Kappa coefficients are commonly used as a measure of classification accuracy [44]. The kappa coefficient would usually be between 0 and 1, and the larger the value, the better the consistency. It is derived using the following formula:

$${\mathrm{p}}_{\mathrm{e}}=\frac{{{\displaystyle \sum }}_{\mathrm{i}}^{\mathrm{K}}\left({{\displaystyle \sum }}_{\mathrm{j}}^{\mathrm{K}}\mathrm{C}\left(\mathrm{i},\mathrm{j}\right)\ast {{\displaystyle \sum }}_{\mathrm{j}}^{\mathrm{K}}\mathrm{C}\left(\mathrm{j},\mathrm{i}\right)\right)}{{\mathrm{N}}^2}$$

(2)

$$\mathrm{Kappa}=\frac{\mathrm{OA}-{\mathrm{p}}_{\mathrm{e}}}{1-{\mathrm{p}}_{\mathrm{e}}}$$

(3)

where N denotes the total number of samples.

Mean squared error (MSE) is the mean value of the sum of squares of errors between the predicted data and the actual value. The formula is:

$$\mathrm{MSE}=\frac{1}{\mathrm{N}}{\displaystyle \sum }_{\mathrm{i}=1}^{\mathrm{N}}{\left({\mathrm{y}}_{\mathrm{i}}-{\widehat{\mathrm{y}}}_{\mathrm{i}}\right)}^2$$

(4)

where ${\mathrm{y}}_{\mathrm{i}}$ refers to the actual value, ${\widehat{\mathrm{y}}}_{\mathrm{i}}$ refers to the predicted value, and N denotes the total number of samples.

3. Results and Discussion

3.1. Model Trained by Ludian Dataset Can Be Transferred to Yushu Dataset

The VGG-OR(LD) model trained on the “Ludian dataset” was directly applied to the test set of the “Yushu dataset” to evaluate the performance of directly transferring the established “historical” model to the new task. The overall accuracy and kappa coefficients of the VGG-OR(LD) model for the test set of the “Yushu dataset” are shown in

Table 4. Performance of the VGG-OR(LD) model applied to the Yushu dataset.

Sets	Subclass	Damage Grade	Overall Accuracy	Kappa Coefficient
Set 1	Nearly intact	D0, D1, D2	90.14%	0.80
	Damaged	D3
Set 2	Nearly intact	D0, D1	74.43%	0.60
	Severe damage	D2
	Complete collapse	D3
Set 3	No observable damage	D0	64.28%	0.49
	Light damage	D1
	Heavy damaged	D2
	Collapse	D3

in three sets of damage classes, including two classes (D0–2, D3), three classes (D0–1, D2, D3), and four classes (D0, D1, D2, D3). In general, the direct transfer model performs relatively well with the new dataset. The overall accuracy of the VGG-OR(LD) model in the new task was above 90% with a kappa coefficient value of 0.8 in the dichotomous classification (Set 1: nearly intact and damaged), which outperforms the existing related studies. The performance of the VGG-OR(LD) model in the new task decreases significantly when the damage classes are divided into three categories (Set 2: nearly intact, severe damage, and complete collapse), yet the overall accuracy is 74% and the kappa coefficient value is 0.6, which is comparable to the performance of the existing models in the literature [28,45,46]. Since the performance of the “historical” model is directly applied to the “new” dataset from a different geographical region, without using the training samples from the new dataset in the model, this is a good demonstration that the VGG-OR(LD) model performs very well in the test set of the new task.

The overall accuracy of the VGG-OR(LD) model with the Yushu test set is approximately 64% when the damage classes are divided into four categories (Set 3: no observable damage, light damage, severe damage, and complete collapse), which is significantly lower than the overall accuracy of the VGG-OR(LD) model with the “Ludian dataset”, which was 78%, indicating that the performance of the direct transfer model in more detailed classification is still somewhat inadequate. The confusion matrix of the VGG-OR(LD) model directly applied to the Yushu testing dataset with four classifications of building damage levels is shown in

Table 5. Confusion matrix of the VGG-OR(LD) model applied to the Yushu dataset.

Observation	Prediction
	D0	D1	D2	D3	Total
D0	46	46	19	0	111
D1	25	74	56	10	165
D2	8	27	48	21	104
D3	0	4	34	282	320
Total	79	151	157	313	700

. From Table 5, the classification accuracy of the model for collapsed buildings (D3 level) is significantly higher than that of other building damage levels, with the producer’s accuracy reaching 88% and the user’s accuracy reaching 90%. At the same time, the values of the lower left and upper right corners of the confusion matrix are both zero, which suggests that the model is less likely to misclassify across damage levels. As this “historical” model is implemented straightforwardly on the “new” dataset from a geographically distinct region, this indicates that the straightforward transfer model performs fairly well. However, the direct application of the VGG-OR(LD) model to the Yushu test set still has more misclassification and omission in the two damage levels of D0 and D1, and further improvement of the model is needed. For this reason, this study further explores the use of data transfer methods to attempt to enhance the classification accuracy.

3.2. Data Transfer Improves the Performance of a New Task with Insufficient Data

To further explore the applicability of the data transfer algorithm for improving the model performance, three model schemes are designed in this study, including VGG-OR(YS), VGG-OR(FT), and VGG-OR(DT). VGG-OR(YS) uses the VGG-OR model architecture to retrain the model with the training data in the Yushu dataset. In VGG-OR(FT), the VGG-OR(LD) model is used as the pre-trained model, and the training data in the “Yushu dataset” are used to improve the model through fine-tuning (FT). For VGG-OR(DT), the data transfer (DT) algorithm proposed in this study is used to select samples from the “Ludian dataset” that are beneficial to the new task, and these beneficial historical samples are used together with the training samples from the “Yushu dataset” to fine-tune the parameters of the VGG-OR(LD) model.

Figure 4 indicates the accuracy of the VGG-OR(YS), VGG-OR(FT), and VGG-OR(DT) models when the training sample of “Yushu dataset” accounts for 10% of the sample size of the “Ludian dataset”. The results in Figure 4 showed that VGG-OR(DT) performed better in the test set of the Yushu dataset with an overall accuracy of 74%, which was significantly improved compared with the other three models, especially compared with the VGG-OR(LD) model, where the overall accuracy was improved by approximately 10%, the kappa coefficient was improved by approximately 0.12, and the MSE was lowered by approximately 0.15. This demonstrates that the data transfer algorithm proposed in this study is effective at improving the performance of the model for prediction in new tasks.

It is also worth noting that the VGG-OR(FT) models also showed remarkable improvements compared to the VGG-OR(LD) model, with the overall accuracy increasing from 64% to approximately 71%, the kappa coefficient increasing from 0.49 to 0.58, and the MSE decreasing from 0.52 to 0.39. The fairly good performance of the VGG-OR(FT) models sufficiently demonstrates that the new target data could significantly enhance and improve the performance of the transfer model with the new task. In addition, both the VGG-OR(YS) and VGG-OR(FT) models show a certain degree of performance improvement compared to the VGG-OR(LD) model that was directly applied to the test set of the Yushu dataset. This suggests that using a portion of the samples from the new task for model training or parameter fine-tuning could improve the classification accuracy of the models in the test set of the new task. Therefore, it is greatly important to acquire the new dataset as soon as possible after a new disaster occurs, which would help to improve the classification accuracy of the model.

In order to explore the influence of the training data sample size on the accuracy of the transfer learning models in the new task, this study selects the data size of the “Yushu dataset” equal to 2–20% of the “Ludian dataset” as the training data, and the remaining data are used as testing data to evaluate the classification accuracy of the models. It is shown in Figure 5 that VGG-OR(DT) and VGG-OR(FT) always performed better than VGG-OR(YS). In particular, when the data size of the new task is quite limited, the improvement of the classification accuracy of the VGG-OR(DT) model built with the data transfer algorithm proposed in this study is more significant for the new task test set. When the data size approaches 20%, the classification accuracy of the three models gradually improves. This would be very helpful for the emergency response assessment when new disasters occur. It is usually inconvenient to acquire a large number of new data samples for the affected area immediately after a disaster, and the data transfer approach developed in this study could be employed to select samples from the existing historical data that are beneficial to the new task, which can be combined with a small amount of new sample data from the new disaster scenario to significantly improve the classification accuracy of the model.

3.3. The Applicability of Transfer Learning in New Scenarios

The data transfer methods proposed in this study could efficiently improve the accuracy of seismic building damage assessment. When new earthquake events occur, a building damage class assessment model applicable to the region of new earthquake events can be derived from a small number of labeled damaged building samples from the stricken region, combined with the already established model and historical dataset. The results of this study show that when the training set of the new task reaches about 10% of the historical data, i.e., 1000 samples, combined with the data augmentation method, the model has been able to achieve an overall accuracy of about 74% for the VGG-OR(DT) model derived from transfer learning in four building damage categories (Set 3 in Table 4: no observable damage, light damage, severe damage, and complete collapse). For the same number of new datasets, the overall accuracy is 66% when used to retrain the model VGG-OR(YS). The improvement of the classification accuracy of the transfer learning approach compared with the re-training model is remarkable. With this amount of data, it only requires a relatively short time to complete the manual labeling, which can greatly reduce the time required for disaster damage assessment in the disaster emergency response phase. Therefore, we believe that the reasonable use of historical data through transfer learning methods could greatly improve the efficiency of remote sensing-based damage assessment and reserve more time for emergency response decisions.

In addition to the implementation of the data transfer model in the new seismic scenario, the data transfer model may be further extrapolated in other research areas in the future to address the challenge of a lack of data samples in areas where modeling is not possible. For example, for landslide and debris flow susceptibility modeling in the mountainous areas of the Tibetan Plateau, where data are not available, the transfer of applicable landslide samples or experiences from more abundant regions may be explored. Related studies in regions with less data, such as in Africa, may also attempt to assist modeling with applicable samples from other data-rich regions.

Moreover, local site effects can have a significant impact on building damage in different locations [47]. Site effects lead to the amplification of seismic waves through stratigraphic and topographic effects, which may lead to a concentration of building damage in certain locations [48,49,50]. It would be beneficial for the building damage assessment in this study if site effects are taken into account. However, the purpose of this study is to propose a new data transfer model to derive beneficial samples from existing historical data through evaluation to assist in assessing rapid building damage assessment in areas where new seismic hazards have occurred. In this case, the goal of transferring learning to new areas may not be achieved by considering site effect factors. In future research on high-accuracy building damage assessment modeling in a given region, the site effect factor will be incorporated to improve the accuracy of the model.

4. Conclusions

This study investigates the effective use of historical data and the extrapolation performance of the existing trained models to new or unknown data through transfer learning approaches, attempting to address the limitations of deep learning models in practical seismic building damage assessment applications. For this purpose, a data transfer algorithm is proposed and applied to two historical earthquake building damage datasets (the Ludian earthquake and the Yushu earthquake). The main findings of this study are as follows: (1) The model built based on historical data can be effectively transferred to the new task, and could assist in addressing the situation where there is not yet data available for the new task. (2) The data transfer algorithm proposed in this study can filter data samples suitable for the target task from historical data, and using these historical data could significantly improve the reliability of the classification model. (3) The data transfer algorithm can effectively improve the classification accuracy of the model in the case of limited data, especially in the case of limited new data in the affected area during the disaster emergency response phase, and the data transfer model would be helpful for improving the efficiency and accuracy of rapid damage assessment.