Improving Landslide Recognition on UAV Data through Transfer Learning

Abstract

As a frequent geological disaster, landslides cause serious casualties and economic losses every year. When landslides occur, rapid access to disaster information is the premise of implementing disaster relief and reduction. Traditional satellite remote sensing may not be able to timely obtain the image data from the disaster areas due to orbital cycle and weather impacts. Visual interpretation of remote sensing data and machine learning methods need to be improved the detection efficiency. This paper studies landslide recognition based on the UAV remote sensing image. The affected area of the Zhangmu Port region in Tibet by the Nepal earthquake occurred on 25 April 2015 was selected to carry out the landslide investigation. Aiming at the problem of insufficient training sample data of landslides, we adopt the transfer learning method. The evaluation indexes show that the proposed method can automatically identify landslide disasters. Comparing with the SSD model, our new approach has better detection performance, providing thus accurate data support for scientific decision-making of disaster rescue.

1. Introduction

China is a country with frequent geological disasters, causing serious casualties and economic losses every year. The statistics showed that the direct economic loss caused by geological disasters in China is 50.2 billion yuan in 2020. In order to protect the safety of people’s lives and property, the management of all emergency processes coming into effect after the disaster occurs is crucial to the disaster relief work. Satellite remote sensing has the advantage of obtaining information in a large area. However, the timeliness of obtaining image data is subject to the satellite orbit, which cannot ensure that the high-quality remote sensing images are available in the disaster area in a short time [1]. In recent years, UAV technology has been developing rapidly. The advantages of the UAV system are mainly related to fast data collection, short period taken, high safety and all-day observation, and it has been widely used in the disaster monitoring [2]. UAV can obtain disaster images when it is difficult for the rescue team to reach situations on the ground, which provides an important technical means to fully grasp the disaster situation.

Whether it is by use of remote sensing images or UAV images, a traditional expert visual interpretation is the mainstream method for post-disaster assessment and reconstruction [3,4]. However, the method has inefficiency and high expert resource costs. The interpretation accuracy may be poor [5,6,7] and may delay the emergency rescue. Therefore, new technologies and methods must be adopted to automatically identify landslide disasters [8,9,10]. Automated extraction methods such as the object-oriented and machine learning-based image classification, in recent years, have been used alongside support vector machines (SVM), random forests (RF), artificial neural networks (ANNs), convolutional neural networks (CNN), deep learning and other methods for landslide detection [11,12,13,14].

Li Qiang et al. [15] took landslides near the Panda Sea caused by the Jiuzhaigou earthquake as the research area and introduced the object-oriented analysis method to realize landslide extraction. Based on the object-oriented image analysis, Stumpf et al. [16] extracted landslide disaster information using a random forest classification algorithm. Mondini et al. [17] used the Bayesian statistical framework to semi-automatically realize landslide information extraction and classification.

In recent years, deep learning technology, especially the deep convolutional neural network, has attracted the attention of many scholars [18,19]. Deep learning (DL) processes complex images and extracts features from the original images very effectively [20,21]. There are two typical types of deep learning approaches for image recognition: the first are region-based methods, such as RCNN, fast-cnn, fast-rcnn and r-fcn [22,23,24,25]. The second type is typified by the regression-based methods, such as YOLO and SSD algorithms [26,27]. The second type of method is faster than the first type, but with lower accuracy because it generates bounding boxes in a single network. These methods have become the mainstream deep learning approaches for the extraction of the information from remote sensing images [28].

Deep learning methods can automatically detect landslides in a large area, which can better meet the emergency response requirements. Lei et al. [29] and Omid et al. [30] extracted landslide information from remote sensing images using a depth convolution neural network. The combination of artificial intelligence technology and remote sensing monitoring is expected to improve the automation level of landslide extraction and provide first-hand information for post-disaster emergency rescue [31].

However, target detection based on deep learning often requires massive amounts of training data [32,33]. Few public datasets are open for landslide disasters. The data we acquired in the study area was not enough to train a model with good adaptability. In addition, the deep learning algorithms often have complex structures, large model parameters and a long training time, which have high requirements for computing resources and hardware equipment.

In this paper, we proposed a transfer learning approach, which can share knowledge from the source domain to the target domain [34]. The transfer learning method was based on similar characteristics shared between the source and the target domains [35,36]. However, there are few source domain datasets available for landslide detection. In this work, the Places365 dataset [37], the UC Merced land use dataset [38], and the RSSCN7 dataset [39] were selected as the source domain for feature transferring. By fine-tune existing model parameters and using a small number of field samples, the computing resources and training time could be greatly reduced. The efficiency of landslide extraction is improved, and the workload of field manual interpretation is reduced [40].

The organization of this work is as follows: Section 2 introduces the study area, and the process of the image acquisition is also elaborated. Section 3 focusses on the research methodology, and we introduce the principles and methods of the transfer learning model. The data processing and experimental results are given in Section 4. Finally, we discuss the overall achievements and summarize the full text.

2. Study Site and Datasets

2.1. Study Site

The study site is the Zhangmu Port area adjacent to Nepal. The Zhangmu Port is located in Kaze City, in the Tibet Autonomous Region of China, as shown in Figure 1. On 25 April 2015, Nepal (north latitude 28.2 degrees, east longitude 84.7 degrees) suffered an earthquake of a magnitude of 8.1, namely the 4·25 Nepal earthquake. The earthquake caused a massive landslide near Nepal in Tibet, China.

2.2. Data Collection

We collected the image data of the Zhangmu Port area with the UAV platform. The F200 UAV system was used for image acquisition. The system can provide centimeter-level POS data to realize no control point or sparse control point mapping. The UAV platform can be thrown off by hand and is easy to control. The onboard remote sensor is a Sony ILCE-5100 camera, which has a pixel resolution of 6000 × 4000. The focal length is 20 mm, and the exposure time is 1/1600 s. The technical specifications of the UAV system are listed in

Table 1. The technical specifications of the UAV system.

Parameter	Value
UAV type	Fixed-wing
Airframe length	1.07 m
Wing span	1.9 m
Maximum flight altitude	4500 m
Maximum flight duration	1.5 h
Cruising speed	60 km/h
Payload	3.5 kg

.

Some of the UAV images were used as training dataset, and others were used for image processing and mosaic. Figure 2 shows the training dataset samples.

2.3. The Source Domain Datasets

Because of the limited landslide datasets, it is difficult to train a robust classification model. This paper uses the transfer learning method for small sample target detection. At present, there are few public datasets similar with landslide images. From the characteristics of the currently public datasets, there are three source domain datasets we used for this work: the Places365 dataset, the UC Merced land use dataset and the RSSCN7 dataset.

The Places365 dataset is a scene image dataset published by MIT. The dataset contains a total of 10 million images, covering more than 400 different types of scene environments. The Places365 dataset includes a wide range of and huge data, which are suitable for model training of scene classification [41].

The UC Merced dataset includes 21 types of land use images. The data were selected from the aerial remote sensing images of different cities from the United States Geological Survey. The dataset has 21 categories of images about agricultural fields, airports, baseball fields, sandy beaches, and buildings, each with 100 images and 2100 images. Each image has a resolution of 0.3 m and an image size of 256 × 256 pixels.

The RSSCN7 dataset is a remote sensing image dataset released by Wuhan University in 2015. It collected scene images of seven categories, including forests, grasslands, farmland, residential areas, river and lakes etc. Each category has 400 images, and the image size is 400 × 400 pixels [33]. Figure 3 shows the samples of the source domain datasets.

3. Methodology

In this work, we propose the transfer learning method to improve landslide detection and compare it with the traditional deep learning SSD method. The main research framework is shown in the following Figure 4.

3.1. UAV Data Acquisition and Processing

3.1.1. Route Planning

The route planning is designed using the ground station software matched with the UAV system. The study area covers more than 30 square kilometers with 8 routes designed. The designed heading overlap is 75%, and the lateral overlap is 65%. The flight height is 460 m, and the ground resolution is better than 15 cm. The routes are straight in the southwest–northeast direction, and the flight test is conducted before the flight to ensure the normal operation of the UAV system [42]. After completing the flight in the study area, the image data are exported to the ground station, quickly checked and analyzed to determine whether there are aerial photography loopholes, and ensure that the acquired data is valid.

3.1.2. Data Preprocessing

The original data of each photograph includes the POS data, image data and camera file data. The POS data of each picture includes picture number, latitude, longitude, geodetic height, altitude, heading azimuth and other information. Mainly, check the photo number at the turn of the flight belt, so that the photo number in the POS data corresponds to the image data one by one.

After acquiring the original aerial images, check whether it is clear, whether there is blurring, and whether the color tone of the image is roughly the same. For aerial photos with large chromatic aberration, the overall uniform light and color should be carried out to ensure that the images meet the data requirements of image mosaicking.

3.1.3. DOM Production

The Pix4Dmapper software was used for DOM (Digital Orthophoto Map) production. The POS data and UAV images are the original data of the aerial triangulation solution. The stereo-modeling differential correction method is adopted. The DEM (Digital Elevation Model) data of the study area is superposed after the original image is processed with uniform light and color. After the digital differential correction, a single ortho image is generated, and then all the map frames are spliced to obtain the ortho image map of the entire study area. Figure 5 shows the DOM production of the study area.

3.2. Landslide Detection Based on SSD Model

We select the current mainstream SSD model for landslide detection and compare it with the transfer learning method. The SSD model is currently the most widely used deep learning algorithm in the target detection. In order to improve the accuracy of the results of different feature maps, the model makes the prediction using the feature pyramid prediction mode. The default box with multiple ratios is set. The model combines both speed and accuracy, with faster detection than Faster-R-cnn and higher accuracy than YOLO. At the same time, the end-to-end training method was adopted, and the classification results were very accurate, even for the pictures with relatively low resolution.

The objective function of the SSD algorithm is based on the Multibox objective detector. The loss function consists of the location loss and confidence loss [27].

$$L\left(x,c,l,g\right)=\frac{1}{N}\left(L_{conf}\left(x,c\right)+\alpha L_{loc}\left(x,l,g\right)\right)$$

(1)

where x = {1, 0} represents whether the target exists; c represents confidence; l is the prediction box; $g$ represents the ground-truth box; N represents the number of prediction boxes that match the real box; α is the weight coefficient.

The location loss function is defined as follows:

$$L_{loc}(x,l,g)={\displaystyle \sum }_{i\in Pos}^N{\displaystyle \sum }_{m\in \left\{cx,cy,w,h\right\}}^N{x}_{ij}^{p}smoot{h}_{L1}(l_i^m-{\widehat{g}}_j^m)$$

(2)

where ${\widehat{g}}_j^{cx}=\frac{(g_j^{cx}-d_i^{cx})}{d_i^w},{\widehat{g}}_j^{cy}=\frac{(g_j^{cy}-d_i^{cy})}{d_i^h}{\widehat{g}}_j^w=log(\frac{g_j^w}{d_j^w}),{\widehat{g}}_j^h=log(\frac{g_j^h}{d_j^h})$; if $x_{ij}^p$ = 1, the i prediction box matches with the j real box of class p, if $x_{ij}^p$ = 0, does not match.

The confidence loss is defined as follows:

$$L_{conf}\left(x,c\right)=-{\displaystyle \sum }_{i\in Pos}^N{x}_{ij}^{p}log({\widehat{c}}_i^p)-{\displaystyle \sum }_{i\in Neg}^{N}log({\widehat{c}}_i^0)$$

(3)

where ${\widehat{c}}_i^p=\frac{exp\left({\widehat{c}}_i^p\right)}{{{\displaystyle \sum }}_p^{i}exp\left({\widehat{c}}_i^p\right)}$, ${\widehat{c}}_i^p$ is the probability that the i prediction-box matches with the class p.

3.3. Target Detection Based on Transfer Learning

Transfer learning can improve the model performance by transferring knowledge from the source domain. In this way, the reliance on large amounts of labeled data could be reduced. For landslide detection, no similar large-scale open dataset is available. Therefore, we proposed a feature-based transfer learning algorithm. As shown in the Figure 6 below, generally the characteristics of landslides include that the mountain is covered with green vegetation, and that the falling soil and rocks will form a certain accumulation. Such color, texture, and morphological features would be helpful if could be extracted from the source domain dataset and transferred to the target domain dataset for landslide detection.

Figure 7 shows the transfer learning flow chart. The goal of the transfer learning is to transfer knowledge from the existing source domain data and shorten the training time of the model. The convolutional neural network (CNN) for pre-trained models is composed of convolutional layers and fully connected layers, and the image data processing is as follows.

Step 1.

Feature extraction: the input image is a two-dimensional matrix, using a trainable filter group to compute the convolution with a step size. Each filter corresponds to a specific feature.

Step 2.

Feature maps: an activation function is used to map the results of the filter to ensure the validity of the feature, and the feature map is obtained.

Step 3.

Feature pooling: the feature dimensions of any image are in the millions after convolution. If the output was used to train the classifier directly, it would cause overfitting. Therefore, the feature pooling or downsampling is needed. The feature map is divided into disjoint regions with a size, and the average value of these regions is calculated and taken as the pooling feature. The pooled feature dimension is greatly reduced and could be used to train the classifier.

Step 4.

Feature parameter transfer: the feature weights obtained from the source domain data pre-training are applicable to the target domain datasets. We choose to freeze all the layers outside of the fully connected layers and retrain an output layer through the target data.

Step 5.

Feature classifier: the feature vector obtained based on the transfer learning serves as the input of an SVM categorizer and is used for training the landslide detection model.

The feature extraction is performed to obtain a set of feature vectors that can describe the image and is important for image classification and model training. The initial several layers of the networks capture basic and low-level features. These feature weights can be guaranteed and perform well in the model classification. By using pre-trained models trained on the source images, we can directly apply the corresponding structure and weights to the target detection. Finally, the images are recognized by the retrained model and improve the work efficiency.

The loss function of the transfer learning model is composed by the classification loss and domain transfer loss. Generally, a cross-entropy function is used to calculate the classification loss.

The domain transfer loss function minimizes the distance between the target domain and the source domain, so that to constrain the extracted features shared by these domains. We use the maximum mean discrepancy (MMD) to measure the domain distance. The domain transfer loss is expressed as follows:

$$L_D=MMD\left(X,Y\right)=\Vert \frac{1}{n}{\displaystyle \sum }_{i=1}^n\phi \left(x_i\right)+\frac{1}{m}{\displaystyle \sum }_{j=1}^m\phi \left(y_j\right)\Vert ,$$

(4)

where n and m are the numbers of instances of two different domains, x_i and y_j are the instances in the source domain and target domain respectively, and φ (·) is the kernel that converts two sets of features to a reproducing kernel Hilbert space (RKHS).

4. Experimental Results and Discussion

4.1. Data Labeling

The deep learning methods rely on a large amount of labeled data to train the model.

In the experiment, manually labeled the regions of interest (ROIs) and classified the images as the prior knowledge are necessary. We used the software LabelImg [43] to label the training data. The ROIs of the images are labeled by the green ground-truth boxes, manually removes the frame to select the area and adds the text label “Landslide”.

After a training sample is produced, it would be saved as the xml file. The number of the ground-truth boxes and each box’s name in the corresponding image are recorded. A labeling example is presented in Figure 8. After the training samples are annotated, the training dataset is built, and we can conduct the model training.

4.2. Model Training and Optimization

The experiment is based on the TensorFlow. Developed and maintained by Google Brain, TensorFlow is a symbolic mathematical system based on dataflow programming. It is widely used in the programming implementation of various machine learning algorithms [44]. The essence of training a deep learning model is to make good use of the loss function, which is used to evaluate the difference between the predicted value of the hypothetical function and the real value. The fitting effect refers to the size of the loss value. The better the fitting effect, the smaller the loss. The step of training the model is gradient descent which is to minimize the loss value. The size of the loss value can be used to measure the fitting effect. The loss curve will drop at the beginning during the training process. Then it will begin to converge, and gradually stabilize after a period of time. The loss curves of the SSD model and the transfer learning model can be seen in Figure 9.

The SSD model takes 25 epochs to converge, while the transfer learning model tends to smooth after five epochs to reach equilibrium. The transfer learning model requires less time to complete the model training and is more efficient.

4.3. Recognition Result and Evaluation

The examples of the test results of the SSD and the transfer learning model are shown below. As shown in Figure 10, landslides in the single images (a, b, c, d) can be detected correctly by the two models. However, there are two incorrect results detected by the SSD model in the ortho image (e) of the entire study area. The yellow ellipse is the landslide, which was not detected, and the red ellipse is not the landslide but was wrongly detected.

To validate the effect of landslide detection, we cite the evaluation indicators to compare the detection effects of the SSD and the transfer learning model. The indicators include: precision rate (Precision), recall rate (Recall), P-R curve and F1 score. The indicators are all calculated based on the following concepts.

• True positives (TP): the number of positive samples that are correctly identified as positive;
• False positives (FP): the number of negative samples that are incorrectly identified as positive;
• False negatives (FN): the number of positive samples that are not recognized;
• True negatives (TN): the number of samples correctly classified as negative.

4.3.1. Precision Rate

The indicator is the ratio of the number of positive samples correctly identified as positive samples. The metric is calculated as follows:

$$\mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP} }$$

(5)

Figure 11 shows the precision curves. Comparing with the two recognition precisions, it is obvious that the transfer learning model is better.

4.3.2. Recall Rate

The evaluation indicator reflects whether all positive examples have been confirmed. The calculation formula is as follows:

$$\mathrm{Recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$

(6)

The greater the value of the recall, the better the effect, the value 1 being the ideal state. We can see from Figure 12; the transfer learning model outperforms the SSD model.

4.3.3. P-R Curve (Precision-Recall Curve)

The P-R curve is a curve composed of precision and recall, with recall as the horizontal axis and precision as the ordinate axis. When the P-R curve is closer to the top right, the model performance is better. As shown in Figure 13, the model of the transfer learning is better than the SSD model.

4.3.4. F1-Score

The F1 score is the harmonic mean of recall and precision, which is a more comprehensive evaluation index. Values above 0.6 are considered satisfactory. The calculation formula is as follows:

$$\mathrm{F}1=2\ast \frac{precision\ast recall}{precision+recall}$$

(7)

As shown in Figure 14, the F1 score of the transfer learning model is 0.84, which is higher than that of the SSD model.

In terms of the comprehensive indicators, it is concluded that the transfer learning model performs better than the SSD model, with 0.95 precision, 0.9 recall, and 0.84 F1 measures. This shows that transfer learning is more effective for small sample detection. It not only improves the training efficiency, but also enhances the accuracy of the target detection. The experimental evaluation results are presented in

Table 2. Experimental evaluation results.

Index		SSD	Transfer Learning
	Model
Precision		90.24%	95.1%
Recall		35.38%	90%
P-R Curve		0.64	0.88
F1-Score		51%	84%

.

5. Conclusions

At present, UAV remote sensing is widely used, especially in the field of natural disasters and emergency rescue. Comparing it with the traditional methods, UAV has incomparable advantages. However, there are still some problems of poor effect of object detection and automatic information extraction from the remote sensing image. Deep learning methods are a research hotspot for artificial intelligence. In recent years, object detection and information extraction with deep learning has been used in many fields, and significantly improved efficiency.

This paper focuses on landslide detection. We proposed a fast image-processing method based on UAV data and transfer learning for landslide recognition. The transfer learning method is proposed to overcome the shortcomings of small-sample datasets. We collected the training data by the UAV system and selected three public datasets suitable for scene detection as the source domain data. Comparing with the SSD model and the transfer learning model, the evaluation metrics showed that the transfer learning model outperforms the SSD model.

For future work, we would like to integrate lightweight model to improve the detection speed. Since disaster relief requires faster response, it would be more efficient if the UAV flight control board could detect landslides in real time instead of data post processing. In addition, the rescue departments can make decisions faster based on the data and deploy forces to carry out disaster relief operations.

The dataset, on the other hand, is the key for deep learning detection. With the wide application of UAV technology, the UAV remote sensing data is also enriched. The establishment of a public UAV remote sensing dataset will provide a good platform for researchers. We are trying to make some attempts in the field of environment and disaster.