Transfer Learning-Based Automatic Hurricane Damage Detection Using Satellite Images

Abstract

After the occurrence of a hurricane, assessing damage is extremely important for the emergency managers so that relief aid could be provided to afflicted people. One method of assessing the damage is to determine the damaged and the undamaged buildings post-hurricane. Normally, damage assessment is performed by conducting ground surveys, which are time-consuming and involve immense effort. In this paper, transfer learning techniques have been used for determining damaged and undamaged buildings in post-hurricane satellite images. Four different transfer learning techniques, which include VGG16, MobileNetV2, InceptionV3 and DenseNet121, have been applied to 23,000 Hurricane Harvey satellite images, which occurred in the Texas region. A comparative analysis of these models has been performed on the basis of the number of epochs and the optimizers used. The performance of the VGG16 pre-trained model was better than the other models and achieved an accuracy of 0.75, precision of 0.74, recall of 0.95 and F1-score of 0.83 when the Adam optimizer was used. When the comparison of the best performing models was performed in terms of various optimizers, VGG16 produced the best accuracy of 0.78 for the RMSprop optimizer.

1. Introduction

There has been a steady increase in the occurrence of natural disasters since 1980 globally. The number of people that are prone to disasters is also increasing [1]. Amongst natural disasters, the most catastrophic disaster includes hurricanes that occur in areas with warm seawaters that are in tropical and subtropical areas. The sun heats seawater, leading to the formation of enormous clouds, which cause excessive rainfall, floods and very fast-moving winds [2]. Damage of approximately USD 265 billion was estimated in the US in the year 2017 due to three major hurricanes (Harvey, Maria and Irma). These hurricanes affected thousands of people and caused many fatalities. During such difficult times, the affected people required assistance. Hence, it is very essential assess the destruction brought about due to hurricanes [1].

Satellite images have been used to determine whether there has been damage inflicted by the hurricane or not. Satellite images have been gaining immense popularity for monitoring hurricanes. Ground surveys are time-consuming and also labor-intensive [3].

In artificial intelligence, transfer learning is a technique that involves reusing an already trained model on a different but related problem. This technique is now being popularly used in deep learning when the dataset is not large. This technique helps in the reduction in resources and the labeled data required for training newer models. It helps reduce training times [4].

A convolutional neural network consists of feature extraction as the first stage and classification as the final stage. In transfer learning, the classification stage is altered. The initialization of the network has been performed with weights from the ImageNet dataset [4]. The convolutional layers and the max-pooling layers are frozen so that no modification of weights takes place. Only the dense or the fully connected layers are left free to be altered. After this, the retraining of the model is performed. The advantage of the feature extraction stage is taken, and only the final classifier is tuned, which works better with smaller datasets. This is the reason for why it is called transfer learning, as the advantage of the knowledge of one problem can solve the second problem [5].

This paper involves the study of hurricane damage detection using satellite images. The estimation of the intensity of hurricanes was performed by using deep convolutional neural networks. Infrared images were used for estimation and were obtained from the satellite source. The adopted method is known as Deep Phurie, and it produced a very low root mean square (RMS) value in comparison with the method adopted earlier, which is known as Phurie. Deep Phurie is completely automatic, but this paper does not evaluate the damage post-disaster [2]. Furthermore, deep convolutional neural networks were used to estimate of the intensity of the tropical cyclones or hurricanes that took place over a period from 1998 to 2012. Regularization techniques were used along with many convolutional and dense layers. This technique helped in extracting features from hurricane images effectively. A low RMS value and an improved accuracy were obtained [6]. However, these data were noisy and not of good quality.

A multilayer perceptron was proposed for the determination of the connection between the appearance of hurricanes and the high-energy particles that flow out from the sun. A multilayer perceptron is an artificial neural network accompanied by backpropagation. It was found that hurricane appearances could take place a few days before the breakout of a solar wind [7].

As a deep learning method, a single-shot multibox detector (SSD) was employed for the calculate of the destruction inflicted on buildings due to hurricane Sandy, which occurred in the year 2012. The Vgg16 model and the SSD model were used, and improvements of 72% and 20% in mAp and mF1, respectively, were observed [8]. The CNN model was used to determine areas that were severely affected by Hurricane Harvey. Satellite images were used for the extraction of man-made features such as roads and buildings before and after the occurrence of the disaster. An F1 score of 81.2% was achieved [9].

Damage assessment after a hurricane is of utmost importance. In this paper, the author created a benchmark dataset for the property that became damaged by Hurricane Harvey. The dataset consisted of both undamaged and impaired building images, and they were obtained from satellite imagery. FEMA and TOMNOD were the sources of this dataset [10].

The destruction brought about because of hurricane Dorian has been determined using satellite imagery and artificial intelligence. The austerity of the destruction caused due to the hurricanes has been determined, and an accuracy of 61% was achieved [11].

Earlier studies focused on finding the intensity of hurricanes and providing a benchmark dataset for damage detection. Fewer studies have focused on classifying hurricane images into damaged and undamaged classes. In this paper, a comparative analysis of the four transfer learning models that include DenseNet121 [12], VGG16 [13], MobileNetV2 [14] and InceptionV3 [15] has been performed with respect to confusion matrix parameters. These models have also been used for determining the destruction brought about on buildings because of Hurricane Harvey.

The objectives of this study include the following:

• Addition of a newer set of layers to the pre-trained models for classification of the satellite images of hurricanes into damaged and undamaged categories;
• To generalize the model by applying data augmentation techniques to images;
• To perform a comparative study based on accuracy, precision, recall and F1-score for the four pre-trained models, which include VGG16, MobileNetV2, InceptionV3 and DenseNet121, at a learning rate of 0.0001 and 40 epochs.
• To compare the best performing models for various optimizers, which include SGD, Adadelta, Adam and RMSprop.

The rest of the paper is organized as follows: proposed methodology in Section 2; results and discussion in Section 3; conclusion and future scopes in Section 4.

2. Proposed Methodology

The model that has been presented for automatic damage detection due to hurricanes is shown in Figure 1. The platform used to create and run the algorithm is Kaggle. The model classifies satellite images into damaged and undamaged categories. The methodology comprises two main steps: The first is preprocessing [16,17], which is further divided into normalization and data augmentation, and the second is classification using the pre-trained CNN models. Each stage has been described below.

2.1. Preprocessing

The satellite images of the Houston region used in this study were captured by optical sensors. The images could be covered with clouds either partially or fully. This implies that the images obtained from the satellites have been corrupted by noise. The nature of the noise is unknown, meaning that it could be a result of fluctuations in light, the sensor of the camera or artifacts. Improving the quality of the images so that good results can be obtained is imperative. For this purpose, a denoising operation needs to be performed, which could be based on wavelets [18] or can be acquired from a compressive sensing method [19].

For the suppression of unwanted distortions or enhancement of some of the features of the images, pre-processing steps such as resizing were used. The original size of the satellite images of hurricanes is 128 × 128. The resizing of the satellite images of the hurricane was performed. The resizing of the images was performed at 224 × 224 when Vgg16, MobileNetV2 and DenseNet121 transfer learning techniques were applied. The images were resized to 299 × 299 on the application of the InceptionV3 technique.

The two main steps of the preprocessing stage, which include normalization and data augmentation, have also been explained in this section.

2.1.1. Normalization

Normalization is a very important step for maintaining numerical stability in a model. Normalization helps in learning faster and brings about stability in gradient descents. The input image pixels have, thus, been normalized in the values between 0 and 1. Normalization is brought about by multiplying pixel values by 1/255.

2.1.2. Data Augmentation

The augmentation of data is a technique utilized to generalize the model by applying random transformations with respect to input images [20,21]. It increases the variability and robustness of the model as the model becomes new and modified versions of the input data. An image data generator is utilized for augmenting the data, which is an on-the-fly data augmentation method because augmentation is performed during training time. The image data generator returns only the randomly modified images and not the original images. Data augmentation has been applied only to training images and not to testing images.

The techniques adopted for data augmentation in this study are rotation, width shifting, height shifting, horizontal flipping and zoom operation.

2.2. Hurricane Damage Detection Using Pre-Trained CNN Models

In this paper, four pre-trained models, which include VGG16 [22], MobileNetV2 [23], InceptionV3 [24] and DenseNet121 [25], have been used for classifying satellite images into damaged and undamaged classes.

Transfer learning models are models trained on very large datasets that include millions of images. As the models have been trained on such a large dataset, a generalization of the model takes place. The features that have been learned from the larger datasets help in solving a different problem consisting of lesser data or a smaller dataset. This helps eliminate the need to train a model from scratch.

The description of the architectures of these models is shown in

Table 1. Description of architecture of pre-trained CNN models.

Name of the Model	No. of Layers	Parameters (Millions)	Size of Input Layer
VGG 16	16	138	(224, 224, 3)
MobileNetV2	53	3.4	(224, 224, 3)
InceptionV3	42	24	(299, 299, 3)
DenseNet121	121	8	(224, 224, 3)

.

The VGG16 model comprises 16 layers that have weights and has approximately 138 million parameters. There are 13 convolutional layers and 3 fully connected layers. The VGG16 model is widely used because of its ease of implementation [22]. MobileNetV2 consists of 53 layers and 3.4 million parameters. It has been derived from the MobileNetV1 model, which utilizes depth-wise convolution as the building block of the model. However, the additional feature from the previous models is that it has an additional inverted residual layer [23]. This model is used because it is smaller in size and also cost-effective. There are nineteen bottleneck layers that were residual. There are 42 layers in the InceptionV3 model and 24 million parameters. InceptionV3 is an advanced version of InceptionV2. It reduces the amount of computations as it uses factorization methods [24]. For InceptionV3, the input image size is (299, 299, 3). Densenet121 consists of 121 layers with trainable weights. DenseNet121 has 8 million parameters. In this model, the network proceeds deeper as each layer is connected to all the other layers; for example, the first layer is connected to the second, third, fourth and so on layers. This leads to a improved maximum flow of information amongst the layers [25].

2.3. Tuning the Hyper-Parameters

The training of the four models has been performed for 40 epochs and a batch size of 100. The total epochs refer to how often the learning algorithm will be working through the complete dataset. Batch size refers to the number of training examples that would be utilized in a single iteration [26]. A batch size of 100 implies that 100 samples from the training dataset would be used for the estimation of the error gradient before the weights of the models are updated. The learning rate (LR) is another important hyperparameter that should not be either too big or too small [27]. It is used for finding the learning speed of the proposed models. The model would take a lot more time to reach the minimum loss if the LR is too small and if the LR is too high due to the fact that overshooting the low loss areas can take place. A learning rate of 0.0001 has been chosen in this paper. The batch size, number of epochs [28] and the learning rate have all been decided empirically.

Furthermore, the activation function [29] used is a rectified linear unit (ReLU) [30]. The fully connected head, used along with all the four pre-trained models, is shown in Figure 2. The pre-trained block is followed by a flattening layer and two dense layers. The flattening layer size of the DenseNet121 model is 50,176; for VGG16, the size is 25,088. For the MobileNetV2 model, the flattening layer size is 62,720; for InceptionV3, the size is 131,072. After the flattening layer, a dense layer of size 256 is applied. Finally, a dense layer with two classes that is damaged and undamaged is used.

The block diagram of the four pre-trained models, which include DenseNet121, VGG16, MobileNetV2 and InceptionV3, is displayed in Figure 3.

The block diagram of DenseNet121 is shown in Figure 3a. The input is of 224 × 224 × 3 sizes. This is applied to the DenseNet121 model, and the output obtained is of 7 × 7 × 1024 size. This is then applied to the new fully connected head, which comprises the flattening and dense layers. The output of the flattening layer is of 50,176 in size. The output of the first dense layer is 256, and the last dense layer classifies the images into two classes, which include damaged and undamaged classes.

The block diagram of the VGG16 model is demonstrated in Figure 3b. An input size of 224 × 224 × 3 was applied to the model, and an output of 7 × 7 × 512 was obtained. The output of the flattening layer is 25,088, and the outputs of the dense layers are 256 and 2 in size.

The block diagram of MobileNetV2 has been demonstrated in Figure 3c, for which its input image size is 224 × 224 × 3. This is applied to the model and an output of 7 × 7 × 1280 is obtained. The output after the application of the flattening layer is 62,720, and the outputs of the two dense layers are 256 and 2, respectively.

Figure 3d presents the InceptionV3 model, for which its input image size is 299 × 299 × 3. The output when this input is applied to the model is 8 × 8 × 2048. The output obtained after the flattening layer is of size 131,072.

3. Results and Discussion

This section includes the results performed on the four pre-trained models that are DenseNet121, VGG16, MobileNetV2 and InceptionV3, considering various parameters and then comparing the results of all four models.

3.1. Result Analysis in Terms of Loss and Accuracy

The results of the four transfer learning models, which include DenseNet121, VGG16, MobileNetV2 and InceptionV3, in terms of training performance parameters are shown in

Table 2. Training Performance of the four pre-trained models.

Model Name	Epoch	Training Loss	Training Accuracy	Training Recall	Validation Loss	Validation Accuracy	Validation Recall
DenseNet121	1	0.4231	0.8259	0.8266	0.2266	0.9171	0.9130
		…	…	…	…	…	…
	39	0.0690	0.9736	0.9732	0.1141	0.9559	0.9588
	40	0.0666	0.9727	0.9735	0.0956	0.9652	0.9635
VGG 16	1	0.3726	0.8309	0.8269	0.2628	0.8835	0.8829
		…	…	…	…	…	…
	39	0.1335	0.9435	0.9429	0.1505	0.9409	0.9391
	40	0.1191	0.9480	0.9480	0.1373	0.9455	0.9443
MobileNetV2	1	0.5143	0.8462	0.8416	0.2155	0.9072	0.8986
		…	…	…	…	…	…
	39	0.0838	0.9660	0.9662	0.1341	0.9438	0.9420
	40	0.0801	0.9662	0.9659	0.1341	0.9467	0.9478
InceptionV3	1	1.0183	0.7676	0.7526	0.2388	0.9055	0.9090
		…	…	…	…	…	…
	39	0.0923	0.9634	0.9638	0.0962	0.9571	0.9594
	40	0.0866	0.9648	0.9651	0.0972	0.9670	0.9658

. Training loss, training accuracy, training recall, validation loss, validation accuracy, and validation recall values for the models have been shown for various epochs in Table 2.

As per Table 2, the highest training accuracy of 0.9727 and training recall of 0.9735 were obtained by the DenseNet121model at the 40th epoch and learning rate of 0.0001. The highest validation accuracy of 0.9670 and validation recall of 0.9658 were obtained by the InceptionV3 model at the 40th epoch. The lowest training loss of 0.0666 and validation loss of 0.0956 has been obtained by the DenseNet121 model at the 40th epoch.

The training and validation accuracies for all the proposed models are demonstrated in Figure 4. The model accuracy for the DenseNet121 model has been displayed in Figure 4a. The model accuracy for the VGG16 model has been shown in Figure 4b; Figure 4c demonstrates the model’s accuracy for the MobileNetV2 model, and Figure 4d demonstrates the model accuracy for the InceptionV3 model.

Figure 5 demonstrates the training and validation loss for all four models. Figure 5a shows the model loss for DenseNet121, Figure 5b shows the model loss for the VGG16 model, Figure 5c shows the model loss for the MobileNetV2 model and Figure 5d displays the model loss for InceptionV3.

3.2. Confusion Matrix Parameter Result Analysis

Figure 6 shows the confusion matrix for all four models, which include DenseNet121, VGG16, MobileNetV2 and InceptionV3. The confusion matrix parameters are accuracy, precision, recall and F1-score [31,32], and they can be calculated with the help of the confusion matrix.

The results of the classification report or the confusion matrix parameters of the four original and modified models in terms of precision, recall, F1-score and accuracy have been displayed in

Table 3. Comparison of confusion matrix parameters of the original and modified pre-trained models.

Model Name	Original Transfer Learning Model				Modified Transfer Learning Model
	Precision	Recall	F1-Score	Accuracy	Precision	Recall	F1-Score	Accuracy
DenseNet121	0.82	0.08	0.15	0.40	0.92	0.10	0.17	0.40
MobileNetV2	0.69	0.41	0.52	0.50	0.64	0.63	0.64	0.53
InceptionV3	0.65	0.95	0.77	0.64	0.65	1.00	0.79	0.65
VGG16	0.75	0.85	0.80	0.72	0.74	0.95	0.83	0.75

. From Table 3, it is concluded that the best precision of 0.92 is obtained by the modified DenseNet121 model as compared to the 0.82 precision obtained by the original DenseNet121 model. The best recall of 1.00 is obtained by the modified InceptionV3 model. The best F1-score of 0.83 and accuracy of 0.75 are obtained by the modified VGG16 model. An improvement in the recall is 0.95, and it is obtained by the modified VGG16 model as compared to the recall of 0.85 obtained by the original VGG16 model.

Figure 7 compares the original and modified models in terms of the classification report parameters, which include precision, recall, F1-score and accuracy. The best accuracy of 0.75 is obtained by the VGG16 model followed by an accuracy of 0.65 obtained by the InceptionV3 model. The highest F1 score of 0.83 is obtained by the VGG16 model. The highest recall of 1.00 was obtained from the InceptionV3 model followed by a recall of 0.95 in the VGG16 model.

3.3. Comparison of Results of Various Optimizers

Different optimizers, which include SGD [33], Adadelta [34], Adam [35] and RMSprop [36], have been compared for the two best performing models, which include VGG16 and InceptionV3, as shown in

Table 4. Comparison of the original and modified VGG16 model for the various optimizers.

Optimizer	Original VGG16 Model				Modified VGG16 Model
	Precision	Recall	F1-Score	Accuracy	Precision	Recall	F1-Score	Accuracy
SGD	0.97	0.30	0.45	0.54	0.98	0.34	0.51	0.56
Adadelta	0.86	0.35	0.51	0.56	0.95	0.61	0.71	0.68
Adam	0.75	0.85	0.80	0.72	0.74	0.95	0.83	0.75
RMSprop	0.87	0.69	0.80	0.77	0.96	0.76	0.81	0.78

and

Table 5. Comparison of the original and modified InceptionV3 model for the various optimizers.

Optimizer	Original InceptionV3 Model				Modified InceptionV3 Model
	Precision	Recall	F1-Score	Accuracy	Precision	Recall	F1-Score	Accuracy
SGD	0.65	0.99	0.79	0.65	0.65	0.99	0.79	0.65
Adadelta	0.65	1.00	0.79	0.65	0.65	1.00	0.79	0.65
Adam	0.65	0.95	0.77	0.64	0.65	1.00	0.79	0.65
RMSprop	0.65	1.00	0.79	0.65	0.65	1.00	0.79	0.65

and Figure 8 and Figure 9.

3.3.1. Comparison of Original and Modified VGG16 Model

Table 4 shows the comparison of the original and modified VGG16 model for the four optimizers, which include SGD, Adadelta, Adam and RMSprop. The highest precision of 0.98 is obtained for the modified VGG16 model for the SGD optimizer. The highest recall of 0.95 is obtained by the modified model for the Adam optimizer. An improved F1-score of 0.83 is obtained for the Adam optimizer. The highest accuracy of 0.78 is obtained by the RMSprop optimizer.

Figure 8 displays the comparison of the original and modified VGG16 model. There is an improvement in accuracy from 0.77 for the original model to 0.78 for the modified model. An improved F1-score of 0.83 is obtained by the modified VGG16 model. The highest recall of 0.95 and precision of 0.98 were obtained by the modified model.

3.3.2. Comparison of Original and Modified InceptionV3 Model

Table 5 presents the comparison of the original and modified InceptionV3 model for four optimizers. It can be inferred from the table that, in the case of the InceptionV3 model, all optimizers produced almost the same results in terms of precision, F1-score and accuracy. For the modified model, an equal precision of 0.65, F1-score of 0.79 and accuracy of 0.65 were obtained for all four optimizers.

Figure 9 compares the results of the original and the modified InceptionV3 model, and it was observed that almost the same results were obtained for the model for all four optimizers.

3.4. Classification and Misclassification Results

Figure 10 shows the classification and misclassification results for the VGG16 model. Figure 10a shows that the actual class is undamaged and the predicted class is also undamaged. Figure 10b shows that the actual class is damaged and the predicted class is also damaged.

Figure 10c,d display the misclassification results. For Figure 10c, the actual class is undamaged and the predicted class is damaged. For Figure 10d, the actual class is damaged and the predicted class is undamaged.

3.5. Comparison with Present State-of-Art Deep Learning Models

Table 6. Comparison with present state-of-the-art deep learning models.

Reference/Year	Number of Images	Technique Used	Accuracy
[35]/2019	1128	VGG16	64.61%
[36]/2021	-	Stacked CNN	61%
[37]/2017	61,000	VGG16	74%
[38]/2018	48,828	CNN model with 5 Convolutional layers	77.85%
Proposed Transfer Learning Model	23,000	VGG16	78%

presents the comparison of the best Transfer learning model (VGG 16) obtained in this paper with the state-of-the-art deep learning models. The study was performed on 23,000 satellite images of the hurricane. VGG16 obtained an accuracy of 78%, which is more than the other deep learning models presented in Table 6. In reference number [37], work was performed on 1128 hurricane images using a VGG16 model and an accuracy of 64.61% was obtained. A stacked CNN model was used in reference number [11], and an accuracy of 61% was obtained. Work on 61,000 hurricane images was performed in reference [38] using a VGG16 model, and an accuracy of 74% was achieved. Accuracy at 77.85% was obtained by the CNN model comprising five convolutional layers in reference [6]. The hurricane images used were 48,828 in number.

3.6. Comparison with Present State-of-Art Machine Learning Models

This section compares the best Transfer learning model (VGG16) obtained in this paper with commonly used machine learning algorithms. VGG16 achieved an accuracy of 78%, and the best accuracy was achieved with the RMSProp optimizer and 23,000 satellite images. The hurricanes are accompanied by floods. Most authors have worked on floods using machine learning algorithms. Naive Bayes achieved an accuracy of 78.51% and Support Vector Machine achieved an accuracy of 91% when applied to 7500 images [39]. Random forest attained an accuracy of 82% when applied to 201 images [40]. Random forest attained an accuracy of 92% on 255 flood images [41]. The machine learning results are better since the analysis has been conducted on lower numbers of images, whereas the deep learning transfer learning models proposed in this paper have worked on a greater number of images, which involved 23,000 images.

4. Conclusions and Future Scope

In this paper, four pre-trained models, includingDenseNet121, VGG16, MobileNetV2 and InceptionV3, based on transfer learning have been put forward for the detection of destruction inflicted on buildings due to Hurricane Harvey, which took place in the Greater Houston region in the year 2017. The comparison of the four models has been performed based on training accuracy, training recall, training loss, validation accuracy, validation recall and validation loss. The highest training accuracy of 0.9727 and training recall of 0.9735 was obtained by the DenseNet121model at the 40th epoch and learning rate of 0.0001. The highest validation accuracy of 0.9670 and validation recall of 0.9658 was obtained by the InceptionV3 model at the 40th epoch. The lowest training loss of 0.0666 and validation loss of 0.0956 was obtained by the DenseNet121 model at the 40th epoch.

A comparison was also performed in terms of the classification report’s parameters, and it was found that VGG16 outperformed other models by obtaining an accuracy of 0.75, an F1 score of 0.83 and a recall of 0.95.

When the comparison was performed for the best-performing models for various optimizers in terms of the classification report parameters, it was found that VGG16 performed better by obtaining an accuracy of 0.78 for the RMSprop optimizer.

Furthermore, an improvement could be brought in with values of the confusion matrix parameters. Moreover, the model could be made more generalizable by including images of other hurricanes.