Deep Learning-Based Detection of Oil Spills in Pakistan’s Exclusive Economic Zone from January 2017 to December 2023

Abstract

Oil spillages on a sea’s or an ocean’s surface are a threat to marine and coastal ecosystems. They are mainly caused by ship accidents, illegal discharge of oil from ships during cleaning and oil seepage from natural reservoirs. Synthetic-Aperture Radar (SAR) has proved to be a useful tool for analyzing oil spills, because it operates in all-day, all-weather conditions. An oil spill can typically be seen as a dark stretch in SAR images and can often be detected through visual inspection. The major challenge is to differentiate oil spills from look-alikes, i.e., low-wind areas, algae blooms and grease ice, etc., that have a dark signature similar to that of an oil spill. It has been noted over time that oil spill events in Pakistan’s territorial waters often remain undetected until the oil reaches the coastal regions or it is located by concerned authorities during patrolling. A formal remote sensing-based operational framework for oil spills detection in Pakistan’s Exclusive Economic Zone (EEZ) in the Arabian Sea is urgently needed. In this paper, we report the use of an encoder–decoder-based convolutional neural network trained on an annotated dataset comprising selected oil spill events verified by the European Maritime Safety Agency (EMSA). The dataset encompasses multiple classes, viz., sea surface, oil spill, look-alikes, ships and land. We processed Sentinel-1 acquisitions over the EEZ from January 2017 to December 2023, and we thereby prepared a repository of SAR images for the aforementioned duration. This repository contained images that had been vetted by SAR experts, to trace and confirm oil spills. We tested the repository using the trained model, and, to our surprise, we detected 92 previously unreported oil spill events within those seven years. In 2020, our model detected 26 oil spills in the EEZ, which corresponds to the highest number of spills detected in a single year; whereas in 2023, our model detected 10 oil spill events. In terms of the total surface area covered by the spills, the worst year was 2021, with a cumulative 395 sq. km covered in oil or an oil-like substance. On the whole, these are alarming figures.

1. Introduction

Oil spills are a major contributor to marine pollution and continue to pose a significant threat to marine and coastal ecosystems. They are mainly caused by ship accidents, bilge dumping and seepage from natural oil reservoirs [1]. In the last few decades, Synthetic-Aperture Radar (SAR) has been widely used for detection and classification of oil spills and look-alikes, because of its operational capability in all-day, all-weather conditions. Oil spills dampen the capillary waves on the waters and reduce electromagnetic backscatter, making the oil spill appear as a dark stretch in SAR images [2]. These dark stretches can also occur as a result of natural phenomena, such as low-wind areas, shallow waters, algae blooms and grease ice, etc. [1]. These are generally known as look-alikes, and they render the classification problem even more complex.

Several classification methods have been proposed in the literature, to discriminate between oil spills and look-alikes in remote sensing data. In most of the cases, a three-step procedure is followed: (1) detection of dark spots, (2) features extraction and (3) classification of oil spills and look-alikes regions. Solberg et al. [2] presented an automated framework based on the three-step procedure to classify oil spills and look-alikes. However, this approach requires prior probability of the existence of the oil spill. A similar probability-based approach was followed in [3], where subsequent images were compared, to define templates and classify them as oil spills or look-alikes. The authors in [4] proposed a pre-processing step that improves the classifier’s performance by using a level-set method for image segmentation, as compared to the previous edge detection or threshold-based approaches. CNNs are an efficient alternative to probabilistic and threshold-based approaches. They are more robust in handling classification problems. Towards this end, several studies [5,6,7,8,9] were conducted to address the oil spill classification problem, using CNNs. The advantage of using CNNs was that they could be trained end-to-end and learn the input–output mapping from examples [10]. This end-to-end training would simplify the task of defining critical thresholds and parameters, and would thereby reduce the human effort in an otherwise solely expert analyst-based approach. To reduce the human effort, both shallow and deep neural networks were exploited [11,12]. These studies claimed high accuracy, but the results were based on only two classes, viz., oil spill and look-alikes. SegNet [13] is a popular Deep Convolutional Neural Network (DCNN) for semantic segmentation. An oil spill detection method based on SegNet was proposed and applied to SAR images of a pre-confirmed oil spill [14]. The model performed well under high-clutter conditions. However, the model was again based on and limited to classification of SAR images into two (binary) classes, i.e., oil spill and look-alikes. Krestenitis et al. [15] proposed a deep DCNN based on the architecture of DeepLab [8] for semantic segmentation of SAR images into regions of interest, such as sea surface, oil spills, look-alikes, ships and land. They showed that among the existing semantic segmentation models, DeepLabV3+ [16,17,18] achieved the best performance.

Satellite-based SAR remote sensing makes oil spill detection possible from a regional to a global scale. This is exemplified by the CleanSeaNet service [19] provided by the European Maritime Security Agency (EMSA), which offers regular coverage over European waters and reports oil spills. Similarly, several countries around the world that have dedicated satellite missions for environmental monitoring are observing their waters for any oil spills. However, to the best of our knowledge, there has been no systematic study on the detection and classification of oil spills in Pakistani waters that could serve as a motivation for this work. Due to the lack of a formal remote sensing-based early warning system, typically oil spill events in these waters remain undetected. At times, oil spills have not even been reported until the oil had spread to the beaches and local residents had lodged complaints. We aimed to find out the spills that remained mostly undetected in Pakistan’s Exclusive Economic Zone (EEZ)—wherein lie the primary maritime trade routes of the country—in the Arabian Sea for the last seven years, i.e., January 2017–December 2023. Towards this end, we employed a DCNN-based method for multi-class semantic segmentation of SAR images.

2. Data and Methods

In this work, we have developed an oil spill detection framework based on deep learning and spaceborne SAR images, for the study area as shown in Figure 1. Our framework relies on two datasets. The first dataset comprises selected verified oil spillages, as observed in Sentinel-1 imagery and confirmed by EMSA authorities. The second dataset is the one that we have prepared exclusively for Pakistan’s EEZ.

2.1. Benchmark Dataset for Model Training

Krestenitis et al. [20] developed a labeled dataset of several oil spill events; it is publicly available at URL: (https://mklab.iti.gr/, accessed on 23 September 2020). The dataset contains spaceborne SAR acquisitions containing oil spill events verified by the EMSA through the CleanSeaNet service. These SAR images are from the Sentinel-1 constellation operated by the European Space Agency (ESA). The images cover a ground range of approximately 250 km in Interferometric Wide (IW) swath mode with a resolution of 10 m. The images are dual-polarized, i.e., VV and VH, but only VV polarized images were used in developing the dataset. After a series of pre-processing steps, the authors retained 1112 SAR images, which were split into training and test data subsets comprising 1002 and 110 images [20], respectively. The dataset contains manually annotated ground truth masks with a distinct RGB color assigned to each of the classes, viz., sea surface, land area, oil spill, look-alikes and ships.

2.2. Dataset for Oil Spill Monitoring in Pakistan’s EEZ

Sentinel-1 acquires images regularly over the Arabian Sea, and the imagery is available freely and openly under the Copernicus Program. We downloaded Sentinel-1 Ground Range Detected High-resolution (GRDH) imagery over Pakistan’s EEZ, available from the Copernicus Open Access Hub, for the duration January 2017–December 2023. The data were in the IW mode, with a resolution of 10 m. The data were dual-polarized, i.e., VV and VH, but we retained only VV polarization (as for the training dataset). We used the data of specific orbital paths (13, 78 and 151) covering the seaward as well as the coastal territories of Pakistan. The aim was to detect potential oil spills in the Arabian sea, with particular emphasis on the regions close to the coastline. In order to prepare the dataset for testing, a series of pre-processing steps were applied, as shown in Figure 2. The images were radiometrically calibrated and a $7\times 7$ median filter was applied, to reduce speckle. Employing visual inspection, areas containing potential oil spills were cropped from the images, and a patch size of $320\times 320$ was used for testing. This test set contained the relevant five classes akin to the public (training) dataset.

2.3. Model Development and Training

Our methodology for oil spill detection is based on the semantic segmentation of SAR images. Due to irregularity in oil slick shapes and textures, a single label for the entire image is not sufficient to detect potential oil spills. Similarly, other approaches, like object-based detection [21] and assigning multiple labels to a single image [22], do not perform well in oil spill detection cases. On the contrary, semantic segmentation classifies the multiple classes of interest in a single image at pixel level, making it suitable for complex problems like oil spill detection and classification.

UNet stands out as a widely recognized architecture initially designed for segmenting biomedical images. Its encoder–decoder structure has proven effective in delineating objects of interest within images, and it has been used in many remote sensing studies ranging from land cover classification to object detection [5,6,7,23]. Concurrently, ResNet-101 architecture has emerged as an efficient feature extractor among the deep convolutional networks.

Motivated by the strengths of both the architectures, this study proposes the use of the ReU-Net model for oil spill detection in SAR imagery. In ReU-Net, the encoder (contracting) part of the original UNet architecture is replaced by the ResNet-101 feature extractor. This integration aims to harness the efficient feature extraction capabilities of ResNet-101 while retaining the robust segmentation abilities of UNet. The proposed architecture is presented in Figure 3.

At its core, the ResNet-101 begins with a batch normalization layer, succeeded by a convolutional layer featuring a filter size of $7\times 7$, batch normalization and Rectified Linear Unit (ReLU) activation. Subsequently, a maxpool layer is applied to downsample the feature map, followed by 33 residual blocks. Each residual block comprises three stages: stage 1, stage 2 (bottleneck) and stage 3. Within each stage, convolutional layers of sizes $1\times 1$, $3\times 3$ and $1\times 1$ are sequentially applied. Batch normalization and ReLU activation are integrated within stage 1 and stage 2, as delineated in the schematic presented at the bottom-right of Figure 3. The numerical annotations within the boxes, named as Res-x, denote the number of residual blocks sequentially connected within the respective box. Further comprehensive insights into the ResNet-101 can be found in [24].

In adherence to the original UNet architecture, the decoder part employs a series of Up-convolution layers at five different stages. These layers, featuring a $2\times 2$ filter size, serve to reconstruct features by doubling the height and width while keeping the number of feature maps the same. This process of upsampling is crucial for restoring spatial information lost during the encoding phase. The upsampled features are then fused (via skip connections) with their corresponding features (shown as dark-blue) from the encoder through a copy-and-concatenate operation, facilitating the integration of both high-level semantic information and detailed spatial features. A critical component of the decoder is the Conv-B block, highlighted in green in Figure 3. This block consists of two consecutive $3\times 3$ convolutional layers, each followed by batch normalization and ReLU activation. In the final decoder stage, a $1\times 1$ convolutional layer is applied to map the aggregated features to five channels (corresponding to number of classes in our case), consolidating the information extracted throughout the network into a compact representation. A softmax activation function (presented in yellow) is used in the last step to produce a probability distribution in a range between 0 and 1. Finally, the output is a semantic segmentation mask of the assigned class labels. A deeper dive into the model development can be found in our previous work [25].

2.4. Experimental Setup

We trained ReU-Net on the benchmark dataset from Krestenitis et al. [20]. The model was trained using input images of shape $320\times 320$ and batch sizes of 12. To improve the model performance and avoid overfitting, we applied random data augmentation [26]. We used 1002 labeled images for training and 110 images for validation and testing. The model was trained with a combination of three loss functions, viz., categorical focal loss, Jaccard loss and gradient profile loss. The evaluation metric used for assessing the performance was mean Intersection over Union (mIoU).

Categorical focal loss function is a useful loss function to address the class imbalance problem. During training, it pays more attention to reducing a large number of false negatives, which, in turn, improves overall classification performance. Mathematically, the categorical focal loss is defined by adding a modulating factor ${(1-p_t)}^{\gamma }$ to the cross-entropy loss function [27],

$${\mathcal{L}}_{\mathrm{FL}}\left(p_t\right)=-{\alpha }_t{(1-p_t)}^{\gamma }log\left(p_t\right)$$

(1)

where $\alpha$ and $\gamma$ are the hyperparameters.

The Jaccard index is a commonly used performance metric for semantic segmentation, which measures the similarity between ground truth and predicted class labels. Considering y as the ground truth and $\widehat{y}$ as the predicted class labels, the Jaccard loss function can be mathematically defined as follows [28]:

$${\mathcal{L}}_{\mathrm{jac}}(y,\widehat{y})=1-{\displaystyle \frac{\left(y·\widehat{y}\right)+ϵ}{(y+\widehat{y}-y·\widehat{y})+ϵ}}$$

(2)

where $ϵ$ is used to prevent division by zero. The subtrahend corresponds to the Intersection over Union (IoU) value. Therefore, the use of the Jaccard loss function for the training aims to directly increase the IoU value.

Gradient profile loss is a new loss function, recently presented in our earlier work [25]. It computes the similarity between ground truths and predicted class labels by considering rows and columns as spatial profiles. Mathematically, the similarity over each image channel can be computed as follows:

$$\begin{array}{c}\hfill \mathcal{S}(y,\widehat{y})=\sum _c\left(\frac{1}{H}tr\left(y_c·{\widehat{y}}_c^{\tau }\right)+\frac{1}{W}tr\left(y_c^{\tau }·{\widehat{y}}_c\right)\right)\end{array}$$

(3)

where y and $\widehat{y}$ are the ground truth mask and the predicted class labels, respectively; $tr(.)$ is the trace of a matrix and ${(.)}^{\tau }$ corresponds to the transpose of a matrix; and the subscript c represents each image channel. The loss function is computed in the image gradients space and we call it the gradient profile loss:

$${\mathcal{L}}_{\mathrm{GP}}(y,\widehat{y})=-\mathcal{S}(\nabla y,\nabla \widehat{y}).$$

(4)

For further details, readers are referred to [25].

3. Results

The performance of the proposed ReU-Net framework was assessed on the test split of the annotated benchmark dataset. Then, the trained and validated model was used to detect potential oil spills in Pakistan’s EEZ, imaged by Sentinel-1 in its orbital paths 13, 78 and 151.

3.1. Performance on the Benchmark Dataset

As stated earlier, the oil spill detection dataset contains 1112 labeled SAR images. The original train and test split of this dataset was 1002 and 110 SAR images, respectively. We tested 110 SAR images, using our trained model to obtain a semantic segmentation mask containing the predicted class labels, viz., oil spill, look-alikes, ship, land and sea surface. The qualitative results showing the ground truth masks and predicted class labels are shown in Figure 4. As shown in row 1 of Figure 4, the model accurately detected the oil spill as well as all the ships near the spill. Moreover, an additional patch of oil was detected that corresponded to a missed ground truth label or misclassification of sea-surface pixels as oil-spill pixels. As shown in row 2, the coastal area was correctly classified, and most of the look-alike pixels were also correctly classified. As shown in row 3, the model also classified the look-alikes and land area accurately. Moreover, a large ship near the look-alikes was also identified. We then computed the Intersection over Union (IoU) value for each class and also the mIOU value for all the classes. The IoU values for sea surface, oil spill, look-alikes, ship and land were 96.00%, 63.95%, 60.87%, 74.61% and 96.81%, respectively. The comparison of the ReU-Net model with other segmentation models is presented in our earlier work, representing nearly a 13% improvement, compared to the state of the art, for this dataset, with an mIOU score of 78.45% for all classes [25].

3.2. Classification and Detection of Spills in Pakistan EEZ

To detect potential oil spills in Pakistan territorial waters, we tested the unseen data of Pakistan’s EEZ containing potential oil spills. Our classifier suggested 92 potential cases as oil spills from January 2017 to December 2023. Each detection was followed by SAR expert validation. Four SAR images from the dataset developed for testing and their predicted class labels are shown in Figure 5. In columns 1, 2 and 3, we can spot a probable bilge-dumping scenario by visual inspection, where a ship is releasing oil into the sea. The model also suggested the dark stretch on the ship’s trail as an oil spill. Referring to column 4, the model detected multiple oil slicks on the surface of the ocean and a nearby ship.

3.3. Discussion

Having trained the deep learning model (ReU-Net) on a dataset of verified spills—and exceeding the performance benchmark by nearly 13%—we were assured that the model could be reliably used to detect oil spills in unseen data from the same sensor, processed similarly to the benchmark training data. We therefore proceeded with the application of the trained classifier on the test imagery over Pakistan’s EEZ. This led to the identification of 92 oil spill incidents in the EEZ over the last seven years (2017–2023). The yearly breakdown of these incidents is shown in Figure 6. The highest number of spills during this period was detected in 2020, but in terms of the total surface area covered by the spills the worst year was 2021, with a cumulative 395 sq. km found to be covered in oil. Figure 7 shows the spatial distribution and other physical characteristics of the spills (length of the spills, distance from the shoreline and spill surface area). It can be seen that several spills were detected in each path of the sensor. The longest spill detected was up to 175 km long, while most of the spills (up to 75%) were as long as 25 km. The mean distance from the shoreline was 87 km. Details on individual spill incidents are listed in Appendix A.

4. Outlook

This study aims to bring to light the problem of increasing marine pollution in Pakistan’s EEZ. Continued and mostly unattended oil spills are a major contributor. In general, it is expected that concerned authorities are more vigilant. Understandably, due to the vastness of the territorial waters, it is practically challenging to locate spill incidents solely through patrolling activities. A remote sensing-based detection system, as presented in this work, is indeed productive. We have demonstrated the successful use of a deep learning-based system that ingests SAR imagery and detects oil spills, while classifying stretches of oil among other classes, comprising normal sea surface, land, ships and look-alikes. In the future, we aim to extend our work towards operationally deployable models, to seek further improvement in classification accuracy and, thereafter, to extend our analysis to further paths overlapping Pakistan’s EEZ and neighboring waters.