SAR Image Fusion Classification Based on the Decision-Level Combination of Multi-Band Information

Abstract

Synthetic aperture radar (SAR) is an active coherent microwave remote sensing system. SAR systems working in different bands have different imaging results for the same area, resulting in different advantages and limitations for SAR image classification. Therefore, to synthesize the classification information of SAR images into different bands, an SAR image fusion classification method based on the decision-level combination of multi-band information is proposed in this paper. Within the proposed method, the idea of Dempster–Shafer evidence theory is introduced to model the uncertainty of the classification result of each pixel and used to combine the classification results of multiple band SAR images. The convolutional neural network is used to classify single-band SAR images. Calculate the belief entropy of each pixel to measure the uncertainty of single-band classification, and generate the basic probability assignment function. The idea of the term frequency-inverse document frequency in natural language processing is combined with the conflict coefficient to obtain the weight of different bands. Meanwhile, the neighborhood classification of each pixel in different band sensors is considered to obtain the total weight of each band sensor, generate weighted average BPA, and obtain the final ground object classification result after fusion. The validity of the proposed method is verified in two groups of multi-band SAR image classification experiments, and the proposed method has effectively improved the accuracy compared to the modified average approach.

1. Introduction

Remote sensing image processing has been a hot research issue recently [1,2,3]. A synthetic aperture radar (SAR) is a high-resolution imaging radar that is not affected by climate and day and night. It has great application value. SAR image classification is used to classify each pixel of the SAR image into its corresponding category [4,5,6,7,8]. Single-band SAR images can obtain limited target information, while multi-band SAR systems can simultaneously perform high-resolution imaging in multiple bands [9,10], which can describe the characteristics of the surface more comprehensively. By fusing the classification results of multi-band SAR images, a more accurate and reliable classification result can be obtained than using only single-band image information.

In recent decades, the classification of SAR images has flourished [11,12,13,14,15]. According to whether the labeled data participates in the training process, the existing algorithms can be roughly divided into three categories: unsupervised learning-based, supervised learning-based and semi-supervised learning-based. Zhao et al. [16] proposed a discriminative deep belief network, which is used to build multiple weak classifiers to make decisions and input the decision features into the deep belief network to learn deep features for high-resolution SAR image classification. Hou et al. [17] proposed an algorithm combining a superpixel segmentation algorithm with stacked autoencoders for the classification of polarimetric SAR image features. The data processing uses a multilayer autoencoder to extract deep features and uses a superpixel segmentation method to optimize image classification results. Among them, a superpixel is a small area composed of a series of adjacent pixels with similar color, brightness, texture, and other characteristics. Most of these small areas retain effective information for further image segmentation and generally do not destroy the boundary information of objects in the image. Zhang et al. [18] proposed a deep ensemble model combining the gradient boosting method and convolutional neural network (CNN). This method performs well in optical remote sensing images. Zhou et al. [19] proposed a deep convolutional neural network that can automatically learn hierarchical polarization spatial features from the data, which can classify SAR images. Shang et al. [20] proposed a densely connected and deeply separable convolutional neural network, using depth separable convolution to replace standard convolution, and independently extracting features on each channel of the polarization image so as to achieve SAR image classification. Ni et al. [21] use long short-term memory networks for recurrent learning, random adjacent pixel blocks for data augmentation, and conditional random fields for post-processing, which performs well.

In general, the development of SAR image classification technology has become more and more mature, but how to comprehensively consider the features of multiple band images and accurately realize the classification of SAR images is still an unsolved problem. In this paper, in order to fully exploit the complementarity of multi-band classification information to complete the SAR classification of the same scene, a new decision fusion method called the SAR image classification method based on the decision-level combination of multi-band information is proposed. Within the proposed method, the Dempster–Shafer theory [22,23,24] is employed to model the uncertainty of the classification result of each pixel and used to combine the classification results of multiple band SAR images. Firstly, multi-band SAR image data are collected by sensors that are input to the CNN to obtain single band classification results. Secondly, the belief entropy [25] of each pixel classification is calculated to measure the uncertainty of the classification, and a basic probability assignment (BPA) is generated after the normalization for each band. Then, in terms of the idea of term frequency-inverse document frequency (TF-IDF) [26,27] and the neighborhood influence, the total weight for every band of each pixel is calculated to implement the weighted average combination of BPAs coming from multiple band images. Finally, the final classification result can be obtained according to the combined BPA. Our method uses decision fusion under the framework of evidence theory to measure the uncertainty of classification results of different bands. The evidence combination is used to fuse the classification results of different bands, which can reduce the uncertainty of the classification results of different bands and improve the classification accuracy. The difficulty of the decision fusion method is how to better measure the complementarity between the evidence. By introducing the idea of TF-IDF text mining into the conflict coefficient, we propose a new method to measure the similarity of evidence. Using this new method combined with neighborhood information can well measure the complementarity between pixels and, thus, obtain more accurate decision fusion results.

The rest of the paper is organized as follows: Section 2 introduces a single-band SAR image classification based on convolutional neural networks. Section 3 proposes a SAR image classification method based on the decision-level combination of multi-band information. In Section 4, two sets of multi-band SAR images are used to verify the effectiveness of the proposed method. In Section 5, a summary of the work in this paper is made.

2. Single-Band SAR Image Classification Based on Convolutional Neural Networks

2.1. Convolutional Neural Networks (CNN)

CNN adopts a weight-sharing network structure to reduce the number of weights and the connections between various layers of the network, which has been widely used in image processing [28], facial recognition [29,30], natural language processing [31,32], and other fields [33,34,35,36,37,38,39,40].

2.1.1. Convolutional Layers

Convolutional layers of CNN mainly conduct convolution operations, where image features can be extracted through convolution operations. The convolution operation is to slide the convolution kernels on the input matrix to find the dot product of the current area. After repeating the operation, the convolutional results can be obtained.

2.1.2. Pooling Layers

Pooling layers are connected after the convolutional layers to compress the extracted features, which can highlight the effective information. Maximum pooling and average pooling are generally applied on the pooling layers. Maximum pooling takes the maximum value of the current scan area, and the average pooling takes the average value of the current scan area.

2.1.3. Fully Connected Layer

The fully connected layer is used to integrate features extracted from the previous layer, which can be adopted for classification. The number of outputs is equal to the category number for classification, and all nodes of the fully connected layer are connected with the previous layer.

2.2. Single-Band SAR Image Classification

In our work, the CNN structure for single-band SAR image classification can be shown in Figure 1. The network is composed of three convolutional modules and three fully connected layers (FC layer).

Each convolutional module includes a convolutional layer with a convolution kernel of $3\times 3$, a BatchNorm layer, and a ReLU layer. The output of the convolution module is:

$$Ou{t}_1=ReLU(BN(f_1(x)))$$

(1)

$$Ou{t}_2=ReLU(BN(f_2(Ou{t}_1)))$$

(2)

$$Ou{t}_3=ReLU(BN(f_3(Ou{t}_2)))$$

(3)

where, $x$ is the input, $Ou{t}_k$ is the output of the $k$ convolutional module, $k=1,2,3$, $f_k$ represents convolutional operation, $BN$ is a BatchNorm function, and ReLU is an activation function.

Fully connected layers are used for classification. The number of input and output nodes of the first FC layer is 3872 and 4096. The number of input and output nodes of the second FC layer is 4096 and 1024. The number of input and output nodes of the third FC layer is 1024 and the number of categories. The output of three FC layers is:

$$Out=F{C}_3(F{C}_2(F{C}_1(Ou{t}_3)))$$

(4)

where $F{C}_k$ is fully connected layer, $k=1,2,3$.

Patches of the single-band SAR image are imported to the network, and pixel-level classification results are output from the last fully connected layer.

3. SAR Image Classification Method Based on the Decision-Level Combination of Multi-Band Information

Assuming there are types of sensors with different wavebands, denoted as: $X=\left\{x_1,x_2,\cdots ,x_n\right\}$. Once the image data obtained by the sensors are classified, $h$ types of categories for each pixel $u_{ij}$ will be generated, denoted as: $\Theta =\left\{{\theta }_1,{\theta }_2,\dots ,{\theta }_h\right\}$. The flowchart of an SAR image classification method based on the decision-level combination of multi-band information is shown in Figure 2.

As can be seen from Figure 2, the classification results of each pixel in a single-band SAR image obtained in Section 2.2 are expressed as a probability matrix. The belief entropy uses Shannon entropy to measure the reliability between different pieces of evidence. The belief entropy of each pixel classification in the probability matrix is calculated to measure the uncertainty of the classification, and a BPA is generated for each band. In terms of the idea of TF-IDF, the weight of different sensors is calculated. Then, considering the influence of the classification results on the neighborhood of SAR images in each band, the neighborhood influence weight is calculated. The two weights are multiplied, and the total weight is obtained after normalization. The weighted average of the BPAs of different bands is used to obtain the average BPA, which is combined to obtain the final classification result.

3.1. Construction of Basic Probability Assignment (BPA) Functions of the Sensors’ Classification

3.1.1. Construct the Probability Matrix

By using the method in Section 2.2 to classify each pixel $u_{ij}$ in the image, a probability matrix $Q$ can be obtained: $Q=\left[\begin{array}{ccccc}{q}_{11}& \cdots & q_{1b}& \cdots & q_{1n}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ q_{a1}& \cdots & q_{ab}& \cdots & q_{an}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ q_{h1}& \cdots & q_{hb}& \cdots & q_{hn}\end{array}\right]$, where, $q_{ab}$ represents the probability that the sensor $x_b$ recognizes the pixel $u_i{}_j$ as the feature category ${\theta }_a$, $a=1,2,\cdots ,h$, $b=1,2,\cdots ,n$, $i$, $j$ are the $x$ and $y$ coordinates of the pixel point.

3.1.2. Construct the BPAs

Suppose $\Theta =\left\{{\theta }_1,{\theta }_2,\dots ,{\theta }_h\right\}$ is a frame of discernment, the cardinality of $\Theta$ is $h$, then the number of elements in the power set $2^{\Theta }$ of $\Theta$ is $2^h$, and a basic probability assignment (BPA) function $m$ should satisfy: $m(\varnothing )=0$ and ${\displaystyle \sum _{A\subseteq \Theta }m(A)=1}$, where $\varnothing$ is the empty set, and $A$ is a subset of $\Theta$, $m(A)$ is the basic probability number of proposition $A$. For the probabilities of pixel $u_i{}_j$’s classification result generated by each sensor $x_b$, an unnormalized BPA can be generated as: ${\widehat{m}}_b(\{{\theta }_a\})=q_{ab}$. Then, the belief entropy is used to calculate the uncertainty of ${\widehat{m}}_b$ and assign it to the mass of $\Theta$: ${\widehat{m}}_b(\{\Theta \})=-{\displaystyle \sum _{\theta \subseteq \Theta }m(\{\theta \})}\ln \frac{m(\{\theta \})}{2^{\left|\theta \right|}-1}$, where, $\left|\theta \right|$ represents the cardinality of proposition $\theta$. At last, we normalize ${\widehat{m}}_b$ to obtain $m_b\left(\right\{{\theta }_a\left\}\right)=\frac{{\widehat{m}}_b\left(\right\{{\theta }_a\left\}\right)}{{\displaystyle \sum _{a=1}^h{\widehat{m}}_b\left(\right\{{\theta }_a\left\}\right)+{\widehat{m}}_b(\{\Theta \})}}$, for any ${\theta }_a\in \Theta$ and $m_b\left(\right\{\Theta \left\}\right)=\frac{{\widehat{m}}_b\left(\right\{\Theta \left\}\right)}{{\displaystyle \sum _{a=1}^h{\widehat{m}}_b\left(\right\{{\theta }_a\left\}\right)+{\widehat{m}}_b(\{\Theta \})}}$. Calculating the belief entropy [41,42,43] of the classification results of each pixel can measure the uncertainty of the classification results of SAR images. Since evidence theory is a method that can effectively fuse uncertainty information, the evidence fusion can achieve more accurate classification while reducing uncertainty.

3.2. Calculate the Weights of Sensors in Different Bands

Here, we calculate the conflict coefficient between the sensors’ classification results of each band to obtain the degree of conflict between different sensors. If the sum of the conflict coefficients between a certain sensor and the remaining sensors is larger, it means that the result obtained by this sensor is more inconsistent with the judgment of other sensors. By combining the conflict coefficient and the idea of TF-IDF, the weights of different sensors are obtained.

3.2.1. Calculate the Conflict Coefficient between the BPA of Sensors in Different Bands

For two BPAs, $m_b$ and $m_c$, where $b$ and $c$ represent the serial numbers of the sensors in different bands, the conflict coefficient [44] is used to measure the degree of conflict between $m_b$ and $m_c$. The greater the conflict, the greater the value of the conflict coefficient. The formula is as follows: $K_{bc}=\frac{1}{2}[k_{bc}+d_{BPA}(m_b,m_c)]$, where $k_{bc}$ is the classic conflict coefficient, expressed as: $k_{bc}={\displaystyle \sum _{\begin{array}{l}B\cap D=\varphi \\ B\cup D\ne \varphi \end{array}}{m}_b(B)m_c(D)}$, and $d_{BPA}(m_b,m_c)$ is the Jousselme evidence distance: $d_{BPA}(m_b,m_c)=\sqrt{\frac{1}{2}{({\overrightarrow{m}}_b-{\overrightarrow{m}}_c)}^T\underset{\_}{\underset{\_}{D}}({\overrightarrow{m}}_b-{\overrightarrow{m}}_c)}$,where $\underset{\_}{\underset{\_}{D}}$ is a $2^n\times 2^n$matrix whose elements are $\underset{\_}{\underset{\_}{D}}=\frac{\left|B\cap D\right|}{\left|B\cup D\right|}$, where $B$ and $D$ are the subsets of the frame of discernment.

3.2.2. Calculate the TF-IDF Weights of Sensors in Different Bands

TF-IDF is a keyword extraction method: TF-IDF = TF × IDF, where $TF$ represents the number of occurrences of a term in the article, $IDF$ weights the value of $TF$ according to the importance of the term in the corpus, where $IDF=\log (\frac{C_{total}}{C_{number}+1})$, where $C_{total}$ represents the total number of articles in the corpus, $C_{number}$ represents the number of articles containing the term. Here, in terms of IDF’s weighting idea, suppose there are $n$ BPAs for a certain BPA $m_b,b=1,2,\cdots n$ if it is assumed to completely conflict with the rest of $n-1$ BPAs; that is, the conflict coefficient is 1, the value of $C_{total}$ should be $n-1$. Actually, the sum of the conflict coefficients of $m_b$ with the rest of the $n-1$ BPAs is ${\displaystyle \sum _{b=1,c\ne b}^{n-1}{K}_{bc}}$. Therefore, the weights of sensors in different bands can be obtained: for sensor $b$, an unnormalized weight of $m_b$ is obtained by the following formula: $w_b^{1^{\prime }}=\log (\frac{n-1}{{\displaystyle \sum _{b=1,c\ne b}^{n-1}{K}_{bc}}})$$b=1,2,\cdots n$, where $K_{bc}$ is the conflict coefficient between the BPA of $m_b$ and $m_c$. Then, the unnormalized $n$ are normalized as follows:$w_b^1=\frac{w_b^{1^{\prime }}}{{\displaystyle \sum _{b=1}^n{w}_b^{1^{\prime }}}}$$b=1,2,\cdots n$.

3.3. Calculate the Neighborhood Influence Weight and Total Weight

Given the classification result of a sensor $x_b$ for each pixel $u_i{}_j$ the weight of the influence of $u_i{}_j$’s $\delta \times \delta$ neighborhood block to $u_i{}_j$’s classification result is calculated as follows:$w_b^2=\frac{Num}{{\delta }^2-1}$, $b=1,2,\cdots ,n$, where $Num$ represents the number of classification result of pixels in $u_i{}_j$’s $\delta \times \delta$ neighborhood block whose classification results are same as $u_i{}_j$’s. At last, by synthesizing the two weights $w_b^1$ and $w_b^2$, a total weight $w_b^{}$ (for $u_i{}_j$ in sensor $x_b$’s classification results) can be obtained as: $w_b^{}=\frac{w_b^1{w}_b^2}{{\displaystyle \sum _{b=1}^n{w}_b^1}}$.

3.4. Fusion

For every pixel $u_{ij}$, its classification results from $n$ sensors, $m_b,b=1,\cdots ,n$, are used to generate a weighted average BPA in terms of different weights $w_b^{},b=1,\cdots ,n$:$\overline{m}(\{{\theta }_a\})={\displaystyle \sum _{b=1}^n{w}_{\mathrm{b}}{m}_b(\{{\theta }_a\})},{\theta }_a\in \Theta$. Then, we use Dempster’s rule to fuse $\overline{m}$ for $n-1$ times, $m=\overline{m}\oplus \overline{m}\oplus \cdots \oplus \overline{m}$, where $\oplus$ is the Dempster’s rule for the classification of two BPAs: $\{\begin{array}{l}m\left(E\right)=\frac{1}{1-k_{bc}}{\displaystyle \sum _{B\cap D=E}{m}_b\left(B\right)m_c\left(D\right)},E\subseteq \Theta \mathrm{and} E\ne \mathrm{\varnothing }\\ m(\mathrm{\varnothing })=0\end{array}$. After fusion, the final classification result for pixel $u_i{}_j$ is obtained.

4. Results

Two sets of multi-band SAR image data are used to verify the effectiveness of the proposed method according to the classification accuracy, and the modified average approach [45] is used for quantitative and qualitative comparisons. The modified average approach is a classic evidence combination method, which consists of weighted decisions based on the distance of evidence similar to our proposed method. The intermediate visual effects of the two methods are shown in Figure 3 and Figure 4. For the proposed method, the computational complexity of constructing the BPA part is $\mathrm{O}(h)$, the computational complexity of calculating the conflict coefficient part is $\mathrm{O}(2^h\times 2^h)$, the computational complexity of calculating the Jousselme evidence distance is $\mathrm{O}(2^h\times 2^h)$, and the computational complexity of the fusion part is $\mathrm{O}(2^h\times 2^h)$. For the modified average approach, the computational complexity of constructing the BPA part is $\mathrm{O}(h)$, the computational complexity of calculating the Jousselme evidence distance is $\mathrm{O}(2^h\times 2^h)$, and the computational complexity of the fusion part is $\mathrm{O}(2^h\times 2^h)$. The single-band SAR image classification experiment is completed under the Pytorch framework, Adam is set as the optimizer, CrossEntropyLoss is the loss function, the learning rate is 0.001 and the number of trainings is 100. All decision-level combination experiments are run in a MATLAB R2021a environment. The size of the $\delta \times \delta$ neighborhood block is $9\times 9$. For a single pixel, the time required to fuse the first data set using the proposed algorithm and the comparison algorithm is 0.0201 and 0.0193 s, respectively, and the time required to fuse the second data set using the proposed algorithm and the comparison algorithm is 0.0510 and 0.0384 s, respectively.

4.1. Preprocessing

After the multi-band data are collected by sensors, the size and coordinates are not uniform. Therefore, for the joint classification of the multi-band images, we need image preprocessing including mosaicking, registration and cropping. Taking the images acquired in Dongying City as an example, Figure 5, Figure 6 and Figure 7 show the preprocessing process based on ENVI 5.1.

4.2. Experimental with the Dongying City Dataset

In order to verify the effectiveness of the method, the first set of multi-band SAR images were captured by C-band SAR, L-band SAR, and P-band SAR sensors at the Yellow River estuary in Dongying City, Shandong Province. Among them, the acquisition time of C-band SAR is 25 November 2019, with a resolution of 0.5 m, the L-band SAR was collected on 26 November 2019, with a resolution of 3.0 m and the acquisition time of P-band SAR is 25 November 2019, with a resolution of 3.0 m. Figure 8 shows the images from these sensors.

The labels of these images are manually labeled, which contains three categories, farmlands, buildings, and roads. The label is shown in Figure 9.

At first, we use the single-band SAR image classification method in Section 2.2 to classify the SAR images of the C, L, and P bands, respectively. Then, we use the method mentioned in Section 3 to achieve decision fusion of multi-band SAR images to obtain the final result. The single band classification result and the decision fusion result are obtained, as shown in Figure 10 and Figure 11. The accuracy of single-band and multi-band fusion is shown in

Table 1. Classification accuracy.

	C-Band SAR Classification	L-Band SAR Classification	P-Band SAR Classification	Modified Average Approach	Our Method
Accuracy	66.32%	64.08%	63.17%	68.67%	70.34%

.

It can be seen from Figure 10 and Figure 11 and Table 1 that the classification accuracy of C, L, and P bands are 66.32%, 64.08%, 63.17%, and 68.19%. The classification accuracy of the modified average approach and our decision fusion method is 68.67% and 70.34%. Among them, the highest accuracy is achieved by the decision fusion method proposed in this paper. The accuracy of the proposed method is 7.17% higher than the single-band’s minimum accuracy and 4.02% higher than the single-band’s highest accuracy. Compared with the modified average approach, the accuracy of our method is improved by 1.67%. For the C-band classification results, a large number of farmlands is classified as roads, while for the P-band classification results, the roads are not effectively identified. As can be seen from Figure 11a, the modified average approach incorrectly classifies a large number of farmlands into roads, which significantly reduces the classification accuracy. By contrast, the proposed method can more accurately classify the three categories.

4.3. Experimental with the Baotou City Dataset

The second group of multi-band SAR images was captured by C-band SAR, L-band SAR, P-band SAR, and X-band SAR sensors at the Baotou calibration field in Baotou, Inner Mongolia Autonomous Region, and the acquisition time is 2 November 2019, with a resolution of 0.5 m. Among them, the resolution of the C-band is 0.5 m, and the resolution of the L, P, and X-bands is 3.0 m. Figure 12 shows the images.

Similarly, all pictures are manually labeled, as shown in Figure 13, which contains four categories, pits, bare soil, buildings and roads. For these images, the classification results of the single band and the fusion method proposed in this paper are shown in Figure 14 and Figure 15 and

Table 2. Classification accuracy.

	C-Band SAR Classification	L-Band SAR Classification	P-Band SAR Classification	X-Band SAR Classification	Modified Average Approach	Fusion Result
Accuracy	59.85%	62.39%	62.07%	67.75%	68.97%	69.26%

.

It can be seen from Figure 14 and Figure 15 and Table 2 that the classification accuracy of C, L, P and X bands are 59.85%, 62.39%, 62.07% and 67.75%. The classification accuracy of the modified average approach and our decision fusion method is 68.97% and 69.26%. Among them, the highest accuracy is achieved by the decision fusion proposed in this paper. The accuracy of the proposed method is 9.41% higher than the single-band’s minimum accuracy, and 1.51% higher than the single-band highest accuracy. Compared with the modified average approach, the accuracy of our method is improved by 0.29%. By analyzing the single-band classification result maps, most of the bare soil area in C-band SAR images was identified as buildings. Moreover, the P-band classification does not effectively identify the road. As can be seen from Figure 15, many of the bare soils of (a) were classified into pits, and our method greatly alleviated this phenomenon. In general, the proposed method can more accurately classify the four categories.

5. Conclusions

In this paper, an SAR image fusion classification method based on the decision-level combination of multi-band information is proposed. The idea of evidence theory in decision fusion is introduced into the SAR image classification process, and the results of classification by sensors of different bands are merged to improve the final classification accuracy. Belief entropy is used to measure the uncertainty of the single-band classification result of each pixel, and the uncertainty value is assigned to the full set to obtain the basic probability assignment function. In terms of the idea of TF-IDF in natural language processing and conflict coefficients, the weights of sensors in different bands are obtained. At the same time, considering the neighborhood classification of each pixel in different band sensors, the total weight of each sensor is obtained, and weighted average BPA is generated. After fusion, the final classification result is obtained. Experimental results on two groups of multi-band SAR images demonstrate the effectiveness of our fusion classification method.