A Novel Deep Fully Convolutional Network for PolSAR Image Classification

Abstract

Polarimetric synthetic aperture radar (PolSAR) image classification has become more and more popular in recent years. As we all know, PolSAR image classification is actually a dense prediction problem. Fortunately, the recently proposed fully convolutional network (FCN) model can be used to solve the dense prediction problem, which means that FCN has great potential in PolSAR image classification. However, there are some problems to be solved in PolSAR image classification by FCN. Therefore, we propose sliding window fully convolutional network and sparse coding (SFCN-SC) for PolSAR image classification. The merit of our method is twofold: (1) Compared with convolutional neural network (CNN), SFCN-SC can avoid repeated calculation and memory occupation; (2) Sparse coding is used to reduce the computation burden and memory occupation, and meanwhile the image integrity can be maintained in the maximum extent. We use three PolSAR images to test the performance of SFCN-SC. Compared with several state-of-the-art methods, SFCN-SC achieves promising results in PolSAR image classification.

1. Introduction

Polarimetric synthetic aperture radar (PolSAR), which can utilize SAR complex images in different polarimetric channels to recognize the orientation, geometric shape, configuration and composition of targets [1], has become one of the most advanced technologies [2]. In the past decades, a large amount of PolSAR data has been acquired as a series of PolSAR systems are put into use [3]. The studies on the applications of PolSAR data, especially PolSAR image classification, have attracted more and more attention [4,5]. In order to complete the task of PolSAR image classification, lots of methods have been proposed [5,6]. Some were based on the physical scattering mechanisms, including Pauli decomposition [7], Krogager decomposition [8], Freeman decomposition [9], Huynen decomposition [10], Cloude–Pottier decomposition [11], and the extensions of these methods [12,13]. In addition, some researchers thought the statistical distribution of PolSAR data could be used for PolSAR image classification [14,15]. For example, based on the complex Wishart distributions of the covariance matrix and coherency matrix, Lee et al. [15,16] proposed Wishart distance to classify the PolSAR images. In addition, many machine learning methods, such as k-nearest neighbor (KNN) [17], sparse representation [18], support vector machine (SVM) [19,20,21], Bayes [22] and neural network [23,24,25], have also been applied in PolSAR image classification. In addition, sparse representation has also been successfully applied to the applications related to cloudy MODIS images [26], high resolution satellite images [27] and hyperspectral images [28]. In addition, Ayhan et al. [29] demonstrated that image fusion is helpful to pixel clustering and anomaly detection in multispectral image, which is instructive for PolSAR image classification.

Recently, deep learning methods have made remarkable achievements in many fields. Stacked auto-encoder (SAE) [30] and deep belief network (DBN) [31] have been successfully applied to remote sensing image classification. In particular, the convolutional neural network (CNN) model, based on convolution, pooling and nonlinear transformation operations, has achieved great performance in image classification [32], semantic segmentation [33], scene labeling [34], action recognition [35] and object detection [36,37]. Because of the great performance of the CNN model, it has also been successfully into PolSAR image classification [38,39,40,41,42,43]. However, CNN still has a few shortcomings in PolSAR image classification. As we all know, PolSAR image classification is actually a dense prediction problem. For the classification framework of CNN, the input is the image, and the output is the class of the image. However, the classification result of CNN cannot reflect the details of the image [44]. For example, the input is an image of a cat, CNN can get the class of the image is cat, but it cannot tell which part of the image is cat’s leg and which part is cat’s face. Therefore, for PolSAR image classification, the neighborhood of a pixel is set as the input to get the class of the pixel [38,39,40,41,42,43]. For instance, recording one pixel in the image as ${\mathit{p}}_1$, the neighborhood of ${\mathit{p}}_1$ is set as the input to get the class of ${\mathit{p}}_1$. Therefore, for one pixel (recorded as ${\mathit{p}}_2$) in the neighborhood of ${\mathit{p}}_1$, ${\mathit{p}}_1$ will also be involved in the process of getting the class of ${\mathit{p}}_2$. Assuming the size of the neighborhood is $H_n$ × $H_n$, we need to calculate the information of $H_n$ × $H_n$ pixels to get the classification result of one pixel. That is to say, CNN has the disadvantages of repeated calculation and memory occupation.

Fortunately, Long et al. [44] put forward Fully Convolutional Network (FCN) model, which is shown in Figure 1 and can be trained end-to-end, pixels-to-pixels. FCN transforms fully connected layers into convolutional layers, which enables an efficient classification net for end-to-end dense learning. That is to say, FCN is a promising method in PolSAR image classification. However, to utilize FCN for PolSAR image classification, some problems need to be solved, which can be described as follows:

(1)

As for common PolSAR image classification architectures, only a portion of labeled pixels are selected for training. However, for the FCN architecture, the whole ground truth map is used in the training stage;

(2)

As for the FCN architecture, each pixel in the images must have a corresponding label in the ground truth maps. However, there are some unlabeled pixels in the ground truth maps of the PolSAR images because PolSAR images are generally not fully labeled;

(3)

In general, each PolSAR image is trained individually, and the size of each PolSAR image is different. However, we cannot design an architecture for each image alone. Furthermore, a larger-size input image usually means a more complex architecture.

Recently, Wang et al. [45] have applied FCN to PolSAR image classification. However, they did not give the details of how to solve the problems mentioned. In order to provide a general solution to the problems aforementioned above, we propose sliding window fully convolutional network (SFCN). The sliding window operation of SFCN is similar to the sliding window operation of CNN, and the images obtained by sliding window operation are called as ${\mathit{Image}}_{\mathit{s}}$ in this paper. As for SFCN, there are some details to state. First, the region where sliding window passes is extracted as ${\mathit{Image}}_{\mathit{s}}$ directly, without convolution operation. In addition, the obtained ${\mathit{Image}}_{\mathit{s}}$ is set as the input of FCN. Second, the size of the window is also the size of ${\mathit{Image}}_{\mathit{s}}$; therefore, we can design a network architecture for all PolSAR images. Finally, because the training framework of FCN cannot be applied directly, we have designed a new training framework for SFCN in this paper.

However, the number of ${\mathit{Image}}_{\mathit{s}}$ will increase with the increase of the image size, which leads to heavier computation burden and more memory occupation. Since sparse coding can better model the complex local image structures and it has achieved many promising results in PolSAR image classification [18,46,47], we think sparse coding can be utilized to model the local structures of PolSAR image. In this way, PolSAR image can be downsampled to a smaller size, so that the number of the ${\mathit{Image}}_{\mathit{s}}$ can be limited. Therefore, finding a suitable sparse coding method becomes the key to the problem. Lately, Chen et al. [46] proposed multi-layer projective dictionary pair learning (MDPL), which can be used here to deal with the problem above without obvious image information loss.

Based on SFCN and sparse coding, we propose sliding window fully convolutional network and sparse coding (SFCN-SC) for PolSAR image classification, which can obtain excellent classification results while reducing the computational burden and memory occupation. The remainder of this paper is structured as follows: Section 2 gives the methods used for extracting features of PolSAR images. Our proposed method is presented in Section 3. Section 4 gives the experimental results. Finally, discussion and conclusions are given in Section 5 and Section 6, respectively.

2. Feature Extraction of PolSAR Images

2.1. Coherency Matrix

According to [48], each pixel in PolSAR image can be described by a scattering matrix, which contains the polarimetric information of the PolSAR data. With linear horizontal (H) and vertical (V) polarizations for transmitting and receiving, the scattering matrix can be formulated as:

$$\mathit{S}=\left[\begin{array}{c}\hfill S_{HH}{S}_{HV}\\ \hfill S_{VH}{S}_{VV}\end{array}\right],$$

(1)

where the reciprocity theorem applies in a mono-static system configuration results $S_{HV}$ = $S_{VH}$, the scattering matrix can be described as [49]:

$$\mathbf{S}=A\frac{1}{\sqrt{2}}\left[\begin{array}{c}\hfill 10\\ \hfill 01\end{array}\right]+B\frac{1}{\sqrt{2}}\left[\begin{array}{c}\hfill 10\\ \hfill 0-1\end{array}\right]+C\frac{1}{\sqrt{2}}\left[\begin{array}{c}\hfill 01\\ \hfill 10\end{array}\right],$$

(2)

where

$$\begin{array}{c}\hfill A=(S_{HH}+S_{VV})/\sqrt{2},B=(S_{HH}-S_{VV})/\sqrt{2},C=\sqrt{2}{S}_{HV}.\end{array}$$

(3)

In addition, the coherency matrix $\mathit{T}$ of PolSAR data can be described as [48]:

$$\begin{array}{c}\hfill \mathit{T}={[A,B,C]}^T[A^{\ast },B^{\ast },C^{\ast }]=\left[\begin{array}{c}\hfill T_{11}{T}_{12}{T}_{13}\\ \hfill T_{21}{T}_{22}{T}_{23}\\ \hfill T_{31}{T}_{32}{T}_{33}\end{array}\right]=\left[\begin{array}{c}\hfill {\left|A\right|}^{2}A{B}^{\ast }A{C}^{\ast }\\ \hfill A^{\ast }B{\left|B\right|}^{2}B{C}^{\ast }\\ \hfill A^{\ast }C{B}^{\ast }C{\left|C\right|}^2\end{array}\right].\end{array}$$

(4)

The coherency matrix $\mathit{T}$ has been successfully applied in PolSAR image classification [46,48].

2.2. Cloude–Pottier Decomposition

According to the eigen-decomposition model [11], the coherence matrix can be decomposed as:

$$\mathit{T}={\mathit{U}}_3\left[\begin{array}{c}\hfill {\lambda }_{1}00\\ \hfill 0{\lambda }_{2}0\\ \hfill 00{\lambda }_3\end{array}\right]{\mathit{U}}_3^{\ast }=\sum _{i=1}^3{\lambda }_i{\overrightarrow{e}}_i{\overrightarrow{e}}_i^{\ast },$$

(5)

where ${\mathit{U}}_3=[{\overrightarrow{e}}_1,{\overrightarrow{e}}_2,{\overrightarrow{e}}_3]$ represents the eigenvector matrix of the coherency matrix T, and ${\lambda }_i(i=1,2,3)$ represents the eigenvalue of the coherency matrix $\mathit{T}$. Based on the eigen-decomposition model, Cloude and Pottier proposed the Cloude–Pottier decomposition method, in which the entropy H, the anisotropy A, and the mean alpha angle $\overline{\alpha }$ can be obtained as:

$$\begin{array}{cc}\hfill H=-\sum _{i=1}^3& p_{i}lo{g}_3{p}_i,p_i=\frac{{\lambda }_i}{{\sum }_{k=1}^3{\lambda }_k},\hfill \\ & A=\frac{{\lambda }_2-{\lambda }_3}{{\lambda }_2+{\lambda }_3},\hfill \\ \hfill \overline{\alpha }=\sum _{i=1}^3{p}_i{\alpha }_i=& p_{1}co{s}^{-1}\left(e_1^1\right)+p_{2}co{s}^{-1}\left(e_2^1\right)+p_{3}co{s}^{-1}\left(e_3^1\right),\hfill \end{array}$$

(6)

where $e_i^1$ represents the first element of the eigenvector ${\overrightarrow{e}}_i$ (i = 1, 2, 3). In addition, the Cloude–Pottier decomposition features have also been successfully used in PolSAR image classification [11,18].

In this paper, we utilize the coherency matrix T ($T_{11}$, $T_{22}$, $T_{33}$, Re($T_{12}$), Re($T_{13}$), Re($T_{23}$), Im($T_{12}$), Im($T_{13}$), Im($T_{23})$), where Re($T_{ij}$) and Im($T_{ij}$) represent the real part and the imaginary part of $T_{ij}$ respectively, and the Cloude–Pottier decomposition features (H, A, $\overline{\alpha }$, ${\lambda }_1$, ${\lambda }_2$, ${\lambda }_3$) to construct our extracted features.

3. Methodology

As for the traditional CNN architecture [50], the convolutional layers are used for extracting local features of images, and the fully connected layers are used for predicting the labels of images. In other words, the traditional CNN architecture cannot be trained pixels-to-pixels, which means that the classification results of CNN cannot reflect the labels of all the piexls in the images. However, the task of PolSAR image classification is to make a prediction at each pixel. Therefore, the pixels-to-pixels architecture is more suitable for processing PolSAR image classification task than CNN. Fortunately, Long et al. [44] proposed FCN model, which can be trained end-to-end, pixels-to-pixels. Therefore, FCN has enormous potential in PolSAR image classification.

Because of the advantages of the FCN model, we wish to utilize the FCN model for PolSAR image classification. However, some necessary improvements to the FCN model have to be done according to the problems as described in Section 1. In general, supervised methods used for PolSAR image classification usually select a portion of the labeled pixels as the training data and the remaining labeled pixels as the testing data, but, for the FCN architecture, the whole ground truth map of a certain image will participate in the training stage and all pixels in the image must have their own corresponding labels in the ground truth map. It is difficult to use FCN for classifying the PolSAR images, of which there are some unlabeled pixels in the ground truth maps, because most of the PolSAR images are not fully labeled. Therefore, the training framework of FCN [44] cannot be applied directly for PolSAR image classification. According to this, a new training framework of FCN is redesigned. Furthermore, images of different sizes need different FCN architectures and a larger-size input image usually brings more difficulty in designing network architecture. To design a unified FCN model for different images, a sliding window operator is introduced in this paper. Overall, a method named sliding window fully convolutional network (SFCN) is proposed to tackle the problems referred to above.

The sliding window operation of SFCN is similar to the sliding window operation of CNN, and the number of the ${\mathit{Image}}_{\mathit{s}}$ can be obtained through

$$\begin{array}{c}\hfill \mathit{Num}=\left(ceil\right((Height-W)/S)+1)·\left(ceil\right((Width-W)/S)+1),\end{array}$$

(7)

where ceil represents the upward integer-valued function, Height and Width represent the height and width of the input image, respectively, W is the size of the sliding window, S represents the stride of the sliding window operation, and Num represents the number of the ${\mathit{Image}}_{\mathit{s}}$. The obtained ${\mathit{Image}}_{\mathit{s}}$ is set as the input of FCN. The advantages of SFCN can be described as follows:

(1)

SFCN can avoid repeated calculation and memory occupation which exist in CNN;

(2)

Supposing that there are three categories in the image, then these sliding window images may contain one, two, or three categories. Compared with the whole image being set as the input, we think the sliding window operation makes SFCN more constructive to learn the difference between different categories in ${\mathit{Image}}_{\mathit{s}}$;

(3)

${\mathit{Image}}_{\mathit{s}}$ obtained by SFCN can contain more spatial information since the size of sliding window for SFCN is larger than that of CNN;

(4)

For SFCN with overlapping sliding window, there are multiple windows passing through almost all the pixels of the images except the edge parts, and each ${\mathit{Image}}_{\mathit{s}}$ will give its own predicted results of the pixels. Combining multiple classification results can make the final result more reasonable.

The architecture of the SFCN is shown in Figure 2. From Figure 2, we can see that two architectures, namely SFCN-128 and SFCN-256, have been designed for FCN in this paper. In addition, the loss function can be described as:

$$\begin{array}{c}\hfill Loss=-\frac{1}{N}\sum _{i=1}^N\sum _{j=1}^{Numclass}{y}_{ij}log\left(y_{ij}^{\prime }\right),\end{array}$$

(8)

where y and $y^{\prime }$ represent the ground truth and the predicted result of ${\mathit{Image}}_{\mathit{s}}$, respectively, N represents the number of training samples in ${\mathit{Image}}_{\mathit{s}}$, and Numclass represents the classes number of $\mathit{Image}$.

Furthermore, there are some details to describe for the new training framework for SFCN. First, the training pixels are randomly selected. Second, only the training samples are involved in modifying the network parameters. The new training framework for SFCN is shown in Algorithm 1.

Algorithm 1 The training framework of SFCN.

Input: Images $\mathit{X}$, ground truth $\mathit{G}$, training samples ratio $\mathit{p}$;   1: Initialize model parameters ${\Theta }_{SFCN}$;   2: Randomly select training samples from labeled samples, and the samples ratio of each class is p;   3: Utilize sliding window operation to get the sliding window image, recorded as ${\mathit{Image}}_{\mathit{s}}$;   4: Initialize a zero matrix $\mathit{g}$ as the ground truth of each ${\mathit{Image}}_{\mathit{s}}$;   5: if the pixel in ${\mathit{Image}}_{\mathit{s}}$ is selected as training data then   6:     Get the label of the pixel according to $\mathit{G}$, recorded as L;   7:     Set the element in g corresponding to the pixel to L;   8: end if   9: Set ${\mathit{Image}}_{\mathit{s}}$ and $\mathit{g}$ as the input of SFCN; 10: Utilize ADAM [51] algorithm to minimize the loss function of SFCN, namely Equation (8), and only the non-zero points in $\mathit{g}$ are involved in model parameter updating.   Output: SFCN learnt parameters: ${\widehat{\Theta }}_{SFCN}$.

According to Equation (7), the Num will increase with the increase of Height and Width under the condition that W and S are fixed. A larger number of ${\mathit{Image}}_{\mathit{s}}$ will bring heavier computational burden and more memory occupation. As Section 1 described, sparse coding can be utilized to model the local structures of PolSAR image. Through sparse coding, PolSAR image can be downsampled to a smaller size. In this way, the number of the ${\mathit{Image}}_{\mathit{s}}$ can be limited. Thus, the recently proposed MDPL model, as one of the sparse coding methods, is chosen to accomplish the downsampling task. MDPL model can be formulated as:

$$\begin{array}{c}\hfill F=arg\underset{\mathit{P},\mathit{D}}{min}{\parallel {\mathit{X}}_l-{\mathit{D}}_l{\mathit{A}}_l\parallel }_F^2+\mu {\parallel {\mathit{P}}_l{\mathit{X}}_l-{\mathit{A}}_l\parallel }_F^2,s.t.{\parallel {\mathit{d}}_i\parallel }_F^2\le 1,\end{array}$$

(9)

where ${\mathit{X}}_l$ represents the input data of the lth layer, ${\mathit{A}}_l$ represents the coding coefficients of the lth layer, {${\mathit{P}}_l$ and ${\mathit{D}}_l$} represents the projective dictionary pair of the lth layer, $\mu$ represents a scalar constant, and ${\mathit{d}}_i$ is the ith atom of ${\mathit{D}}_l$. In addition, $\mathit{X}$ is set as the input of the first layer, and the coding coefficients of the previous layer are set as the input of the next layer. Because MDPL belongs to the sparse coding methods, MDPL can tackle the downsampling task while maintaining the image integrity to the maximum extent.

Based on SFCN and sparse coding, we propose sliding window fully convolutional network and sparse coding (SFCN-SC) in this paper. There are some details to state for SFCN-SC. First, the input image (recorded as $\mathit{X}$) is cut into small blocks, and the features of each small block are flattened into a feature vector, recorded as ${\mathit{x}}_d$. Then, ${\mathit{x}}_d$ is set as the input of MDPL to obtain the coding coefficients, recorded as ${\mathit{x}}_A$. In this way, the input image is downsampled to a smaller size, recorded as ${\mathit{X}}_A$. In other words, each pixel in ${\mathit{X}}_A$ has a corresponding small block in $\mathit{X}$. Second, ${\mathit{X}}_A$ is set as the input of SFCN to obtain the classification result of ${\mathit{X}}_A$, recorded as $\mathit{result}$, and the size of $\mathit{result}$ is the same as that of ${\mathit{X}}_A$, not the same as $\mathit{X}$. Third, the element of $\mathit{result}$ is set as the classification result of the corresponding small block to obtain the final classification result. The algorithm architecture of SFCN-SC is shown in Figure 3, and the implementation details of SFCN-SC are described in Algorithm 2.

Algorithm 2 The framework of SFCN-SC.

Input: Images $\mathit{X}$, ground truth $\mathit{G}$, training samples ratio $\mathit{p}$, the size of small block ${\mathbf{d}}_{\mathbf{h}}$ × ${\mathbf{d}}_{\mathbf{w}}$, the size of the extracted feature through MDPL dim;   1: Get the feature dimension of $\mathit{X}$, recorded as Dim;   2: Cut the image into small blocks, and the size of the small blocks is ${\mathbf{d}}_{\mathbf{h}}$ × ${\mathbf{d}}_{\mathbf{w}}$;   3: Flat all the features of each small block into feature vector, recorded as ${\mathit{x}}_d$, and the dimension of ${\mathit{x}}_d$ is ${\mathbf{d}}_{\mathbf{h}}$·${\mathbf{d}}_{\mathbf{w}}$· Dim;   4: Set ${\mathit{x}}_d$ and $\mathbf{d}$ as the input of MDPL, get the coding coefficients ${\mathit{x}}_A$;   5: Each small block operates 3–4, in this way, $\mathit{X}$ is downsampled to a smaller size, recorded as ${\mathit{X}}_A$, which is the downsampled image;   6: Set the popular label of the pixels in each small block as the label of ${\mathit{x}}_A$, recorded as $\mathit{g}$, each small block operates this to obtain the ground truth of ${\mathit{X}}_A$, recorded as ${\mathit{G}}_S$;   7: Set ${\mathit{X}}_A$, ${\mathit{G}}_S$ and $\mathit{p}$ as the input of SFCN to get the classification result through Algorithm 1, recorded as $\mathit{result}$;   8: Set the element of $\mathit{result}$ as the classification result of the corresponding small block in X to obtain the final classification result of the X, recorded as Result. Output: Classification result Result.

4. Experimental Results

As described in Section 2, the coherency matrix T ($T_{11}$, $T_{22}$, $T_{33}$, Re($T_{12}$), Re($T_{13}$), Re($T_{23}$), Im($T_{12}$), Im($T_{13}$), Im($T_{23})$) and the Cloude–Pottier decomposition features (H, A, $\overline{\alpha }$, ${\lambda }_1$, ${\lambda }_2$, ${\lambda }_3$) are set as our input features in this paper. The refined Lee filter [52] is used for suppressing the speckle noise. Three PolSAR images, including images for Xi’an, China; Oberpfaffenhofen, Germany; San Francisco, CA, USA are utilized to test the performances of different methods. In addition,

Table 1. Information about Three PolSAR Images.

Images	Radar	Band	Year	Resolution	Polarimetric Information	Size	Classes
Xi’an	E-SAR	C	2010	$8\times 8$ m	full polarimetric, multilook	$512\times 512$ pixels	3
Oberpfaffenhofen	E-SAR	L	1991	$3\times 2.2$ m	full polarimetric, multilook	$1300\times 1200$ pixels	3
San Francisco	RADARSAT-2	C	2008	$10\times 5$ m	full polarimetric, multilook	$1800\times 1380$ pixels	5

gives the detail information of the three PolSAR images. We utilize overall accuracy (OA) and Kappa coefficient [53] to evaluate the performances of different methods. In the following experiments, SVM [20], sparse representation classifier (SRC) [54], CNN [55] and SAE [30] are used as the compared methods. In this paper, all methods (including our method and compared methods) are implemented in a 2.20-GHz machine with a 64.00-GB RAM and a NVIDIA GTX 1080 GPU.

4.1. Parameter Setting

For the SVM method, the radial basis function (RBF) kernel is used. As for the SRC, the number of dictionary atoms is set to 15. For the CNN method, we use a $21\times 21$ neighborhood of each pixel as the small image, and the architecture of CNN is shown in Figure 4. As for SAE, the dimensions of middle layers are 300 and 100, respectively, and Softmax classifier [56] is utilized to get the classification result. As to our proposed method SFCN, we design two architectures, namely SFCN-128 and SFCN-256, which are shown in Figure 2. For the SFCN-SC method, which is shown in Figure 3, we downsample Oberpfaffenhofen and San Francisco to $650\times 600$ and $600\times 690$, recorded as ${\mathit{X}}_A$, and the dimension of the pixel in ${\mathit{X}}_A$ is set to 10. Then, ${\mathit{X}}_A$ is set as the input of SFCN. We set stride in Equation (7) to 32 and 64 for the window images of 128 and 256 sizes, respectively. In the following experiments, the number of training samples of each class corresponding to all the three PolSAR images is 5%, and the training samples are randomly selected.

4.2. Classification Performance

4.2.1. Xi’an Image

Figure 5a,b give the Pauli RGB (red-green-blue) image and the ground truth of Xi’an, respectively. There are 237,416 labeled pixels in the ground truth. As Section 4.1 mentioned, we use 5% of the labeled pixels in the ground truth as the training set and the remaining 95% as the testing set. Figure 5c–h and

Table 2. Classification results of Xi’an with different methods.

Methods	Water	Grass	Building	OA	Kappa
SVM	0.8429	0.9050	0.9000	0.8939	0.8242
SRC	0.5922	0.9201	0.9029	0.8648	0.7697
SAE	0.8269	0.9019	0.9207	0.8973	0.8297
CNN	0.9233	0.9551	0.9509	0.9488	0.9156
SFCN-128	0.8813	0.9580	0.9739	0.9521	0.9208
SFCN-256	0.8281	0.9439	0.9695	0.9355	0.8932

give the classification results, which can be utilized to compare the performances of different methods.

From Figure 5, we can see that the classification result of SFCN-128 is more reasonable than other methods. The classification results of SVM, SRC and SAE are relatively worse, because there are many misclassified pixels in Figure 5c–e. From Table 2, we can get the same conclusion. The accuracy values and Kappa coefficient of SFCN-128 are the highest among all the methods, and accuracy values and Kappa coefficient of SRC are the lowest. For that mentioned above, the effectiveness of our method can be verified.

4.2.2. Oberpfaffenhofen Image

Figure 6a,b gives the Pauli RGB image and the ground truth of Oberpfaffenhofen, respectively. There are 1,374,298 labeled pixels in the ground truth. Just as Section 4.1 mentioned, 5% of the labeled pixels in the ground truth are set as the training set and the remaining 95% are set as the testing set. Figure 6c–j and

Table 3. Classification results of Oberpfaffenhofen with different methods.

Methods	Built-Up Areas	Wood Land	Open Areas	OA	Kappa
SVM	0.7189	0.8908	0.9721	0.8937	0.8169
SRC	0.7253	0.7503	0.9463	0.8534	0.7478
SAE	0.8036	0.8682	0.9679	0.9078	0.8423
CNN	0.9284	0.9639	0.9694	0.9582	0.9294
SFCN-128	0.9870	0.9853	0.9905	0.9886	0.9807
SFCN-256	0.9741	0.9876	0.9957	0.9888	0.9809
SFCN-SC-128	0.9763	0.9747	0.9857	0.9812	0.9681
SFCN-SC-256	0.9814	0.9864	0.9854	0.9846	0.9740

show the classification results to compare the effectiveness of different methods.

As shown in Figure 6, the number of the misclassified pixels corresponding to SFCN-128 and SFCN-256 is smaller than other methods, which can demonstrate the effectiveness of SFCN. As to SFCN-SC (including SFCN-SC-128 and SFCN-SC-256), the classification results are comparable to SFCN. In agreement with Xi’an image, the classification results corresponding to SVM, SRC and SAE are relatively worse than other methods. We can get the same conclusion from Table 3. As shown in Table 3, the accuracy values and Kappa coefficient of SFCN are the highest among all the methods, and the accuracy values and Kappa coefficient of SFCN-SC are a little lower than but comparable to SFCN. Therefore, we can draw a conclusion that our sparse coding method can maintain the image integrity to the maximum while accomplishing the downsampling task. In summary, the effectiveness of our method can be made evident by the reasons mentioned above.

4.2.3. San Francisco Image

Figure 7a,b shows the Pauli RGB image and the ground truth of San Francisco, respectively. The number of labeled pixels in the ground truth is 1,804,087. Just as Section 4.1 said, 5% of the labeled pixels in the ground truth are invoked as the training set and the remaining 95% are used as the testing set. Figure 7c–j and

Table 4. Classification results of San Francisco with different methods.

Methods	Ocean	Vegetation	Low Density Urban	High Density Urban	Developed	OA	Kappa
SVM	0.9998	0.9111	0.9006	0.7936	0.8932	0.9317	0.9016
SRC	0.9871	0.8834	0.9334	0.7190	0.5737	0.9025	0.8591
SAE	0.9999	0.9240	0.8924	0.7919	0.9086	0.9323	0.9025
CNN	0.9999	0.9845	0.9790	0.9210	0.9872	0.9809	0.9725
SFCN-128	0.9999	0.9855	0.9869	0.9999	0.9995	0.9955	0.9935
SFCN-256	0.9999	0.9821	0.9967	0.9977	1	0.9966	0.9951
SFCN-SC-128	0.9999	0.9651	0.9849	0.9961	1	0.9919	0.9883
SFCN-SC-256	0.9981	0.9871	0.9913	0.9967	0.9726	0.9940	0.9914

show the classification results of all the methods. The classification results of San Francisco image are similar to the two above mentioned images. SFCN obtains the best classification results among all the methods, and the classification results of SFCN-SC are comparable to SFCN. The classification results corresponding to SVM, SRC and SAE are relatively worse. For that mentioned above, the effectiveness of our method can be proved.

5. Discussion

5.1. The Comparison of Performance of Architectures with Different Window Sizes

In this paper, two window sizes are adopted, namely 128 and 256. It can be seen from Table 2, Table 3 and Table 4 that the classification results corresponding to 128 are usually better than or comparable to that corresponding to 256. However, the network architecture using the 128 size of window has lower complexity compared to the other one. Therefore, 128 is a more ideal window size for SFCN and SFCN-SC.

5.2. The Effect of Sliding Window on the Classification Results

There are three PolSAR images in this paper, including Xi’an, Oberpfaffenhofen, and San Francisco, and the sizes of the three PolSAR images are $512\times 512$, $1300\times 1200$, and $1800\times 1380$, respectively. The FCN architectures for Oberpfaffenhofen and San Francisco are too hard to design because of their larger image sizes. Therefore, as shown in Figure 8, we design a FCN architecture for Xi’an as the contrast method of SFCN-128 and SFCN-256, namely FCN-512, to validate the effect of sliding window on the classification results.

Table 5. Classification Results of Xi’an with FCN-512 and SFCN as a function of the rate of training samples.

Rate	FCN-512	SFCN-128	SFCN-256
5%	0.6372	0.9521	0.9355
10%	0.7725	0.9686	0.9665
15%	0.9281	0.9779	0.9737
20%	0.9634	0.9787	0.9766

shows the classification results of Xi’an with FCN-512, SFCN-128 and SFCN-256 as a function of the rate of training samples. It can be seen that the OA of FCN-512 is 0.9634, a little smaller than that of SFCN-128 and SFCN-256, under 20% of training samples. When the rate of training samples decrease from 20% to 5%, the OA of FCN-512 falls sharply to 0.6372 but the OA of SFCN-128 and SFCN-256 remain over 0.93. Therefore, SFCN can obtain better classification results even using less training samples compared to FCN-512, which proves that the sliding window is useful to improve the classification results.

5.3. The Effect of Sparse Coding

As presented in Section 3, the number of the sliding window images obtained by SFCN will increase with the image size, which increases the computational burden in the training stage. In this paper, we use sparse coding to resolve this problem. As two of the most popular downsampling methods, the bilinear interpolation (BILI) [57] and bicubic interpolation (BICI) [58] methods have achieved great performance in image processing in the past decades. Therefore, BILI and BICI are used to compare the performance of sparse coding. In addition, the compared methods are recorded as SFCN-BILI and SFCN-BICI.

Table 6. Classification results of Oberpfaffenhofen and San Francisco with different methods.

Image Size	Methods	Oberpfaffenhofen	San Francisco
128	SFCN	0.9886	0.9955
	SFCN-BILI	0.9635	0.9878
	SFCN-BICI	0.9511	0.9893
	SFCN-SC	0.9812	0.9919
256	SFCN	0.9888	0.9966
	SFCN-BILI	0.9606	0.9709
	SFCN-BICI	0.9492	0.9907
	SFCN-SC	0.9846	0.9940

gives the classification results of Oberpfaffenhofen and San Francisco with SFCN, SFCN-BILI, SFCN-BICI, and SFCN-SC. We can see that the classification results of SFCN-BILI and SFCN-BICI are much lower than SFCN. However, the classification results of SFCN-SC are a little lower than but comparable to SFCN. Based on the above discussion, we can draw a conclusion that the popular downsampling method, such as BILI and BICI, may result in the loss of details of the PolSAR image, but sparse coding can maintain the details of the PolSAR image in the maximum extent.

Table 7. Execution time of the three PolSAR images with different methods (S).

Methods	Xi’an			Oberpfaffenhofen			San Francisco
	Train	Predict	Total	Train	Predict	Total	Train	Predict	Total
SVM	1.34	22.16	23.50	80.16	1316.89	1397.05	39.95	1744.05	1784.00
SRC	1.52	0.41	1.93	3.07	2.15	5.22	4.52	2.61	7.13
SAE	19.26	0.13	19.39	109.84	0.35	110.19	140.89	0.55	141.44
CNN	100.28	1.48	101.76	286.10	9.47	295.57	462.27	15.33	477.60
SFCN-128	80.37	3.04	83.41	271.32	23.89	295.21	452.87	39.76	492.63
SFCN-256	164.89	1.39	166.28	515.07	15.98	531.05	1206.42	27.41	1233.83
SFCN-SC-128	-	-	-	159.88	5.17	165.05	274.37	5.46	279.83
SFCN-SC-256	-	-	-	165.37	3.11	168.48	290.79	3.12	293.91

summarizes the execution time of various methods, where “Train” represents the training time, “Predict” represents the time taken to classify entire image, and “Total” represents the total time taken to train plus predict. The training time consumption of SFCN-SC can be divided into two parts, namely SFCN and sparse coding. Though sparse coding consumes a certain amount of time at the training stage, the training time of SFCN-SC is less than SFCN. In summary, SFCN-SC still can obtain competitive classification results while reducing the computational burden in the training stage.

5.4. Data Memory Consumed by Various Methods

Table 8. Data memory consumed by various methods corresponding to three PolSAR images (G).

Methods	Xi’an	Oberpfaffenhofen	San Francisco
SVM	0.026	0.14	0.24
SRC	0.026	0.14	0.24
SAE	0.026	0.14	0.24
CNN	11.9	66.1	110.5
SFCN-128	0.24	1.72	2.90
SFCN-256	0.14	1.50	2.60
SFCN-SC-128	-	0.30	0.30
SFCN-SC-256	-	0.23	0.22

gives the data memory consumed by various methods corresponding to the three PolSAR images, and the “G” in the caption of Table 8 represents gigabytes. From Table 8, we can see that the data memory consumed by CNN is the highest among all the methods, which demonstrates that CNN does have the disadvantage of repeated memory occupation. The data memory consumed by SFCN-SC is smaller than SFCN, since the data memory occupation can be reduced by sparse coding. In summary, SFCN-SC does have an advantage over CNN and SFCN in memory occupation. black

To sum up, the classification results of SVM, SRC and SAE are relatively worse because these three methods cannot take into account the spatial information of the images. However, they are superior to other methods in memory occupation. As for the time consumption, SRC and SAE performs well. Nonetheless, in order to fit the relationship between the training data and the training label, SVM has caused its structure to be too complicated. Therefore, SVM takes a relatively longer time to predict the images. The neighborhood of the pixel is used as the input of CNN, and in this way the spatial information can be used by CNN. Therefore, the classification result of CNN is acceptable. However, setting the neighborhoods of the pixels as the input of CNN also leads to repeated calculation and memory occupation, which means that CNN has no advantage over other methods in terms of time consumption and memory occupation. As Section 1 described, PolSAR image classification is actually a dense prediction problem, and FCN does have great potential in PolSAR image classification because of its special network architecture. However, the traditional FCN architecture cannot be applied directly to PolSAR image classification. Based on FCN, we utilize sliding window operation to tackle the problem. From the classification results of the three PolSAR images, we can note that the classification results of SFCN are better than CNN and SAE. This is because SFCN takes the spatial information into account and the spatial information of SFCN is more abundant than that of CNN. However, the sliding window operation also increases computational burden and memory occupation. Thus, we utilize sparse coding as the downsampling method to reduce the num of ${\mathit{Image}}_{\mathit{s}}$. In addition, classification results will hardly be affected because sparse coding can find the key information of the image. In summary, SFCN-SC is the only algorithm that can take into account the classification results, time consumption, and memory occupation at the same time.

6. Conclusions

In this paper, we propose a novel method, namely SFCN-SC, for PolSAR image classification. The coherency matrix T and Cloude–Pottier decomposition features are used as the feature vectors, SVM, SRC, SAE and CNN are utilized as the compared methods, and three PolSAR images, including Xi’an, Oberpfaffenhofen, and San Francisco are used to test the performances of different methods. On the basis of the previous FCN model and sliding window operation, we propose SFCN to obtain more competitive classification results. However, time consumption and memory occupation of SFCN will increase with the increase of the image size. In order to address the problem mentioned above, sparse coding is used as the downsampling method in our method. We first downsample the images to a fixed size, and the obtained images through downsampling are set as the input of SFCN. In this way, the time consumption and memory occupation can be reduced. Because of the great performances of SFCN and sparse coding, SFCN-SC can get competitive classification results while reducing the time consumption and memory occupation. Compared with some state-of-the-art methods, SFCN-SC achieves impressive performance in PolSAR image classification.