A Full Tensor Decomposition Network for Crop Classification with Polarization Extension

Abstract

The multisource data fusion technique has been proven to perform better in crop classification. However, traditional fusion methods simply stack the original source data and their corresponding features, which can be only regarded as a superficial fusion method rather than deep fusion. This paper proposes a pixel-level fusion method for multispectral data and dual polarimetric synthetic aperture radar (PolSAR) data based on the polarization extension, which yields synthetic quad PolSAR data. Then we can generate high-dimensional features by means of various polarization decomposition schemes. High-dimensional features usually cause the curse of the dimensionality problem. To overcome this drawback in crop classification using the end-to-end network, we propose a simple network, namely the full tensor decomposition network (FTDN), where the feature extraction in the hidden layer is accomplished by tensor transformation. The number of parameters of the FTDN is considerably fewer than that of traditional neural networks. Moreover, the FTDN admits higher classification accuracy by making full use of structural information of PolSAR data. The experimental results demonstrate the effectiveness of the fusion method and the FTDN model.

1. Introduction

Crop classification plays an important role in the remote sensing monitoring of agricultural conditions. It is also a premise for further monitoring of crop growth and yields. Accurate and reliable discrimination of crop categories based on remotely-sensed images is a significant data source for agricultural monitoring and food security evaluation research.

Optical and radar remote sensing technologies are the two main methodologies for crop classification. Optical remote sensing technology is susceptible to environmental factors, such as weather and light, and is unable to obtain accurate surface information under adverse weather [1]. Radar remote sensing technology is insensitive to weather and light. Hence, the fusion of optical data and synthetic aperture radar (SAR) data is commonly carried out to implement more accurate classifications of crops [2]. For example, Qiao et al. fused SAR data and optical data and adopted a maximum likelihood method for crop classification based on fused data, leading to improved classification accuracy [3]. Shi et al. classified the fused data of multispectral and hyperspectral data through classification and regression tree, showing that the classification accuracies of fused data are better than those of individual data alone [4]. However, these fusion methods simply stack the multisource data or their corresponding features, they are far from in-depth fusion. Seo constructed a fusion model of optical data and SAR data using the random forest regression method, benefiting from the subsequent image analysis and interpretation [5]. This method fused the multisource data in-depth, which enhanced the quality of the optical image with auxiliary SAR data, and provided a new illustration for image fusion. Different from image fusion, the crop classification task focuses on the effective extraction of distinguishable features from remotely sensed data and the subsequent powerful classification model trained by these features. In particular, under the adverse weather condition, multisource data fusion is beneficial to generate enough distinguishable features for crop classification. In order to overcome the adverse weather effect, more polarization scattering features are always superior [6]. Quad PolSAR data usually provide data rich enough for crop classification, as it enables extracting large numbers of polarization scattering features via various polarization decomposition schemes [7]. However, quad PolSAR data are usually unavailable due to their higher costs. Dual PolSAR data can be served as natural substitutes, but fewer polarization decomposition schemes can be adopted, providing inadequate polarization scattering features. Therefore, the extension of dual PolSAR data to quad PolSAR data with the assistance of optical data is promising for crop classification.

Although higher dimensional features usually improve crop classification performance, it is prone to result in the “curse of dimensionality” problem in real applications. As for a regular cognitive model, the higher dimensional features extracted from raw data contain more information, and the trained model will be more capable to deal with sophisticated issues. However, Hughes [8] shows that the prediction ability of the model is enhanced at the beginning and then deteriorated with the increase of feature dimension for a fixed number of learning samples. This is called the Hughes phenomenon, which is one of the most significant drawbacks caused by the “curse of dimensionality”. This eventually results in a contradiction between the feature space and sample space below. To solve this problem, dimensionality reduction methods, including Principal Component Analysis (PCA) [9], Locally Linear Embedding (LLE) [10], and Autoencoder [11], are generally adopted before classification. Moreover, the subsequent classifiers, such as support vector machine (SVM) [12], K-nearest neighbor (KNN) [13] and neural networks [14,15,16] worked on the compressed features. However, these two-stage methods destroyed the structural information of high-dimensional data, and the hard connection between the dimensionality reduction model and classification model usually leads to a degraded performance. Fortunately, tensor decomposition techniques, such as Tucker decomposition and CP decomposition [17], are efficient tool to extract features from higher dimensional data with fewer parameters [18]. Moreover, it preserves the structural information in the decomposed core tensor. These two merits make it suitable to serve as the feature extractor or data compressor in higher dimensional scenario. However, tensor decomposition always works in an offline mode, where the transformation parameters such as the loading matrices depend only on the individual tensor sample. They cannot be properly learned online by a set of data samples. Therefore, the tensor decomposition-based learning method is promising to incorporate compressor and classifier in an end-to-end network.

In order to solve the above problems, we propose a polarization extension method for multisource data fusion and a full tensor decomposition network (FTDN) for end-to-end crop classification. The polarization extension method extends dual PolSAR to synthetic quad PolSAR data, then providing more complementary polarization scattering features to optical data. The FTDN is a one-stage network with fewer parameters, which exploits tensor decomposition technique in each layer to implement feature extraction and pattern classification. It is able to extract structural information from the original high-dimensional data, and circumvent the curse of dimensionality through tensor decomposition.

The rest of the paper is organized as follows. The second part presents the polarization extension-based fusion method, feature collection themes, and the architecture of the FTDN. The third part provides experimental results to verify the feasibility of the proposed method. The fourth part discusses the experimental results. Finally, our conclusion is drawn in Section 5.

2. Proposed Methodology

2.1. Multisource Data Fusion via Polarization Extension

Multispectral data and SAR data provide complementary features. The fusion of SAR data and multispectral data contributes to the better visual perception of objects and compensates for spatial information. Intensity hue saturation (IHS) and PCA methods are often used to merge multisensor data [19]. Chavez investigated the feasibility of three methods for using panchromatic data to substitute spatial features of multispectral data (both statistically and visually) [20]; Chandrakanth demonstrated the feasibility of the fusion of SAR data and multispectral data [21]. A basic assumption concerning these methods is that the SAR amplitude is closely related to the intensity and principal component of multispectral images with high correlation coefficients. Therefore, SAR data can replace either of these two images while re-transforming data back into the original image space. Inspired by this assumption, we propose to consider an antithetical assumption that the principal component of multispectral images can be regarded as PolSAR data from the point of view of intensity, which yields the simulated SAR data in a certain polarization mode. Then the dual PolSAR data can be extended to construct synthetic quad PolSAR data.

For quad PolSAR data, each pixel is represented by a 2 × 2 complex matrix as follows

$$\mathbf{S}=\left[\begin{array}{cc}{S}_{HH}& S_{HV}\\ S_{VH}& S_{VV}\end{array}\right]$$

(1)

where $S_{HV}$ denotes the scattering factor of horizontal transmitting and vertical receiving polarization and the others have similar definitions. Here we demonstrate the fusion of multispectral data and dual PolSAR data generated in VH and VV polarization modes. Note that the reciprocity condition $S_{HV}=S_{VH}$ is commonly assumed for quad PolSAR data, then we only need to construct the scattering factor $S_{HH}$ to form the quad PolSAR data. Firstly, we perform the registration of imageries collected by different sensors in advance. The overall fusion process is then illustrated in Figure 1. On one hand, we extract the principal component of multispectral data using PCA. The extracted principal component represents the spatial information, which is used to generate the amplitude of $S_{HH}$. On the other hand, the phase of $S_{HH}$ is simulated by using the phase of $S_{HV}$ instead. This is because they shared the same horizontal polarization transmitting antenna. Moreover, most of the distinguishable features (see

Table 1. The 36-dimensional features of quad PolSAR data.

Feature Extraction	Features	Dimension
Features based on measured data	Polarization intensities $\left|S_{HH}\right|,\left|S_{HV}\right|,\left|S_{VV}\right|$	3
	Amplitude of HH-VV correlation $\left|\frac{S_{VV}{S}_{HV}^{*}}{\sqrt{{\left|S_{HH}\right|}^2{\left|S_{VV}\right|}^2}}\right|$	1
	Phase difference of HH-VV ${tan}^{-1}(\frac{\mathrm{Im}\left(S_{HH}{S}_{VV}^{*}\right)}{\mathrm{Re}\left(S_{HH}{S}_{VV}^{*}\right)})$	1
	Co-polarized ratio $10{log}_{10}(2{\left|S_{HV}\right|}^2/\phantom{2{\left|S_{HV}\right|}^2{\left|S_{VV}\right|}^2}{\left|S_{VV}\right|}^2)$	1
	Cross-polarized ratio $10{log}_{10}({\left|S_{VV}\right|}^2/\phantom{{\left|S_{VV}\right|}^2{\left|S_{HH}\right|}^2}{\left|S_{HH}\right|}^2)$	1
	Co-polarization ratio $10{log}_{10}(2{\left|S_{HV}\right|}^2/\phantom{2{\left|S_{HV}\right|}^2{\left|S_{HH}\right|}^2}{\left|S_{HH}\right|}^2)$	1
	Degrees of polarization ${\left|S_{VV}\right|}^2/\phantom{{\left|S_{VV}\right|}^2{\left|S_{HH}\right|}^2}{\left|S_{HH}\right|}^2$ $2{\left|S_{HV}\right|}^2/\phantom{2{\left|S_{HV}\right|}^2({\left|S_{HH}\right|}^2+{\left|S_{VV}\right|}^2)}({\left|S_{HH}\right|}^2+{\left|S_{VV}\right|}^2)$	2
Incoherent decomposition	Freeman–Durden decomposition	5
	Yamaguchi decomposition	7
	Cloude decomposition	3
	Huynen decomposition	9
Other decomposition	Null angle parameters	2
Total	36

below) for crop classification is irrelevant to the phase of scattering factor, and our experimental results show that the crop classification performance is insensitive to the phase of $S_{HH}$. Finally, the simulated $S_{HH}$ is incorporated into the dual polarization data $S_{VH}$ and $S_{VV}$ to form synthetic quad PolSAR data, which enables us to extract higher dimensional features of the target through various polarimetric decomposition schemes. High-dimensional features extracted from synthetic quad PolSAR data provide more complementary information to multispectral data for accurate classification applications.

2.2. Polarization Decomposition and Feature Collection

Polarization scattering feature extraction of PolSAR data can obtain more target feature information and further improve the classification, detection, and identification capabilities of a data-driven model. A quad PolSAR image can obtain the 36-dimensional polarimetric scattering features by using a variety of classical decomposition algorithms. As shown in Table 1, 10 features were directly obtained from the measured data by simple transformations and combinations. We obtained 24 features using incoherent decomposition algorithms, including the Huynen decomposition [22], Freeman–Durden decomposition [23], Yamaguchi decomposition [24], and Cloude–Pottier decomposition [25]. The last 2 null angle parameters were computed via polarimetric matrix rotation [26]. A quad PolSAR image was processed and the 36-dimensional features were obtained. The resulting features involved all of the potential data of the primitive PolSAR data.

2.3. Full Tensor Decomposition Network

(1) Tucker decomposition-based feature extraction layer: When using the traditional neural network to classify high dimensional crop data, it is often necessary to perform dimensionality reduction operations to avoid the curse of dimensionality. Moreover, the extracted features from the data compressor may not be suitable for the classification model because the compressor and classification model are separately trained. We propose a Tucker decomposition-based feature extraction (TDFE) layer to extract hidden information from high-dimensional data. The TDFE layer can be used as the hidden layer of a tensor network, which performs the data compression or feature extraction for further processing.

The TDFE layer, different from the fully connected layer, uses tensor decomposition to realize the forward propagation. For a 3-way tensor $\mathcal{X}=\left\{x_{i_1{i}_2{i}_3}\right\}\in R^{I_1\times I_2\times I_3}$, a compressed feature tensor $\mathcal{Z}=\left\{z_{j_1{j}_2{j}_3}\right\}\in R^{J_1\times J_2\times J_3}$ can be obtained by transforming $\mathcal{X}$ as follows

$$\mathcal{Y}=\mathcal{X}{\times }_1{\mathbf{M}}_1{\times }_2{\mathbf{M}}_2{\times }_3{\mathbf{M}}_3$$

(2)

$$\mathcal{Z}=f\left(\mathcal{Y}\right)$$

(3)

where ${\mathbf{M}}_n\in R^{J_n\times I_n},n=1,2,3$ are factor matrices, ${\times }_1$ performs the operation that multiplies each column fiber of $\mathcal{X}$ with ${\mathbf{M}}_1$, ${\times }_2$ performs the operation that multiplies each row fiber of $\mathcal{X}$ with ${\mathbf{M}}_2$, and ${\times }_3$ performs the operation that multiplies each tube fiber of $\mathcal{X}$ with ${\mathbf{M}}_3$, $\mathcal{Y}=\left\{y_{j_1{j}_2{j}_3}\right\}\in R^{J_1\times J_2\times J_3}$ is the multi-linear transformed tensor and $f(·)$ is the activation function. Note that the outcome $\mathcal{Y}$ is independent of the calculation order of mode product, each component $y_{j_1{j}_2{j}_3}$ in tensor $\mathcal{Y}$ is calculated by

$$y_{j_1{j}_2{j}_3}=\sum _{i_1=1}^{I_1}\sum _{i_2=1}^{I_2}\sum _{i_3=1}^{I_3}{x}_{i_1{i}_2{i}_3}{m}_{1_{j_1{i}_1}}{m}_{2_{j_2{i}_2}}{m}_{3_{j_3{i}_3}}$$

(4)

where $m_{1_{j_1{i}_1}}$, $m_{2_{j_2{i}_2}}$, $m_{3_{j_3{i}_3}}$ are corresponding entries in matrices ${\mathbf{M}}_1$, ${\mathbf{M}}_2$ and ${\mathbf{M}}_3$. The forward propagation of the TDFE layer can be regarded as the inverse process of Tucker decomposition. The TDFE layer allows us to decompose the input tensor without destroying its coupling structure between each mode.

The factor matrices ${\mathbf{M}}_n$ are learned end-to-end by error backpropagation. When the TDFE layer is regarded as a hidden layer of a tensor neural network, we define the neuron error tensor $\mathcal{E}$ of the TDFE layer as $\mathcal{E}\stackrel{\Delta }{=}\frac{\partial L}{\partial \mathcal{Y}}=\left\{e_{j_1{j}_2{j}_3}\right\}\in R^{J_1\times J_2\times J_3}$,where L represents the loss function of the network. The update of each entry $m_{1_{j_1{i}_1}}\in {\mathbf{M}}_1$ is derived by finding the gradient

$$\begin{array}{cc}\hfill \frac{\partial L}{\partial m_{1_{j_1{i}_1}}}& =\sum _{j_2=1}^{J_2}\sum _{j_3=1}^{J_3}\frac{\partial L}{\partial y_{j_1{j}_2{j}_3}}\frac{\partial y_{j_1{j}_2{j}_3}}{\partial m_{1_{j_1{i}_1}}}\hfill \\ \hfill & =\sum _{j_2=1}^{J_2}\sum _{j_3=1}^{J_3}\sum _{i_2=1}^{I_2}\sum _{i_3=1}^{I_3}{e}_{j_1{j}_2{j}_3}{x}_{i_1{i}_2{i}_3}{m}_{2_{j_2{i}_2}}{m}_{3_{j_3{i}_3}}\hfill \end{array}$$

(5)

The updates of ${\mathbf{M}}_2$ and ${\mathbf{M}}_3$ are similar to that of ${\mathbf{M}}_1$.

(2) Tucker decomposition-based classification layer: A flattened layer is always necessary for classical networks to vectorize the resulting features extracted by the hidden layer when classification or regression is performed. However, the structural information from the high-dimensional feature tensor will be discarded after the flattened layer. Instead, we propose to use a higher order weight tensor $\mathcal{W}$ to project the feature tensor into a class vector without discarding the multimodal structure. The higher order weight tensor $\mathcal{W}$ usually contains a large number of parameters, the update of which will lead to extremely high computational complexity. Here we proposed a Tucker decomposition-based classification (TDC) layer, which used the low-rank representation to replace the original weight tensor.

Figure 2 illustrates the structure of the TDC layer. Assuming that the 3-way feature tensor $\mathcal{Z}=\left\{z_{k_1{k}_2{k}_3}\right\}\in R^{K_1\times K_2\times K_3}$ is fed to the TDC layer, an output vector $\mathbf{t}=\left\{t_{k_4}\right\}\in R^{K_4}$ can be generated by a 4-way weight tensor $\mathcal{W}=\left\{w_{k_1{k}_2{k}_3{k}_4}\right\}\in R^{K_1\times K_2\times K_3\times K_4}$, where $K_4$ is the number of classes and each component $t_{k_4}$ is calculated by the inner product of $\mathcal{Z}$ and the slices of $\mathcal{W}$

$$\begin{array}{cc}\hfill t_{k_4}& =〈\mathcal{Z},{\mathcal{W}}_{k_4}〉\hfill \\ & =\sum _{k_1=1}^{K_1}\sum _{k_2=1}^{K_2}\sum _{k_3=1}^{K_3}{z}_{k_1{k}_2{k}_3}{w}_{k_1{k}_2{k}_3{k}_4}\hfill \end{array}$$

(6)

where $\langle \rangle$ denotes the inner product of two tensors, and ${\mathcal{W}}_{k_4}$ is the slice of $\mathcal{W}$ along the fourth way. In order to reduce the parameters of the TDC layer, the weight tensor $\mathcal{W}$ is decomposed and represented by a core tensor $\mathcal{R}\in R^{Q_1\times Q_2\times Q_3\times Q_4}$ and factor matrices ${\mathbf{U}}_n\in R^{K_n\times Q_n}$

$$\mathcal{W}=\mathcal{R}{\times }_1{\mathbf{U}}_1{\times }_2{\mathbf{U}}_2{\times }_3{\mathbf{U}}_3{\times }_4{\mathbf{U}}_4$$

(7)

Then substituting (6) into (5) leads to

$$\begin{array}{cc}\hfill \begin{array}{c}\hfill t_{k_4}\end{array}& \begin{array}{c}\hfill ={\displaystyle \sum _{k_1=1}^{K_1}}{\displaystyle \sum _{k_2=1}^{K_2}}{\displaystyle \sum _{k_3=1}^{K_3}}{\displaystyle \sum _{q_1=1}^{Q_1}}{\displaystyle \sum _{q_2=1}^{Q_2}}{\displaystyle \sum _{q_3=1}^{Q_3}}{\displaystyle \sum _{q_4=1}^{Q_4}}{z}_{k_1{k}_2{k}_3}{r}_{q_1{q}_2{q}_3{q}_4}\\ \hfill ·u_{1_{k_1{q}_1}}{u}_{2_{k_2{q}_2}}{u}_{3_{k_3{q}_3}}{u}_{4_{k_4{q}_4}}\end{array}\hfill \\ \hfill & \begin{array}{c}\hfill ={\displaystyle \sum _{q_1=1}^{Q_1}}{\displaystyle \sum _{q_2=1}^{Q_2}}{\displaystyle \sum _{q_3=1}^{Q_3}}({\displaystyle \sum _{k_1=1}^{K_1}}{\displaystyle \sum _{k_2=1}^{K_2}}{\displaystyle \sum _{k_3=1}^{K_3}}{z}_{k_1{k}_2{k}_3}{u}_{1_{k_1{q}_1}}{u}_{2_{k_2{q}_2}}{u}_{3_{k_3{q}_3}})\\ \hfill ·({\displaystyle \sum _{q_4=1}^{Q_4}}{r}_{q_1{q}_2{q}_3{q}_4}{u}_{4_{k_4{q}_4}})\end{array}\hfill \\ \hfill & \begin{array}{c}\hfill =〈{\mathcal{Z}}^{\prime },{{\mathcal{R}}^{\prime }}_{k_4}〉\end{array}\hfill \end{array}$$

(8)

where ${\mathcal{Z}}^{\prime }$ and ${\mathcal{R}}_{k_4}^{\prime }$ are 3-way smaller tensors of dimension $Q_1\times Q_2\times Q_3$, since $Q_n$ can be always chosen far less than $K_n$. Note that the parameters of the TDC layer are converted to smaller core tensor $\mathcal{R}$ and the factor matrices ${\mathbf{U}}_1,\cdots ,{\mathbf{U}}_4$. Therefore, the number of parameters of the network can be greatly reduced.

When the TDC layer is used as the classification layer of a tensor neural network, the parameters of the TDC layer are learned end-to-end by error backpropagation. Let us define the neuron error vector $\mathbf{e}$ of the TDC layer as $\mathbf{e}\stackrel{\Delta }{=}\frac{\partial L}{\partial \mathbf{t}}=\left\{e_{k_4}\right\}\in R^{K_4}$, then the update of $\mathcal{R}$ and ${\mathbf{U}}_1$ are derived by finding the gradient

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \frac{\partial L}{\partial r_{q_1{q}_2{q}_3{q}_4}}={\displaystyle \sum _{k_1=1}^{K_1}}{\displaystyle \sum _{k_2=1}^{K_2}}{\displaystyle \sum _{k_3=1}^{K_3}}{\displaystyle \sum _{k_4=1}^{K_4}}{e}_{k_4}{z}_{k_1{k}_2{k}_3}\\ \hfill ·u_{1_{k_1{q}_1}}{u}_{2_{k_2{q}_2}}{u}_{3_{k_3{q}_3}}{u}_{4_{k_4{q}_4}}\end{array}\end{array}$$

(9)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \frac{\partial L}{\partial u_{1_{k_1{q}_1}}}={\displaystyle \sum _{k_2=1}^{K_2}}{\displaystyle \sum _{k_3=1}^{K_3}}{\displaystyle \sum _{k_4=1}^{K_4}}{e}_{k_4}{z}_{k_1{k}_2{k}_3}{\displaystyle \sum _{q_2=1}^{Q_2}}{\displaystyle \sum _{q_3=1}^{Q_3}}{\displaystyle \sum _{q_4=1}^{Q_4}}\\ \hfill r_{q_1{q}_2{q}_3{q}_4}{u}_{2_{k_2{q}_2}}{u}_{3_{k_3{q}_3}}{u}_{4_{k_4{q}_4}}\end{array}\end{array}$$

(10)

The update of each entry ${\mathbf{U}}_2$, ${\mathbf{U}}_3$, and ${\mathbf{U}}_4$ is similar to that of ${\mathbf{U}}_1$.

(3) FTDN architecture and learning: Based on TDFE layer and TDC layer, we propose a full tensor decomposition network (FTDN) for high-dimensional data classification. The architecture of the FTDN is shown in Figure 3, which contains two TDFE layers and one TDC layer. The input tensor sample $\mathcal{X}=\left\{x_{i_1{i}_2{i}_3}\right\}\in R^{I_1\times I_2\times I_3}$ that corresponds to a certain pixel is formed by collecting all features of its local neighborhood pixels, where $I_3$ is the number of features and $I_1\times I_2$ is the size of the neighborhood. For TDFE layer-1, the feedforward computation is straightforward, and its output feature tensor is denoted by ${\mathcal{Z}}^{\left\{1\right\}}=\{z_{j_1{j}_2{j}_3}^{\left\{1\right\}}\}\in R^{J_1\times J_2\times J_3}$, where the superscript in the bracket denotes the layer index. For TDFE layer-2, the input tensor comes from the output of the previous layer, and the output feature tensor is denoted by ${\mathcal{Z}}^{\left\{2\right\}}=\{z_{k_1{k}_2{k}_3}^{\left\{2\right\}}\}\in R^{K_1\times K_2\times K_3}$. For the TDC layer, the output vector $\mathbf{t}$ is transformed by a softmax layer to calculate the class posteriors $\mathbf{z}$, where $g\left(x\right)$ represents the softmax function.

We now derive the error backpropagation-based learning of the FTDN. We see that a stochastic gradient descent learning for the FTDN is straightforward according to Equation (3), (7) and (8), where the remained issue is to compute the neuron errors of each layer. As for layer TDC layer, the neuron error vector $\mathbf{e}$ depends on the specific loss function L, which can be chosen from typical cross-entropy or mean square error between the network output and target label. Taking the cross entropy loss, for example, the neuron error vector $\mathbf{e}$ reads $\mathbf{e}=\mathbf{z}-\mathbf{g}$, where $\mathbf{g}$ denotes the corresponding label vector. As for TDFE layer-2, the entries $e_{k_1{k}_2{k}_3}^{\left\{2\right\}}$ of neuron error tensor ${\mathcal{E}}^{\left\{2\right\}}$ can be computed from $\mathbf{e}$ via the error backpropagation technique

$$\begin{array}{cc}\hfill \begin{array}{c}\hfill e_{k_1{k}_2{k}_3}^{\left\{2\right\}}\end{array}& \begin{array}{c}\hfill ={\displaystyle \sum _{k_4=1}^{K_4}}\frac{\partial L}{\partial t_{k_4}}\frac{\partial t_{k_4}}{\partial z_{k_1{k}_2{k}_3}^{\left\{2\right\}}}\frac{\partial z_{k_1{k}_2{k}_3}^{\left\{2\right\}}}{\partial y_{k_1{k}_2{k}_3}^{\left\{2\right\}}}\end{array}\hfill \\ \hfill & \begin{array}{c}\hfill =f^{\prime }\left(y_{k_1{k}_2{k}_3}^{\left\{2\right\}}\right){\displaystyle \sum _{k_4=1}^{K_4}}{e}_{k_4}{\displaystyle \sum _{q_1=1}^{Q_1}}{\displaystyle \sum _{q_2=1}^{Q_2}}{\displaystyle \sum _{q_3=1}^{Q_3}}{\displaystyle \sum _{q_4=1}^{Q_4}}\\ \hfill r_{q_1{q}_2{q}_3{q}_4}{u}_{1_{k_1{q}_1}}{u}_{2_{k_2{q}_2}}{u}_{3_{k_3{q}_3}}{u}_{4_{k_4{q}_4}}\end{array}\hfill \end{array}$$

(11)

Similarly, the error backpropagation from ${\mathcal{E}}^{\left\{2\right\}}$ to ${\mathcal{E}}^{\left\{1\right\}}$ is represented as

$$\begin{array}{cc}\hfill \begin{array}{c}\hfill e_{j_1{j}_2{j}_3}^{\left\{1\right\}}\end{array}& \begin{array}{c}\hfill ={\displaystyle \sum _{k_1=1}^{K_1}}{\displaystyle \sum _{k_2=1}^{K_2}}{\displaystyle \sum _{k_3=1}^{K_3}}\frac{\partial L}{\partial y_{k_1{k}_2{k}_3}^{\left\{2\right\}}}\frac{\partial y_{k_1{k}_2{k}_3}^{\left\{2\right\}}}{\partial z_{j_1{j}_2{j}_3}^{\left\{1\right\}}}\frac{\partial z_{j_1{j}_2{j}_3}^{\left\{1\right\}}}{\partial y_{j_1{j}_2{j}_3}^{\left\{1\right\}}}\end{array}\hfill \\ \hfill & \begin{array}{c}\hfill =f^{\prime }\left(y_{j_1{j}_2{j}_3}^{\left\{1\right\}}\right){\displaystyle \sum _{k_1=1}^{K_1}}{\displaystyle \sum _{k_2=1}^{K_2}}{\displaystyle \sum _{k_3=1}^{K_3}}{e}_{k_1{k}_2{k}_3}^{\left\{2\right\}}{m}_{1_{k_1{j}_1}}^{\left\{2\right\}}\\ \hfill ·m_{2_{k_2{j}_2}}^{\left\{2\right\}}{m}_{3_{k_3{j}_3}}^{\left\{2\right\}}\end{array}\hfill \end{array}$$

(12)

3. Experimental Results

3.1. Experimental Data

(1) Dali Dataset: The study area in Dali dataset is a farm (${109}^{\circ }{10}^{\prime }{49}^{″}\mathrm{E},$$34^{\circ }{47}^{\prime }{60}^{″}\mathrm{N}$), located in Dali County, Weinan City, Shaanxi Province, which has 1106 × 514 pixels. There are mainly five different kinds of targets and an `unknown’ class in this area; the data are shown in

Table 2. Crop information of the study area.

Targets	Pixel	Proportion
Wheat	142,450	25.06
Bare soil	37,549	6.60
Corn	25,227	4.44
Alfalfa	3009	0.53
Greenhouse	7100	1.25
Unknown	353,149	62.12
Sum	568,484	100

, and the corresponding images of this area are shown in Figure 4.

As shown in Figure 4, the study area is covered by clouds. Five types of data are used in the following experiments: dual PolSAR data, multispectral data, stacking-based fusion data, textural feature fusion data, and polarization extension-fused data. The dual PolSAR data are obtained from the Sentinel-1 satellite with VV and VH polarization modes, and the multispectral data are obtained from the Sentinel-2 satellite. Stacking-based fusion data are formed by stacking multispectral data and original dual PolSAR data. In order to construct textural feature fusion data, we adopt gray-level co-occurrence matrix (GLCM) [27] method to extract textural features of dual PolSAR data, including energy, homogeneity and entropy. Then textural feature fusion data are obtained by combining the extracted features of VV and VH data respectively with multispectral data.

As for the polarization extension-fused data, the generation procedure is illustrated in Figure 5. Firstly, the principal component of the multispectral information is obtained by the standard PCA. Then we use the principal component to form the amplitude of $S_{HH}$, and the phase of $S_{HH}$ is simulated by using the phase of $S_{HV}$ instead. Consequently, the synthetic HH channel data $S_{HH}$ and the dual PolSAR data are combined to obtain synthetic quad PolSAR data. Finally, the 36-dimensional features can be computed according to Table 1, and the combination of the extracted features with multispectral data yields the polarization extension-fused data.

(2) Flevoland Dataset:The Flevoland dataset was acquired by the NASA/JPL AIRSAR system in August 1989. This area has 15 classes of different types of crops and an `unknown’ class with 750 × 1024 pixels. The number of pixels and total area for each crop are summarized in

Table 3. Main Crops in the Study Area.

Crop Type	Crop Code	Number of Pixels	Ratio/%
Stem beans	C1	6338	0.83
Rapeseed	C2	13,863	1.81
Bare soil	C3	5109	0.67
Potatoes	C4	16,156	2.10
Beet	C5	10,033	1.31
Wheat 2	C6	11,159	1.45
Peas	C7	9582	1.25
Wheat 3	C8	22,241	2.90
Lucerne	C9	10,181	1.33
Barley	C10	7595	0.99
Wheat	C11	16,386	2.13
Grasses	C12	7058	0.92
Forest	C13	18,044	2.35
Water	C14	13,232	1.72
Building	C15	735	0.10
Unknown	UN	600,288	78.16
Sum	16	768,000	100

. Figure 6a is the Pauli RGB image of the study area. The true distribution of crops is shown in Figure 6b [28,29]. As described in Section 2, the 36-dimensional features were collected for the Flevoland dataset, in the following experiments, we performed the model training and prediction based on the 36-dimensional features.

3.2. Classification Result of Different Methods Using Polarization Extension-Fused Data

We conducted an experiment on the Dali dataset to confirm the validity of the proposed method for crop classification. We compared FTDN with the state of art SAE-CNN [30], 3D CNN [31], classifiers, and other traditional deep learning methods [32] based on polarization extension-fused data to confirm the validity of the proposed methods.

For fair comparisons, the dimensions of the input data were reduced to 5 through the corresponding dimensionality reduction models for SAE-CNN and 3D CNN. For the FTDN, SAE-CNN, and 3D CNN classifiers, the sizes of the input samples were set to 5 × 5, while for other classifiers, the input single samples were without neighborhood pixels. The learning rate was set to 0.001 and the optimizer we adopted was adaptive moment estimation. for the FTDN, we used the activation function hyperbolic tangent for all TDFE layers. All methods were trained and implemented on our computing server equipped with two Intel Xeon E5-2620 CPUs and four Nvidia TITAN XP GPUs. Overall accuracy (OA) and Kappa coefficient (Kappa) were used to evaluate the classification accuracies of these methods. Each experiment repeated 10 times with randomly selected training samples.

The classification results of SAE-CNN, 3D CNN, 1D CNN, LSTM, and FTDN methods using polarization extension-fused data are shown in Figure 7.

Table 4. Comparison between polarization extension-fused data classified.

Methods	OA	Kappa	Total Parameters
SAE-CNN	95.81	91.78	63,493
3D CNN	96.20	92.57	30,800
1D-CNN	95.53	91.29	157,573
LSTM	96.72	94.35	99,973
FTDN	98.37	96.69	3346

presents a quantitative comparison of the classification accuracy and the number of parameters of the networks. Stratified random sampling was commonly implemented for accuracy assessment [33]; the training samples were selected using stratified random sampling with a 1% ratio.

We can conclude from Table 4 that the FTDN behaves the best performance. From the perspective of OA, the FTDN method is 2.17% higher than 3D CNN, 2.56% higher than SAE-CNN, 2.84% higher than 1D CNN, and 1.65% higher than LSTM. As for Kappa coefficient, the FTDN method is 4.12% higher than 3D CNN, 4.91% higher than SAE-CNN, 5.40% higher than 1D CNN and 2.34% higher than LSTM. Comparing the number of parameters of the networks, the FTDN is 89.14% fewer than 3D CNN, 94.73% fewer than SAE-CNN, 99.98% fewer than 1D CNN and 99.97% fewer than LSTM. These results indicate that classification using polarization extension-fused data achieves excellent effects, and the FTDN method has the best classification accuracy.

It should be pointed out that the proposed FTDN, 1D CNN, and LSTM are single end-to-end models for crop classification free of the specialized dimensionality compression model. The FTDN allows us to extract features without destroying its coupling structure between each mode, then the extracted structural information yields better classification accuracy. The fewer network parameters demonstrate the reason why FTDN can circumvent the curse of dimensionality [29]. However, for 3D CNN and SAE-CNN methods, the hard coupling between the feature compression model and the classification model is inevitable. After dimensionality reduction by data compression model, the input features are not necessarily discriminative for the classifier. As for 1D CNN and LSTM methods, although these classifiers maintain the end-to-end structure, these classifiers are usually used to process sequence data. The features of polarization extension-fused data contain few continuous sequence features and the characteristics of their network architectures determine that these classifiers cannot extract sufficient information for classification. The FTDN is a single end-to-end model for crop classification without hard coupling, and the special multi-way feature extraction technique based on the Tucker decomposition along each mode is efficient at capturing the structural distinctions between different crops. These results demonstrate the superiority of the FTDN method.

3.3. Classification Result of Different Methods Using the Flevoland Dataset

In the previous experiment, the FTDN method is used for the first time to classify polarization extension data, the results validate its superiority. However, Dali dataset is a relatively simple dataset with only five different categories, which cannot demonstrate the classification capability sufficiently. In order to investigate the performance of the FTDN method in high-dimensional crop classification from multiple aspects, we present the experimental results on Flevoland dataset.

The classification results and error maps for the SVM, SAE-CNN, 3D CNN, 1D CNN, LSTM, vision transformer (ViT) [34], and FTDN methods are shown in Figure 8. It is noticeable that the FTDN behaves the best performance.

By further summarizing the recall rates of crops of 15 categories individually in

Table 5. The recall rates of different classifiers for each crop.

Crop Type	The Number of Train Samples (1%)	Accuracy (%)
		SVM	SAE-CNN	3D CNN	1D CNN	LSTM	ViT	FTDN
C1	63	9.94	92.30	94.11	92.43	96.79	97.05	98.09
C2	139	8.36	95.24	95.28	87.28	96.79	94.99	95.95
C3	51	93.13	98.61	99.77	99.08	99.54	99.12	99.84
C4	162	84.37	93.52	93.96	94.99	95.07	97.46	99.16
C5	100	81.33	96.74	92.85	89.68	97.14	95.83	96.24
C6	112	48.67	97.99	93.54	93.82	96.69	98.05	98.64
C7	96	95.68	95.09	93.22	95.97	99.14	97.70	99.47
C8	222	99.84	99.22	95.55	96.27	98.30	99.15	99.50
C9	102	90.71	98.89	94.75	85.64	97.85	97.39	97.73
C10	76	87.02	94.81	90.60	89.41	96.36	96.59	99.57
C11	164	41.36	93.52	94.54	81.86	91.05	95.92	98.88
C12	71	73.52	90.12	92.94	84.92	95.80	91.92	94.86
C13	180	97.56	96.71	97.91	89.59	97.75	98.27	99.16
C14	132	99.92	99.74	98.19	99.67	98.50	99.93	99.95
C15	7	77.69	85.56	83.27	76.46	87.34	90.07	90.61

, we can see that the recall rates of the FTDN for each category are universally higher than that of other methods. It is noticeable that the training samples are randomly selected with a 1% ratio for each category. Although the training sample numbers between different categories are unbalanced, the statistical distribution of the training samples was consistent with the distribution of the overall samples, the classification results will obey the prior distribution of crops. Extra experiments were also carried out, the results show that the overall accuracies under balanced samples case and unbalanced case are close to each other. In particular, for some categories with small planting areas, i.e., C1, C10, C12, and C15, the FTDN still yields excellent classification performances. The recall rates of other classifiers for these crops are poorer than the FTDN. Note that this corresponds to a small-sample learning problem; the results imply that the proposed FTDN is qualified for the small-sample learning problem to some extent. The low overall accuracy of other classifiers arises from the poorer classification performance for small planting areas. We believe that large scale networks, such as SAE-CNN and ViT, are not suitable for handling the small-sample learning scenario because the small samples are not adequate to train the large parameter amounts. The FTDN model is a simple network with only two TDFE layers and one TDC layer, which makes it more suitable for the small-sample learning problem. Therefore, the classification accuracy of small sample crops can be improved.

The number of parameters for a crop classification model is also very important because it affects the efficiency of the model implemented in a specified platform. The number of parameters of the FTDN, ViT, LSTM, 1D CNN, 3D CNN, and SAE-CNN are shown in

Table 6. The total parameters of different classifiers.

Competitors	Total Parameters
SAE-CNN	64,795
3D CNN	110,559
1D CNN	226,703
LSTM	86,927
ViT	15,305,790
FTDN	5792

. It can be observed that the total number of parameters for the FTDN is fewer than that of 3D CNN, SAE-CNN, 1D CNN, LSTM, and ViT. The sizes of the factor matrices determine the number of parameters of the FTDN, which are far less than convolution kernels. The FTDN can circumvent the increasing parameters caused by high-dimensional input data, which is why the FTDN can integrate the dimensionality reduction model and the classification model in a single network. The fewer parameters imply that the FTDN model is more suitable to efficiently be implemented in resource-limited-embedded platforms.

4. Discussion

4.1. Classification Results of the SVM Using Different Datasets

We can conclude from Section 3 that all classifiers achieved excellent classification accuracies using polarization extension-fused data. In order to further explore the superiorities of the polarization extension method, we used the SVM classifier here to classify different types of fusion data and compare the classification results.

The classification results of the five different datasets using the SVM method are shown in Figure 9,

Table 7. Contrast between the classification results before and after data fusion.

Methods	OA	Kappa
SAR data	73.18	62.32
Multispectral data	87.93	74.72
Stacking-based fusion data	88.03	74.96
Textural features fusion data	91.76	83.63
Polarization extension-fused data	93.68	87.30

presents a quantitative comparison of the classification accuracy. Training samples were randomly selected with 1% ratio, and the remaining samples were used for the test. The regularization parameter for the SVM was set to 0.8.

We can conclude from Table 7 that the OA of the SVM method, which used the polarization extension-fused data, was 1.92% higher than the textural feature fusion data, 5.65% higher than the stacking-based fusion data, 5.75% higher than multispectral data, and 20.5% higher than dual PolSAR data. The Kappas of the SVM method that used the polarization extension-fused data were 3.67% higher than textural feature fusion data, 12.34% higher than stacking-based fusion data, 12.58% higher than multispectral data, and 24.98% higher than dual PolSAR data.

Since the study area is covered by clouds, it is difficult to achieve outstanding accuracy by directly classifying multispectral data using the SVM method. Furthermore, the dual PolSAR data provide few features, they are insufficient for the SVM classifier to yield good classification performance. Simply stacking dual PolSAR data and multispectral data cannot take full use of the complementary advantages of multisource data, which only provide few accuracy improvements for crop classification. Textural features are more distinguishable for the classifier than the original dual PolSAR data, and fusing these features with multispectral data can improve classification accuracy. Therefore, the classification accuracies of textural feature fusion data outperform stacking-based fusion data. The proposed polarization extension method extracted higher dimensional radar features by generating synthetic quad PolSAR data, which provided more complementary data to multispectral data. The spatial data of multispectral data were very similar to SAR data, and could be regarded as good alternates to a certain polarization channel for PolSAR data. Hence, the synthetic quad PolSAR data can be used to approximate the real quad PolSAR data. We can obtain the 36-dimensional features from synthetic quad PolSAR data by various polarization decomposition schemes; the extracted 36-dimensional features are more distinguishable than textural features extracted by the GLCM from dual PolSAR data. Therefore, the classification accuracies of the polarization extension-fused data outperformed textural feature fusion data.

4.2. Performance of the FTDN on Different Tensor Samples and Train Ratios

We tested the FTDN classification performance and compared it with the SAE-CNN network and 3D CNN network under different input sizes. It should be noted that the 36-dimensional features are directly fed to the proposed FTDN, whereas the dimensions of the input features are reduced to 9 through corresponding dimensionality reduction models for SAE-CNN and 3D CNN.

Table 8. Classification performances of different classifiers under different input sizes.

Input Size	Classifier	Classification Accuracy (%)
		OA	Kappa
3 × 3	SAE-CNN	90.86	90.04
	3D CNN	90.93	90.12
	FTDN	96.23	95.89
7 × 7	SAE-CNN	93.00	92.38
	3D CNN	92.69	92.04
	FTDN	97.61	97.51
15 × 15	SAE-CNN	96.17	95.83
	3D CNN	94.95	94.50
	FTDN	98.48	98.34
25 × 25	SAE-CNN	96.98	96.72
	3D CNN	96.36	96.03
	FTDN	98.82	98.71

shows the OA and Kappa coefficients of these classifiers with different input sizes under the 1% train ratio.

By observing Table 8, we conclude that these classifiers trained by larger input sizes perform better than smaller ones. The FTDN classifier can achieve superior classification accuracy under various input sizes; however, other classifiers are susceptible to the input sizes. In particular, the OA of the FTDN classifier is 5.37% higher than that of SAE-CNN and 5.30% higher than that of 3D CNN when the input size is set to 3 × 3, and 4.61% higher than that of SAE-CNN and 4.92% higher than that of 3D CNN when the input size is set to 7 × 7. The gap of OA (of different classifiers) will diminish as the input size increases. As for various input sizes, the classification performance of the FTDN is always better than SAE-CNN and 3D CNN. The excellent performance of the FTDN arises from the special multi-way feature extraction technique that is based on the TDFE layers, which can capture the structural distinctions in smaller tensor samples. However, the corresponding network architectures of 3D CNN and SAE-CNN cannot perform better in feature extracting when inputting smaller sample sizes. The input size 15 × 15 achieved excellent performance for all classifiers; therefore, there is no need to further increase the sizes of the input tensors because the computational burden increases accordingly. So, in the following experiments, we set the input sizes of these classifiers to 15 × 15.

The classification accuracy of different methods can be improved with the increase in the training ratio. The increase of the training ratio will increase the computational burden accordingly; therefore, it is important to explore the performances of different models under different training ratios. We evaluated the crop classification performances of different classifiers by using different ratios of training samples. Here, four ratios (0.5%, 1%, 5%, and 10%) were considered.

Table 9. Classification performances of different classifiers under different train ratios.

Train Ratio	Classifier	Classification Accuracy (%)
		OA	Kappa
0.5%	SVM	55.76	50.97
	SAE-CNN	93.08	92.47
	3D CNN	91.69	90.92
	1D CNN	93.22	92.62
	LSTM	95.33	94.91
	ViT	96.69	96.40
	FTDN	97.75	97.55
1%	SVM	73.35	71.87
	SAE-CNN	96.17	95.83
	3D CNN	94.95	94.50
	1D CNN	95.53	91.29
	LSTM	96.72	94.35
	ViT	97.26	97.02
	FTDN	98.48	98.34
5%	SVM	86.27	87.36
	SAE-CNN	98.68	98.35
	3D CNN	97.99	97.81
	1D CNN	97.44	97.21
	LSTM	98.17	98.01
	ViT	98.84	98.62
	FTDN	99.34	99.28
10%	SVM	92.96	92.32
	SAE-CNN	99.07	98.93
	3D CNN	98.49	98.35
	1D CNN	97.99	97.81
	LSTM	99.06	98.98
	ViT	99.37	99.13
	FTDN	99.81	99.79

shows the OA and Kappa coefficients of different classifiers with different train ratios.

It can be seen from Table 9 that the classification accuracies for all classifiers increased as did the growth of the training ratios. The FTDN always performed better than the other classifiers among these ratios. Note that the proposed FTDN model under a low training ratio performed excellent, and the other model could not perform well in these cases. The SVM classifier is obviously inferior to other competitors in terms of OA and Kappa because the SVM usually performs the crop classification based on the samples formed by individual pixels; the others are based on the samples formed by local neighborhood pixels. We believe that the poorer performances of the 3D CNN and SAE-CNN arise from the hard coupling between the feature compression model and the classification model. The feature compression operation may lead to insufficient information for discrimination; the parameters of the classification model are difficult to train adequately under low training ratios. As for 1D CNN and LSTM classifiers, as mentioned earlier, although these classifiers maintain end-to-end structures, the characteristics of their network architectures determine that these classifiers cannot extract sufficient information for classification. The ViT network is commonly applied to large data sets. In this application, it nearly achieves the best performance. However, the ViT model holds a large number of parameters, and the proposed FTDN accounts for merely 0.04% of ViT parameters. The excellent performance of the FTDN arises from the end-to-end network architecture. The extracted features by the TDFE layers are discriminative for the TDC layer; hence, the classification accuracy can be improved.

5. Conclusions

In order to extract high-dimensional polarization scattering features from dual PolSAR data, this paper proposes a deep fusion method of SAR and multispectral data based on the polarization extension, which can extract high-dimensional polarization scattering features to provide more complementary information for multispectral data. Furthermore, to solve the curse of dimensionality problem in high-dimensional data classification, a full tensor decomposition network (FTDN) is proposed. Compared with other fusion methods, the polarization extension method can make better use of the advantages of multisource data. The experimental results verify that the polarization extension method is effective and can significantly improve the classification accuracy of crops. Moreover, the FTDN model can effectively deal with high-dimensional features through end-to-end learning. Experimental results show that the FTDN outperforms other traditional methods for classifying high-dimensional remote sensing data in terms of classification accuracy and the number of network parameters.