SMOTE-Based Weighted Deep Rotation Forest for the Imbalanced Hyperspectral Data Classification

Abstract

Conventional classification algorithms have shown great success in balanced hyperspectral data classification. However, the imbalanced class distribution is a fundamental problem of hyperspectral data, and it is regarded as one of the great challenges in classification tasks. To solve this problem, a non-ANN based deep learning, namely SMOTE-Based Weighted Deep Rotation Forest (SMOTE-WDRoF) is proposed in this paper. First, the neighboring pixels of instances are introduced as the spatial information and balanced datasets are created by using the SMOTE algorithm. Second, these datasets are fed into the WDRoF model that consists of the rotation forest and the multi-level cascaded random forests. Specifically, the rotation forest is used to generate rotation feature vectors, which are input into the subsequent cascade forest. Furthermore, the output probability of each level and the original data are stacked as the dataset of the next level. And the sample weights are automatically adjusted according to the dynamic weight function constructed by the classification results of each level. Compared with the traditional deep learning approaches, the proposed method consumes much less training time. The experimental results on four public hyperspectral data demonstrate that the proposed method can get better performance than support vector machine, random forest, rotation forest, SMOTE combined rotation forest, convolutional neural network, and rotation-based deep forest in multiclass imbalance learning.

1. Introduction

Hyperspectral imagery is simultaneously obtained by remote sensors in dozens or hundreds of narrow and contiguous wavelength bands [1,2,3,4,5]. Compared with traditional panchromatic and multispectral remote sensing images, hyperspectral imagery carry a wealth of spectral information, which enables more accurate discrimination of different objects. Consequently, in recent years, hyperspectral imagery has gained extensive attention for a variety of applications in Earth observations [1,6,7,8,9,10], such as urban mapping, precision agriculture, and environmental monitoring [11,12,13,14,15]. The hyperspectral image classification is a significant research topic and it centers on assigning class labels to pixels. Class distribution, i.e., the proportion of samples belonging to each class, plays an extremely important part in classification research. Some traditional classification methods, such as maximum likelihood classification [16], support vector machine (SVM) [17] and artificial neural network [18], have acquired satisfactory performance on the balanced hyperspectral data.

However, since the hyperspectral image scene usually contains many objects of various sizes and sample labeling is difficult in the real world, the class imbalanced is a fundamental problem in hyperspectral image classification [19]. Generally, the majority classes are defined as the classes with a large number of instances while minority classes are the classes with a small number of samples [9]. Because the cost of misclassifying the minority class is usually much higher than the cost of majority classes [20]. With the skewed class distribution, the classifier is inclined to predict that the input instances belong to the majority class to keep high prediction accuracy [20,21,22,23,24]. Such a strategy is not effective for distinguishing the minority classes, even if they are usually foreground classes of interest. Therefore, one of the biggest challenges that machine learning and remote sensing face is how to classify imbalanced data effectively.

Generally, the aim of imbalance learning is to strive for acquiring a classifier that can provide high classification accuracy for the minority class without heavily compromising the accuracy of the majority classes [25,26,27]. Traditionally, the class-imbalance problem has been dealt with either in the data level [28,29,30] or in the algorithm level [31,32,33,34]. The data level focuses on modifying the sample distribution of classes in training sets to reduce the degree of class imbalance, which makes it fit for the classification prediction of the standard algorithm model. The most common method to deal with the imbalance problem in data level is resampling whose major advantages are that no modification to the classifier is needed and the balanced data can be reused in other applications or classification tasks [35,36]. Resampling can be further divided into two types: undersampling [37] and oversampling [38].

• Undersampling methods: Undersampling alters the size of training sets by sampling a smaller majority class, which reduces the level of imbalance [37] and is easy to perform and have been shown to be useful in imbalanced problems [39,40,41,42]. The major superiority of undersampling is that all training instances are real [35]. Random undersampling (RUS) is a popular method that is designed to balance class distribution by eliminating the majority class instances randomly. However, the main disadvantage of undersampling is that it may neglect potentially useful information, which could be significant for the induction process.
• Oversampling methods: Over-sampling algorithms increase the number of samples either by randomly choosing instances from the minority class and appending them to the original dataset or by synthesizing new examples [43], which can reduce the degree of imbalanced distribution. Random oversampling is simply copying the sample of the minority class, which easily leads to overfitting [44] and has little effect on improving the classification accuracy of the minority class. The synthetic minority oversampling technique (SMOTE) is a powerful algorithm that was proposed by Chawla [29] and has shown a great deal of success in various applications [45,46,47]. SMOTE will be described in detail in Section 2.1.

The main idea at the algorithm level is to modify the existing classification algorithm model appropriately in combination with the actual data distribution. The typical methods include active learning [48], cost-sensitive learning [49,50], and Kernel-based learning [51].

• Active learning methods: Traditional active learning methods are utilized to deal with problems with the unlabeled training dataset. In recent years, various algorithms on active learning from imbalanced data problems have been presented [48,52,53]. Active learning is a kind of learning strategy that selects samples from a random set of training data. It can choose more worthy instances and discard the instances which have less information, so as to enhance the classification performance. The large computation cost for large datasets is the primary disadvantage of these approaches [48].
• Cost-sensitive learning methods: Cost-sensitive learning solves class imbalance problems by using different cost matrices [50]. Currently, there are three commonly used cost-sensitive strategies. (1) The cost-sensitive sample weighting: converting the cost of misclassification into the sample weights on the original data set. (2) The cost-sensitive function is directly incorporated into the existing classification algorithm, which will ameliorate internal structure of the algorithm. (3) The cost-sensitive ensemble: cost-sensitive factors are integrated into the existing classification methods and combine with ensemble learning. Nevertheless, cost-sensitive learning methods require the knowledge of misclassification costs, which are hard to obtain in the datasets in the real world [54,55].
• Kernel-based learning methods: Kernel-based learning is focused on the theories of statistical learning and Vapnik-Chervonenkis (VC) dimensions [56]. The support vector machines (SVMs), which is a typical kernel-based learning method, can obtain the relatively robust classification accuracy for imbalanced data sets [51,57]. Many methods that combine sampling and ensemble techniques with SVM have been proposed [58,59] and effectively improve performance in the case of imbalanced class distribution. For instance, a novel ensemble method, called Bagging of Extrapolation Borderline-SMOTE SVM (BEBS) was proposed to incorporate the borderline information [60]. However, as this method is based on SVM, it is difficult to implement in a large dataset.

The classification approaches only using the spectral information cannot capture the crucial spatial variability perceived for data, which usually leads to lower performance, especially for the hyperspectral data [61]. Recently, approaches based on deep learning have been developed for the spectral-spatial hyperspectral datasets classification and exhibited their high effectiveness and performance [61,62]. Deep learning is an emerging method that has achieved excellent performance in hyperspectral image classification with sufficient well-labeled data sets [63,64]. Generally, a deep graph structure includes a cascade of layers which is consists of multiple linear and non-linear transformations. Compared with traditional machine learning approaches, deep learning methods can automatically extract informative features from the original hyperspectral dataset by a sequence of hierarchical layers [63]. In addition, deep learning has stronger robustness and higher accuracy than machine learning methods with shallower structures. However, most deep learning approaches, such as the convolutional neural network (CNN), have no algorithmic strategy for dealing with imbalanced data [63,65,66]. As the data set grows larger, the detrimental impact of class imbalance on deep learning methods augments. As mentioned before, the imbalance problem has been comprehensively researched in classical machine learning approaches, nevertheless, it has acquired less attention in the context of deep learning [66]. Besides, the training process of traditional deep learning methods generally consumes much time. The rotation-based deep forest [67], a novel deep learning method, is proposed for the classification of hyperspectral images and achieves satisfactory results with less training time. Nevertheless, this method does not solve the classification problem when the data distribution is imbalanced.

To improve the classification ability of non-ANN based deep-learning approaches for imbalanced hyperspectral datasets, a novel SMOTE-Based Weighted Deep Rotation Forest(SMOTE-WDRoF) algorithm is proposed in this paper. First, the neighboring pixels of instances are introduced as spatial information and multiple new synthetic balanced datasets are created by using the SMOTE algorithm. And then, these datasets are fed into the WDRoF model that consists of the rotation forest and the multi-level cascaded random forests. Specifically, the rotation forest is utilized to generate rotation feature vectors, which are input into the subsequent cascade forest. Moreover, the output probability of each level and the original data are stacked as the dataset of the next level. And the sample weights are automatically adjusted according to the dynamic weight function constructed by the classification results of each level. In summary, the proposed algorithm integrates the advantages of SMOTE, spatial information, and adaptive sample weights. The main contributions of this paper are as follows:

(1)

The proposed SMOTE-WDRoF based on deep ensemble learning combines deep rotating forest and SMOTE internally. It can obtain higher accuracy and faster training speed for the imbalanced hyperspectral data.

(2)

Besides, the introduction of the adaptive weight function can alleviate the defect of SMOTE, which is that SMOTE would generate additional noise when synthesizing new samples.

The remaining part of this paper is summarized as follows. Section 2 describes the related work. Section 3 presents the detail information about the proposed methodology. Then, Section 4 shows the results and discussion. Finally, the conclusions are given in Section 5.

2. Related Works

2.1. Synthetic Minority Over-Sampling Technique (SMOTE)

SMOTE, presented by Chawla et al. [29], is the most popular oversampling approach which can solve the overfitting problem. Its main idea is to randomly synthesize new minority samples in the k nearest neighborhood of the selected one through interpolation. It should be noted that the artificial samples are created in the feature space instead of in the data space. The detail process of SMOTE is as follow:

(1)

Calculate k nearest neighbors with minority class samples in accordance with Euclidean distance for each minority instance $x_i$.

(2)

A neighbor $x_j$ is randomly chosen from the k nearest neighbors of $x_i$.

(3)

Create a new instances $x_{new}$ between $x_j$ and $x_i$:

$$x_{new}=x_i+\delta |x_i-x_j|$$

(1)

where $\delta$ is the random number between 0 and 1.

2.2. Random Forest (RF)

Inspired by the bagging algorithm [68], Breiman first proposed random forests [69] in 2001. Its main idea is random sample selection and random feature selection. In RF, all trees are independent of each other, so that the training and testing process are in parallel. Let us suppose a dataset $D_m$ with m samples $(X,Y)$, where $X\in R^D$. First of all, n instances are randomly selected from the original data set $D_m$ with replacement. These instances are utilized to build the current decision tree. Second, f features ($f<D$) are first randomly chosen from the original D features. Based on the criterion of Gini impurity or mean squared error (MSE), Classification and Regression Trees (CART) are created. Finally, the classification result is obtained according to the majority voting criterion.

2.3. Rotation Forest (RoF)

Drawing upon the idea of RF, Rodriguez proposed RoF in 2006 [70]. Based on the idea of feature transformation, this algorithm focuses on improving the difference and accuracy of the base classifier. An RoF model of T size is constructed by implementing the following steps.

(1)

Firstly, the feature space $\mathbb{F}$ is split into K feature sets which are disjoint and each subset includes $N=\mathbb{F}/K$ number of features.

(2)

Secondly, a new training set is obtained by using bootstrap algorithm to randomly selected the 75% of the training data.

(3)

Then, the coefficients $a_{t,g}(g\le G,t\le T)$ is obtained by employing the principal component analysis (PCA) on each subspace $F_{t,g}(g\le G,t\le T)$ and the coefficients of all subspaces are organized in a sparse “rotation” matrix $R_t(t\le T)$.

$$R_t=\left[\begin{array}{cccccccccc}{e}_{t,1}^1& \cdots & e_{t,1}^{N_1}& & 0& & \cdots & & 0& \\ & 0& & e_{t,2}^1& \cdots & e_{t,2}^{N_2}& \cdots & & 0& \\ & ⋮& & & ⋮& & \ddots & & ⋮& \\ & 0& & & 0& & & e_{t,g}^1& \cdots & e_{t,g}^{N_g}\end{array}\right]$$

(2)

(4)

The columns of $R_t$ is rearranged by matching the order of original features F to build the rotation matrix $R_t^{{}^{\prime }}$. Then, construct the new training set $S_t^{{}^{\prime }}=[S_t{R}_t^{{}^{\prime }},Y_t]$, which is used to train an individual classifier.

(5)

Repeat the aforementioned process on all diverse training sets and generate a series of individual classifiers. Finally, the results are obtained by the majority vote rule.

2.4. Rotation-Based Deep Forest (RBDF)

As a simple deep learning model, the rotation-based deep forest (RBDF) includes L level random forests and each level contains w RF models. This approach adopts the output probability of each level as the supplement feature of the next level [67]. The RBDF model contains three steps. First, spatial information is acquired by using a sliding window to extract the neighboring pixels of training samples. Second, the training samples and its neighboring pixels are fed into the RoF model. And each RoF will generate rotation matrices and construct the rotation feature vector. Third, feed the rotation feature vector into an RF model and obtain the classification probability. Then, all the classification probability vectors of level l are averaged to acquire the averaged probability vector which is stacked into the original dataset as the input data of the next level. Finally, the result is generated by finding out the maximum classification probability.

3. Method

In this section, the SMOTE-WDRoF method is proposed to deal with hyperspectral imbalanced data. Firstly, the local spatial structure of instances is introduced and balance datasets are generated by SMOTE, which allows more wealthy information to be obtained from hyperspectral images and alleviates class imbalance in the data level. Then, multiple levels of the forest are utilized to construct the WDRoF model which is the key ingredient of the whole algorithm. More specifically, the rotation forest is utilized to generate rotating feature vectors, which are input into the subsequent cascade forest. Moreover, the output probability of each level and the original data are stacked as the dataset of the next level. And the sample weights are automatically adjusted according to the dynamic weight function constructed by the classification results of each level. The details of the algorithm are as follows.

3.1. Spatial Information Extraction and Balanced Datasets Generation

The object in the image usually contains consistent spatial structure, i.e., neighbor pixels are likely to have the same label. Consequently, spatial-contextual information should be taken into account when classifying. The proposed algorithm combines spatial neighborhood information extraction strategy with SMOTE approach to select informative spatial neighbors and balance the dataset distribution to increase classification accuracy.

First, spatial information is extracted by using a sliding window. Let us assume $\mathbb{X}\in R^{M\times N\times D}$ is the hyperspectral image, where $M,N,D$ represent the height, width, and the number of spectral bands of the image respectively. The $a_{m,n,d}$ represents the value of the pixel that is located in line m, column n and band d. To obtain the spectral and spatial information of the hyperspectral dataset, the patch is constructed by extracting pixels in a window of size $w_1\times w_2\times D$ and step size 1 with the central pixel. Suppose the spectral vector of a pixel is $x\in R^D$, the patch $A_i$ can be defined as

$$A_i=\left[\begin{array}{ccccc}{a}_{(w_1-b)(w_2-b)}& \cdots & a_{w_1(w_2-b)}& \cdots & a_{(w_1+b)(w_2-b)}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ a_{(w_1-b)w_2}& \cdots & a_{w_1{w}_2}& \cdots & a_{(w_1+b)w_2}\\ ⋮& \ddots & ⋮& \ddots & ⋮\\ a_{(w_1-b)(w_2+b)}& \cdots & a_{w_1(w_2+b)}& \cdots & a_{(w_1+b)(w_2+b)}\end{array}\right]$$

(3)

where $w_1=w_2=2b+1$, $1\le b\le min(w_1,w_2)/2$, $i\in (M-w_1)(N-w_2)$. After scanning the whole hyperspectral image, it can be obtained K patches, where $K=(M-w_1)(N-w_2)$. Take a $3\times 3\times D$ sliding window, for example, each sample and its 8 neighboring pixels are extracted, which is shown in Figure 1a. Due to spatial similarity, each instance is generally the same as those of its spatial neighbors and the material fractions are close to each other. Therefore, they have the same label. The hyperspectral imbalance datasets $\{s_1,s_2,\dots ,s_9\}$ denoted as S are formed by extracting the pixels of corresponding positions in all patches and combining the sample labels Y.

Second, according to the proportion of majority class instances to minority class instances, SMOTE oversamples each imbalance data $s_w(w\in 9)$. As is shown in Figure 1b, the circle and star stand for the majority class samples and minority class instances, respectively. Suppose that the new sample is created from sample $x_i$ with $\mathbb{T}=5$. SMOTE will randomly choose a sample from the minority class and its nearest five neighbors. Assuming sample $x_j$ is selected. The newly synthesized instance highlighted by the square shape is generated between $x_i$ and $x_j$ by Equation (1). And then the balanced datasets $\{s_1^{{}^{\prime }},s_2^{{}^{\prime }},\dots ,s_9^{{}^{\prime }}\}$ denoted $S^{{}^{\prime }}$ can be obtained.

3.2. Weighted Deep Rotation Forest (WDRoF)

In this part, we propose the WDRoF algorithm, which is shown in detail in Figure 2. This algorithm adopts a multi-level random forest cascade to classify the hyperspectral dataset. Each level of the random forest produces the classification probabilities and misclassification information of the data which are used as guidance information for the next level. More specifically, the classification probabilities form a class vector that is concatenated with original data to constitute input of the next level. And the classification probability of each layer will be applied to all subsequent layers. Furthermore, the misclassification probability is employed to update the sample weight adaptively. When growing a new level, the performance of this level will be evaluated on the test set. If there is no obvious performance gain, the training procedure will finish. Consequently, the number of levels for RF is automatically identified. The implementation steps of WDRoF are as follows.

(1)

The datasets $\{s_1^{{}^{\prime }},s_2^{{}^{\prime }},\dots ,s_W^{{}^{\prime }}\}$ that have been generated by SMOTE are fed into the RoF models where $W=w_1\times w_2$. The $s_w^{{}^{\prime }}(w\in W)$ can be written as $s_w^{{}^{\prime }}=\{\mathbf{X},Y\}=\{(x_1,y_1),(x_2,y_2),\dots ,(x_K,y_K)\}$, where K stands for the number of instances. In RoF, we apply PCA for features transformation which is a mathematical transformation method that transforms a set of variables into a set of unrelated ones. Its goal is to obtain the projection matrix $Q=[q_1,q_2,\dots ,q_K]$:

$$\begin{array}{cc}\hfill & \underset{Q}{min}tr\left(Q^T\mathbf{X}{\mathbf{X}}^{T}Q\right)\hfill \\ \hfill & s.t.Q^{T}Q=I\hfill \end{array}$$

(4)

First of all, the self-correlation matrix for $\mathbf{X}$ is computed:

$$cov\left(\mathbf{X}\right)=E\left[(\mathbf{X}-E\left[\mathbf{X}\right]){(\mathbf{X}-E\left[\mathbf{X}\right])}^T\right]$$

(5)

where $E\left[\mathbf{X}\right]$ is the expected number of $\mathbf{X}$ and ${[·]}^T$ represents transposition. Second, eigen decomposition is applied on $cov\left(\mathbf{X}\right)$ to calculate its eigenvalues: ${\lambda }_1,{\lambda }_2,\cdots ,{\lambda }_K$ and corresponding eigenvectors: ${\alpha }_1,{\alpha }_2,\cdots ,{\alpha }_K$. Finally, the principal component coefficient can be calculated by the following:

$$E=[e_1,e_2,\cdots ,e_K]={[{\alpha }_1,{\alpha }_2,\cdots ,{\alpha }_K]}^T\mathbf{X}$$

(6)

Construct the rotation matrix with Equation (2) and then generate the rotation feature vectors $\{f_1,f_2,\dots ,f_W\}$ by the RoF.

(2)

The rotation feature vectors $\{f_1,f_2,\dots ,f_W\}$ are fed into the first level of the random forest and the weight of the sample $Weigh{t}_{w,l-1}\left(x_k\right)$ is set to 1. In level 1, each RF will generate the classification probability and classification error information of each instance for the dataset. All the classification probabilities vector $P=\{p_1,p_2,\dots ,p_W\}$ of level 1 are averaged to obtain a robust estimation $\overline{P}$:

$$\begin{array}{cc}\hfill \overline{P}& =\frac{1}{W}\sum _{w=1}^W{p}_w\hfill \\ \hfill & =\frac{1}{W}\sum _{w=1}^W\left[\frac{1}{N_{tree}}\sum _{i=1}^{N_{tree}}I(h_i\left(\mathbf{X}\right)=Y)\right]\hfill \end{array}$$

(7)

where $h_i$ represents the ith decision tree output and $N_{tree}$ stands for number of decision tree in RF. In addition, according to the classification error, the weights of the sample $(x_k,y_k)$ can be computed

$$\begin{array}{c}\hfill Weigh{t}_{w,l}\left(x_k\right)=Weigh{t}_{w,l-1}\left(x_k\right)exp[\frac{1}{C}\sum _{c=1,c\ne y_k}^C\left(v_w(x_k,c)\right]\end{array}$$

(8)

where $v_w(x_k,c)$ is the number of votes of any other class with the wth RF model. The weight of a sample will be increased if it is misclassified by the previous level, which makes the sample play a more significant role in the next level and forces the classifier to focus attention on the misclassified samples.

(3)

In the last level, after the average probability vector is calculated, the prediction label is acquired by finding the maximum probability.

$$y^{*}=argmax\sum _{w=1}^{W}I(v_w\left(x_k\right)=c,c\in 1,2,\dots ,C)$$

(9)

The process of the novel SMOTE-WDRoF method is summarized in Algorithm 1.

Algorithm 1: SMOTE-Based Weighted Deep Rotation Forest (SMOTE-WDRoF)
1	Input: $\mathbb{X}\in R^{M\times N\times D}$: the hyperspectral image; M: the height of the image; N: the width of the image; D: the spectral bands of the image; $w1\times w_2\times D$: the size of sliding window; $W=w1\times w_2$;
2	Process:
3	form = 1:M do
4	for n = 1:N do
5	Obtain K patches $\{A_1,\dots ,A_K\}$ by scanning the image using the sliding
	window with (3)
6	end for
7	end for
8	forw = 1:$\mathbf{W}$ do
9	Acquire the imbalanced data $s_w$ by extracting the pixels of corresponding
	positions in K patches
10	Input $s_w$ into the SMOTE algorithm
11	Construct the balanced data $s_w^{{}^{\prime }}$
12	end for
13	Get the balanced datasets $\{s_1^{{}^{\prime }},\dots ,s_w^{{}^{\prime }},\dots ,s_W^{{}^{\prime }}\}$
	Classification:
14	forl = 1:$\mathbf{L}$ do
15	for w = 1:$\mathbf{W}$ do
16	Construct the rotation feature vector $f_w$ by utilizing RoF algorithm
17	Train the RF model with $f_w$
18	Update each sample weight: $Weigh{t}_{w,l}\left(x_k\right)\leftarrow Weigh{t}_{w,l-1}\left(x_k\right)$ with (8)
19	Calculate the the classification probability $p_w$
20	end for
21	Obtain the average probability vector $\overline{P}$ with (7)
22	Concatenate $\overline{P}$ with the input feature vector to constitute input of the next
	level
23	end for
24	Output: The prediction label $y^{*}=argmax{\sum }_{w=1}^{W}I(v_w\left(x_k\right)=c,c\in 1,2,\dots ,C)$

4. Experimental Results

4.1. Datasets

Four hyperspectral imagery (http://www.ehu.eus/ccwintco/index.php?title=Hyperspectral_Remote_Sensing_Scenes) with a high imbalance ratio (IR), including Indian Pines AVRIS, Kennedy Space Center (KSC), Salinas and University of Pavia scenes, are adopted to assess the effectiveness of the proposed WDRoF. For the sake of assessing the performance of the classification algorithms objectively, the training data and the test data should be independent. For Indian Pines AVRIS and KSC, 30% of samples of each class are randomly selected to construct the training set, and the remaining 70% of samples from each class constitute the test set. For Salinas and University of Pavia scenes, 5% of samples of each class are chosen to construct the training set, and the remaining samples constituted the test set. Furthermore, if the number of samples in a certain class is less than 100, half of the samples in that class are selected for training and the remaining half for testing. More detailed information related to the number of training and testing instances is listed in

Table 1. Data information of Indian Pines AVRIS, Salinas, KSC and University of Pavia ROSIS.

The Dataset	Indian Pines AVRIS			Salinas
	Class No.	Train	Test	Class No.	Train	Test
1	Alfalfa	23	23	Brocoli_green_weeds_1	100	1909
2	Corn-notill	428	1000	Brocoli_green_weeds_2	186	3540
3	Corn-mintill	249	581	cFallow	98	1878
4	Corn	71	166	cFallow_rough_plow	68	1325
5	Grass-pasture	144	339	Fallow_smooth	133	2545
6	Corn-trees	219	511	Stubble	197	3762
7	Corn-pasture-mowed	14	14	Celery	178	3401
8	Hay-windrowed	143	335	Grapes_untrained	563	10,708
9	Oats	10	10	Soil_vinyard_develop	310	5893
10	Soybeans-notill	291	681	Corn_senesced_green_weeds	163	3115
11	Soybeans-mintill	736	1719	Lettuce_romaine_4wk	53	1015
12	Soybeans-clean	177	416	Lettuce_romaine_5wk	96	1831
13	Wheat	61	144	Lettuce_romaine_6wk	45	871
14	Woods	379	886	Lettuce_romaine_7wk	53	1017
15	Buildings-Grass-Trees-Drivers	115	271	Vinyard_untrained	363	6905
16	Stone-steel-Towers	46	47	Vinyard_vertical_trellis	90	1717
Total		3106	7143		2697	51,432
The Dataset	KSC			University of Pavia ROSIS
	Class No.	Train	Test	Class No.	Train	Test
1	Scrub	229	532	Asphalt	331	6300
2	Willow swamp	73	170	Meadows	932	17,717
3	Cabbage palm ham	80	179	Gravel	104	1995
4	Cabbage palm/oak ham	76	176	Trees	153	2911
5	Slash pine	49	112	Painted metal sheets	67	1278
6	Oak/broadleaf ham	69	160	Bare Soil	251	4778
7	Hardwood swamp	32	73	Bitumen	66	1264
8	Graminoid marsh	130	301	Self-Blocking Bricks	184	3498
9	Spartina marsh	157	363	Shadows	47	900
10	Cattail marsh	122	282
11	Salt marsh	126	293
12	Mud flats	151	352
13	Water	279	648
Total		1573	3635		2135	40,641

.

• Indian Pines AVRIS were obtained employing the National Aeronautics and Space Administration’s Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) sensor and gathered over northwest Indiana’s Indian Pine test site in June 1992. As the high imbalance dataset, Indian Pines AVRIS consists of $145\times 145$ pixels and 220 bands covering the range from 0.4 to 2.5 $\mathsf{\mu }$m with a spatial resolution of 20 m. There are 16 different land-cover classes and 10,249 samples in the original ground truth. 30% of original reference data are chosen randomly to constitute training dataset and the remaining part constructs test dataset. For Indian Pines AVRIS, if the number of samples is less than 100, such as Oats, half of the samples are randomly chosen to construct training sets. The IR on the training set is 73.6.
• KSC was acquired by the Airborne Visible/Infrared Imaging Spectrometer instrument over the Kennedy Space Center (KSC), Florida, on 23 March 1996. The image consists of $512\times 614$ pixels with a spatial resolution of 18 m. After removing noisy bands, 176 spectral bands were used for the analysis. Approximately 5208 instances with 13 classes from the ground-truth map. Similar to the setup in the Indian Pines AVRIS image, 30% of pixels per class are randomly selected to constitute the training set, and the others are utilized to construct the test set. The IR on the training set is 8.71.
• Salinas was gathered by the AVIRIS sensor over Salinas Valley, California with 224 spectral bands. This image consists of $512\times 217$ pixels with a spatial resolution of 20 m. The original ground truth also has 16 classes mainly including vegetables, vineyard fields, and bare soils. Training sets are constructed by 8% of its samples chosen randomly from original reference data. The IR on the training set is 12.51.
• University of Pavia scenes covering the city of Pavia, Italy, was gathered by the reflective optics system imaging spectrometer sensor. The data sets consist of $610\times 340$ pixels covering the range from 0.43 to 0.86 $\mathsf{\mu }$m with a spatial resolution of 1.3 m. There are 16 classes and 42,776 instances in the original ground truth. The training dataset is constituted by 8% of samples that are chosen randomly from original data without replacement. The IR on the training set is 19.83.

4.2. Experiment Settings

In order to demonstrate the advantages of the proposed SMOTE-WDRoF, six popular methods, SVM, RF, RoF, SMOTE combined rotation forest (SMOTE-RoF), Convolutional neural network(CNN) [71], and RBDF is utilized in the comparative analysis. The settings of the six methods are introduced as follows. (1) In the SVM algorithm, the Gaussian function is employed. (2) For RF, the number of trees is twenty. (3) The RoF adopts the PCA transformation and includes 5 trees. The feature dimension of each sample subset is set to 10 and the number of trees. (4) For SMOTE-RoF, the parameter setting is the same as RoF. (5) The settings of CNN are based on reference [71]. (6) For RBDF, there are 20 features in each sample subset of RoF and each RF contains 20 trees. (7) In the proposed SMOTE-WDRoF, each RF also contains 20 trees and 20 features are included in each sample subset for RoF. In addition, for Indian Pines AVRIS and Kennedy Space Center (KSC), the $7\times 7$ neighborhood pixels are utilized to classify in RBDF and SMOTE-WDRoF. For Salinas and University of Pavia scenes, these two algorithms use $5\times 5$ neighborhood pixels for classification. All the programs are implemented using Python language. The results are generated on a PC equipped with an Intel(R) Core(TM) i5-10200H CPU with 2.4 GHz.

4.3. Assessment Metric

Because the Overall Accuracy (OA) can reflect the overall classification performance of the classifier, it is often adopted to evaluate traditional machine learning classification algorithms. However, when there is a serious imbalance between the data classes, the classification model may be strongly biased towards the majority classes, which results in poor recognition of the minority classes. Therefore, OA is not the most appropriate index to evaluate the model since it might result in inaccurate conclusions [72]. Consequently, this paper adopts five main metrics as performance measures, including the precision, average accuracy, Recall, F-measure, and Kappa.

• Precision: Precision is employed to measure the classification accuracy of each class in the imbalanced data. The $precisio{n}_i$ measures the prediction rate when testing only samples of class i

  $$precisio{n}_i=\frac{m_{ii}}{{\sum }_{i=1}^C{m}_{ji}}$$

  (10)

  where $m_{ii}$ and $m_{ji}$ stand for the true prediction of the ith class and the false prediction of the ith class into ith class, respectively.
• Average Accuracy (AA): As a performance metric, AA provides the same weight to each of the classes in the data, independently of the number of instances it has. It can be defined as

  $$\mathrm{AA}=\frac{{\sum }_{i=1}^{C}precisio{n}_i}{C}$$

  (11)
• Recall: True Positive Rate is defined as recall denoting the percentage of instances that are correctly classified. Recall is particularly suitable for evaluating classification algorithms that deal with multiple classes of imbalanced data [73]. It can be computed as the following equation:

  $$\mathrm{Recall}=\frac{1}{C}\sum _{i=1}^C\frac{m_{ii}}{{\sum }_{i=1}^C{m}_{ij}}$$

  (12)

  where $m_{ij}$ stand for the false prediction of the ith class into jth class.
• F-measure: F-measure, an evaluation index obtained by integrating precision and Recall, has been widely used in the imbalance data classification [55,74,75]. In the process of classification, precision is expected to be as high as possible, and it is also expected to Recall as large as possible. In fact, however, the two metrics are negatively correlated in some cases. The introduction of F-measure synthesizes the two, and the higher F-measure is, the better the performance of the classifier is. F-measure can be calculated as the following equation:

  $$\mathrm{F}-\mathrm{measure}=\frac{2}{C}\frac{{\sum }_{i=1}^{C}Recal{l}_i{\sum }_{i=1}^{C}precisio{n}_i}{{\sum }_{i=1}^{C}Recal{l}_i+{\sum }_{i=1}^{C}precisio{n}_i}$$

  (13)

  where $Recal{l}_i$ can be calculated by $\frac{n_{ii}}{{\sum }_{i=1}^C{n}_{ij}}$.
• Kappa: The metric that assesses the consistency of the predicted results is Kappa, which checks if the consistency is caused by chance. And the higher Kappa is, the better the performance of the classifier is Kappa can be defined as

  $$\mathrm{Kappa}=\frac{OA-{\sum }_{i=1}^C{p}_i{\widehat{p}}_i}{1-{\sum }_{i=1}^C{p}_i{\widehat{p}}_i}$$

  (14)

  where $p_i$ and ${\widehat{p}}_i$ stand for the actual sample size of class i and the predicted sample size of class i, respectively.

4.4. Performance Comparative Analysis

In the experiments, the results acquired according to precision, AA, Recall, F-measure, and Kappa are exhibited in

Table 2. Classification results (%) of the Indian pines AVRIS image, respectively obtained by SVM, RF, RoF, SMOTE-RoF, CNN, RBDF and the proposed SMOTE-WDRoF in the case of the imbalance ratio of 73.6.

IR: 73.6	SVM	RF	RoF	SMOTE-RoF	CNN	RBDF	SMOTE-WDRoF
1	$67.70\pm 6.36$	$85.88\pm 8.14$	$67.97\pm 9.16$	$66.86\pm 8.69$	$70.37\pm 16.15$	$83.46\pm 2.46$	$\mathbf{96.88}\pm \mathbf{2.28}$
2	$75.48\pm 1.55$	$73.78\pm 0.94$	$72.49\pm 0.82$	$71.84\pm 0.98$	$83.60\pm 11.93$	$83.39\pm 0.46$	$\mathbf{86.46}\pm \mathbf{0.61}$
3	$70.27\pm 1.06$	$77.12\pm 2.10$	$70.32\pm 0.97$	$72.07\pm 1.41$	$80.45\pm 6.33$	$\mathbf{85.69}\pm \mathbf{0.42}$	$82.97\pm 0.64$
4	$67.79\pm 3.93$	$60.64\pm 3.12$	$61.11\pm 2.20$	$62.58\pm 2.96$	$64.50\pm 11.29$	$78.65\pm 1.52$	$\mathbf{81.18}\pm \mathbf{0.49}$
5	$91.15\pm 1.58$	$92.97\pm 0.57$	$91.61\pm 1.20$	$87.89\pm 0.84$	$88.33\pm 6.12$	$\mathbf{95.72}\pm \mathbf{0.28}$	$94.30\pm 0.46$
6	$92.88\pm 0.97$	$88.05\pm 0.75$	$92.09\pm 0.63$	$91.52\pm 1.42$	$96.35\pm 3.53$	$95.16\pm 0.36$	$\mathbf{96.77}\pm \mathbf{0.08}$
7	$85.36\pm 5.16$	$93.00\pm 10.95$	$74.91\pm 10.83$	$\mathbf{93.71}\pm \mathbf{10.05}$	$81.25\pm 8.37$	$93.12\pm 1.77$	$93.29\pm 4.19$
8	$98.56\pm 0.90$	$95.71\pm 0.82$	$98.78\pm 0.36$	$98.25\pm 0.39$	$\mathbf{99.39}\pm \mathbf{0.66}$	$97.95\pm 0.27$	$98.67\pm 0.06$
9	$43.09\pm 6.20$	$73.57\pm 10.28$	$79.64\pm 16.58$	$57.75\pm 12.67$	$47.06\pm 6.56$	$81.89\pm 5.93$	$\mathbf{96.39}\pm \mathbf{1.33}$
10	$76.41\pm 0.90$	$75.85\pm 1.70$	$79.13\pm 0.54$	$78.09\pm 1.51$	$78.17\pm 7.79$	$\mathbf{84.89}\pm \mathbf{0.46}$	$84.20\pm 0.34$
11	$78.89\pm 0.43$	$76.07\pm 0.63$	$81.42\pm 0.69$	$83.03\pm 3.23$	$78.17\pm 4.26$	$84.28\pm 0.32$	$\mathbf{90.54}\pm \mathbf{0.22}$
12	$81.32\pm 1.94$	$73.21\pm 1.01$	$77.49\pm 2.78$	$77.39\pm 3.80$	$80.25\pm 7.90$	$84.54\pm 0.30$	$\mathbf{84.84}\pm \mathbf{0.40}$
13	$94.53\pm 3.06$	$91.93\pm 1.78$	$94.34\pm 2.84$	$87.68\pm 12.42$	$95.30\pm 2.29$	$95.26\pm 0.40$	$\mathbf{98.95}\pm \mathbf{0.15}$
14	$93.96\pm 1.37$	$91.16\pm 0.75$	$92.16\pm 0.70$	$92.55\pm 0.90$	$97.15\pm 2.69$	$95.99\pm 0.11$	$\mathbf{97.51}\pm \mathbf{0.13}$
15	$72.87\pm 2.73$	$76.75\pm 1.40$	$76.28\pm 2.93$	$75.57\pm 2.16$	$72.52\pm 12.45$	$\mathbf{86.23}\pm \mathbf{0.51}$	$82.35\pm 0.32$
16	$95.71\pm 3.26$	$99.00\pm 2.23$	$98.64\pm 1.23$	$97.36\pm 0.95$	$93.75\pm 1.81$	$96.84\pm 0.48$	$\mathbf{99.48}\pm \mathbf{0.56}$
AA (%)	$80.37\pm 0.54$	$82.80\pm 0.40$	$81.77\pm 1.26$	$80.88\pm 1.24$	$81.66\pm 0.91$	$88.94\pm 0.44$	$\mathbf{91.55}\pm \mathbf{0.44}$
Recall (%)	$81.71\pm 1.77$	$75.78\pm 1.62$	$81.42\pm 0.82$	$81.35\pm 2.20$	$89.77\pm 0.71$	$87.72\pm 0.20$	$\mathbf{91.67}\pm \mathbf{0.40}$
F-measure (%)	$80.58\pm 0.90$	$78.45\pm 1.31$	$81.26\pm 1.04$	$81.05\pm 1.40$	$85.46\pm 0.64$	$88.21\pm 0.29$	$\mathbf{91.51}\pm \mathbf{0.34}$
Kappa (%)	$79.26\pm 0.20$	$77.62\pm 0.16$	$79.33\pm 0.22$	$79.57\pm 0.30$	$84.88\pm 2.47$	$86.35\pm 0.18$	$\mathbf{88.64}\pm \mathbf{0.18}$

,

Table 3. Classification results (%) of the KSC image, respectively obtained by SVM, RF, RoF, SMOTE-RoF, CNN, RBDF and the proposed SMOTE-WDRoF in the case of the imbalance ratio of 8.71.

IR: 8.71	SVM	RF	RoF	SMOTE-RoF	CNN	RBDF	SMOTE-WDRoF
1	$74.23\pm 1.20$	$91.96\pm 0.62$	$89.89\pm 0.99$	$93.12\pm 1.26$	$85.57\pm 15.13$	$90.37\pm 1.25$	$\mathbf{97.25}\pm \mathbf{0.26}$
2	$70.96\pm 1.91$	$80.20\pm 1.99$	$90.17\pm 1.75$	$88.03\pm 1.59$	$80.46\pm 6.76$	$86.35\pm 3.42$	$\mathbf{93.20}\pm \mathbf{1.12}$
3	$60.86\pm 10.39$	$88.66\pm 0.73$	$89.21\pm 1.00$	$86.29\pm 2.24$	$73.30\pm 19.32$	$87.76\pm 4.19$	$\mathbf{91.63}\pm \mathbf{1.29}$
4	$35.80\pm 4.34$	$60.49\pm 0.50$	$62.58\pm 2.14$	$66.53\pm 1.98$	$61.40\pm 19.21$	$72.92\pm 2.42$	$\mathbf{80.45}\pm \mathbf{2.06}$
5	$64.98\pm 37.91$	$79.39\pm 3.99$	$71.26\pm 6.81$	$72.58\pm 4.36$	$79.45\pm 11.57$	$75.20\pm 4.16$	$\mathbf{82.36}\pm \mathbf{3.31}$
6	$55.29\pm 9.64$	$70.10\pm 4.24$	$66.44\pm 4.32$	$66.17\pm 2.92$	$65.22\pm 37.52$	$78.91\pm 3.19$	$\mathbf{79.50}\pm \mathbf{0.71}$
7	$0.00\pm 0.00$	$73.79\pm 1.87$	$80.70\pm 2.10$	$85.09\pm 3.89$	$75.00\pm 10.62$	$86.76\pm 3.04$	$\mathbf{90.19}\pm \mathbf{2.48}$
8	$65.71\pm 4.01$	$85.72\pm 1.43$	$86.74\pm 1.84$	$85.07\pm 1.42$	$79.73\pm 4.96$	$86.64\pm 3.24$	$\mathbf{90.70}\pm \mathbf{1.37}$
9	$71.79\pm 1.84$	$89.97\pm 0.61$	$91.75\pm 0.95$	$94.86\pm 0.50$	$82.08\pm 1.79$	$91.40\pm 0.36$	$\mathbf{94.88}\pm \mathbf{0.69}$
10	$\mathbf{99.12}\pm \mathbf{1.09}$	$96.41\pm 1.20$	$98.77\pm 0.31$	$98.28\pm 0.67$	$98.13\pm 5.91$	$97.27\pm 1.52$	$98.43\pm 0.35$
11	$95.15\pm 1.61$	$99.04\pm 0.28$	$99.12\pm 0.83$	$97.52\pm 0.78$	$98.25\pm 0.33$	$98.64\pm 0.57$	$\mathbf{99.84}\pm \mathbf{0.09}$
12	$76.74\pm 1.07$	$93.67\pm 1.01$	$\mathbf{96.91}\pm \mathbf{1.59}$	$96.19\pm 1.69$	$93.53\pm 7.68$	$95.00\pm 0.83$	$95.97\pm 0.73$
13	$98.86\pm 0.29$	$\mathbf{100.00}\pm \mathbf{0.00}$	$\mathbf{100.00}\pm \mathbf{0.00}$	$99.93\pm 0.13$	$99.08\pm 0.72$	$99.82\pm 0.09$	$99.97\pm 0.01$
AA (%)	$66.89\pm 3.55$	$85.34\pm 0.76$	$86.42\pm 0.85$	$86.90\pm 0.51$	$82.40\pm 3.27$	$88.24\pm 1.86$	$\mathbf{91.87}\pm \mathbf{0.17}$
Recall (%)	$63.69\pm 1.73$	$85.43\pm 2.66$	$85.31\pm 0.55$	$86.20\pm 0.38$	$80.43\pm 4.60$	$87.20\pm 1.84$	$\mathbf{92.40}\pm \mathbf{0.18}$
F-measure (%)	$62.58\pm 2.42$	$84.47\pm 0.46$	$85.68\pm 0.64$	$86.41\pm 0.33$	$80.47\pm 4.26$	$87.59\pm 1.87$	$\mathbf{92.13}\pm \mathbf{0.00}$
Kappa (%)	$74.71\pm 1.09$	$88.63\pm 0.31$	$89.50\pm 0.49$	$90.17\pm 0.55$	$85.37\pm 3.12$	$90.42\pm 1.39$	$\mathbf{93.78}\pm \mathbf{0.13}$

,

Table 4. Classification results (%) of the Salinas image, respectively obtained by SVM, RF, RoF, SMOTE-RoF, CNN, RBDF and the proposed SMOTE-WDRoF in the case of the imbalance ratio of 12.51.

IR: 12.51	SVM	RF	RoF	SMOTE-RoF	CNN	RBDF	SMOTE-WDRoF
1	$\mathbf{100.00}\pm \mathbf{0.00}$	$99.83\pm 0.04$	$99.92\pm 0.06$	$99.57\pm 0.16$	$96.22\pm 7.64$	$99.74\pm 0.14$	$99.81\pm 0.04$
2	$98.57\pm 0.39$	$99.65\pm 0.04$	$98.88\pm 0.38$	$\mathbf{99.66}\pm \mathbf{0.13}$	$97.78\pm 2.24$	$98.56\pm 0.19$	$99.08\pm 0.37$
3	$89.20\pm 1.96$	$94.62\pm 0.55$	$95.66\pm 0.37$	$95.51\pm 0.60$	$94.03\pm 2.79$	$95.49\pm 0.28$	$\mathbf{97.08}\pm \mathbf{0.42}$
4	$95.62\pm 0.92$	$98.19\pm 0.14$	$\mathbf{98.63}\pm \mathbf{0.43}$	$98.46\pm 0.18$	$98.15\pm 0.17$	$95.01\pm 1.18$	$98.52\pm 1.07$
5	$89.39\pm 3.05$	$97.63\pm 0.06$	$98.03\pm 0.54$	$97.10\pm 0.83$	$90.24\pm 6.20$	$98.67\pm 0.16$	$\mathbf{99.30}\pm \mathbf{0.09}$
6	$99.85\pm 0.17$	$\mathbf{99.94}\pm \mathbf{0.06}$	$99.91\pm 0.05$	$99.85\pm 0.12$	$99.76\pm 0.18$	$99.90\pm 0.04$	$99.38\pm 0.22$
7	$99.37\pm 0.44$	$99.40\pm 0.25$	$99.40\pm 0.32$	$\mathbf{99.44}\pm \mathbf{0.21}$	$97.97\pm 1.70$	$99.30\pm 0.08$	$99.17\pm 0.31$
8	$66.94\pm 0.75$	$75.58\pm 0.26$	$79.42\pm 0.68$	$80.15\pm 0.55$	$71.74\pm 4.84$	$79.22\pm 0.23$	$\mathbf{82.80}\pm \mathbf{0.19}$
9	$96.98\pm 0.92$	$97.27\pm 2.42$	$98.79\pm 0.35$	$98.68\pm 0.37$	$97.91\pm 0.83$	$99.09\pm 0.16$	$\mathbf{99.62}\pm \mathbf{0.31}$
10	$87.11\pm 0.78$	$93.02\pm 0.60$	$\mathbf{94.96}\pm \mathbf{1.32}$	$92.56\pm 0.87$	$89.27\pm 3.76$	$94.05\pm 0.47$	$91.17\pm 1.15$
11	$83.67\pm 1.99$	$93.50\pm 0.77$	$94.79\pm 1.32$	$94.07\pm 0.70$	$78.61\pm 7.71$	$94.94\pm 0.71$	$\mathbf{95.70}\pm \mathbf{0.27}$
12	$94.54\pm 0.59$	$95.46\pm 0.69$	$97.50\pm 0.91$	$98.40\pm 0.23$	$90.17\pm 9.00$	$96.68\pm 0.34$	$\mathbf{99.25}\pm \mathbf{0.24}$
13	$93.49\pm 0.30$	$96.05\pm 0.21$	$97.17\pm 1.69$	$94.66\pm 0.64$	$93.69\pm 2.31$	$96.91\pm 0.60$	$\mathbf{97.92}\pm \mathbf{0.35}$
14	$95.52\pm 1.26$	$92.55\pm 0.87$	$92.65\pm 1.25$	$96.21\pm 1.52$	$94.51\pm 2.21$	$97.64\pm 1.40$	$\mathbf{98.81}\pm \mathbf{0.16}$
15	$\mathbf{78.45}\pm \mathbf{2.04}$	$75.23\pm 0.41$	$78.42\pm 11.22$	$72.38\pm 1.28$	$78.40\pm 6.68$	$77.79\pm 0.87$	$74.90\pm 0.96$
16	$99.35\pm 0.37$	$97.76\pm 0.51$	$98.43\pm 0.44$	$\mathbf{99.67}\pm \mathbf{0.17}$	$96.62\pm 1.40$	$99.21\pm 0.14$	$98.95\pm 0.19$
AA (%)	$91.75\pm 0.24$	$94.11\pm 0.14$	$95.16\pm 0.26$	$94.77\pm 0.16$	$91.57\pm 0.66$	$95.14\pm 0.03$	$\mathbf{95.72}\pm \mathbf{0.05}$
Recall (%)	$90.18\pm 0.22$	$93.98\pm 0.27$	$94.75\pm 0.15$	$94.87\pm 0.17$	$91.61\pm 1.09$	$95.11\pm 0.05$	$\mathbf{96.05}\pm \mathbf{0.04}$
F-measure (%)	$90.22\pm 0.20$	$93.99\pm 0.09$	$94.03\pm 1.85$	$94.89\pm 0.16$	$91.09\pm 0.93$	$95.07\pm 0.02$	$\mathbf{95.73}\pm \mathbf{0.05}$
Kappa (%)	$84.90\pm 0.98$	$88.83\pm 0.09$	$89.86\pm 0.12$	$90.04\pm 0.62$	$85.40\pm 1.43$	$90.44\pm 0.05$	$\mathbf{91.01}\pm \mathbf{0.05}$

and

Table 5. Classification results (%) of the University of Pavia ROSIS image, respectively obtained by SVM, RF, RoF, SMOTE-RoF, CNN, RBDF and the proposed SMOTE-WDRoF in the case of the imbalance ratio of 19.83.

IR: 19.83	SVM	RF	RoF	SMOTE-RoF	CNN	RBDF	SMOTE-WDRoF
1	$76.73\pm 1.01$	$91.13\pm 0.40$	$88.96\pm 0.32$	$89.58\pm 0.31$	$\mathbf{96.12}\pm \mathbf{1.57}$	$90.51\pm 0.94$	$95.68\pm 0.16$
2	$84.25\pm 0.48$	$90.18\pm 0.21$	$91.87\pm 0.57$	$92.32\pm 0.28$	$93.09\pm 1.66$	$88.68\pm 0.35$	$\mathbf{96.09}\pm \mathbf{0.17}$
3	$\mathbf{81.64}\pm \mathbf{3.97}$	$69.90\pm 1.12$	$76.30\pm 8.67$	$73.55\pm 0.93$	$79.00\pm 7.55$	$72.97\pm 1.78$	$76.73\pm 0.64$
4	$91.77\pm 2.88$	$87.63\pm 0.78$	$90.06\pm 1.06$	$90.71\pm 0.84$	$73.99\pm 5.34$	$\mathbf{95.07}\pm \mathbf{0.84}$	$88.35\pm 0.43$
5	$99.04\pm 0.47$	$96.06\pm 0.45$	$99.22\pm 0.32$	$99.55\pm 0.26$	$98.78\pm 0.12$	$99.10\pm 0.10$	$\mathbf{99.71}\pm \mathbf{0.11}$
6	$\mathbf{92.96}\pm \mathbf{1.48}$	$76.64\pm 0.75$	$79.96\pm 0.76$	$78.66\pm 1.22$	$73.97\pm 9.20$	$88.50\pm 0.37$	$75.09\pm 0.95$
7	$0.87\pm 1.94$	$82.89\pm 1.28$	$85.45\pm 0.33$	$82.31\pm 1.32$	$74.49\pm 10.90$	$88.05\pm 0.81$	$\mathbf{88.50}\pm \mathbf{1.47}$
8	$71.08\pm 4.04$	$79.64\pm 0.65$	$82.47\pm 0.65$	$82.70\pm 0.40$	$80.93\pm 6.64$	$84.33\pm 1.15$	$\mathbf{88.60}\pm \mathbf{0.18}$
9	$99.97\pm 0.04$	$99.86\pm 0.04$	$\mathbf{100.00}\pm \mathbf{0.00}$	$\mathbf{99.89}\pm \mathbf{0.13}$	$99.91\pm 0.04$	$98.44\pm 1.19$	$86.63\pm 0.63$
AA (%)	$77.59\pm 0.79$	$85.99\pm 0.23$	$88.25\pm 0.20$	$87.70\pm 0.22$	$85.59\pm 1.46$	$\mathbf{89.52}\pm \mathbf{0.07}$	$88.38\pm 0.28$
Recall (%)	$70.80\pm 0.64$	$85.27\pm 0.37$	$85.89\pm 0.30$	$86.83\pm 0.23$	$88.69\pm 0.92$	$85.65\pm 0.31$	$\mathbf{91.28}\pm \mathbf{0.23}$
F-measure (%)	$71.42\pm 1.11$	$85.52\pm 0.25$	$86.88\pm 0.18$	$71.54\pm 35.11$	$86.52\pm 1.20$	$87.23\pm 0.18$	$\mathbf{89.44}\pm \mathbf{0.26}$
Kappa (%)	$76.61\pm 0.63$	$82.64\pm 0.19$	$84.72\pm 0.39$	$84.86\pm 0.36$	$82.56\pm 1.67$	$85.00\pm 0.17$	$\mathbf{87.32}\pm \mathbf{0.36}$

for SVM, RF, RoF, SMOTE-RoF, CNN, RBDF and the proposed SMOTE-WDRoF on the four imbalanced hyperspectral datasets. The best results in each hyperspectral dataset are highlighted in bold.

4.4.1. Experimental Results on Indian Pines AVRIS

The results of seven algorithms for Indian Pines AVRIS are listed in Table 2. The first 16 rows are the results of precision, AA, Recall, F-measure, and Kappa coefficients are shown in the last four rows. Among the seven methods, SMOTE-WDRoF achieves the best classification performance in most cases. Because it not only introduces spatial neighborhood pixels and synthesizes samples to increase the sample size and balance the dataset but also adjusts the sample weights adaptively. The proposed method obtains AA of 91.55%, Recall of 91.67%, F-measure of 91.51%, and Kappa of 88.64%, which are the best classification results among the seven methods. Compared with other methods, SMOTE-WDRoF enhances at least 2.61% in AA, 1.90% in Recall, 3.30% in F-measure, and 2.29% in Kappa. Moreover, the SMOTE-WDRoF algorithm obtains 10 of the highest class accuracies among 16 classes in all. Besides, for the class with the least number of training samples, namely Class 9, the accuracy of the proposed algorithm achieves 96.39%, which is at least 14.50% higher than other methods and 53.30% higher at most. The proposed algorithm is superior to other methods in both precisions of the minority classes and overall performance. Figure 3 shows the classification maps obtained by different classification methods for Indian Pines AVRIS. It shows that the proposed SMOTE-WDRoF acquires the best performance on Indian Pines AVRIS dataset.

4.4.2. Experimental Results on KSC

For KSC dataset, the statistical classification results are summarized in Table 3, and the classification results of different methods are shown in Figure 4. As can be observed in Table 3, SMOTE-WDRoF is superior to the other six comparison methods by generating balanced data sets and multi-level forests feature learning. For KSC data containing 13 classes, SMOTE-WDRoF obtained the highest classification accuracy of 10 classes, including multiple minority classes, such as Class 2, Class 4, and Class 7. Furthermore, among all the methods, SMOTE-WDRoF acquires the best statistical results in terms of the AA, Recall, F-measure, and Kappa, and the accuracy of the four metrics is improved by at least 3.63%, 5.20%, 4.54%, and 3.36% respectively. Although RF and RoF algorithms achieve 100.00% accuracy for Class 16, they are far less effective than SMOTE-WDRoF in terms of other performance, especially for the minority classes. In addition, although algorithm SMOTE-RoF has balanced the dataset by synthesizing new samples, its classification performance is worse than SMOTE-WDRoF. In addition, worth noting that SVM algorithm is the worst performer as it pays no attention to the recognition of the minority classes, and its classification accuracy for Class 7 is 0. Therefore, it is demonstrated that the proposed SMOTE-WDRoF has the best classification performance when processing the KSC dataset.

4.4.3. Experimental Results on Salinas

The classification results of seven different methods on the Salinas dataset are shown in Table 4. SMOTE-WDRoF is superior to the other six comparison methods and acquires AA of 95.92%, Recall of 96.05%, F-measure of 95.73%, and Kappa of 91.01%. In addition, SMOTE-WDRoF obtained the highest accuracy for half of the classes on Salinas dataset. For the two classes with the least number of training samples, namely Class 13 and Class 14, the precision of SMOTE-WDRoF reaches 97.92% and 98.81% respectively, which proves its ability to handle the minority classes better than the other comparison methods. Although SMOTE-RF has the highest accuracy in the two classes, the other classes performance of it is not superior. The corresponding classification maps on the data set are illustrated in Figure 5. The experimental results on this dataset testify that the SMOTE-WDRoF shows better classification performance than traditional methods when dealing with class imbalanced data.

4.4.4. Experimental Results on University of Pavia scenes

The results for the proposed SMOTE-WDRoF and six comparison methods on the University of Pavia ROSIS are exhibited in Table 5. Compared with the other methods, SMOTE-WDRoF improves the classification performance by creating new samples to construct the balanced dataset and automatically updating the sample weights based on the classification error information. The proposed SMOTE-WDRoF surpasses RBDF by 2.59%, 2.21%, and 2.32% in terms of Recall, F-measure, and Kappa. Although the AA of the RDBF algorithm is slightly higher than SMOTE-WDRoF, its F-measure that is the synthesis of recall and AA, is significantly lower than SMOTE-WDRoF. While dealing with the minority classes, such as Class 5 and Class 7, the SMOTE-WDRoF performs better than CNN, RDBF, and the other four traditional methods. For visual comparisons, Figure 6 shows the categorization maps of the classification results for all these methods. From Figure 6, we can observe that the proposed method exhibits the best result with least noise. It is obvious that SMOTE-WDRoF obtains the best effect on the University of Pavia ROSIS dataset.

4.4.5. Training Time of Different Deep Learning Methods

The training times of CNN and SMOTE-WDRoF are shown in

Table 6. Training times (seconds) of CNN and SMOTE-WDRoF for four hyperspectral image datasets.

Data	Indian Pines AVRIS	KSC	Salinas	University of Pavia Scenes
CNN	30,830	5958	11,430	21,030
SMOTE-WDRoF	3942	1389	1809	1752

. For the CNN, the model needs to continuously adjust the parameters through backpropagation to achieve good performance. Consequently, a large number of parameters need to be calculated in the time-consuming training process. Different from traditional deep learning methods which require backpropagation, the SMOTE-WDRoF needs much less training time. For Indian Pines AVRIS, the training time of CNN is 30,830 s, while the training time of the proposed algorithm is only 3942 s. For KSC, the training time of the proposed algorithm is only one-fourth of that of CNN. For Salinas and University of Pavia ROSIS, SMOTE-WDRoF spends one-sixth and one-twelfth as much time training as CNN, respectively.

4.5. Influence of Model Parameters on Classification Performance

4.5.1. Influence of Level

In order to study the influence of level on SMOTE-WDRoF, we present in Figure 7 the evolution of the AA and Recall on Indian Pines AVRIS, KSC, Salinas and University of Pavia ROSIS. Similar to the traditional depth model, the deep forest structure of SMOTE-WDRoF is of great significance to improve the classification performance. When the output of each level is used as the feature and stacked with the original feature as the input of the next level, the sample weights are adjusted accordingly. Consequently, the classification accuracy is enhanced with the growth of the level. As can be seen from Figure 7a, AA of four hyperspectral datasets is increased significantly when the level increased from 1 to 3. When the level is 4, the growth rate of AA slowed down gradually. And when the level number exceeds 5, the AA of the four datasets reaches a stable value. For Indian Pines AVRIS, KSC, Salinas and University of Pavia ROSIS, the stable values are 91.55%, 91.87%, 95.44% and 88.37% respectively. The evolution of the Recall on four hyperspectral datasets is shown in Figure 7b. It can be observed that at first recall is increased greatly. With the increase in levels, the recall turns to a relatively stable value. When the level is set as 5, the stable values are 91.67%, 92.40%, 96.05% and 91.28% respectively on Indian Pines AVRIS, KSC, Salinas and University of Pavia ROSIS. These results demonstrate that when there are too many levels in the proposed model, the output of the last several levels cannot afford helpful information for classification anymore. Therefore, statistically better performance can be achieved when L is equal to 5. And in other experiments, the level is set as 5.

4.5.2. Influence of the Window Size

Due to the spatial homogeneity of hyperspectral images, neighboring samples are likely to belong to the same class. Consequently, neighboring pixels are introduced to be the local spatial information by a sliding window in SMOTE-WDRoF. To study the influence of window size on classification accuracies, we vary this parameter from $1\times 1\times D$ to $7\times 7\times D$ for four hyperspectral datasets to introduce different numbers of spatial neighbor pixels. D represents the number of bands for hyperspectral data. For Indian Pines AVRIS, KSC, Salinas and University of Pavia ROSIS, D are 220, 176, 224 and 103 respectively. The results with different window sizes are shown in Figure 8. While the window size increases, the classification accuracy also presents an upward trend. More specifically, for Indian Pines AVRIS, AA, Recall, F-measure and Kappa increase from 87.69%, 71.27%, 74.81% and 85.05% to 91.71%, 91.12%, 91.29% and 88.41% respectively when the size of window is changed from $1\times 1\times 1\times 220$ to $7\times 7\times 220$. And the highest precision of Indian Pines AVRIS is obtained at $7\times 7\times 220$. For KSC, the three indexes, Recall, AA, and F-measure, reach the highest precision at $7\times 7\times 176$. Besides, the value of Kappa first rises and then falls, and it achieves the highest value at $5\times 5\times 176$. For Salinas, the high precision is obtained at $5\times 5\times 224$, and then the precision almost no longer increases with the expansion of the window size. In addition, SMOTE-WDRoF with a window size of $5\times 5\times 103$ delivers the best performance for University of Pavia ROSIS. This phenomenon is not surprising. More useful spatial information can be introduced by a relatively large window, which is beneficial to improve classification performance. However, if the window size is too large, samples that do not belong to the same class as the central pixel will be extracted, which will result in the decreased accuracy.

5. Conclusions

In this paper, the SMOTE-Based Weighted Deep Rotation Forest(SMOTE-WDRoF) algorithm is proposed for the imbalanced hyperspectral data classification. First of all, the local spatial structure of samples is extract to enrich the data information, and the balanced datasets are built by SMOTE. Second, RoF and multi-layer cascade RF form the WDRoF model which uses the output probability of each layer as a supplement feature of the next layer and updates the sample weights adaptively to improve classification performance. The proposed method is validated on four public hyperspectral image datasets. Compared with the traditional deep learning models, SMOTE-WDRoF consumes much less training time. Experimental results show that the proposed SMOTE-WDRoF is effective for dealing with multi-class imbalanced data and significantly outperforms SVM, RF, RoF, SMOTE-RoF, CNN, and RBDF and Besides, the parameter analysis has also been implemented and the results have demonstrated the advantages of our algorithm in terms of accuracy and robustness.