Hyperspectral Image Classification with Imbalanced Data Based on Semi-Supervised Learning

Abstract

Hyperspectral remote sensing image classification has been widely employed for numerous applications, such as environmental monitoring, agriculture, and mineralogy. During such classification, the number of training samples in each class often varies significantly. This imbalance in the dataset is often not identified because most classifiers are designed under a balanced dataset assumption, which can distort the minority classes or even treat them as noise. This may lead to biased and inaccurate classification results. This issue can be alleviated by applying preprocessing techniques that enable a uniform distribution of the imbalanced data for further classification. However, it is difficult to add new natural features to a training model by artificial combination of samples by using existing preprocessing techniques. For minority classes with sparse samples, the addition of sufficient natural features can effectively alleviate bias and improve the generalization. For such an imbalanced problem, semi-supervised learning is a creative solution that utilizes the rich natural features of unlabeled data, which can be collected at a low cost in the remote sensing classification. In this paper, we propose a novel semi-supervised learning-based preprocessing solution called NearPseudo. In NearPseudo, pseudo-labels are created by the initialization classifier and added to minority classes with the corresponding unlabeled samples. Simultaneously, to increase reliability and reduce the misclassification cost of pseudo-labels, we created a feedback mechanism based on a consistency check to effectively select the unlabeled data and its pseudo-labels. Experiments were conducted on a state-of-the-art representative hyperspectral dataset to verify the proposed method. The experimental results demonstrate that NearPseudo can achieve better classification accuracy than other common processing methods. Furthermore, it can be flexibly applied to most typical classifiers to improve their classification accuracy. With the intervention of NearPseudo, the accuracy of random forest, k-nearest neighbors, logistic regression, and classification and regression tree increased by 1.8%, 4.0%, 6.4%, and 3.7%, respectively. This study addresses research a gap to solve the imbalanced data-based limitations in hyperspectral image classification.

1. Introduction

Hyperspectral images provides hundreds of continuous narrow spectral bands, which facilitate the discrimination of different surface features [1,2,3,4]. The rich spectral information obtained from hyperspectral remote sensing images has been widely employed in numerous applications, such as environmental monitoring, agriculture, and mineralogy [1,2]. Hyperspectral image classification is essential in applications that focus on assigning class labels to each pixel [2]. The number of selected training samples in each class may be significantly different in some cases. Such a dataset is called an imbalanced dataset [2,5,6]. Conventionally, the rarest classes are named minority classes, and the remaining ones are called majority classes. In remote sensing classification, dataset imbalance is commonly observed during field sampling owing to the uneven distribution of objects with various sizes seen in nature [2,5,7].

Handling an imbalanced dataset frequently encounters degradation of the prediction accuracy or poor accuracy for the minority class, which are called imbalanced problems [5,6]. The lack of capacity of most machine-learning classifiers to handle imbalanced datasets further contributes to these problems. With the rapid development of computer technology, machine-learning classifiers, such as logistic regression (LR) [8] and random forest (RF) [9], have been widely utilized in hyperspectral image classification, and they have exhibited satisfactory classification performance and excellent generalization [10,11]. However, most classifiers are designed under the assumption that the sample sizes of each class are approximately equal. When using imbalanced data, prejudicial and biased classification results may be obtained [5]. Consequently, the minority classes may be distorted or treated as noise. Even if the prediction model achieves a relatively high overall accuracy, the minority classes may remain unknown [12,13,14]. Additionally, some common classifiers in hyperspectral images that use small spectral signatures are also insensitive to the imbalanced data problem [15,16]. Thus, the classification of imbalanced data remains a significant challenge [17].

Several studies have highlighted the imbalanced data problem and its research value [2,5,7]. Most approaches that mitigate imbalanced data problems can be categorized as: preprocessing techniques [18] or modified classifiers [19]. Preprocessing techniques usually alter an imbalanced dataset into balanced distributions before training classifiers, whereas modified classifiers are often associated with a specific algorithm. Preprocessing techniques can be applied to most algorithms and are more flexible. Among the preprocessing techniques, the resampling methods are widely used to compensate for problems arising due to imbalanced data [5]. Resampling methods balance the dataset by adding samples of the minority classes or deleting samples of majority classes [2,3]. Studies have shown that the classification performance on a balanced distribution is better than that of an imbalanced distribution [20].

Resampling methods are typically categorized as: oversampling, undersampling, and hybrid methods [5]. The most popular approach, oversampling, generates new minority class samples to balance the dataset [21]. This involves the random duplication of minority samples, termed as the synthetic minority oversampling technique (SMOTE) [22]. For each minority sample ${\mathit{x}}_1$, SMOTE calculates the distances from ${\mathit{x}}_1$ to each sample and selects the k-nearest neighbor set $K=\{{\mathit{x}}_1,{\mathit{x}}_2,\dots ,{\mathit{x}}_k\}$. Subsequently, artificial samples ${\widehat{\mathit{x}}}_a$ are generated by ${\widehat{\mathit{x}}}_a={\mathit{x}}_1+({\sum }_{j=1}^k{\mathit{x}}_j-{\mathit{x}}_1)\times \delta$, where $\delta \in [0,1]$ is a random number. Artificial samples are clearly generated by a linear combination of existing training samples, which may not add new features to the training process. Conversely, the undersampling method randomly eliminates the majority of samples to balance the data. The most common approaches are random undersampling (RUS) and NearMiss [23]. NearMiss selects the majority of samples with the smallest average distance from the minority classes to balance the training data [23]. The main limitation of these methods is the loss of information from the majority classes [24]. Thus, it is difficult to add new natural features to build classifiers by using existing preprocessing techniques. In general, abundance features are required to improve generalization in classification. Minority classes in an imbalanced dataset lack sufficient features for classification owing to sparse samples. This lack of features results in the failure of specific classifications in remote sensing.

For such an imbalanced problem, semi-supervised learning is a creative solution that utilizes the rich natural features in the unlabeled data. In the remote sensing classification, large amounts of unlabeled image data can be obtained through remote sensors at a low cost [10]. However, acquiring a large amount of labeled data is challenging because of the expensive and time-consuming manual labeling process [5]. Particularly in hyperspectral image classification, the categories are often matched to the species level, which increases the difficulty of collecting ground-truth labels. Massive datasets with few labeled samples are commonly used [5]. Therefore, semi-supervised learning, which leverages the rich natural features of unlabeled images to improve the performance of classifiers, has attracted significant attention [25,26]. Commonly used semi-supervised learning methods have made progress on hyperspectral image classification, including the self-learning [27,28], graph-based [29,30], and active learning [31,32,33] methods. Recently, several studies have focused on the application of generative adversarial networks [34,35,36,37,38] and graph convolutional networks [39,40,41] for semi-supervised hyperspectral image classification. In natural images classification, several semi-supervised learning methods aim to find the most representative class-specific samples from both labeled and unlabeled samples. For example, pseudo-label selects the maximum predicted probability class as the true label for unlabeled samples [42]. The $\mathrm{\Pi }$-model combines the labeled and unlabeled data by applying consistency regularization [43]. These methods improve the classification performance and prediction ability of the scene classification data. Furthermore, abundant natural information can be extracted from unlabeled samples. However, these semi-supervised methods assume that the labels cover all classes and that each class is relatively balanced. Indeed, only a few studies have attempted to address the imbalanced data problem by using semi-supervised learning [44,45,46]. The key challenge is the effective utilization of the natural features of unlabeled data to improve the classification performance of hyperspectral image classifiers when dealing with imbalanced data. Furthermore, understanding the extraction of effective labels from a minority class is crucial to improving the performance of the classifier.

To address this problem, we propose a novel semi-supervised learning-based solution called NearPseudo. In NearPseudo, an initialization classifier is built based on training data with label, and then creates the corresponding pseudo-labels for unlabeled data. In order to reduce the misclassification cost and improve the quality of the pseudo-labels, the optimal unlabeled samples are selected according to the nearest distance to minority sample and added to unbalanced datasets with the corresponding pseudo-labels. Thus, the unlabeled data can be used for model training, which ensures the automatic extraction of natural features by the classifiers. Simultaneously, we created a feedback mechanism based on a consistency check, which requires that the adding of pseudo-labels to unlabeled data should be consistent with the minority class. It further improves the quality of the pseudo-label. To verify the proposed method, multiple independent experiments were conducted on the publicly available Xiongan hyperspectral dataset. Unlike the smaller datasets commonly used in the hyperspectral image literature, abundant natural unlabeled samples are available in the Xiongan dataset. The Xiongan dataset is a state-of-the-art high-resolution hyperspectral dataset covering a large area. The corresponding labels for the images were accurate to the species levels. The three contributions of this study are summarized as follows.

• We propose a novel preprocessing solution called NearPseudo, to utilize the natural features of unlabeled data to improve the classification of imbalanced data; NearPseudo generates pseudo-labels for unlabeled samples and creates a feedback mechanism based on a consistency check to increase its reliability.
• Compared to other common processing methods in different imbalanced environments, NearPseudo performs better in hyperspectral image classification, especially in the case of an extremely imbalanced dataset. Simultaneously, NearPseudo can effectively improve the accuracy of most minority classes.
• We report the simplicity and universality of NearPseudo. After balancing by NearPseudo, the classification accuracy of most common classifiers, including RF, classification and regression tree (CART), LR, and k-nearest neighbors (kNN), was improved.

The remainder of this paper is organized as follows. In Section 2, NearPseudo and its pseudocode are described in detail. Section 3 describes the materials and experimental setup. Section 4 presents the experimental results for the Xiongan dataset. Finaly, Section 5 concludes the paper.

2. Proposed Method

We propose an oversampling approach, NearPseudo, in which the minority class is oversampled from unlabeled samples instead of creating samples from the training data. This approach is designed to use a large amount of unlabeled data, which contains rich, useful information. It may be essential to overcome the limitation of a few new features being added to the training process by only combining existing training samples. Our algorithm constantly iterates and oversamples until it obtains a balanced dataset, in which the number of selected training samples in each class is equal. During each loop, new samples from unlabeled data and their manufactured pseudo-labels are added to a single minority class.

The pseudocode for NearPseudo is shown in Algorithm 1. Suppose we have an imbalanced training dataset ${\mathit{T}}_r$ and an unlabeled dataset ${\mathit{T}}_u$. ${\mathit{T}}_r=\{({\mathit{x}}^{\left(1\right)},y^{\left(1\right)}),({\mathit{x}}^{\left(2\right)},y^{\left(2\right)}),\dots ,({\mathit{x}}^{\left(n\right)},y^{\left(n\right)})\}$, where $y^{\left(i\right)}\in \left\{1,2,3,\dots ,m\right\}$ is the label corresponding to ${\mathit{T}}_r$, m is the number of classes, n is the number of samples, and v is the number of features/bands. ${\mathit{T}}_u=\{{\mathit{x}}^{(n+1)},{\mathit{x}}^{(n+2)},\dots ,{\mathit{x}}^{(n+z)}\}$, where z is the number of unlabeled samples. During each loop, a subset ${\mathit{S}}_u\in {\mathit{R}}^{q\times v}$ is randomly extracted from the unlabeled samples ${\mathit{T}}_u$, where q is the number of samples in the random subset ${\mathit{S}}_u$. Simultaneously, a random sample ${\mathit{x}}^{\left(i\right)}$ is taken from ${\mathit{T}}_r$ with the label $y^{\left(i\right)}$. Then, the algorithm computes the distance between ${\mathit{x}}^{\left(i\right)}$ and each sample in ${\mathit{S}}_u$ as follows:

$$\begin{array}{c}\hfill D^{\left(e\right)}=\sum _{j=1}^v\left|{\mathit{x}}_j^{\left(i\right)}-{\mathit{x}}_j^{\left(e\right)}\right|,\end{array}$$

(1)

where $D^{\left(e\right)}$ is the distance between ${\mathit{x}}^{\left(i\right)}$ and ${\mathit{x}}^{\left(e\right)}$ in ${\mathit{S}}_u$. The algorithm then identified a group of k objects in ${\mathit{S}}_u$ that were closest to ${\mathit{x}}^{\left(i\right)}$. These objects also need to simultaneously undergo a consistency check. In this process, the training dataset ${\mathit{T}}_r$, which labels $\mathit{y}$, was utilized to model multi-classifier $G(\mathit{x};\mathbf{\theta })$, as shown in (2). Then, the consistency check model $F(y^{\left(i\right)},{\mathit{x}}^{\left(e\right)})$ was constructed by using (3). When $F(y^{\left(i\right)},{\mathit{x}}^{\left(e\right)})=0$, ${\mathit{x}}^{\left(e\right)}$ passes the conformance check, and vice versa,

$$\begin{array}{c}\hfill \mathit{y}=G(\mathit{x};\mathbf{\theta })\end{array}$$

(2)

$$\begin{array}{c}\hfill F(y^{\left(i\right)},{\mathit{x}}^{\left(e\right)})=y^{\left(i\right)}-G({\mathit{x}}^{\left(e\right)};\mathbf{\theta }).\end{array}$$

(3)

When ${\mathit{x}}^{\left(e\right)}$ belongs to k objects and also passes the consistency check, ${\mathit{x}}^{\left(e\right)}$ is added to the training dataset with the pseudo-label $y^{\left(e\right)}$. The algorithm was recirculated until the training sample was balanced.

Algorithm 1: Pseudocode for selecting samples and generating labels for NearPseudo.

Require:${\mathit{T}}_r$, imbalance training set with known labels, where v is the number of bands, n is the number of training samples. Require:${\mathit{T}}_u$, unlabeled set, where z is the size of the unlabeled set. Require:q, the number of samples in a random subset. Require:k, the number of the nearest neighbour samples we obtain for each iteration.   1:  $G(\mathit{x};\mathbf{\theta })\leftarrow {\mathit{T}}_r▹$ initialization classifier   2:  $\mathit{Repeat}:$   3:    ${\mathit{S}}_u\in {\mathit{R}}^{q\times v}\leftarrow {\mathit{T}}_u▹$ select a random subset   4:    $({\mathit{x}}^{\left(i\right)},y^{\left(i\right)})\leftarrow {\mathit{T}}_r▹$ select a random sample in minority classes   5:    compute distances between ${\mathit{x}}^{\left(i\right)}$ and each sample in ${\mathit{S}}_u$, search k closet samples.   6:    initialization: $j=1$   7:    $\mathit{While}j<k:$   8:        ${\mathit{x}}^{\left(e\right)}\leftarrow {\mathit{S}}_u▹$ select $j\mathrm{th}$ nearest neighbour   9:        $y^{\left(e\right)}\leftarrow G(\mathit{x};\mathbf{\theta })▹$ generate its pseudo-label 10:       $\mathbf{if}F(y^{\left(i\right)},{\mathit{x}}^{\left(e\right)})\equiv 0:▹$ check the consistency 11:           add $({\mathit{x}}^{\left(e\right)},y^{\left(e\right)})$ to ${\mathit{T}}_r▹$ formulate a new sample 12:          $j=j+1$ 13:  $\mathit{Until}:{\mathit{T}}_r$ is the balance training dataset 14:  $\mathit{Return}:{\mathit{T}}_r$

3. Materials and Experimental Setup

3.1. Datasets

In our study, the Xiongan dataset was used to evaluate and compare the performance of the proposed method with that of other existing methods [47]. Xiongan is located in Matiwan village of the Xiongan New Area in Hebei Province, China. The Xiongan dataset is a state-of-the-art hyperspectral remote sensing image dataset (Figure 1). The Xiongan New Area is the recently developed special economic zone established by the Chinese government in 2017. The area mainly consists of plains with a warm semi-humid continental monsoon climate [48].

The Xiongan dataset was obtained through a flight campaign experiment, which was completed by the Shanghai Institute of Technical Physics (SITP) and the Institute of Remote Sensing Applications (IRSA). SITP and IRSA are both affiliated with the Chinese Academy of Science (CAS). In this flight campaign experiment, data were collected by using an airborne multi-modular imaging spectrometer (AMMIS), which is the latest airborne hyperspectral imaging system developed by SITP [49,50,51]. The AMMIS possesses three modules: visible near-infrared (VNIR), shortwave infrared (SWIR), and long-wave infrared (LWIR). Each module was equipped with three spectrometers and integrated into a hyperspectral imaging system [10]. Based on this system, remote sensing images with high spectral and spatial resolutions can be acquired by using AMMIS.

The hyperspectral images in the Xiongan dataset were collected during the period from 15:40 to 16:03 on 3 October 2017, by using the VNIR module of AMMIS carried by an aircraft. The aircraft was operated at an altitude of 2000 m, with a coverage area of 1320 km${}^2$. After implementing high-precision geometric correction, band registration, radiometric calibration, and residual strip-noise removal [10,49], 1-level hyperspectral image data with 256 bands and a spectral resolution of 2.4 nm were obtained. As the hyperspectral images were acquired in clear, cloud-free weather conditions, they were less affected by clouds and the atmosphere.

Simultaneously, to manually acquire labels of the Xiongan dataset, IRSA conducted ground-truth investigations and accurately mapped the geometry of each site. In the Matiwan village of Xiongan, the research team investigated 57 sample areas and obtained 39 pictures. Finally, 12 categories were defined based on the ground investigations, as shown in

Table 1. Labeled samples of Xiongan.

NO.	Common Name/Latin Name	T1	T2	T3	TEST
1	Box elder/$Acernegundo$	1200	2000	2800	800
2	Willow/$Sali\mathit{x}$	2800	1600	2000	800
3	Paddy/$Oryzasativa$	200	2400	2600	800
4	Chinese scholar tree/$Sophorajaponica$	2400	2800	1800	800
5	Ash/$Fraxinuschinensis$	2800	1600	2600	800
6	Water	200	1200	2000	800
7	Post-harvest field	1200	2000	2200	800
8	Corn/$Zeamays$	1600	800	2400	800
9	Pear/$Pyrus$	2400	2800	1800	800
10	White poplar/$Populus$	800	1200	2400	800
11	Weeds	1600	2400	2800	800
12	Peach/$Amygdalus$	1800	800	1800	800
R (Max/Min)		14.0	3.5	1.6	1.0

and Figure 1. After geographic registration, the ground-truth labels and hyperspectral images constituted a complete Xiongan dataset with a pixel resolution of 3750 × 1580 and spatial resolution of 0.5 m.

3.2. Experimental Setup

To evaluate the performance of NearPseudo, three distinct experiments were performed as follows. (1) We performed multiple sampling to generate data subsets with variable degrees of imbalance for comparing the performance of NearPseudo with several other well-known preprocessing methods. The significance of differences between NearPseudo and others was evaluated by the paired t-test. (2) We utilized several types of common hyperspectral image classifiers to classify the unbalanced and balanced datasets and evaluated the performance of NearPseudo. (3) We modulated the different hyperparameters of NearPseudo and performed classification experiments to determine the mechanism for establishing the optimum values of hyperparameters in NearPseudo.

3.2.1. Sampling the Imbalanced Data Subset

In the case of hyperspectral image methods, the processing of imbalanced datasets with varying degrees of imbalance may vary. Thus, to compare the influence of hyperspectral image methods on the classification of datasets with different degrees of imbalance, we designed three training subsets: T1, T2, and T3 (Table 1). R (Max/Min), the ratio of maximum to minimum number of classes, is the difference between the three training subsets, which can be used to measure the degree of imbalance. T1 represents an extremely imbalanced training dataset with R (Max/Min) = 14.0, T2 represents a comparatively imbalanced training dataset with R (Max/Min) = 3.5, and T3 represents a slightly imbalanced training dataset with R (Max/Min) = 1.6 (Table 1). In addition, as shown in Table 1, when designing the three training subsets, we aimed to avoid the repetition of minority classes appearing in the same class. Thus, the results remain unaffected by the data quality of a certain class.

According to the designs in Table 1, we randomly performed multiple sampling to generate imbalanced training and testing datasets on a per-pixel basis without being returned to the dataset. The training and test datasets were two disjoint datasets. In addition, the number of testing data selected for each class was fixed at 2/7 of the number of classes with the maximum training data. As shown in Table 1, we sampled 800 entities from each class equally to construct the testing dataset and avoid the effects of imbalanced data during the testing process.

3.2.2. Processing Method and Classifiers

Following the preprocessing for balancing the data, the dataset was classified by using classifiers. In experiment (1), the classification without processing was compared with classification results from three representative imbalanced processing methods—NearMiss, RUS [23], and SMOTE [22]. These comparison methods are publicly accessible via the imbalance-learn package (https://imbalanced-learn.org/, accessed on 25 August 2017) and are commonly used to compare imbalance processing methods used for hyperspectral image classification [2,3]. Numerous algorithms require manual setting of the hyperparameters. Generally, the range recommended by the algorithm developers is reliable [52]. In our study, most hyperparameters were set to the default values, which were recommended based on experience. In particular, for SMOTE, the number of nearest neighbors was set to 5. For NearMiss, the number of nearest neighbors was 1. RF, as a multi-classifier, was used in NearPseudo. RF is a simple and useful classifier that exhibits excellent performance and better complexity in hyperparameter classification [11,53]. Thus, the RF classifier was used to classify the hyperspectral images for all methods after data was balanced in experiment (1).

Further analysis was conducted to compare the performance and improvement of classifiers after the preprocessing by NearPseudo in experiment (2). The representative classifiers for comparison included RF, CART [54], LR, and kNN [55]. These classifiers are commonly used for multispectral and hyperspectral image classification [10,53] and do not require a large number of labeled datasets to build the model. Similarly, for the above preprocessing methods, these classifiers are publicly accessible through the scikit-learn package (https://scikit-learn.org/, accessed on 1 October 2018) and can be quickly applied for hyperspectral image classification. Numerous papers have detailed the mechanism of these classifiers and have recommended ranges of hyperparameters suitable for remote sensing classification [53,56]. In this study, we adopted these recommended hyperparameter ranges. For RF, the number of trees was set to 180. For kNN, the number of neighbors was set to 11. The other parameters were set to default values. In addition, all experiments were programmed by using Python 3.6 and run on the Ubuntu 14.0 platform with an Intel Xeon e5-2620 CPU and four TITAN XP graphics cards.

3.3. Evaluation Criteria

To assess the accuracy of the hyperspectral image classification, three evaluation metrics are commonly used: (1) overall accuracy (OA), (2) F1-score, and (3) average F1-score (AF) [57]. OA is related to the ratio of correctly classified samples to the total samples, which provides an intuitive interpretation of the accuracy (Equation (5)). In contrast, to evaluate the per-class accuracy, the F1-score of each class was used in our study, which takes into account both recall and precision (Equation (4)) [57]. The F1-score helps to evaluate the accuracy of the majority and minority classes. Additionally, AF, a supplementary accuracy evaluation index, is also listed in our results.

$$\begin{array}{c}\hfill \mathrm{F}1=\frac{2\times \mathrm{P}\times \mathrm{R}}{\mathrm{P}+\mathrm{R}},\mathrm{P}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}},\mathrm{R}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}\end{array}$$

(4)

$$\begin{array}{c}\hfill \mathrm{OA}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{FP}+\mathrm{TN}+\mathrm{FN}}\end{array}$$

(5)

where FN is the false negative, FP is the false positve, TN is the true negative, and TP is the true positive.

4. Results

4.1. Comparison of Classification Results

In this experiment, we compared the accuracy of classification of four preprocessing methods, namely, NearMiss, RUS, SMOTE, and NearPseudo, by recording their respective accuracies before and after processing. We mainly discuss four specific issues: (1) the accuracy of classification of the aforementioned imbalanced preprocessing methods; (2) whether the differences between NearPseudo and others are statistically significant; (3) the enhancement in the accuracy of minority classification after preprocessing, and (4) in the case of training subsets with different degrees of imbalance, whether the different preprocessing methods remain consistent in terms of improving the accuracy. The results are presented in

Table 2. Classification accuracy (%) of different solutions on sub-training data T1.

CLA.	Unbalanced	NearMiss	RUS	SMOTE	NearPseudo
1	66.3 ± 0.3	20.6 ± 0.4	58.4 ± 0.5	66.7 ± 0.4	68.9 ± 0.3
2	68.0 ± 0.4	18.6 ± 0.4	57.2 ± 0.4	68.7 ± 0.3	71.4 ± 0.5
3	92.0 ± 0.2	50.5 ± 0.4	91.1 ± 0.2	92.8 ± 0.3	94.0 ± 0.2
4	59.2 ± 0.4	11.3 ± 0.7	44.2 ± 1.0	60.0 ± 0.6	62.7 ± 0.4
5	74.9 ± 0.5	21.6 ± 1.4	66.1 ± 0.4	75.0 ± 0.7	78.0 ± 0.2
6	89.3 ± 0.1	89.3 ± 0.1	92.7 ± 0.2	90.4 ± 0.2	92.4 ± 0.1
7	92.4 ± 0.1	28.6 ± 0.3	90.8 ± 0.3	92.1 ± 0.1	92.4 ± 0.1
8	74.3 ± 0.3	29.6 ± 0.6	64.2 ± 0.5	73.5 ± 0.6	73.4 ± 0.3
9	52.7 ± 0.3	9.3 ± 0.5	44.0 ± 0.5	53.2 ± 0.3	53.6 ± 0.5
10	54.8 ± 0.4	24.7 ± 0.1	53.9 ± 0.9	55.4 ± 0.4	68.0 ± 0.7
11	68.6 ± 0.1	20.9 ± 1.5	61.4 ± 0.6	68.9 ± 0.5	68.2 ± 0.2
12	74.8 ± 0.4	30.3 ± 0.2	63.2 ± 0.7	75.3 ± 0.3	75.2 ± 0.5
AF	72.3 ± 0.3	29.6 ± 0.6	65.6 ± 0.5	72.7 ± 0.4	74.8 ± 0.3
OA	72.3 ± 0.1	32.9 ± 0.2	65.8 ± 0.1	72.7 ± 0.2	74.8 ± 0.1

,

Table 3. Classification accuracy (%) of different solutions on sub-training data T2.

CLA.	Unbalanced	NearMiss	RUS	SMOTE	NearPseudo
1	69.8 ± 0.6	53.5 ± 0.3	66.1 ± 0.5	69.7 ± 0.5	70.1 ± 0.6
2	69.1 ± 0.5	55.4 ± 0.7	65.6 ± 0.4	67.7 ± 0.4	69.7 ± 0.7
3	94.6 ± 0.1	79.8 ± 0.8	92.9 ± 0.2	94.4 ± 0.2	95.3 ± 0.2
4	60.8 ± 0.5	43.0 ± 0.4	56.7 ± 0.8	60.7 ± 0.6	62.9 ± 0.5
5	73.2 ± 0.4	65.6 ± 0.4	72.7 ± 0.5	72.6 ± 0.7	75.3 ± 0.4
6	94.8 ± 0.2	89.5 ± 0.3	94.9 ± 0.1	94.7 ± 0.1	95.7 ± 0.2
7	92.9 ± 0.1	74.3 ± 0.8	93.1 ± 0.2	92.9 ± 0.2	93.7 ± 0.2
8	71.5 ± 0.3	49.9 ± 0.4	72.5 ± 0.4	71.2 ± 0.3	73.9 ± 0.3
9	53.7 ± 0.3	37.3 ± 0.6	49.7 ± 0.5	54.5 ± 0.5	53.6 ± 0.5
10	67.0 ± 0.4	58.0 ± 0.3	67.1 ± 0.6	65.7 ± 0.3	69.5 ± 0.5
11	70.3 ± 0.2	59.0 ± 0.5	68.7 ± 0.5	69.8 ± 0.4	70.0 ± 0.2
12	63.5 ± 0.4	65.5 ± 0.5	73.0 ± 0.3	64.3 ± 0.9	70.4 ± 0.7
AF	73.4 ± 0.3	60.9 ± 0.5	72.8 ± 0.4	73.2 ± 0.4	75.0 ± 0.4
OA	73.3 ± 0.1	60.9 ± 0.1	72.9 ± 0.2	73.1 ± 0.2	75.1 ± 0.1

and

Table 4. Classification accuracy (%) of different solutions on sub-training data T3.

CLA.	Unbalanced	NearMiss	RUS	SMOTE	NearPseudo
1	70.0 ± 0.3	63.2 ± 0.5	70.0 ± 0.4	69.8 ± 0.1	71.4 ± 0.4
2	72.3 ± 0.5	69.5 ± 0.4	72.0 ± 0.4	71.9 ± 0.4	72.2 ± 0.5
3	95.2 ± 0.2	92.0 ± 0.3	95.3 ± 0.2	95.3 ± 0.1	95.5 ± 0.3
4	60.1 ± 0.6	58.4 ± 0.4	61.6 ± 0.6	60.3 ± 0.3	62.0 ± 0.3
5	76.8 ± 0.4	72.7 ± 0.4	77.7 ± 0.5	76.7 ± 0.4	77.5 ± 0.3
6	95.7 ± 0.2	93.9 ± 0.2	95.7 ± 0.2	95.1 ± 0.1	95.7 ± 0.2
7	93.9 ± 0.1	90.5 ± 0.3	93.4 ± 0.1	93.8 ± 0.2	94.0 ± 0.2
8	77.2 ± 0.5	73.1 ± 0.6	76.5 ± 0.4	77.0 ± 0.3	77.0 ± 0.4
9	52.7 ± 1.1	50.5 ± 0.6	54.1 ± 0.5	52.6 ± 1.0	53.4 ± 0.7
10	71.9 ± 0.3	66.7 ± 0.4	71.6 ± 0.4	71.8 ± 0.5	71.8 ± 0.6
11	69.4 ± 0.2	64.1 ± 0.2	69.3 ± 0.3	69.1 ± 0.6	70.5 ± 0.3
12	74.5 ± 0.3	74.8 ± 0.2	75.8 ± 0.6	74.7 ± 0.6	74.8 ± 0.4
AF	75.8 ± 0.4	72.5 ± 0.4	76.1 ± 0.4	75.7 ± 0.4	76.3 ± 0.4
OA	76.1 ± 0.1	72.5 ± 0.1	76.2 ± 0.2	76.0 ± 0.2	76.5 ± 0.1

. The final results include the standard deviation values and average of the five experiments.

Table 2 lists the classification results of the different methods on training subset T1, which is an extremely imbalanced training subset with R (Max/Min) = 14.0 (Table 1). We can observe that the performance and accuracy of the classifier increased after preprocessing. The highest AF and OA were achieved by NearPseudo (AF = 74.8 ± 0.3% and OA = 74.8 ± 0.1%), followed by SMOTE (AF = 72.7 ± 0.4% and OA = 72.7 ± 0.2%), RUS (AF = 65.6 ± 0.5% and OA = 65.8 ± 0.1%), and NearMiss (AF = 29.6 ± 0.6% and OA = 32.9 ± 0.2%). Compared with the unbalanced data, NearPseudo improved both AF and OA by 2.5%, respectively, whereas SMOTE improved both AF and OA by 0.4%. The RUS and NearMiss method exhibited low classification performance. The significant result from the paired t-test shows that the AF of NearPseudo was significantly higher than that of NearMiss by 45.2% $(p\le 0.01)$, significantly higher than that of RUS by 9.2% $(p\le 0.01)$, and higher than that of SMOTE by 2.1% $(p\le 0.05)$. As shown in Table 2, after preprocessing by NearPseudo, the F1 scores of nine classes (75%) were improved by varying degrees. Only three classes (25%) obtained the highest F1 value when SMOTE was used to balance the training data, whereas the F1 values of NearMiss and RUS for each species decreased. Notably, the F1 of $Populus$ (NO. 10), which is a minority class with 800 training samples, increased by nearly 13.2% after preprocessing through NearPseudo. Moreover, NearPseudo exhibited outstanding performance on $Oryzasativa$ (NO. 3) and $Zeamays$ (NO. 6), which are minority classes and consist of only 200 training samples. The F1 scores of $Oryzasativa$ and $Zeamays$ improved by approximately 2.0% and 3.1%, respectively.

Table 3 presents the classification results of different balanced methods on the sub-training set T2, with R (Max/Min) = 3.5 (Table 1), which is a relatively imbalanced training subset. These results are similar to those in Table 2. It can be observed that the performance and accuracy of the classifier increased after preprocessing. The best classification performance was achieved after processing by NearPseudo (AF = 75.0 ± 0.4% and OA = 75.1 ± 0.1%), followed by SMOTE (AF = 73.2 ± 0.4% and OA = 73.1 ± 0.2%), RUS (AF = 72.8 ± 0.4% and OA = 72.9 ± 0.2%), and NearMiss (AF = 60.9 ± 0.5% and OA = 60.9 ± 0.1%). Compared to the imbalanced dataset, the classification accuracy was improved by 1.8% after being balanced by NearPseudo. The significant result from the paired t-test shows that the AF of NearPseudo was extremely significantly higher than that of NearMiss by 14.1% $(p\le 0.001)$, significantly higher than that of RUS by 2.2% $(p\le 0.01)$, and significantly higher than that of SMOTE by 1.8% $(p\le 0.01)$. In addition, the F1 score of the 10 classes was improved by 83.33%, and the best accuracy was achieved by NearPseudo. Similarly, the accuracy of these minority classes can be improved by using NearPseudo to various degrees. For example, it can be observed in Table 3 that the classifier exhibited outstanding performance on the $Amygdalu$ (NO. 12) and $Zeamays$ (NO. 8) classes. These two classes are minority classes and have the fewest samples (only 800 training samples). The F1 score of $Amygdalu$ improved by 6.9%, which is the greatest improvement compared with the other classes. Furthermore, the F1 score of $Zeamays$ improved by 2.4%.

Table 4 presents the classification results on sub-training set T3, with R (Max/Min) = 1.6 (Table 1). The results are similar to those in Table 2 and Table 3. They show that the best classification performance was achieved by NearPseudo (AF = 76.3 ± 0.4% and OA = 76.5 ± 0.1%), followed by RUS (AF = 76.1 ± 0.4% and OA = 76.2 ± 0.2%), SMOTE (AF = 75.7 ± 0.4% and OA = 76.0 ± 0.2%), and NearMiss (AF = 72.5 ± 0.4% and OA = 72.5 ± 0.1%). Compared to the imbalanced dataset, the classification accuracy improved by 0.4% after being balanced by NearPseudo. The significant result from the paired t-test shows that the AF of NearPseudo was extremely significantly higher than that of NearMiss by 3.8% $(p\le 0.001)$ and significantly higher than that of SMOTE by 0.6% $(p\le 0.01)$. The difference was not significant between the AF of NearPseudo and that of RUS $(p=0.27)$. The F1 score of 9 classes (75%) improved, and the best performance was exhibited by NearPseudo. It can be observed from Table 4 that the F1 score of three minority classes, $Sophorajaponica$ (NO. 4), $Pyrus$ (NO. 9), and $Amygdalus$ (NO. 12), increased by 1.9%, 0.7%, and 0.3%, respectively, thereby verifying the performance of NearPseudo.

Figure 2 depicts the discrepancy in the accuracy of the preprocessing methods in the case of training subsets with different degrees of imbalance. After balancing with NearPseudo, the accuracies of the extremely imbalanced sub-training set T1, relatively imbalanced sub-training set T2, and slightly imbalanced sub-training set T3 improved by 2.6%, 1.9%, and 0.4%, respectively. However, after balancing with SMOTE, the accuracies of T1, T2, and T3 improved by 0.4%, −0.2%, and −0.1%, respectively. With an increase in the degree of imbalance of the dataset, the classification accuracy of the two processing methods increased. This change was most prominent in the case of NearPseudo. The results indicate that the greater the imbalance in the sub-training data, the higher the improvement in the accuracy of the imbalanced data after being balanced by NearPseudo.

Table 2, Table 3 and Table 4 show that the accuracy of most minority classes can be improved by NearPseudo. In sub-training set T1, the average accuracy of minority classes with minimum sample size (NO. 3 and NO. 6) was improved by 2.6%. In sub-training set T2, the average accuracy of minority classes with minimum sample size (NO. 8 and NO. 12) was improved by 4.7%. In sub-training set T3, the average accuracy of minority classes with minimum sample size (NO. 4, NO. 9, and NO. 12) was improved by 1.0%.

4.2. Comparison of Different Multi-Classification Algorithms

NearPseudo is a preprocessing technique that can improve the capacity of most machine-learning classifiers to handle imbalanced data. As described in the previous subsections, RF was used as the main classifier in NearPseudo. In this experiment, we utilized several types of common classifiers in the hyperspectral images to classify the unbalanced and balanced dataset after preprocessing by NearPseudo. These classifiers included RF, CART, LR, and kNN. Based on the results of classification accuracy (

Table 5. Classification accuracy (%) of different solution.

CLA.	Unbalanced				NearPseudo
	CART	kNN	LR	RF	CART	kNN	LR	RF
1	45.0%	62.0%	52.5%	69.8%	48.7%	64.2%	54.3%	70.1%
2	45.4%	60.5%	63.4%	69.1%	49.2%	62.7%	64.9%	69.7%
3	90.9%	93.0%	95.2%	94.6%	89.9%	93.4%	95.4%	95.3%
4	39.3%	50.1%	49.9%	60.8%	41.0%	52.3%	48.2%	62.9%
5	54.2%	66.2%	67.7%	73.2%	59.7%	68.6%	71.0%	75.3%
6	90.2%	92.3%	93.0%	94.8%	91.0%	93.7%	93.2%	95.7%
7	88.6%	91.1%	91.0%	92.9%	89.9%	92.0%	92.5%	93.7%
8	47.0%	54.6%	45.7%	71.5%	58.9%	65.6%	65.7%	73.9%
9	39.9%	46.7%	43.9%	53.7%	37.6%	46.9%	50.4%	53.6%
10	44.4%	55.8%	68.2%	67.0%	52.0%	64.3%	68.5%	69.5%
11	55.5%	60.9%	56.6%	70.3%	56.8%	64.4%	59.4%	70.0%
12	46.4%	57.9%	8.0%	63.5%	57.5%	68.5%	66.6%	70.4%
AF	57.2%	65.9%	61.3%	73.4%	61.0%	69.7%	69.2%	75.0%
OA	57.0%	65.8%	63.1%	73.3%	60.7%	69.8%	69.5%	75.1%

), we quantitatively evaluated the improvement in the imbalanced ability of these classifiers after the processing of NearPseudo. For this experiment, we illustrate our results by using a detailed example of the imbalanced dataset T2.

As shown in Table 5, before preprocessing, the RF classifier exhibited the best classification accuracy (OA = 73.3%), followed by kNN (OA = 65.8%), LR (OA = 63.1%), and CART (OA = 57.0%). After processing by NearPseudo, the best classification accuracy was still achieved by the RF (OA = 75.1%), followed by kNN (OA = 69.8%), LR (OA = 69.5%), and CART (OA = 60.7%). As can be seen, after balancing the imbalanced training set T2 using NearPseudo, the classification performance of the four commonly used multi-classifiers improved. Compared with the unbalanced condition, the OA values of RF, kNN, LR, and CART increased by 1.8%, 4.0%, 6.4%, and 3.7%, respectively. The effect of NearPseudo on accuracy is related to the capacity of machine-learning algorithms to handle imbalanced datasets. The classification performance of most multi-classifiers with imbalanced data can be improved by using NearPseudo.

A significant improvement was observed in F1 scores of minority classes in each classifier before and after processing by NearPseudo. In imbalanced dataset T2, the $Amygdalu$ (NO. 12) and $Zeamays$ (NO. 8) classes were the minority classes and had the fewest samples (only 800 training samples) (Table 1). The F1 scores of these two classes using four common classifiers improved after processing by NearPseudo. In particular, for $Zeamays$ (NO. 8), the F1 scores of RF, kNN, LR and CART increased by 2.4%, 11.1%, 20%, and 11.9%, respectively. For $Amygdalu$ (NO. 12), the F1 scores of RF, kNN, LR, and CART increased by 6.9%, 11.1%, 20%, and 11.9%, respectively. This further illustrates the improvement in the accuracy of most minority classes and the reduction of bias of most models by NearPseudo.

4.3. Impact of Different Hyperparameters

NearPseudo has two hyperparameters: number of random subset samples q and the number of nearest neighbor samples obtained for each iteration k. The two hyperparameters were selected manually. In this experiment, for NearPseudo, we adopted different values of q = [5000, 10,000, 15,000, 20,000, 25,000, 30,000] and k = [2, 3, 4, 5, 6, 7, 8] (Figure 3). We set different hyperparameters of NearPseudo and performed classification experiments on training subsets T1, T2, and T3 (Table 1). This helped evaluate the impact of the two hyperparameters on the classification accuracy in NearPseudo and further reveal the mechanism underlying the changes in hyperparameters to guide their manual setting in NearPseudo.

All three imbalanced training subsets exhibited a decrease in the OA curve of the classifier with the hyperparameter k of NearPseudo (Figure 3a–c). Specifically, when the parameter k was varied from 2 to 8, the classification OA based on T1 decreased by 1.1% from 74.8% to 73.7% (Figure 3a). Under the same conditions, the classification OA based on T2 decreased by 0.8% from 75.1% to 74.2% (Figure 3b), whereas the classification OA based on T3 decreased by 0.6% from 76.5% to 75.9% (Figure 3c). Thus, it can be deduced that the improvement in the accuracy of the classifier by NearPseudo is affected by the hyperparameter k. The larger the hyperparameter k, the worse the accuracy. The intuitive explanation of this is included in Section 5.2.

In contrast to hyperparameter k, all results of using three imbalanced training subsets showed that the OA curve of the classifier increased with the hyperparameter q (Figure 3d–f). Specifically, when the parameter q was varied from 5000 to 30,000, the classification OA based on T1 increased by 0.7% from 73.4% to 74.1% (Figure 3d). Under the same conditions, the classification OA based on T2 increased by 0.7% from 73.8% to 74.5% (Figure 3e), whereas the classification OA based on T3 increased by 0.4% from 75.8% to 76.2% (Figure 3f). Therefore, improvement in the accuracy of the classifier by NearPseudo is affected by the hyperparameter q; the larger the hyperparameter q, the better the accuracy. The intuitive explanation of this is included in Section 5.2.

5. Discussion

5.1. Effect of NearPseudo on Different Multi-Classification Algorithms

As shown in Table 5, the impacts of NearPseudo on different multi-classification algorithms vary. To reveal the mechanism, Pearson correlation analysis was used to analyze the relationship between two variables: (1) mean change in overall accuracy from baseline unbalanced dataset to balanced dataset after processing in the sub-training set T2 and (2) variance of F1-score for each class before processing. Notably, lower values for variance correlated with better capacity of most machine-learning algorithms to handle imbalanced datasets, and vice versa. For example, the F1-score of minority class (NO. 12) by using classifier LR was 8.0%, which is high variance (variance = 24.7%), whereas the F1 score of minority class (NO. 12) by using classifier RF was 63.2%, which is low variance (variance = 13.64%) (Figure 4 and Table 5). Compared with RF, LR obtained prejudicial and biased classification results in minority, which increased the difference in F1 scores between each class. Consequently, the variance of the F1 score for each class in the LR classifier was larger.

As shown in Figure 4, the values of mean change increased as the variance of F1 score for each class without balance. That is, the effect of NearPseudo on accuracy is related to the capacity of most machine-learning algorithms to handle imbalanced datasets. The more negative the impact of the imbalance data on performance of algorithms, the more likely it is that NearPseudo improves the classification accuracy. Conversely, when the algorithm is less affected by imbalanced datasets, there is no additional balanced method required. For the LR classifier, the goal is to find the parameters with optimal values when the minimum value of loss function is obtained. Each class in LR requires sufficient samples to fit the fixed number of parameters. If lacking samples in a minority class, the classifier will perform poorly in this class.

5.2. Effect of Hyperparameters on Performance

In this study, we generated corresponding pseudo-labels for unlabeled data, which allowed us to include unlabeled data in the training process. However, evaluating the accuracy or quality of pseudo-labels is challenging. Existing models to build pseudo-labels have a certain probability of generating false labels. If the samples have false labels or are located in the edge areas to supplement the imbalanced dataset, the classifiers can easily adopt a bias. Therefore, we introduced the nearest neighbor method and a consistency check in NearPseudo to evaluate the quality of the pseudo-labels and the remote sensing data, consequently improving the quality of the selected samples. Notably, the values of hyperparameters k and q in NearPseudo could influence the credibility and accuracy judgments of those pseudo-labels.

Hyperparameter q refers to the search range of the supplemental sample. The larger the hyperparameter q, the greater the search range for NearPseudo to screen suitable samples. Therefore, the accuracy of NearPseudo increases with q. When the nearest neighbor method and consistency check are introduced, the selected samples can be filtered according to the quality of the sample. Hyperparameter k controls the filtering conditions. Smaller k values indicate that the selection conditions are harsher. The selected pseudo-labels have a higher quality under these conditions. Therefore, the accuracy of NearPseudo increases with the value of hyperparameter k decreases.

Notably, the pursuit of maximized q and minimized k may help to improve the predictive accuracy. However, it is not always feasible. The run time may increase significantly when parameter q is extremely large or parameter k is extremely small. The full parameter time complexity can be utilized to reveal the relationship between various hyperparameters and run time [53]. The optimal full parameter time complexity of the balanced process in NearPseudo is $O(n^{\prime }q(v+lo{g}_{2}k)/k+n^{\prime }M)$, where n is the number of training data points and $n^{\prime }$ is the number of supplemented samples, v is the number of bands/features, and M is the time for initialization classifier to predict a sample. It can be seen that the run time of NearPseudo increases with q. Simultaneously, the run time of NearPseudo increases as k decreases. Thus, these discrepancies should be noted when selecting the optimal hyperparameters.

5.3. Limitations

There were a few limitations to this study. The excellent performance of NearPseudo can be achieved under certain conditions. A large amount of unlabeled data is required for efficient processing in NearPseudo. With the availability of a significant amount of unlabeled data, suitable samples can be readily selected to complement the imbalanced data. In this case, q can be set to a large value, and the performance of NearPseudo will improve accordingly. Contrarily, a lack of sufficient unlabeled samples for selection will limit the performance of our proposed method.

Generally, in most flight campaign experiments, a large number of unlabeled remote sensing images are relatively easy to access. From the flight campaign experiment in the Xiongan New Area, the coverage area by the aircraft was 1320 km${}^2$, and a large number of remote sensing images with high spectral and spatial resolution could be acquired. However, the coverage area of ground investigations accounts for a very small fraction of airborne remote sensing images. Thus, most airborne remote sensing data are not equipped with the corresponding ground-truth labels. This is because ground-true labels acquisition consumes substantial amounts of resources and effort. Specifically, in the classification of hyperspectral remote sensing images, the categories are often matched to the species level, which increases the difficulty of collecting ground-truth labels.

In this study, the Xiongan dataset was chosen as a sample dataset to evaluate the performance of NearPseudo. This dataset was chosen because it could provide a large number of samples as unlabeled data to validate our proposed method. Compared to other publicly available benchmark datasets (e.g., Pavia data, Purdue Indian Pines, etc.), the Xiongan dataset possesses a high resolution and larger sample size. However, there are still some limitations with this dataset. For example, the same species are clustered together, and there is not much crossover between different species. Therefore, we will aim to collect more data from different areas and sources and improve the proposed method in subsequent analyses.

6. Conclusions

The most common classifiers have a weak ability to deal with imbalanced data problems. They often provide prejudicial and biased classification results when an imbalanced dataset is used. In this paper, a novel method based on semi-supervised learning called NearPseudo is proposed to solve this problem. This solution makes full use of abundant information and features of unlabeled remote sensing images, balances imbalanced data, and improves the classification accuracy at a low cost. Experimental evaluation using a hyperspectral dataset from Xiongan proved the effectiveness of this method. The experiments were designed to analyze the following issues: (1) performance comparison of NearPseudo with other popular preprocessing methods in different imbalanced environments, (2) the effect of NearPseudo on the classification accuracy of these common classifiers, and (3) the mechanism of change in the hyperparameters of NearPseudo. The experimental results demonstrate the simplicity and universality of NearPseudo. The major conclusions of this study are as follows:

• Compared with other methods, NearPseudo exhibits enhanced efficiency and superiority in balancing data with semi-supervised learning. In addition, NearPseudo is more suitable for extremely imbalanced datasets. After using NearPseudo, the RF accuracies of the sub-training set T1, T2, and T3 were improved by 2.6%, 1.9%, and 0.4%, respectively.
• The classification performance of most minority classes can be improved by using NearPseudo. After using NearPseudo, the average accuracy of minority classes with fewest samples in T1, T2, and T3 were improved by 2.6%, 4.7%, and 1.0%, respectively.
• The classification performance of most multi-classifiers with imbalanced data can be improved by using NearPseudo. After using NearPseudo, the accuracy of RF, kNN, LR, and CART increased by 1.8%, 4.0%, 6.4% and 3.7% in sub-training sets T2, respectively.
• The performance of NearPseudo increases with hyperparameter q and decreases with hyperparameter k.

In future research, we intend to further modify NearPseudo by improving its search efficiency and introducing ensemble learning to improve the classification performance.