Instance-Level Scaling and Dynamic Margin-Alignment Knowledge Distillation for Remote Sensing Image Scene Classification ^†

Abstract

Remote sensing image (RSI) scene classification aims to identify semantic categories in RSI using neural networks. However, high-performance deep neural networks typically demand substantial storage and computational resources, making practical deployment challenging. Knowledge distillation has emerged as an effective technique for developing compact models that maintain high classification accuracy in RSI tasks. Existing knowledge distillation methods often overlook the high inter-class similarity in RSI scenes, leading to low-confidence soft labels from the teacher model, which can mislead the student model. Conversely, overly confident soft labels may discard valuable non-target information. Additionally, the significant intra-class variability in RSI contributes to instability in the model’s decision boundaries. To address these challenges, we propose an efficient method called instance-level scaling and dynamic margin-alignment knowledge distillation (ISDM) for RSI scene classification. To balance the target and non-target class influence, we apply an entropy regularization loss to scale the teacher model’s target class at the instance level. Moreover, we introduce dynamic margin alignment between the student and teacher models to improve the student’s discriminative capability. By optimizing soft labels and enhancing the student’s ability to distinguish between classes, our method reduces the effects of inter-class similarity and intra-class variability. Experimental results on three public RSI scene classification datasets (AID, UCMerced, and NWPU-RESISC) demonstrate that our method achieves state-of-the-art performance across all teacher–student pairs with lower computational costs. Additionally, we validate the generalization of our approach on general datasets, including CIFAR-100 and ImageNet-1k.

1. Introduction

Rapid advancements in remote sensing technology have spurred the development of diverse algorithms designed to efficiently manage extensive Earth observation data. Scene classification, a core component of remote sensing image analysis, is crucial for accurately interpreting land use changes [1,2,3,4], optimizing agricultural practices [5,6,7,8], managing forest resources [9,10], and monitoring hydrological dynamics [11].

Traditional machine learning methods like SIFT [12,13], HOG [14], and LBP [15] have been employed to determine land cover types. While effective, these methods often require expert knowledge and manual feature extraction, making them costly and inefficient. In contrast, recent advances in deep neural networks have significantly enhanced the performance of tasks such as image classification [16,17,18], object recognition and tracking [19,20,21], and person re-identification [22,23,24]. Specifically, deep learning has revolutionized remote sensing image (RSI) scene classification, yielding substantial performance improvements [25,26,27,28,29,30].

Despite their effectiveness, high-performance models often entail significant computational and storage demands, posing substantial challenges for deployment in real-world applications. To mitigate these issues, knowledge distillation has been adopted. This method facilitates the transfer of knowledge from a large, complex model (the teacher) to a smaller, more efficient model (the student). Through this technique, the student model can reach performance levels comparable to that of the teacher by minimizing the Kullback–Leibler divergence.

However, RSI scene classification is marked by high inter-class similarity and large intra-class variability, presenting significant challenges to existing knowledge distillation methods. In particular, samples with high inter-class similarity often yield soft labels with diminished confidence in the target class, potentially leading to misguidance of the student model.

Conversely, samples with low inter-class similarity may lead to overly sharp soft target distributions, which suppress the effectiveness of logit knowledge distillation [31,32] (the reason overly sharp soft target distributions can diminish the potential benefits of knowledge distillation is explained in Appendix A). Similar to one-hot labels, these soft labels exhibit low confidence for non-target classes and lack rich category knowledge. This underscores the importance of non-target logits, which may contain valuable “dark knowledge” essential for effective model training, as illustrated by the blue dashed box in Figure 1.

Additionally, as shown in the green dashed box within Figure 1, large intra-class variability can compel the student model to develop poor decision boundaries to accommodate diverse within-class samples, adversely affecting the model’s performance during testing.

To address these challenges, we propose a straightforward and efficient logits-based distillation technique termed instance-level scaling and dynamic margin-alignment knowledge distillation (ISDM) tailored for RSI scene classification. ISDM scales the target class of the teacher’s soft label at instance level via an entropy regularization loss. As shown in the blue solid box within Figure 1, for hard samples with high inter-class similarity, the instance-level scaling can improve the teacher model’s confidence in the target class to prevent inadvertent misguidance of the student. For easy samples with low inter-class similarity, the instance-level scaling reduces the excessive confidence of the target class to amplify the difference of the non-target class and preserve the difference of the target class simultaneously. Moreover, a dynamic margin is adopted for alignment to accommodate the large intra-class variability, as shown in the green solid box within Figure 1. It can dynamically yield more rational and stable decision boundaries based on sample differences.

It should be noted that this article is an expanded version of the paper “Instance-level Scaling and Dynamic Margin-alignment Knowledge Distillation” [33] accepted for presentation at the 7th Chinese Conference on Pattern Recognition and Computer Vision (PRCV 2024). It analyzes the impact of RSI data on existing knowledge distillation methods, introduces a framework for RSI scene classification, and presents additional datasets and experimental validation to demonstrate the effectiveness of the proposed ISDM approach.

In summary, our contributions are as follows:

• We scale the target class of the teacher’s soft label at instance level via an entropy regularization loss to address the high inter-class similarity inherent in RSI.
• We introduce dynamic margin alignment for the probabilistic prediction scores, allowing the student model to establish more logical and adaptive decision boundaries, effectively addressing large intra-class variability.
• We propose an effective logits-based distillation method named ISDM, which achieves state-of-the-art performances across all datasets with minimal additional computational costs.

2. Related Work

2.1. Remote Sensing Image Scene Classification

The primary goal of scene classification is to automatically determine land cover types within remote sensing image patches. Scene classification using supervised learning algorithms can be divided into three categories: low-level, mid-level, and high-level methods. Low-level methods focus on extracting handcrafted features, such as SIFT (scale-invariant feature transform) [12,13], HOG (histogram of oriented gradients) [14], and LBP (local binary patterns) [15], and use classifiers like support vector machines (SVM) [34] or K-nearest neighbors (KNN) [35] for scene classification. Although these methods work well for specific structures and arrangements, they often fall short when dealing with the varied and intricate spatial distributions present in images. Mid-level methods create scene representations by encoding low-level local features. The Bag-of-Visual-Words (BoVW) [36] model is frequently utilized and is often augmented with different low-level descriptors [37], along with Gaussian mixture models (GMMs) [38] and pyramid-based approaches [39,40,41]. Moreover, topic models are used to integrate higher-order spatial relationships between local visual words [42,43,44]. High-level methods leverage deep learning models, which have set new benchmarks in image recognition, speech recognition, semantic segmentation, and remote sensing scene classification. Prominent deep learning models, such as VGG [45], ResNet [16], and WRN [46], have demonstrated superior performance in remote sensing scene classification, surpassing both shallow models and low-level techniques by extracting deep visual features from large-scale training datasets.

However, deep neural networks require substantial computational resources and storage due to their large number of parameters. This makes them impractical for resource-constrained environments like embedded systems or real-time processing. To address this, knowledge distillation techniques can be used to compress models, balancing efficiency with performance.

2.2. Knowledge Distillation

As an effective model compression method, knowledge distillation is firstly proposed by [31], which aims to transfer the knowledge of the teacher model to the student model. Based on the type of transferred knowledge, knowledge distillation is predominantly divided into two categories: (1) logits-based distillation methods and (2) feature-based distillation methods.

The logits-based methods merely employ the teacher model’s logits to transfer the knowledge. The origin knowledge distillation [31] transfers the knowledge by minimizing the Kullback–Leibler (KL) divergence between the probabilistic prediction scores of the teacher and student, which is adopted by many subsequent works due to its simplicity and efficiency. To use the knowledge better, much research attention is drawn to transfer the knowledge from the deep intermediate layers [47,48,49,50]. Thus, the feature-based methods are normally well performing. However, they are always infeasible when confronted with many practical problems. In some practical applications, the intrinsic architecture and the intermediate features of the teacher model may not be available due to safety and privacy issues. Moreover, feature-based methods are computationally expensive.

Research on applying knowledge distillation to RSI scene classification is still limited. A key challenge is that existing KD methods struggle with the significant intra-class variability and inter-class similarity present in RSI data. Traditional distillation techniques often generate low-quality soft labels because they do not effectively manage these variations and similarities, which adversely affects the performance of the student model. Additionally, the limited capacity of student models can exacerbate this issue, resulting in overly complex decision boundaries and diminished accuracy. To address these challenges, we propose an effective approach that combines improved soft label techniques with dynamic-alignment distillation methods. This approach aims to enhance both the quality of the soft labels and the decision boundaries in RSI scene classification.

2.3. Optimization of Soft Labels

Soft labels are the predicted probability scores obtained by applying softmax to logits at temperature $\tau$, where the target class reflects the confidence in the correct category and the non-target classes reflect the confidence in other categories. In logit-based methods, student models rely heavily on soft labels for training, making their quality crucial. Enhancing soft label quality has, thus, become a key area of research.

When soft labels are overly confident in the target class, they degenerate into one-hot labels, preventing full utilization of the knowledge in non-target classes. Many studies have addressed this issue [51,52,53,54,55,56,57]. For smoother soft labels, ATS [51] and NTCE-KD [52] reduce the target class values, while SFKD [53] uses attention mechanisms for smoothing. Label smoothing research [54,55,56,57] shows that it provides benefits similar to knowledge distillation (KD) in optimizing soft labels. For instance, [57] applied label smoothing to achieve softer labels, improving student model performance. On the other hand, insufficient confidence in the target class also hinders student training [58], as it misleads the student model into incorrect classifications. CKD [58] addresses this by replacing incorrect soft labels with hard labels, avoiding the transfer of erroneous knowledge.

However, existing studies have primarily focused on either excessive or insufficient confidence in soft labels, often relying on manual optimizations. To address this, we propose an IS module that comprehensively optimizes soft labels at instance level, ensuring that they capture both the correct knowledge from the target class and rich information from the non-target classes.

3. Materials and Methods

3.1. Preliminaries

In an image classification task with C classes, the logits can be denoted as $\mathbf{Z}=[z_1,z_2,\dots ,z_i,\dots ,z_t,\dots ,z_C]\in {\mathbb{R}}^{1\times C}$, where $z_i$ represents the logit of the i-th class and $z_t$ represents the target class. The probabilistic prediction scores $\mathbf{P}$ are defined by applying the softmax function to the logits as follows:

$$\begin{array}{c}\hfill p_i={\displaystyle \frac{exp(z_i/\tau )}{{\sum }_{j=1}^C(exp(z_j/\tau ))}},\end{array}$$

(1)

where $p_i$ is the i-th probabilistic prediction score and $\tau$ is the temperature hyperparameter to scale the soft labels. In original knowledge distillation, the student model is forced to mimic the teacher model’s behavior by minimizing the KL divergence between the probabilistic prediction scores of the teacher model and the student model:

$$\begin{array}{cc}\hfill L_{KD}=& \hfill KL({\mathbf{P}}^{tea}\left|\right|{\mathbf{P}}^{stu})=\sum _{i=1}^C{p}_i^{tea}log(p_i^{tea}/p_i^{stu}),\hfill \end{array}$$

(2)

where $L_{KD}$ is the knowledge distillation loss and $p_i^{tea}$ and $p_i^{stu}$ are the i-th class of the probabilistic prediction scores of the teacher model and the student model. And the ${\mathbf{P}}^{tea}$ are known as soft labels.

3.2. Instance-Level Scaling (IS)

In practical scenarios, traditional knowledge distillation methods often produce low-quality soft labels due to inter-class similarity and intra-class variability in remote sensing imagery. Furthermore, due to their limited capacity, student models may struggle to effectively manage intra-class variability, leading to complex decision boundaries and, consequently, reduced accuracy of the student network.

Therefore, we propose a novel instance-level scaling (IS) method to tailor an improved soft label for each instance, as shown in Figure 2.

Specifically, the IS module, a single-layer perceptron, generates a new target class of logits ${\tilde{z}}_t$ as follows:

$$\begin{array}{cc}\hfill \widetilde{z_t}=& \hfill F_{IS}({\theta }_{IS},\mathbf{Z}).\hfill \end{array}$$

(3)

Let optimized logits $\tilde{\mathbf{Z}}=[z_1,z_2,\dots ,z_i,\dots ,{\tilde{z}}_t,\dots ,z_C]\in {\mathbb{R}}^{1\times C}$, where $z_t$ is replaced by $\widetilde{z_t}$. We normalize $\tilde{\mathbf{Z}}$ through the softmax function and obtain the scaled soft labels $\tilde{\mathbf{P}}$:

$$\begin{array}{cc}\hfill \widetilde{p_i}=& \hfill Softmax\left(\tilde{\mathbf{Z}}\right)=\left\{\begin{array}{ccc}\hfill {\displaystyle \frac{exp(z_i/\tau )}{{\sum }_{j=1,j\ne t}^C(exp(z_j/\tau ))+exp(\widetilde{z_t}/\tau )}}& \mathrm{for}& \hfill i\ne t,\hfill \\ \hfill {\displaystyle \frac{exp\widetilde{z_t}/\tau }{{\sum }_{j=1,j\ne t}^C(exp(z_j/\tau ))+exp(\widetilde{z_t}/\tau )}}& \mathrm{for}& \hfill i=t,\hfill \end{array}\right.\hfill \end{array}$$

(4)

where t is the target index of a training sample. Different from existing methods that utilize the same soft label scaling strategy for all samples in the dataset, our proposed IS module generates an instance-level target class for each sample based on the perceptron. Therefore, the design of the perceptron loss is crucial to our IS module, and we carefully analyze the optimization objectives of the IS module in the following two aspects.

For easy samples, the teacher model generates a high value on the target class due to overconfidence. Therefore, directly aligning the output probability of teacher and student models through KL divergence may lead to the depreciation of the information about the non-target class. To tackle this, we aim to increase the entropy of the soft labels to enrich the information about the non-target class. In the implementation, this can be achieved by minimizing the negative entropy of the soft labels outputted by the IS module as follows:

$$\begin{array}{cc}\hfill L_{NE}=& \hfill \underset{{\theta }_{IS}}{min}-(-\mathbf{P}log{\mathbf{P}}^{tea})=\underset{{\theta }_{IS}}{min}\sum _{i=1}^C{p}_i^{tea}log{p}_i^{tea},\hfill \end{array}$$

(5)

where ${\theta }_{IS}$ is parameters of the IS module.

For hard samples, the teacher model tends to generate a relatively small value on the target class, which is insufficient for the student model to obtain discriminative representations. To tackle this, the cross-entropy loss is utilized as the optimization objective for the IS module to further enhance the discrimination of its soft labels as follows:

$$\begin{array}{cc}\hfill L_{CE}=& \hfill \underset{{\theta }_{IS}}{min}-\mathbf{Y}log{\mathbf{P}}^{tea}=\underset{{\theta }_{IS}}{min}\sum _{i=1}^C-y_{i}log{p}_i^{tea},\hfill \end{array}$$

(6)

where $y_i$ is the one-hot representation of the ground truth:

$$\begin{array}{cc}\hfill y_i=& \hfill \left\{\begin{array}{ccc}\hfill 0& \mathrm{for}& \hfill i\ne t,\hfill \\ \hfill 1& \mathrm{for}& \hfill i=t.\hfill \end{array}\right.\hfill \end{array}$$

(7)

To address both of the above situations, we optimize the soft labels via an entropy regularization loss consisting of $L_{NE}$ and $L_{CE}$:

$$\begin{array}{cc}\hfill L_{IS}=& \hfill L_{NE}+\omega L_{CE}=\sum _{i=1}^C{p}_i^{tea}log{p}_i^{tea}+\omega \sum _{i=1}^C{y}_{i}log{p}_i^{tea},\hfill \end{array}$$

(8)

where $\omega$ is the balance weight.

In summary, we design the entropy regularization loss to obtain a better target class of logits ${\tilde{z}}_t$ and then obtain an optimized soft label $\tilde{\mathbf{P}}$ via softmax function.

3.3. Dynamic Margin-Alignment (DM)

Benefitting from the IS module, the refined soft labels from the teacher model can balance the effect of the target and non-target classes for each sample, thereby mitigating the impact of high inter-class similarity to some extent. High intra-class variability can cause smaller student models to struggle with fitting most of the samples effectively. To address this issue, we propose the dynamic margin-alignment (DM) module (see Figure 2), which generates specific margins for each sample to create a more reasonable boundary. Specifically, for simple samples, the margin can be large to ensure sufficient discrimination by the student model, while for hard samples that are difficult to classify, the margin is relatively small to prevent overfitting.

3.3.1. Margin-Alignment for KD

For better understanding, we first introduce the margin-alignment module, which explicitly encourages intra-class compactness and inter-class separability and produces a reasonable decision boundary for the target class with a marginal region.

Firstly, we introduce a hyperparameter $\Delta$ for setting the marginal region and subtract $\Delta$ from the target class within logits while keeping the non-target class unchanged.

$$\begin{array}{cc}\hfill {\widehat{z}}_i^{stu}=& \hfill \left\{\begin{array}{ccc}\hfill z_i& \mathrm{for}& \hfill i\ne t,\hfill \\ \hfill z_t-\Delta & \mathrm{for}& \hfill i=t.\hfill \end{array}\right.\hfill \end{array}$$

(9)

And the margin-enhanced probabilistic prediction scores of the student model ${\tilde{\mathbf{P}}}^{stu}$ can be calculated via the softmax function.

$$\begin{array}{cc}\hfill {\widehat{\mathbf{P}}}^{stu}=& \hfill Softmax\left({\widehat{\mathbf{Z}}}^{stu}\right),\hfill \end{array}$$

(10)

where ${\widehat{z_i}}^{stu}$ is logits’ value on the i-th class and ${\widehat{\mathbf{P}}}^{stu}$ is the probability vector.

Secondly, we align the margin-enhanced student’s probabilistic prediction scores ${\widehat{\mathbf{P}}}^{stu}$ with optimized soft labels ${\tilde{\mathbf{P}}}^{tea}$ via the KL divergence.

$$\begin{array}{cc}\hfill L_{KD}=& \hfill KL({\tilde{\mathbf{P}}}^{tea}\left|\right|{\widehat{\mathbf{P}}}^{stu})=\sum _{i=1}^C{\widetilde{p_i}}^{tea}log({\widehat{p_i}}^{tea}/{\widehat{p_i}}^{stu}).\hfill \end{array}$$

(11)

3.3.2. Dynamic Adjustment of Margin

Taking into account the influence of the difficulty level of samples on the decision boundary, the adaptive adjustment of the hyperparameter $\Delta$ is adopted for dynamic margin.

In order to clearly explore the relationship between the margin and the difficulty of samples, we conduct a geometric analysis on binary classification tasks focusing on both easy and hard samples. Considering features $\mathit{x}$ of a sample belonging to class 1, the fixed alignment criterion is used to ensure that $z_1-\Delta >z_2$ (in Equation (9)) where $z_i={\mathbf{W}}_i^T\mathbf{x}$ and ${\mathbf{W}}_i$ is the parameters of classifier for class i, that is,

$$\begin{array}{cc}\parallel {\mathbf{W}}_2\parallel \parallel \mathbf{x}\parallel cos\left({\theta }_2\right)\hfill & <\parallel {\mathbf{W}}_1\parallel \parallel \mathbf{x}\parallel cos\left({\theta }_1\right)-\Delta \hfill \\ & =\parallel {\mathbf{W}}_1\parallel \parallel \mathbf{x}\parallel (cos({\theta }_1)-{\displaystyle \frac{\Delta }{\parallel {\mathbf{W}}_1\parallel \parallel \mathbf{x}\parallel }}).\hfill \end{array}$$

(12)

For simplicity, we analyze the scenario when $\parallel {\mathbf{W}}_1\parallel =\parallel {\mathbf{W}}_2\parallel$, as shown in Figure 3. Thus, the marginal region m can be calculated as follows:

$$\begin{array}{c}\hfill m={\displaystyle \frac{\Delta }{\parallel {\mathbf{W}}_1\parallel \parallel \mathbf{x}\parallel }}<cos\left({\theta }_1\right)-cos\left({\theta }_2\right).\end{array}$$

(13)

Specifically, for easy samples, a fixed margin can result in a decision boundary, as shown in Figure 3a; the model can classify them correctly but lacks sufficient discrimination. Thus, the margin needs to be increased to further reduce the intra-class distance and expand the inter-class distance, which can yield a more stringent decision boundary (in Figure 3b), expressed as

$$\begin{array}{c}\hfill cos\left({\theta }_1\right)-m>cos\left({\theta }_1\right)-m^{\prime }>cos\left({\theta }_2\right).\end{array}$$

(14)

For hard samples, the model struggles with accurate classification. Maintaining the margin fixed in such a case may lead to model overfitting:

$$\begin{array}{c}\hfill cos\left({\theta }_1\right)-m<cos\left({\theta }_1\right)<cos\left({\theta }_2\right),\end{array}$$

(15)

resulting in overlapping decision boundaries, as depicted in Figure 3c. Therefore, it becomes necessary to set the margin negative for hard samples. This adjustment encourages the model to sacrifice the accuracy of certain hard samples,

$$\begin{array}{c}\hfill cos\left({\theta }_1\right)-m^{″}>cos\left({\theta }_2\right),\end{array}$$

(16)

ultimately enhancing overall accuracy during the testing phase, as illustrated in Figure 3d.

In summary, due to $m\propto \Delta$, increasing $\Delta$ for easy samples makes ${\tilde{z}}_t$ greater than $z_t$. For difficult samples, relaxing $\Delta$ makes ${\tilde{z}}_t$ approaching or less than $z_t$.

Because the target class value reflects the difficulty of a sample [32], we use its variation as the delta value. In the implementation, $\Delta$ (in Equation (9)) can be adjusted dynamically as follows:

$$\begin{array}{c}\hfill {\Delta }_j={\tilde{z}}_{t,j}-z_{t,j},\end{array}$$

(17)

where t is the target class index and j is the sample index.

The overall workflow of our proposed ISDM can be found in Algorithm 1.

Algorithm 1 Instance-level scaling and dynamic margin-alignment distillation

Require:  Dataset D, Teacher model $F_T$ Ensure:  Student model $F_S$, IS module $F_{IS}$    1: Initialize student model $F_S$, IS module $F_{IS}$ with parameters ${\theta }_S$, ${\theta }_{IS}$    2: Set hyper-parameters: learning rate $\eta$, batch size B, number of epochs E, weight of KD $\alpha$, weight of Entrop $\omega$,    3: for $epoch=1$ to E do    4:    Shuffle dataset D    5:    for each $(x$, $y)$ in D with size B do    6:      % Forward pass    7:      ${\mathbf{Z}}^{tea}\leftarrow F_T\left(x\right)$, ${\mathbf{Z}}^{stu}\leftarrow F_S\left(x\right)$    8:      % Instance-level Scaling    9:      ${\tilde{z}}_t^{stu}\leftarrow F_{IS}\left({\mathbf{Z}}^{tea}\right)$   10:      ${\tilde{\mathbf{Z}}}^{tea}\leftarrow Concat({\mathbf{Z}}_n^{tea},{\tilde{z}}_t^{stu})$   11:      ${\tilde{\mathbf{P}}}^{tea}\leftarrow Softmax\left({\tilde{\mathbf{Z}}}^{tea}\right)$, ${\mathbf{P}}^{tea}\leftarrow Softmax\left({\mathbf{Z}}^{tea}\right)$   12:      % Dynamic Margin-alignment   13:      $\Delta \leftarrow {\tilde{z}}_{tar}^{stu}-z_{tar}^{tea}$   14:      ${\widehat{z}}_{tar}^{stu}\leftarrow z_{tar}^{stu}-\Delta$   15:      ${\widehat{\mathbf{Z}}}^{stu}\leftarrow Concat({\mathbf{Z}}_{non}^{stu},{\widehat{z}}_{tar}^{stu})$   16:      ${\widehat{\mathbf{P}}}^{stu}\leftarrow Softmax\left({\widehat{\mathbf{Z}}}^{stu}\right)$   17:      % Compute loss and update parameters   18:      $L_{IS}\leftarrow L_{NE}+\omega L_{CE}$   19:      $L_{CE}\leftarrow CE({\mathbf{P}}^{tea},y)$, $L_{KD}\leftarrow KL({\tilde{\mathbf{P}}}^{tea},{\widehat{\mathbf{P}}}^{stu})$   20:      $L_{total}\leftarrow L_{CE}+\alpha L_{KD}$   21:      ${\theta }_S\leftarrow {\theta }_S-\eta \nabla L_{total}$, ${\theta }_{IS}\leftarrow {\theta }_{IS}-\eta \nabla L_{IS}$   22:    end for   23: end for   24: Return trained student model $F_S$

4. Results

4.1. Datasets

We evaluate our method on three popular RSI scene classification benchmark datasets and two widely-used general image classification benchmark datasets.

4.1.1. NWPU-RESISC45 Dataset

The NWPU-RESISC45 dataset [59] is a comprehensive resource for remote sensing image classification, featuring 31,500 images from over 100 countries. It includes 45 scene categories with 700 images per category, each image sized at 256 × 256 pixels in RGB format. The dataset’s challenge lies in its varying spatial resolutions (300 cm to 20 cm per pixel), which can result in significant inter-class similarities, necessitating advanced classification methods.

4.1.2. Aerial Image Dataset (AID)

The AID dataset [60] contains 10,000 high-resolution aerial images across 30 scene types. Each type is represented by 200 to 400 images at a resolution of 600 × 600 pixels in RGB format. The images have varying spatial resolutions (800 cm to 50 cm per pixel), adding complexity and relevance for testing sophisticated aerial image classification algorithms.

4.1.3. UC Merced Land-Use Dataset (UCM)

The UCM dataset [61] comprises 2100 images across 21 land-use categories, with 100 images per category. The images are uniformly sized at 256 × 256 pixels and are in RGB format. With a consistent spatial resolution of 30 cm per pixel, this dataset simplifies analysis while providing detailed land-use patterns for research purposes.

4.2. Settings and Implementation Details

For the three remote sensing (RS) datasets, ResNet34 is used as the teacher model and ResNet18 as the student model. We apply two data splitting ratios: one with 80% of the data for training and 20% for testing, and another with 50% for both training and testing, to evaluate the model performance with less training data. The top-1 and top-5 accuracy on the test set are used as evaluation metrics.

We compare our methods with various SOTA methods, including logit-based methods, such as KD [31], DKD [32], MLLD [62], LS [63], and SDD [64], and feature-based methods, such as FitNet [47], AT [50], RKD [65], OFD [66], CRD [67], ReviewKD [49], and CAT [68].

We set the training batch size to 64 and the testing batch size to 128. The temperature parameter is set as 4. The initial learning rate is 0.1, with a total of 200 epochs. The learning rate is reduced by a factor of 10 at 60, 120, and 160 epochs. We use the SGD optimizer with a momentum of 0.9. We apply a weight decay of 5 × 5 × ${10}^{-4}$. The base weight for the cross-entropy loss is set to 1, and the base weight for the knowledge distillation loss $\alpha$ is set to 4. The base balance weight for the entropy regularization loss $\omega$, as defined in Equation (8), is set to 0.1. Experiments are conducted using Python 3.7 with PyTorch on an NVIDIA V100 GPU.

4.3. Main Results

In our experiments across the NWPU-RESISC45, AID, and UCM datasets, the ISDM method consistently outperforms other techniques, using ResNet34 as the teacher network and ResNet18 as the student network. On the NWPU-RESISC45 dataset presented in

Table 1. Results of NWPU-RESISC45 validation. ResNet34 and ResNet18 are adopted as the teacher network and the student network. The best and second-best results are emphasized in bold and underlined cases.

Method	Teacher/Student	Split Ratio (8:2)		Split Ratio (5:5)
		Top 1	Top 5	Top 1	Top 5
	Teacher	96.13	99.92	94.18	99.64
	Student	93.48	98.54	91.86	98.47
Feature	AT (ICLR17)	94.71	99.81	92.23	99.53
	RKD (CVPR19)	95.02	99.67	93.20	99.56
	ReviewKD (CVPR21)	94.84	99.81	93.25	99.57
Logits	KD (NIPS14)	95.63	99.89	94.11	99.59
	DKD (CVPR22)	95.79	99.84	94.25	99.64
	MLLD (CVPR23)	95.00	99.73	93.47	99.16
	LSKD (CVPR24)	95.97	99.79	94.28	99.59
	SDD (CVPR24)	96.05	99.81	94.17	99.66
	ISDM (ours)	96.27	99.96	94.58	99.68

, ISDM achieves the highest top-1 and top-5 accuracy across both split ratios, demonstrating its robustness in knowledge distillation.

Similarly, on the AID dataset in

Table 2. Results of AID validation. ResNet34 and ResNet18 are adopted as the teacher network and the student network. The best and second-best results are emphasized in bold and underlined cases.

Method	Teacher/Student	Split Ratio (8:2)		Split Ratio (5:5)
		Top 1	Top 5	Top 1	Top 5
	Teacher	95.00	99.40	91.66	99.24
	Student	92.95	99.25	88.86	98.56
Feature	AT (ICLR17)	94.15	99.45	91.08	99.12
	RKD (CVPR19)	93.70	99.50	88.84	98.88
	ReviewKD (CVPR21)	93.85	99.65	90.94	99.02
Logits	KD (NIPS14)	94.55	99.55	91.84	99.24
	DKD (CVPR22)	94.60	99.55	92.14	99.16
	MLLD (CVPR23)	93.25	99.30	90.16	98.84
	LSKD (CVPR24)	94.30	99.55	91.76	99.12
	SDD (CVPR24)	94.60	99.45	92.06	99.18
	ISDM (ours)	95.55	99.75	92.92	99.34

, ISDM surpasses all competing methods, including those with feature-based and logits-based distillation, with significant improvements in top-1 accuracy (95.55%) and top-5 accuracy (99.75%) under the 8:2 split ratio.

The results are even more pronounced on the UCM dataset in

Table 3. Results of UCM validation. ResNet34 and ResNet18 are adopted as the teacher network and the student network. The best and second-best results are emphasized in bold and underlined cases.

Method	Teacher/Student	Split Ratio (8:2)		Split Ratio (5:5)
		Top 1	Top 5	Top 1	Top 5
	Teacher	90.95	99.52	89.71	99.71
	Student	88.10	99.29	83.81	98.45
Feature	AT (ICLR17)	90.81	99.05	85.90	99.24
	RKD (CVPR19)	89.29	99.29	84.33	98.38
	ReviewKD (CVPR21)	91.19	99.52	84.64	98.57
Logits	KD (NIPS14)	90.24	99.29	86.86	98.76
	DKD (CVPR22)	90.76	99.52	87.42	99.05
	MLLD (CVPR23)	86.14	98.81	85.38	98.95
	LSKD (CVPR24)	87.33	98.80	84.71	97.24
	SDD (CVPR24)	90.48	99.29	87.95	99.14
	ISDM (ours)	92.62	99.76	88.29	99.43

, where ISDM sets new benchmarks with a top-1 accuracy of 92.62% and a top-5 accuracy of 99.76%, surpassing the second-best method by 1.43%. This highlights its ultimate efficacy in leveraging teacher–student networks. This superior performance across multiple datasets underscores ISDM’s effectiveness in distilling knowledge.

4.4. Ablation Study

The results of ablation experiments are shown in

Table 4. Results of ablation study on NWPU-RESISC45 with split ratio of 8:2. ResNet34 and ResNet18 are adopted as the teacher network and the student network. IS: instance-level scaling. DM: dynamic margin-alignment. FM: fixed margin-alignment. ✔ and × present whether the module is adopted or not.

#	Ablation	IS	DM	FM	Acc
①	ISDM	✔	✔	×	96.27
②	DM only	×	✔	×	95.62
③	IS only	✔	×	×	95.54
④	Baseline	×	×	×	93.48
⑤	IS + FM	✔	×	✔	95.95

. The first row presents the experimental results of ISDM. Removing the IS component from ISDM, there is a performance decrease of 0.65% (see ① and ②). When the DM component is removed, it results in a performance drop of 0.73% (see ① and ③), and removing both IS and DM components leads to a significant performance decrease of 2.79% (see ① and ④).

Under equivalent conditions, we also compare DM and FM (fixed margin-alignment). We set $\Delta$ (seen in Equation (9)) fixed as 3, 5, and 7 in FM and contrast the best result with $\Delta$ as 5 against DM (see ① and ⑤). The results indicate that DM shows performance improvement over FM, proving its superiority over FM.

4.5. Sensitivity of Hyperparameters

The selection of loss weights $\alpha$ and $\beta$ is essential for effectively balancing the cross-entropy loss $L_{CE}$ and distillation loss $L_{KD}$. Based on previous studies [31,32,62,64], we keep $\alpha$ constant at 1.0. To find the best value for $\beta$, we conduct a systematic grid search, testing various options: $\{1.0,2.0,4.0,6.0,8.0\}$. We choose the option that results in the highest accuracy, as shown in

Table 5. Results of different $\beta$ values. The experiments are conducted on UCM, with ResNet34 as the teacher and ResNet18 as the student. The parameter $\alpha$ is set to 1.0 and $\omega$ is set to 0.1. The best results are emphasized in bold.

Split Ratio	$\mathit{\beta }=1.0$	$\mathit{\beta }=2.0$	$\mathit{\beta }=4.0$	$\mathit{\beta }=6.0$	$\mathit{\beta }=8.0$
8:2	90.71	92.14	92.62	91.24	90.71
5:5	87.81	87.62	88.29	87.23	86.57

. Notably, $\beta =4.0$ achieves the highest accuracy, indicating that increasing $\beta$ from 1.0 to 4.0 improves performance, but further increases lead to smaller gains. This highlights the need for careful tuning of $\beta$ to optimize knowledge distillation.

In

Table 6. Results of different $\alpha$ values. The experiments are conducted on UCM, with ResNet34 as the teacher and ResNet18 as the student. The parameter $\beta$ is set to 4.0 and $\omega$ is set to 0.1. The best results are emphasized in bold.

Split Ratio	$\mathit{\alpha }=0.2$	$\mathit{\alpha }=0.5$	$\mathit{\alpha }=1.0$	$\mathit{\alpha }=1.5$	$\mathit{\alpha }=2.0$
8:2	90.95	91.43	92.62	91.19	89.29
5:5	88.00	88.38	88.29	87.14	81.33

, we observe that with an 8:2 dataset split, the model’s accuracy peaks at $\alpha =1.0$. For a 5:5 split, the best performance is at $\alpha =0.5$. Keeping $\alpha$ below 1.0 generally maintains good performance, but it drops sharply if $\alpha$ exceeds 1.0. This suggests that higher $\alpha$ values may overly rely on cross-entropy loss, especially with limited data in a 5:5 split, raising the risk of overfitting. Hence, setting $\alpha$ to 1.0 is a sensible choice.

The results in

Table 7. Results of different $\omega$ values. The experiments are conducted on UCM, with ResNet34 as the teacher and ResNet18 as the student. The parameter $\alpha$ is set to 1.0 and $\beta$ is set to 4.0. The best results are emphasized in bold.

Split Ratio	$\mathit{\omega }=0.05$	$\mathit{\omega }=0.1$	$\mathit{\omega }=0.2$	$\mathit{\omega }=0.3$	$\mathit{\omega }=0.4$
8:2	91.19	92.62	91.43	90.48	90.71
5:5	87.90	88.29	87.33	87.05	87.90

indicate that the parameter $\omega$ achieves optimal performance at a value of 0.1, with the model attaining accuracies of 92.62% and 88.29% for the 8:2 and 5:5 split ratios, respectively. Although slight variations in accuracy are observed across different $\omega$ values, the overall performance remains relatively stable. This suggests a degree of robustness in the performance across various settings of $\omega$.

4.6. Motivation Validation

To better understand the characteristics of the RS dataset, we performed experiments to measure feature similarity between categories in the UCM dataset. We trained ResNet18 models on the UCM dataset using different methods. We then calculated the average logits for each category to use as category centers and computed the cosine similarity between each sample’s logits and all category centers. Figure 4b–d show the inter-class similarity results for the UCM dataset using models trained with Vanilla (“Vanilla” represents the standard ResNet18 model training with only the cross-entropy loss), KD, and MISD methods, respectively. Figure 4a shows the results from Vanilla on the CIFAR-100 dataset. For comparison, we only used data from 21 out of 100 classes.

Comparing Figure 4a,b, the RS dataset has more complex category similarity patterns than CIFAR-100. Some categories, like 12, 19, and 20, are quite similar to other categories, while others like 5 and 7 differ more with others. In contrast, CIFAR-100 has a more balanced similarity across categories, which helps the model learn features better. As shown in Figure 4b–d, KD can somewhat reduce the negative effects of the RS dataset, but ISDM can largely remove these effects, coming close to the results seen with CIFAR-100.

4.7. Effect of Instance-Level Scaling

To further evaluate the effects of our proposed IS module, Figure 5 visualizes the soft labels processed by the IS module for both easy and hard samples, respectively.

Figure 5a illustrates the scaling process of soft labels from easy images in the beach and tennis-court classes. The second column describes the values of the target class before and after scaling for an easy sample. The third column shows the original soft labels; it can generate a high value on the target class due to its low inter-class similarity, thus causing the depreciation of the information about the non-target class. The last column shows the soft labels processed by our IS module. Through the IS module, the overconfidence effect on the target class can be relieved and the information about the non-target class can be enhanced.

Figure 5b illustrates the scaling process of soft labels from easy images in the sparse-residential and baseball-diamond classes. The third column illustrates the raw soft labels, which have a small target class value due to their high inter-class similarity. The last column shows the soft labels processed by our IS module, demonstrating how the insufficiency of the target class can be alleviated via the IS module.

4.8. Effect of Dynamic Margin-Alignment

To assess the effectiveness of DM, we employ visualization of t-SNE. Figure 6a,b showcase features learned by KD and ISDM, respectively. Figure 6a shows that KD results in the features of most samples being mixed together, making them hard to distinguish, with more complex decision boundaries and smaller inter-class distances. As demonstrated in Figure 6b, dynamic margin sets greater and negative margins for simple and difficult samples, respectively. It achieves more dispersed inter-class distances for the vast majority of samples, resulting in more marginal decision boundaries.

4.9. Distillation Fidelity

To provide a comprehensive understanding of distillation fidelity, we follow [32,49] and present our visualizations in Figure 7. Specifically, when focusing on the ResNet34–ResNet18 model pair trained on the UCM dataset, we calculate the absolute distance between the correlation matrices of the teacher and student models. Our findings show that ISDM enhances the alignment of the student model’s predictions with those of the teacher model. Specifically, ISDM results in a maximum difference of 1.71 and a mean difference of 0.35, whereas KD has a maximum difference of 1.95 and a mean difference of 0.42. The lower difference metrics for ISDM indicate that it achieves better alignment between the student and teacher models compared to KD.

4.10. Generalization Exploration

In evaluating the generalization of our ISDM across different datasets, we observed its robust performance on two general image classification datasets, beyond its initial remote sensing application.

Our method consistently outperforms or matches state-of-the-art methods across various teacher–student pairs shown in

Table 8. Results of CIFAR-100 validation. Teachers and students are in the same architecture. The best and second-best results are emphasized in bold and underlined cases.

Method	Teacher	ResNet-56	ResNet-32×4	WRN-40-2	VGG13	ResNet-32×4	VGG13
		72.34	79.42	75.61	74.64	75.61	74.64
	Student	ResNet-20	ResNet-8×4	WRN-16-2	VGG8	ShuffleNetV1	MobileNetV2
		69.06	72.50	73.26	70.36	70.50	64.60
Feature	FitNet (ICLR15)	69.21	73.50	73.58	71.02	73.59	64.14
	AT (ICLR17)	70.55	73.44	74.08	71.43	71.73	59.40
	RKD (CVPR19)	69.61	71.90	73.35	71.48	72.28	64.52
	ReviewKD (CVPR21)	71.89	75.63	76.12	74.84	77.45	70.37
	CAT (CVPR23)	71.62	76.91	75.60	74.65	78.26	69.13
Logits	KD (NIPS14)	70.66	73.33	74.92	72.98	74.07	67.37
	DKD (CVPR22)	71.42	75.97	75.77	74.55	76.23	69.58
	CTKD (AAAI23)	71.19	73.39	75.45	73.52	74.48	68.46
	ISDM(Ours)	72.02	76.98	76.26	74.87	76.72	70.26

, demonstrating competitive results against both feature-based and logits-based methods. ISDM shows notable improvements over other approaches in most configurations, especially in the context of ResNet-56 to ResNet-20 and ResNet-32×4 to ResNet-8×4. This suggests that ISDM not only generalizes well within the CIFAR-100 dataset but also performs comparably to, or better than, existing methods that leverage intermediate feature representations and logits.

On the large-scale ImageNet-1k dataset shown in

Table 9. Results of ImageNet-1k validation. The best and second-best results are emphasized in bold and underlined cases.

Method	Teacher/Student	ResNet34/ResNet18		ResNet50/MN-V1
		Top 1	Top 5	Top 1	Top 5
	Teacher	73.31	91.42	76.16	92.86
	Student	69.75	89.07	68.87	88.76
Feature	AT(ICLR17)	70.69	90.01	69.56	89.33
	OFD(ICCV19)	70.81	89.98	71.25	90.34
	CRD(ICLR20)	71.17	90.13	71.37	90.41
	ReviewKD(CVPR21)	71.61	90.51	72.56	91.00
	CAT(CVPR23)	71.26	90.45	72.24	91.13
Logits	KD(NIPS14)	70.66	89.88	68.58	88.98
	DKD(CVPR22)	71.70	90.41	72.05	91.05
	LS(CVPR24)	71.42	90.29	72.18	90.80
	ISDM (ours)	71.87	90.60	72.98	91.14

, ISDM continues to exhibit superior performance. It outperforms both feature-based and logits-based methods across different teacher–student configurations, including ResNet34 to ResNet18 and ResNet50 to MobileNetV2. This consistent performance across datasets of varying sizes and complexities indicates that ISDM’s effectiveness extends well beyond the original remote sensing tasks, highlighting its strong generalization capability.

In summary, these results confirm that ISDM is not only effective in its primary domain but also shows impressive versatility and robustness in other diverse and challenging scenarios, reinforcing its broad applicability.

4.11. Training Efficiency

We evaluate the training overhead and accuracy for SOTA methods shown in Figure 8. Our approach involves improvements to KD by enhancing soft labels and alignment strategy. Consequently, it exhibits a similar time overhead to KD, providing a substantial advantage over other methods while achieving the highest model performance.

Furthermore, the ISDM method introduces only a perceptron with extra parameters less than 0.01 M, which is negligible compared to the trainable parameters during distillation. Notably, this perceptron is exclusively used to optimize soft labels in training. It does not participate in student’s inference, thereby incurring no overhead.

5. Conclusions

In this paper, we addressed the challenge of high inter-class similarity and large intra-class variance in remote sensing datasets by proposing a distillation method named ISDM. This method optimizes teacher soft labels through instance-level scaling and employs a margin-alignment strategy during the distillation process to enhance model generalization. The ISDM method showed significant improvements on the NWPU-RESISC45, AID, and UCM datasets while maintaining lower costs.

Additionally, we validated the effectiveness of our approach through extensive experiments and demonstrated its generalizability on standard datasets such as CIFAR-100 and ImageNet-1k. We hope that this paper will contribute to advancements in scene classification for remote sensing images and improvements in logits-based distillation methods.