A Scene Classification Model Based on Global-Local Features and Attention in Lie Group Space

Abstract

The efficient fusion of global and local multi-scale features is quite important for remote sensing scene classification (RSSC). The scenes in high-resolution remote sensing images (HRRSI) contain many complex backgrounds, intra-class diversity, and inter-class similarities. Many studies have shown that global features and local features are helpful for RSSC. The receptive field of a traditional convolution kernel is small and fixed, and it is difficult to capture global features in the scene. The self-attention mechanism proposed in transformer effectively alleviates the above shortcomings. However, such models lack local inductive bias, and the calculation is complicated due to the large number of parameters. To address these problems, in this study, we propose a classification model of global-local features and attention based on Lie Group space. The model is mainly composed of three independent branches, which can effectively extract multi-scale features of the scene and fuse the above features through a fusion module. Channel attention and spatial attention are designed in the fusion module, which can effectively enhance the crucial features in the crucial regions, to improve the accuracy of scene classification. The advantage of our model is that it extracts richer features, and the global-local features of the scene can be effectively extracted at different scales. Our proposed model has been verified on publicly available and challenging datasets, taking the AID as an example, the classification accuracy reached 97.31%, and the number of parameters is 12.216 M. Compared with other state-of-the-art models, it has certain advantages in terms of classification accuracy and number of parameters.

1. Introduction

Remote sensing scene classification (RSSC) is a fundamental task for the interpretation of high-resolution remote sensing images (HRRSIs) [1,2,3]. In recent years, with the rapid progress of satellite remote sensing technology, RSSC has also made significant progress [4,5,6]. It has been widely used in various scenarios, such as urban planning, natural disaster prediction, and environmental detection [7,8,9,10,11]. An HRRSI contains a variety of complex information about the structure of the Earth’s surface. The challenge of RSSC is mainly to extract effective features from HRRSI, pay attention to crucial feature information, and realize the differentiation of different scenes.

Deep learning has received widespread attention from scholars due to its autonomous feature learning ability in the field of computer vision (CV), such as image classification, segmentation, and object detection [12,13,14]. Convolutional neural network (CNN) is considered to be the most widely used deep learning model technology, and RSSC based on deep learning has become one of the current mainstream types. Therefore, to improve the accuracy of scene classification, scholars have proposed many improved methods based on CNN models, and these methods have achieved better classification performance [15,16].

In addition to the CNN mentioned above, scholars have also proposed a deep learning model called transformer, which has been widely applied in many fields [17]. The transformer was originally applied in natural language processing (NLP), and its proposed self-attention mechanism can effectively capture the dependency relationships between long input sequences. After the success of NLP, inspired by it, scholars proposed the vision transformer (ViT) model, which can effectively learn the contextual features of images and the correlation relationships between different positions. Furthermore, scholars have also proposed a series of ViT-based RSSC models [18,19,20].

However, the transformer-based models also have limitations. Specifically, firstly, transformer-based models have more computational parameters compared to CNN-based models, which limits the use of transformer-based models in some cases where computational resources are insufficient [1]. Secondly, transformer-based models can effectively extract global relationships in images, but they may ignore smaller local-target-object feature information in HRRSIs [21]. Finally, transformer-based models ignore local induction bias in images, which typically require larger datasets. In the field of remote sensing, the number of HRRSIs is much less than that of natural images, so there may be overfitting issues [1]. Therefore, in this study, we still used a CNN-based model to explore RSSC.

In fields such as HRRSI classification and segmentation, a multi-scale fusion of global and local features is required. Some recent studies, such as ViTAE [22], StoHisNet [23], Transfuse [24], CMT [25], and Comformer [26], have improved the accuracy of classification and segmentation to a certain extent, mainly through the extracted features and attention mechanisms. By studying the successful work of other scholars and the models we proposed earlier, we believe that a better scene classification model should have the following characteristics: (1) integrating multi-scale global features and local features, (2) effective spatial and channel attention mechanisms, and (3) fewer parameters and better computational performance. Therefore, in this study, we pay more attention to the global and local features of the scene, focus on the more crucial regions in the scene through an efficient attention mechanism, eliminate the interference of irrelevant regions, and reduce the number of parameters of the model to improve the computational performance.

Inspired by previous studies such as Swin-Transformer [27], in this study, we propose a novel multi-scale branching model based on Lie Group space. In this model, we design two branches for extracting multi-scale global features and local features, respectively, design efficient spatial and channel attention mechanisms, and perform fusion operations through global and local fusion modules. In summary, the main contributions of this study are as follows:

• We propose a multi-scale branching model for RSSC. In this model, it aims to extract the multi-scale and more discriminative features of the scene in a more fine-grained manner.
• We propose global and local fusion modules to achieve efficient fusion between features at different scales. The module contains spatial and channel attention mechanisms and shortcut connections, which can effectively improve the model to focus on the crucial regions and ignore the irrelevant regions.
• Compared with some existing models, our proposed model is more lightweight and achieves a better balance between classification accuracy and computational performance.

2. Related Works

2.1. RSSC Based on Features of Different Levels

According to the different levels of features, the RSSC model can be divided into four types [2,3,28,29,30,31,32]: (1) RSSC model based on low-level features, (2) RSSC model based on middle-level features, (3) RSSC model based on high-level features, and (4) RSSC model based on object-level features. Early RSSC models were mainly designed from low-level features such as local binary pattern (LBP) [33], color, gradient, and shape features [34]. However, RSSC based on the low-level features cannot effectively handle and represent the features in complex scenes. Then, scholars proposed an RSSC model based on middle-level features. This model mainly creates and encodes local feature dictionaries through local descriptors to achieve a feature representation of scenes. The bag of visual words (BoVW) [35] is one of the representatives of the middle-level model because of its simplicity and ease of implementation [36]. The object-level features model is mainly analyzed from the object-oriented classification results, focusing on the relationship of objects rather than the shallower features of the scene (low-level and middle-level features) [37,38]. Due to the complex geometric structure and spatial layout of HRRSIs, RSSC models based on shallower features do not have advantages in terms of classification accuracy and efficiency.

Different from the mentioned RSSC model, the high-level feature-based RSSC model achieves the autonomous learning of features through the deep neural network, which can effectively improve the accuracy of classification. Xu et al. [29] proposed a scene classification model based on Lie Group manifold space, which effectively extracted shallower and high-level features in the scene. Zhang et al. [39] proposed a gradient-boosting-based random convolution model, which incorporates convolution operations at different depths. Lu et al. [40] proposed a novel aggregated CNN model that incorporates supervised convolutional feature encoding and progressive fusion strategies. Liu et al. [41] proposed a novel multi-scale CNN (MCNN) framework to improve the accuracy of scene classification.

2.2. RSSC Based on Attention Mechanism

The above models have shown high classification accuracy in RSSC. However, the above model does not fully consider the crucial features in the scene, and it is easy to be disturbed by irrelevant features. To address such deficiencies, scholars have proposed the attention mechanism. This mechanism makes the model pay more attention to crucial feature information in crucial regions of the scene and ignore irrelevant regions, and has been successfully applied to many fields, such as scene classification [42,43,44,45,46] and few-shot learning [47,48]. Hu et al. [44] proposed the SENet model, which includes squeeze-and-excite (SE) modules for extracting global contextual feature information. Then, BAM [49] and CBAM [46] attention mechanisms are proposed, which are weighted mainly from channel and spatial dimensions. To further reduce the computational complexity of the model, SA-Net [50] divides the features into several sub-features along the channel dimension and calculates the attention values in channel and spatial dimensions.

Scholars have also proposed a large number of RSSC models based on attention mechanisms. Li et al. [51] proposed a local-global context-aware generative dual-region adversarial model, which introduced globally aware self-attention and locally aware self-attention. Wang et al. [52] proposed the ARCNet model, which is a recurrent neural network based on attention intelligence and can adaptively select crucial regions. Yu et al. [53] proposed an improved model based on SE, mainly by enhancing features of different layers and fusing features. Chen et al. [43] proposed the MBLNet model, which is based on the ResNet50 model and can extract both channel and spatial attention.

In summary, based on our previous research, we decided to thoroughly explore the global and local features of the scene and integrate the above features. The model we propose is based on a hierarchical architecture, which includes both global and local branches that do not interfere with each other, and an efficient feature fusion module is designed. Our model retains the advantages of CNN and transformer models while effectively extracting global and local features.

3. Method

3.1. Overall Framework

In this study, we propose a novel RSSC model, which is a multi-scale global-local feature and attention scene classification model based on Lie Group space, as shown in Figure 1. Different from the existing methods, the model adopts a branch parallel structure, which mainly consists of a global feature extraction (GFE) module, local feature extraction (LFE) module, and global-local fusion (GLF) module. The global features mainly reflect the overall situation of an HRRSI and are statistical features of the entire HRRSI, such as contrast. Local features reflect the characteristics of local regions in an HRRSI, such as texture structure. Local features contain more detailed feature information, which can help us process the details in HRRSIs. Local features can serve as supplements to global features. Some existing methods (such as CNN), when dealing with high-resolution images, increase the number of hidden layers and the number of parameters, slow calculation, and may also encounter overfitting problems. Therefore, in the feature extraction stage, we utilize two independent branches for global and local feature extraction to prevent mutual interference in the above feature extraction process. In the GFE, we use adaptive pooling, parallel dilated convolution, and Lie Group kernel functions to simulate the global space. In the LFE module, we utilize parallel dilated convolution to replace the traditional convolution operation to achieve multi-scale feature extraction. In the GLF module, we adopt efficient global and local fusion methods to achieve feature fusion at different stages, capture the dependency between features in channel dimension and spatial dimension, and combine grouping features with an attention mechanism to enhance semantic features in spatial and channel dimension. More module details are described below.

3.2. Multi-Scale Sampling of HRRIs

The image resolution of HRRSI datasets varies from $0.2m$ to $30m$, which causes a large difference in scale. Considering that the resolution of the actual HRRSI is fixed, it may lead to a significant loss of accuracy in HRRSI transferring. To reduce the scale difference, in this study, we adopt the multi-scale sampling method to obtain different scale data samples of the same scene. Previous studies have verified that uniform grid sampling is effective for the characterization of HRRSIs [28,29]. Therefore, based on previous studies, we first conducted a uniform grid sampling of HRRSIs with block size M and spacing S. Secondly, multi-scale sampling was performed for each region according to the matching of the generated patches with the most overlapping regions. For details, please refer to our previous research results [28].

3.3. Hierarchical Parallel Model

To improve the classification accuracy of RSSC, we integrate global and local features at different levels and scales. Based on the previous research [28,29], we designed a hierarchical parallel framework. In Figure 1, the first branch is used for global feature information extraction, the second branch is used for local feature information extraction, and the third branch is used for the fusion of the above feature information.

The above design is mainly to effectively extract global and local feature information, and preserve the extracted feature information to the greatest extent so that their operations will not affect each other. The previous experimental results show that the branch structure is conducive to feature extraction [28], the parallel branch structure can effectively enhance the characterization of features, and the multi-scale features extracted by the model are more robust and can provide better features for downstream tasks [3,28,29].

3.4. Global Feature Extraction

Due to the shooting angle, shooting distance, and imaging methods, there are various remote sensing scenes with strong inter-class similarity and significant intra-class differences. Therefore, the extraction of global feature information is quite important. The Swin-Transformer [27] model proposed the windows multi-head self-attention (W-MAS) mechanism. Compared with the traditional multi-head attention mechanism (MSA) in transformer, W-MAS divides the feature map into $M\times M$ sizes and then performs self-attention on the divided feature maps, improving calculation efficiency. Therefore, the W-MAS mechanism was introduced in this study.

Traditional models typically utilize global average pooling to simulate the global context environment, but these operations are not comprehensive enough for HRRSIs with complex geometric structures and spatial layouts. To better simulate the global context environment of scenes in HRRSIs, we adopted an adaptive pooling operation. More specifically, in Figure 2, adaptive pooling operation is first used to extract global contextual feature information. To accelerate the convergence of the model, batch normalization (BN) operation was adopted. The W-MAS mechanism is adopted, and its window size is $5\times 5$, $H=3$. Then, a larger kernel parallel dilated convolution is adopted; the kernel size is $5\times 5$. Previous research results have verified that larger kernel parallel dilated convolution has a larger receptive field, and the number of parameters is much smaller than traditional convolution [2,28,29,31]. After a $1\times 1$ convolution operation, in the following operations, nonlinear activation is usually introduced to enhance the representativeness of the model. In our model, the Lie Group Sigmoid activation function is applied instead of the traditional Sigmoid activation function, mainly to improve the robustness and computational performance of the model [54,55]. In addition, we also adopted a residual connection manner to further enhance the representation of features, as shown below.

$$G{F}_i^{\prime }=LGS\left(Conv\left(PDConv(W-MSA\left(BN\left(AP\left(G{F}_{i-1}\right)\right)\right))\right)\right)+G{F}_{i-1}$$

(1)

where $GF$ represents the input feature map, $G{F}_{i-1}$ represents the previous input feature map, $G{F}_i$ represents the $i^{th}$ feature map, $AP$ represents the adaptive pooling, $BN$ represents the batch normalization, $W-MAS$ represents the windows multi-head self-attention, and $PDConv$ represents the parallel dilated convolution with $7\times 7$ convolution kernel. $Conv$ represents a convolution with a convolution kernel of $1\times 1$, and $LGS$ represents a Lie Group Sigmoid activation function. The global feature information extracted from the above will be input into the global and local feature fusion module.

3.5. Local Feature Extraction

Local features in HRRSIs are also quite important. As a supplement to the global features, local features can effectively enhance the features of the scene. As shown in Figure 3, the feature map is firstly divided into four partitions along the channel dimension, as shown below.

$$L{F}_1,L{F}_2,L{F}_3,L{F}_4=S\left(L{F}_i\right)$$

(2)

where $L{F}_i$ represents the $i^{th}$ input feature map, and $L{F}_1$, $L{F}_2$, $L{F}_3$, and $L{F}_4$, respectively, represent the four partitions obtained using the split function $S(·)$.

In the first partition, $L{F}_1$, a larger kernel-parallel-dilated convolution is utilized to extract features, as follows:

$$L{F}_1^{\prime }=PDConv\left(L{F}_1\right)$$

(3)

Then, the extracted feature map $L{F}_1^{\prime }$ is added to the second partition $LF2$, and the new feature map $L{F}_2^{\prime }$ is obtained by using the same convolution operation as above for the added feature. By repeating the above operation, the new feature maps of the four partitions can be obtained, and they are connected as follows:

$$L{F}_i^{\prime }=L{F}_{i-1}^{\prime }+L{F}_i$$

(4)

$$L{F}_i^{\prime }=C_{ch}(L{F}_1^{\prime }+L{F}_2^{\prime }+L{F}_3^{\prime }+L{F}_4^{\prime })$$

(5)

where $C_{ch}(·)$ indicates the channelwise concatenation operation.

Compared with the traditional convolution operation, the above operation has the following advantages:

• The above operation can effectively increase the feature information interaction between different partition modules.
• We have achieved multi-channel and multi-dimensional feature extraction in a finer granularity manner, effectively expanding the receptive field while suppressing the increase in model parameters, improving the computational efficiency of the model, and reducing the number of model parameters.
• More specifically, traditional multi-scale feature extraction mainly adopts multiple parallel branch structures, each branch contains a fixed kernel (such as $3\times 3$, $5\times 5$), without considering the relationship between the feature maps of different branches. Compared with our model, the feature maps obtained from each branch are fed to the next branch, which achieves the reuse of features and promotes the interaction of feature information between different channels. In addition, the receptive field can be effectively expanded in this way, such as in two consecutive $7\times 7$ convolution operations, the receptive field of the first convolution is $7\times 7$ (each output value addresses 49 values in the input feature map), the receptive field of the second convolution is effectively $9\times 9$ (each output value addresses 81 values in the input feature map), with the same size kernel itself, but the receptive field is enlarged.

3.6. Global-Local Fusion

3.6.1. Channel Attention

Each channel in the feature map usually contains the response of the corresponding feature. Channel attention enables the model to pay attention to different regions in the scene, and adaptively assigns different weights to each channel. This process can be regarded as feature screening [56,57,58].

As shown in Figure 4, the feature map is firstly divided into four partitions along the channel dimension, and the four partitions are flattened, which is mainly to mine the correlation between features. $G{F}_1^{\prime }$ and $G{F}_4^{\prime }$ are then operated through average pooling and maximum pooling, respectively. After the $G{F}_2^{\prime }$ and $G{F}_3^{\prime }$ pass through the multi-layer perceptron (MLP), the multiplied operation will be performed with the above feature map, respectively, as shown below.

$$\begin{gathered} GF{A}_i=LGS(Red(Avg\left(Fla\left(G{F}_1^{\prime }\right)\right)·MLP(Fla\left(G{F}_2^{\prime }\right),Fla\left(G{F}_3^{\prime }\right)))+ \\ Red(Max\left(Fla\left(G{F}_4^{\prime }\right)\right)·MLP(Fla\left(G{F}_2^{\prime }\right),Fla\left(G{F}_3^{\prime }\right)))) \end{gathered}$$

(6)

where $G{F}_1^{\prime }$, $G{F}_2^{\prime }$, $G{F}_3^{\prime }$, and $G{F}_4^{\prime }$ represent the four partitions divided along the channel dimension, respectively, $Fla$ represents the flattening operation, $Avg$ represents the average pooling operation, $Max$ represents the maximum pooling operation, and $MLP$ represents the multi-layer perceptron operation. $Red$ indicates the Reordering operation, while $GF{A}_i$ represents the feature maps obtained through channel attention mechanism.

3.6.2. Spatial Attention

Spatial attention can be used as a supplement to the above, mainly to enhance the semantic distribution of features in the spatial dimension, that is, enabling the model to selectively focus on crucial regions in the scene while ignoring irrelevant regions. At the same time, spatial attention is also the process by which the model screens important regions and decides what to focus on. Our model utilizes this mechanism to capture different semantic features for each feature map.

As shown in Figure 3, a $7\times 7$ parallel dilated convolution operation is performed, and a Lie Group Sigmoid kernel function is added to the nonlinear transformation. This operation can effectively preserve the feature information of spatial dimensions, and the use of a $7\times 7$ parallel dilated convolution operation can effectively reduce the number of model parameters and improve computational performance. The specific operation is as follows.

$$LA{F}_i=LGS\left(PDConv\left(L{F}_i^{\prime }\right)\right)$$

(7)

where $LF{A}_i$ represents the feature maps obtained through spatial attention mechanism.

3.6.3. Fusion Module

The main function of this module is to efficiently integrate global features, local features, features obtained through channel attention, features obtained through spatial attention, and features obtained from the previous level, as shown in Figure 3. The fusion operation is as follows:

$$\overline{F{M}_i}=Avg\left(Conv\left(F{M}_{i-1}\right)\right)$$

(8)

$$\widetilde{GF{A}_i}=GF{A}_i\otimes G{F}_i^{\prime }$$

(9)

$$\widetilde{LF{A}_i}=LF{A}_i\otimes L{F}_i^{\prime }$$

(10)

$$\widetilde{F{M}_i}=Conc(LGS\left(Conv\left(BN\left(Conc(G{F}_i^{\prime },L{F}_i^{\prime },\overline{F{M}_i})\right)\right)\right),\widetilde{GF{A}_i},\widetilde{LF{A}_i})$$

(11)

$$F{M}_i=Conc(LGS\left(Conv\left(GELU\left(Conv\left(BN\left(Conc\left(LGS(PDConv\left(\widetilde{F{M}_i}\right),\widetilde{F{M}_i})\right)\right)\right)\right)\right)\right),\overline{F{M}_i})$$

(12)

where $F{M}_{i-1}$ represents the feature map generated in the previous stage of fusion; $F{M}_i$ represents the feature map generated after the fusion; $\widetilde{GF{A}_i}$ represents the feature map after the fusion of channel dimensions, that is, the element-wise multiple of the feature map before the channel attention operation and the feature map obtained after the channel attention mechanism is used; $\widetilde{LF{A}_i}$ represents the feature map after the fusion of spatial dimensions, that is, the element-wise multiple of the feature map before the spatial attention operation and the feature map obtained after using the spatial attention mechanism; $Conc$ represents the concatenation operation; $BN$ represents the batch normalization; and $GELU$ represents the GELU activation function.

4. Experiments

4.1. Experimental Environment

4.1.1. Datasets

We chose three publicly available and representative datasets, such as the aerial image dataset (AID) [10], the remote sensing image classification benchmark (RSICB-256) [59], and the Northwestern Polytechnical University remote sensing image scene classification (NWPU-RESISC45) dataset [37]. The AID is a large-scale aerial imagery dataset collected from Google Earth images, containing 30 scene categories. The RSICB-256 dataset integrates sensor and Google Earth images, containing 35 scene categories. The NWPU-RESISC45 dataset was created by Northwestern Polytechnical University and contains 45 scene categories. The above dataset contains a large number of scenes, and the scenes of different categories have high similarity while the scenes within the same category have significant variations. Therefore, the above dataset poses certain challenges. We compared our proposed model with some classic and state-of-the-art (SOTA) in terms of parameter quantity, classification accuracy, and computational performance.

4.1.2. Experimental Parameter Setting and Evaluation Metrics

All experiments were carried out under the same training parameters; the specific parameters are shown in

Table 1. Setting of experimental environment and other parameters.

Item	Content
CPU	Inter Core i7-4700 CPU with 2.70 GHz $\times 12$
Memory	32 GB
Operating system	CentOS 7.8 64 bit
Hard disk	1TB
GPU	Nvidia Titan-X $\times 2$
Python	3.7.2
PyTorch	1.4.0
CUDA	10.0
Learning rate	${10}^{-3}$
Momentum	0.73
Weight decay	$5\times {10}^{-4}$
Batch	16
Saturation	1.7
Subdivisions	64

. Referring to the previous model [28,29,30,31,42,43,60,61], we adopted different settings for the above three datasets. Specifically, the training ratio is 20% and 50% for the AID, 50% for the RSICB-256 dataset, and 10% and 20% for the NWPU-RESISC45 dataset. The results obtained in the experiment are set according to the parameters in the references. We report the average results and standard deviations of ten replicates performed independently to reduce the effect of randomness.

We chose overall accuracy (OA), confusion matrix (CM), giga multiply-accumulation operations per second (GMACs), and the number of model parameters as evaluation metrics. The above metrics mainly reflect the accuracy of the classification model, the easily confused scenarios in each dataset, and the evaluation model size and computational performance.

4.2. Comparison with SOTA Models

4.2.1. Experimental Results of AID

The experimental results on the AID are shown in Table 2. From the experimental results, we obtain the following findings:

• Our proposed model achieved 95.09% and 97.31% at a training ratio of 20% and 50%, which improved 2.65%, 2.8%, and 1.98% compared to ResNet50 [62], ResNet50+CBAM [1], and ResNet50+HFAM [1], respectively. The experimental results indicate that our proposed model can achieve better classification results.
• The model with an added attention mechanism has higher classification accuracy compared to traditional models without an added attention mechanism. For example, the ResNet50+CBAM [1] model improved by 0.13% compared to the ResNet50 [62] model, and the VGG16+HFAM [1] model improved by 6.25% compared to the VGG-VD-16 [10] model. The experimental results verified the positive role of the attention mechanism in scene classification.
• In our proposed model, crucial feature information of key regions in the scene can be selectively focused on. According to the experimental results, our model improved by 0.67%, 0.4%, and 1.98% compared to the Fine-tune MobileNet V2 [63], DS-SURF-LLC+Mean-Std-LLC+MO-CLBP-LLC [64], and ResNet50+HFAM [1] models, respectively. Therefore, we believe that by combining channel attention and spatial attention, we can obtain more discriminative features and achieve better classification performance.

Table 2. Overall accuracies (%) of thirty-two kinds of methods and our method under the training ratios of 20% and 50% in AID.

Models	Training Ratios
	20%	50%
CaffeNet [10]	$86.72\pm 0.45$	$88.91\pm 0.26$
VGG-VD-16 [10]	$85.81\pm 0.25$	$89.36\pm 0.36$
GoogLeNet [10]	$83.27\pm 0.36$	$85.67\pm 0.55$
Fusion by addition [65]	−	$91.79\pm 0.26$
LGRIN [30]	$94.74\pm 0.23$	$97.65\pm 0.25$
TEX-Net-LF [66]	$93.91\pm 0.15$	$95.66\pm 0.17$
DS-SURF-LLC+Mean-Std-LLC+MO-CLBP-LLC [64]	$94.69\pm 0.22$	$96.57\pm 0.27$
LiG with RBF kernel [55]	$94.32\pm 0.23$	$96.22\pm 0.25$
ADPC-Net [67]	$88.61\pm 0.25$	$92.21\pm 0.26$
VGG19 [62]	$86.83\pm 0.26$	$91.83\pm 0.38$
ResNet50 [1]	$92.16\pm 0.18$	$95.51\pm 0.15$
ResNet50+SE [1]	$92.77\pm 0.18$	$95.84\pm 0.22$
ResNet50+CBAM [1]	$92.29\pm 0.15$	$95.38\pm 0.16$
ResNet50+HFAM [1]	$93.11\pm 0.20$	$95.86\pm 0.15$
InceptionV3 [62]	$92.65\pm 0.19$	$94.97\pm 0.22$
DenseNet121 [68]	$92.91\pm 0.25$	$94.65\pm 0.25$
DenseNet169 [68]	$92.39\pm 0.35$	$93.46\pm 0.27$
MobileNet [69]	$87.91\pm 0.16$	$91.23\pm 0.16$
EfficientNet [70]	$87.37\pm 0.16$	$89.41\pm 0.15$
Two-stream deep fusion Framework [71]	$92.42\pm 0.38$	$94.62\pm 0.27$
Fine-tune MobileNet V2 [63]	$94.42\pm 0.25$	$96.11\pm 0.25$
SE-MDPMNet [63]	$93.77\pm 0.16$	$97.23\pm 0.16$
Two-stage deep feature Fusion [72]	−	$93.87\pm 0.35$
Contourlet CNN [73]	−	$96.87\pm 0.42$
LCPP [74]	$91.12\pm 0.35$	$93.35\pm 0.35$
RSNet [75]	$94.62\pm 0.27$	$96.78\pm 0.56$
SPG-GAN [76]	$92.31\pm 0.17$	$94.53\pm 0.38$
TSAN [77]	$89.67\pm 0.23$	$92.16\pm 0.25$
LGDL [29]	$93.97\pm 0.16$	$97.29\pm 0.35$
VGG16+CBAM [1]	$91.91\pm 0.35$	$95.53\pm 0.07$
VGG16+SE [1]	$91.98\pm 0.31$	$95.45\pm 0.19$
VGG16+HFAM [1]	$92.06\pm 0.16$	$95.78\pm 0.21$
Proposed	$95.09\pm 0.15$	$97.31\pm 0.23$

Table 2. Overall accuracies (%) of thirty-two kinds of methods and our method under the training ratios of 20% and 50% in AID.

Models	Training Ratios
	20%	50%
CaffeNet [10]	$86.72\pm 0.45$	$88.91\pm 0.26$
VGG-VD-16 [10]	$85.81\pm 0.25$	$89.36\pm 0.36$
GoogLeNet [10]	$83.27\pm 0.36$	$85.67\pm 0.55$
Fusion by addition [65]	−	$91.79\pm 0.26$
LGRIN [30]	$94.74\pm 0.23$	$97.65\pm 0.25$
TEX-Net-LF [66]	$93.91\pm 0.15$	$95.66\pm 0.17$
DS-SURF-LLC+Mean-Std-LLC+MO-CLBP-LLC [64]	$94.69\pm 0.22$	$96.57\pm 0.27$
LiG with RBF kernel [55]	$94.32\pm 0.23$	$96.22\pm 0.25$
ADPC-Net [67]	$88.61\pm 0.25$	$92.21\pm 0.26$
VGG19 [62]	$86.83\pm 0.26$	$91.83\pm 0.38$
ResNet50 [1]	$92.16\pm 0.18$	$95.51\pm 0.15$
ResNet50+SE [1]	$92.77\pm 0.18$	$95.84\pm 0.22$
ResNet50+CBAM [1]	$92.29\pm 0.15$	$95.38\pm 0.16$
ResNet50+HFAM [1]	$93.11\pm 0.20$	$95.86\pm 0.15$
InceptionV3 [62]	$92.65\pm 0.19$	$94.97\pm 0.22$
DenseNet121 [68]	$92.91\pm 0.25$	$94.65\pm 0.25$
DenseNet169 [68]	$92.39\pm 0.35$	$93.46\pm 0.27$
MobileNet [69]	$87.91\pm 0.16$	$91.23\pm 0.16$
EfficientNet [70]	$87.37\pm 0.16$	$89.41\pm 0.15$
Two-stream deep fusion Framework [71]	$92.42\pm 0.38$	$94.62\pm 0.27$
Fine-tune MobileNet V2 [63]	$94.42\pm 0.25$	$96.11\pm 0.25$
SE-MDPMNet [63]	$93.77\pm 0.16$	$97.23\pm 0.16$
Two-stage deep feature Fusion [72]	−	$93.87\pm 0.35$
Contourlet CNN [73]	−	$96.87\pm 0.42$
LCPP [74]	$91.12\pm 0.35$	$93.35\pm 0.35$
RSNet [75]	$94.62\pm 0.27$	$96.78\pm 0.56$
SPG-GAN [76]	$92.31\pm 0.17$	$94.53\pm 0.38$
TSAN [77]	$89.67\pm 0.23$	$92.16\pm 0.25$
LGDL [29]	$93.97\pm 0.16$	$97.29\pm 0.35$
VGG16+CBAM [1]	$91.91\pm 0.35$	$95.53\pm 0.07$
VGG16+SE [1]	$91.98\pm 0.31$	$95.45\pm 0.19$
VGG16+HFAM [1]	$92.06\pm 0.16$	$95.78\pm 0.21$
Proposed	$95.09\pm 0.15$	$97.31\pm 0.23$

Figure 5 shows the CM for a training ratio of 50%. Our model can correctly identify most scenes in the AID, and some scenes achieve 100% accuracy, such as “Beach” and “Forest”. However, some scenes have also been confused, such as “School” and “Commercial”. Through further analysis, we find that the features in these two types of scenarios are highly similar, leading to confusion.

4.2.2. Experimental Results of RSICB-256

To further verify the validity of our model, a large number of experiments were conducted on the RSICB-256 dataset, and the experimental results are shown in

Table 3. Overall accuracies (%) of thirty-five kinds of methods and our method under the training ratios of 50% in RSICB-256.

Models	50%
CaffeNet [10]	$91.37\pm 0.23$
VGG-VD-16 [10]	$92.44\pm 0.25$
GoogLeNet [10]	$89.87\pm 0.36$
Fusion by addition [65]	$93.36\pm 0.25$
LGRIN [30]	$97.55\pm 0.23$
TEX-Net-LF [66]	$95.34\pm 0.15$
DS-SURF-LLC+Mean-Std-LLC+ MO-CLBP-LLC [64]	$96.43\pm 0.25$
LiG with RBF kernel [55]	$95.37\pm 0.26$
ADPC-Net [67]	$92.19\pm 0.25$
VGG19 [62]	$92.67\pm 0.35$
ResNet50 [1]	$96.37\pm 0.15$
ResNet50+SE [1]	$96.53\pm 0.29$
ResNet50+CBAM [1]	$95.72\pm 0.26$
ResNet50+HFAM [1]	$97.65\pm 0.22$
InceptionV3 [62]	$94.53\pm 0.22$
DenseNet121 [68]	$94.21\pm 0.26$
DenseNet169 [68]	$93.27\pm 0.28$
MobileNet [69]	$91.33\pm 0.17$
EfficientNet [70]	$92.25\pm 0.18$
Two-stream deep fusion Framework [71]	$94.57\pm 0.25$
Fine-tune MobileNet V2 [63]	$95.83\pm 0.26$
SE-MDPMNet [63]	$96.35\pm 0.26$
Two-stage deep feature Fusion [72]	$94.89\pm 0.39$
Contourlet CNN [73]	$95.39\pm 0.29$
LCPP [74]	$93.72\pm 0.37$
RSNet [75]	$95.89\pm 0.41$
SPG-GAN [76]	$94.57\pm 0.35$
TSAN [77]	$93.12\pm 0.26$
LGDL [29]	$97.36\pm 0.32$
ViT-B-16 [78]	$97.37\pm 0.25$
T2T-ViT-12 [79]	$97.50\pm 0.22$
PVT-V2-B0 [80]	$97.45\pm 0.26$
VGG16+CBAM [1]	$96.17\pm 0.35$
VGG16+SE [1]	$95.87\pm 0.15$
VGG16+HFAM [1]	$97.52\pm 0.20$
Proposed	$97.72\pm 0.25$

. The following can be found from the table:

• When the training ratio is 50%, our proposed model reaches 97.72%. And 0.2%, 1.37%, and 0.07% are, respectively, increased compared with VGG16+HFAM [1], SE-MDPMNet [63], and ResNet50+HFAM [1]. The experimental results further verify the validity of our model.
• In general, the attention mechanism can improve the accuracy of classification. From our experiments, we found that our model improves 1.19%, 2%, and 1.55%, respectively, compared to models that use other attention mechanisms, such as ResNet50+SE [1], ResNet50+CBAM [1], and VGG16+CBAM [1].
• The ViT-based model has achieved better performance because it uses the global attention mechanism to simulate the global environment. In our model, we utilize both the global attention mechanism and the local attention mechanism, and compared with the ViT-based model, the classification accuracy has been improved to some extent.

Figure 6 shows the CM on the RSICB-256 dataset. From Figure 6, we can find that for most scenarios on the RSICB-256 dataset, the classification accuracy of our proposed model is greater than 90%. This experimental result once again verifies the effectiveness of the global attention mechanism and the local attention mechanism in our model.

4.2.3. Experimental Results of NWPU-RESISC45

Compared with the above two datasets, the NWPU-RESISC45 dataset contains more scene categories and has a high degree of similarity between scenes and a large difference within scenes, which puts forward higher requirements for the performance of the model. Table 4 shows the following specific results:

• Since the category of scenes has increased compared with the above two datasets, and the training ratio is only 10% and 20%, the classification results of all models have decreased compared with the above two datasets.
• Compared with other models, our proposed model still has higher classification accuracy. Specifically, when the training ratio is 10%, the ratios of 0.16%, 2.54%, and 1.11% are increased, respectively, compared to ResNet50+EAM [81], ResNet101+HFAM [1], and ViT-B-16 [78]. When the training ratio is 20%, the ratios of 1.44%, 1.14%, and 4.03% are increased, respectively, compared to PVT-V2-B0 [80], LiG with RBF kernel [55], and ResNet101 [1]. Experimental results show that our proposed model is also effective on datasets with multiple scenes.
• Under the same training ratio, the ViT-based model (such as ViT-B-16 [78], T2T-ViT-12 [79], PVT-V2-B0 [80]) achieves higher classification accuracy than the classical CNN model (such as GoogLeNet [82]), mainly because the ViT-based model makes up for the shortcomings of the classical CNN model in the global context. However, in addition to the global context feature information, the proposed model also utilizes a local spatial attention mechanism to extract local detail feature information, further filling the gap in local feature information. Furthermore, since the transformer-based method lacks convolutional inductive bias, it requires more training data samples. However, with a training ratio of 10% and 20%, for example, the classification accuracy of ViT-B-16 [78] is 90.96% and 93.36%, respectively. In terms of classification accuracy, the classification accuracy of the transformer-based method is lower than that of our model. In terms of computational complexity, however, it is more complex than our proposed model.

Table 4. Overall accuracies (%) of twenty-one kinds of methods and our method under the training ratios of 10% and 20% in NWPU-RESISC45.

Models	Training Ratios
	10%	20%
GoogLeNet [82]	$76.19\pm 0.38$	$78.48\pm 0.26$
SCCov [83]	$89.30\pm 0.35$	$92.10\pm 0.25$
ACNet [61]	$91.09\pm 0.13$	$92.42\pm 0.16$
ViT-B-16 [78]	$90.96\pm 0.08$	$93.36\pm 0.17$
T2T-ViT-12 [79]	$90.62\pm 0.18$	$93.19\pm 0.10$
PVT-V2-B0 [80]	$89.72\pm 0.16$	$92.95\pm 0.09$
LGRIN [30]	$91.91\pm 0.15$	$94.43\pm 0.16$
LiG with RBF kernel [55]	$90.23\pm 0.13$	$93.25\pm 0.12$
ResNet50 [1]	$87.43\pm 0.29$	$88.93\pm 0.12$
ResNet50+EAM [81]	$91.91\pm 0.22$	$94.29\pm 0.09$
ResNet50+SE [1]	$89.09\pm 0.14$	$91.37\pm 0.25$
ResNet50+CBAM [1]	$88.11\pm 0.39$	$90.27\pm 0.15$
ResNet50+HFAM [1]	$89.16\pm 0.06$	$91.49\pm 0.23$
ResNet101 [1]	$87.97\pm 0.44$	$90.36\pm 0.17$
ResNet101+SE [1]	$89.39\pm 0.14$	$91.46\pm 0.25$
ResNet101+CBAM [1]	$88.33\pm 0.26$	$90.47\pm 0.15$
ResNet101+HFAM [1]	$89.53\pm 0.29$	$91.67\pm 0.18$
VGG16 [1]	$86.44\pm 0.41$	$88.57\pm 0.16$
VGG16+SE [1]	$86.65\pm 0.26$	$88.75\pm 0.22$
VGG16+CBAM [1]	$86.84\pm 0.24$	$89.32\pm 0.15$
VGG16+HFAM [1]	$87.16\pm 0.22$	$90.21\pm 0.22$
Proposed	$92.07\pm 0.25$	$94.39\pm 0.25$

Table 4. Overall accuracies (%) of twenty-one kinds of methods and our method under the training ratios of 10% and 20% in NWPU-RESISC45.

Models	Training Ratios
	10%	20%
GoogLeNet [82]	$76.19\pm 0.38$	$78.48\pm 0.26$
SCCov [83]	$89.30\pm 0.35$	$92.10\pm 0.25$
ACNet [61]	$91.09\pm 0.13$	$92.42\pm 0.16$
ViT-B-16 [78]	$90.96\pm 0.08$	$93.36\pm 0.17$
T2T-ViT-12 [79]	$90.62\pm 0.18$	$93.19\pm 0.10$
PVT-V2-B0 [80]	$89.72\pm 0.16$	$92.95\pm 0.09$
LGRIN [30]	$91.91\pm 0.15$	$94.43\pm 0.16$
LiG with RBF kernel [55]	$90.23\pm 0.13$	$93.25\pm 0.12$
ResNet50 [1]	$87.43\pm 0.29$	$88.93\pm 0.12$
ResNet50+EAM [81]	$91.91\pm 0.22$	$94.29\pm 0.09$
ResNet50+SE [1]	$89.09\pm 0.14$	$91.37\pm 0.25$
ResNet50+CBAM [1]	$88.11\pm 0.39$	$90.27\pm 0.15$
ResNet50+HFAM [1]	$89.16\pm 0.06$	$91.49\pm 0.23$
ResNet101 [1]	$87.97\pm 0.44$	$90.36\pm 0.17$
ResNet101+SE [1]	$89.39\pm 0.14$	$91.46\pm 0.25$
ResNet101+CBAM [1]	$88.33\pm 0.26$	$90.47\pm 0.15$
ResNet101+HFAM [1]	$89.53\pm 0.29$	$91.67\pm 0.18$
VGG16 [1]	$86.44\pm 0.41$	$88.57\pm 0.16$
VGG16+SE [1]	$86.65\pm 0.26$	$88.75\pm 0.22$
VGG16+CBAM [1]	$86.84\pm 0.24$	$89.32\pm 0.15$
VGG16+HFAM [1]	$87.16\pm 0.22$	$90.21\pm 0.22$
Proposed	$92.07\pm 0.25$	$94.39\pm 0.25$

Figure 7 shows the CM on the NWPU-RESISC45 dataset when the training ratio is 20%. Compared with the first two datasets, the NWPU-RESISC45 dataset is more challenging, so the classification accuracy of the 45 categories of scenes is reduced to a certain extent. Further analysis of the easily confused scenes shows that they contain a large number of the same target objects. For example, “Dense_residential” and “Medium_residential” scenes both contain trees, so their feature maps are relatively similar, resulting in confusion.

4.3. Comparison of the Number of Model Parameters and Computational Performance

In addition to comparing with the SOTA model, we also selected twelve models (such as VGG-VD-16 [10], ResNet50+CBAM [1], and ResNet50+HFAM [1]) for comparison. Specifically, we compared the classification accuracy, number of parameters, GMACs, and velocity of the model. These metrics mainly reflect the size, computational performance, and classification accuracy of the model.

As shown in

Table 5. Evaluation of size of models.

Models	Acc (%)	Parameters (M)	GMACs (G)	Velocity (Samples/s)
CaffeNet [10]	88.91	60.97	3.6532	32
GoogLeNet [10]	85.67	7	0.7500	37
VGG-VD-16 [10]	89.36	138.36	7.7500	35
LiG with RBF kernel [55]	96.22	2.07	0.2351	43
ResNet50 [1]	95.51	25.58	1.8555	38
ResNet50+SE [1]	95.84	26.28	1.9325	38
ResNet50+CBAM [1]	95.38	26.29	1.9327	38
ResNet50+HFAM [1]	95.86	25.58	1.8556	38
Inception V3 [62]	94.97	45.37	2.4356	21
Contourlet CNN [73]	96.87	12.6	1.0583	35
SPG-GAN [76]	$94.53$	87.36	2.1322	29
TSAN [77]	$92.16$	381.67	3.2531	32
Proposed	97.31	12.216	1.2735	45

, the details of the different models and our proposed model are listed. From Table 5, we can see the following:

• When the classification accuracy of most models reaches 90%, they have a large number of parameters. Specifically, the OA of ResNet50+SE [1] is 95.84%, the parameter size is $26.28$ M, the OA of Contourlet CNN [73] is 96.87%, the parameter size is $12.6$ M, and our classification accuracy reaches 97.31%, but the parameters are smaller than theirs.
• Compared with ResNet50+CBAM [1], VGG-VD-16 [10], and ResNet50+HFAM [1], our model decreased by 0.6592, 6.4765, and 0.5825, respectively, in the GMAC metric. The experimental results show that our model achieved better results in the above metrics, and the validity of our model is verified once again.

4.4. Ablation Experiment

Taking the AID with a training ratio of 20% as an example, we evaluated the impact of each module on the accuracy of model classification. Starting from global feature extraction, we added local feature extraction, channel attention, spatial attention, and global-local fusion modules. The experimental results are shown in

Table 6. Component ablation experiment results on AID.

Component	OA
Global Feature Extraction	$88.65\%$
+Local Feature Extraction	$90.25\%$
+Channel Attention	$91.07\%$
+Spatial Attention	$92.26\%$
+Fusion Module	$95.09\%$

. From Table 6, we find that after adding the local feature extraction module, the model improved by 1.6% compared to utilizing only global features. The classification accuracy utilizing dual attention mechanisms (i.e., channel attention and spatial attention) is improved compared to utilizing one attention mechanism. The experimental results also verified the effectiveness of the above modules.

To verify the performance of the global-local fusion module at different stages, we also conducted ablation experiments, and the experimental results are shown in

Table 7. Stage fusion ablation experiment results on AID.

Model	Stage 1	Stage 2	Stage 3	Stage 4	OA
Ours				√	90.32%
			√	√	91.15%
		√	√	√	93.27%
	√	√	√	√	95.09%

. If fusion is only used in the fourth stage, the classification accuracy obtained is lower. From the experimental results, we find that the more stages of fusion there are, the better the classification accuracy. The experimental results have verified the effectiveness of our model fusion mechanism, which can more comprehensively integrate global and local features, and consider the channel attention mechanism and spatial attention mechanism, effectively improving the accuracy of scene classification.

4.5. Visual Comparison of Different Attention Mechanisms

To further verify that our proposed attention mechanism can effectively capture feature information in the scene, we chose different attention mechanisms for comparison. Taking the AID with a training ratio of 20% as an example, the experimental results are shown in Figure 8. From Figure 8, we find that our proposed attention mechanism is more capable of obtaining features of key regions in the scene compared to other attention mechanisms and more accurately covers the key regions. The experimental results show that our proposed model can better integrate global and local features and help the model focus on more important regions, which is beneficial for the model to extract more discriminative features.

5. Conclusions

In this study, we propose a novel framework model for remote sensing scene classification, which mainly includes three branches: global extraction module, local extraction module, and global-local fusion module. The global and local extraction modules we designed can efficiently extract multi-scale features and combine channel attention and spatial attention to effectively focus on key regions in the scene. In addition, to improve computational performance, the model adopts parallel dilated convolution, effectively increasing the receptive field and reducing the number of feature parameters. A large number of experimental results have verified the effectiveness of our model, which can effectively distinguish different scenarios. Our model has significant improvement compared with other models. Taking AID data as examples, the classification accuracy can be improved by 8.4% compared with the traditional model, and it also improves by 1.93% compared with other attention mechanisms. In addition, our model has also decreased by $48.754 \mathrm{M}$ compared to other models in terms of parameter quantity and other indicators. We believe that this research work can provide basic services for other applications in remote sensing image interpretation. In future research, we will explore the fusion model of Lie Group space learning with feature extraction, attention mechanism, and other related methods to further improve the classification accuracy and reduce the complexity of the model.