Multi-Output Network Combining GNN and CNN for Remote Sensing Scene Classification

Abstract

Scene classification is an active research area in the remote sensing (RS) domain. Some categories of RS scenes, such as medium residential and dense residential scenes, would contain the same type of geographical objects but have various spatial distributions among these objects. The adjacency and disjointness relationships among geographical objects are normally neglected by existing RS scene classification methods using convolutional neural networks (CNNs). In this study, a multi-output network (MopNet) combining a graph neural network (GNN) and a CNN is proposed for RS scene classification with a joint loss. In a candidate RS image for scene classification, superpixel regions are constructed through image segmentation and are represented as graph nodes, while graph edges between nodes are created according to the spatial adjacency among corresponding superpixel regions. A training strategy of a jointly learning CNN and GNN is adopted in the MopNet. Through the message propagation mechanism of MopNet, spatial and topological relationships imbedded in the edges of graphs are employed. The parameters of the CNN and GNN in MopNet are updated simultaneously with the guidance of a joint loss via the backpropagation mechanism. Experimental results on the OPTIMAL-31 and aerial image dataset (AID) datasets show that the proposed MopNet combining a graph convolutional network (GCN) or graph attention network (GAT) and ResNet50 achieves state-of-the-art accuracy. The overall accuracy obtained on OPTIMAL-31 is 96.06% and those on AID are 95.53% and 97.11% under training ratios of 20% and 50%, respectively. Spatial and topological relationships imbedded in RS images are helpful for improving the performance of scene classification.

1. Introduction

Remote sensing (RS) image scene classification is an active research area essential for image understanding at the scene level [1]. RS image scene classification focuses on categorizing images using semantic information. The task of RS image scene classification aims to discriminate images of various categories in an image dataset, giving one label to each scene image [1]. Scene classification is different from image segmentation or object-oriented classification, giving one label to each pixel or object. The spatial and topological relationships imbedded in RS images are invariant to image transformation, such as panning, distortion, or rotation, and are valuable for discriminating various categories of RS scenes. The adjacency and the disjointness relationships between objects are two types of commonly used spatial and topological dependency [2]. Some RS scene images contain the same type of geographical entities but have various spatial distributions among these entities and thus have different scene semantic labels. As shown in Figure 1, sparse residential, medium residential, and dense residential scenes consist of buildings, roads, and vegetation but have various adjacency or disjointness relationships among the entities.

Existing methods of RS image scene classification usually neglect or do not exploit the spatial and topological relationships well. In earlier studies, handcrafted features, such as the color, texture, or bag-of-visual words, were widely applied in methods of RS image scene classification [3,4]. Handcrafted features typically require sufficient expert information to be designed or extracted and are not robust enough to various application scenarios [1]. Handcrafted features are usually inadequate in representing high-level semantics of RS images. It is difficult for handcrafted features to achieve promising scene classification results even when used with spatial and topological relationships [1].

In recent years, deep learning methods, such as convolutional neural networks (CNNs), have been commonly applied for RS image scene classification [5,6]. CNNs have a powerful capability to adaptively learn distinctive high-level semantic characteristics [1,7,8,9,10,11]. A bundle of deep CNNs with diverse layouts of convolutional layers, pooling layers, and fully connected layers have been employed in scene classification, such as Visual Geometry Group (VGG)Net [12], Residual(Res)Net [13], and Densely Connected (Dense)Net [14], since AlexNet [15] obtained astonishing success. In recent years, CNN-based architectures have been used to improve classification accuracy [16,17,18,19,20,21,22]. For example, a dual-model architecture by combining two CNN-based branches, ResNet and DenseNet, with a global-attention-fusion strategy was designed in a recent study [23], which was proven effective by gaining higher accuracies than single-model architectures. CNNs use regularly shaped convolutional layers and pooling layers. Thus, it is difficult for CNN-based methods to exploit the spatial and topological relationships imbedded in RS images well [24].

Graph neural networks (GNNs) can implicitly learn the spatial and topological relationships for graphs [25,26,27,28,29], which are non-Euclidean data structures. A critical strategy for exploiting GNNs on RS image scene classification is to transform the image classification task into a graph classification task. Specifically, a graph that consists of nodes and edges is first constructed to express the regularly shaped RS image. The spatial and topological relationships among objects in the RS image are incorporated into the edges of the graph. Graphs expressing RS scene images are then taken as the inputs of the GNN, such as the graph convolutional network (GCN) [26] and graph attention network (GAT) [30], to be labeled as a specified category. The features of corresponding adjacent nodes of graphs are aggregated via the message propagation mechanisms of GNNs. In this way, the spatial and topological relationships imbedded in the RS image are exploited by the GNN-based methods of RS image scene classification, leading to improved scene classification performance.

For an RS image, a scene graph can be constructed in several ways. Yang et al. [31] constructed scene graphs by defining the adjacency relationship between pixels according to their distances in spectral and spatial domains. However, the method may lead to the generation of very large and complex graphs, which may cause a computing burden and negatively affect graph classification accuracy. Liang et al. [32] regarded ground entities detected as nodes with faster region-based CNN (R-CNN), such as planes and ground track fields. Then, the contiguous relationship among nodes is defined according to the spatial distances between these entities. As additional labeled data are required to train the Faster-RCNN, the low-accuracy and limited-efficiency ground entity detection may affect the classification result. Li et al. [24] adopted an unsupervised segmentation algorithm for an image to generate a number of superpixels, which are a group of pixels that share common color, textural or intensity characteristics [33]. In [24], these superpixels were regarded as graph nodes, while graph edges between nodes were created according to the spatial adjacency among corresponding superpixels. The performance of the method in [24] is greatly influenced by the accuracy of the image segmentation step.

GNNs are usually combined with CNNs to perform RS image scene classification to make use of the strength of both on feature representation. One strategy of combining CNNs and GCNs is to treat CNNs as the feature extractor for the nodes of graphs, taking advantage of the outstanding visual feature representation ability of CNNs. Li et al. [24] used feature maps generated from a pretrained CNN model to initialize node features for scene graphs and then performed graph classification for these graphs with a trained GNN model. For this strategy, the representation quality of node features extracted by the pretrained CNN has a substantial impact on the performance of graph classification. Furthermore, some RS image scenes, such as deserts and oceans, may have inaccurate constructed scene graphs due to the lack of salient regions and obvious topological dependencies and, thus, it is difficult for them to effectively perform scene classification. It is difficult for this strategy to achieve promising RS image scene classification for these scenes of RS images.

Another strategy of combining CNNs and GNNs for RS image scene classification is to fuse two global feature vectors produced separately by the CNN and the GNN. In [32], features from GCN and VGG16 were elementwise added up and then delivered to a final classifier to yield the final logits for scene classification. In this strategy, the original GNN output is replaced by a feature-fused output, which is a worthwhile modification compared with the previously mentioned strategy. However, as the single prediction output generated from the fused feature vector is considered the optimized objective, the method may have insufficient robustness. Inspired by the success of recent dual-model architectures [23,34], this strategy can be improved by regarding multi-output predictions as the optimized objective to achieve both high accuracy and great robustness.

In this study, a multi-output network (MopNet) combining GNN and CNN was proposed for RS scene classification, exploiting the spatial and topological relationships imbedded in RS images. A training strategy of jointly learning CNN and GNN was adopted in MopNet. The parameters of the CNN and GNN in MopNet were updated simultaneously with the guidance of a joint loss via the backpropagation mechanism. Experimental results on two datasets illustrated that the proposed MopNet achieved promising performance.

2. Methods

The architecture of our multi-output network (MopNet) combining GNN and CNN for RS scene classification is shown in Figure 2. A graph is created for an RS image through graph structure construction. For the graph, nodes are denoted as corresponding superpixel regions obtained from RS images through image segmentation. The CNN-based branch in MopNet plays the role of a feature extractor for initializing features of graph nodes, and it functions as an encoder of global visual feature representation for RS images. The GNN-based branch in MopNet contributes to aggregating local features into global graph representations by employing the spatial and topological relationships implicit in graph edges. MopNet utilizes two classifiers to output GNN logits and CNN logits. The learnable parameters of the MopNet are optimized by minimizing the joint loss during the training period. In the inference stage, the final prediction of the input scene image is obtained from the GNN-based branch of MopNet. Details of the MopNet are elaborated in the remaining parts of this section.

2.1. Graph Structure Construction

As seen in Figure 3, the graph structure construction procedure abstracts graph-structured information from a regular-shaped image for an RS image. A number of superpixel regions are first obtained from RS images through unsupervised image segmentation. These superpixels are regarded as graph nodes. Graph edges between nodes are created according to the spatial adjacency between corresponding superpixel regions. In this way, each RS image is transformed into a graph.

Mathematically, the graph of an RS image is symbolized as $G=\left(V,E\right)$, where $V$ is the set of nodes representing entities, and $E$ is the set of edges representing the relationships between entities. A graph can be formed from an adjacency matrix $A$, denoting whether an edge between two corresponding nodes exists or not. For an RS image, the adjacency matrix of a graph is symbolized as $A={\left[a_{ij}\right]}_{n\times n}$, where $n$ is the total number of superpixel regions obtained by the unsupervised segmentation method, and $a_{ij}$ is defined as:

$$a_{ij}=\left\{\begin{array}{l}1,if{r}_{i}and{r}_{j}shareaboundary\\ 0,else\end{array},\right.$$

(1)

where $r_i$ and $r_j$ are two superpixel regions.

2.2. Node Feature Matrix Initialization

After the graph structure has been constructed, the node feature matrix is initialized and used for graph classification. The CNN-based branch of MopNet is used as a feature extractor to yield the feature map to initialize the node features. Typically, the CNN generates multiple feature maps with varying spatial sizes and different numbers of channels. The feature map yields from the intermediate block, such as the block “conv4_x” of ResNet [13], and is selected to initialize the node features. As the spatial size of feature maps decreases, feature maps closer to the output layer contain high-level semantic information but have fewer details, while feature maps closer to the input layer have more detailed features that contain low-level semantic information with some noise [32].

The selected feature map is copied and upsampled to the size of the input RS image. By laying the boundary of superpixel regions over the feature map, each node in the scene graph is assigned the maximum feature value of the corresponding superpixel region. The node feature matrix is symbolized as $H={\left[h_{i,j}\right]}_{n\times d}$, where $h_{i,j}$ is the $j$th feature value of the $i$th node, and $d$ is the length of node features. $h_{i,j}$ is calculated as:

$$h_{i,j}=\underset{\left(p,q\right)\in r_i}{\max }F{M}_{j,\left(p,q\right)},$$

(2)

where $r_i$ denotes the $i$th superpixel region, i.e., the $i$th node. ($p,q)$ denotes the location of the pixel in $r_i$, and $F{M}_{j,\left(p,q\right)}$ is the value of the upsampled feature map in the $j$th channel at the location ($p,q)$.

2.3. Encoders and Classifiers in MopNet

MopNet has two encoders, a CNN encoder and a GNN encoder, and their corresponding classifiers. The CNN encoder obtains global visual features for RS images. The GNN encoder obtains global graph representations for scene graphs constructed from RS images.

2.3.1. Global Visual Feature Vector Extraction from the CNN Encoder

For an RS image, two global visual feature vectors are obtained from the CNN encoder using different feature maps of two different blocks. Specifically, feature maps can be obtained via a specified convolutional block of the CNN. Then, global average pooling (GAP) [35] with a fixed output size set at 1 is stacked over these feature maps to calculate global visual feature vectors. The global visual feature vector is symbolized as ${\mathit{x}}^{\left(b\right)}\in {ℝ}^{d^{\left(b\right)}}$, where $d^{\left(b\right)}$ is the number of feature channels of the feature map generated from the $b$th block of the CNN. ${\mathit{x}}^{\left(b\right)}$ is calculated as:

$${\mathit{x}}^{\left(b\right)}=\frac{1}{\left|R^{\left(b\right)}\right|}{\displaystyle \sum }_{\left(p,q\right)\in R^{\left(b\right)}}F{M}_{\left(p,q\right)}^{\left(b\right)}$$

(3)

where $R^{\left(b\right)}$ denotes the corresponding spatial range of the $b$th feature map, ($p,q)$ denotes the location of the pixel in $R^{\left(b\right)}$, $\left|R^b\right|$ is the total number of pixels in $R^{\left(b\right)}$, and $F{M}_{\left(p,q\right)}^{\left(b\right)}$ is the feature vector with the length of $d^{\left(b\right)}$ and filtered from the $b$th feature map at the location ($p,q)$.

Two global visual feature vectors obtained from different feature maps of the CNN are separately used in the two classifiers of MopNet. One is generated from the aforementioned intermediate feature map for initializing node features of scene graphs, denoted as ${\mathit{x}}^{\left(mb\right)}$ in Section 2.3.3, and is later fused with the global graph representation to be fed into a classifier for the GNN output. The other one is generated from the last feature map, denoted as ${\mathit{x}}^{\left(lb\right)}$ in Section 2.3.3, and is fed into a classifier for the CNN output.

2.3.2. Global Graph Representations from the GNN Encoder

In this paper, two GNN models, the graph convolutional network (GCN) [26] and graph attention network (GAT) [30], are adopted as two different GNN backbones to aggregate local node features of scene graphs generated from RS images. GCN aggregates the node information from neighboring nodes to yield a new feature representation.

Mathematically, ${\mathit{h}}_i^{\left(l\right)}\in {ℝ}^{d^{\left(l\right)}}$ is the feature vector of node $v_i$, where $d^{\left(l\right)}$ is the number of output channels of the $l$th GNN layer. Thus, the new feature representation ${\mathit{h}}_i^{\left(l+1\right)}\in {ℝ}^{d^{\left(l+1\right)}}$ of $v_i$ is generated by graph convolution as:

$${\mathit{h}}_i^{\left(l+1\right)}=\sigma \left({\displaystyle \sum }_{v_j\in \tilde{N}\left(v_i\right)}\frac{1}{{\beta }_{ji}}W{\mathit{h}}_j^{\left(l\right)}\right),$$

(4)

where $\sigma$ is the activation function, $W\in {ℝ}^{d^{\left(l+1\right)}\times d^{\left(l\right)}}$ is the learnable weight matrix, and $\tilde{N}\left(v_i\right)$ is the set of neighbors of node $v_i$. ${\mathit{h}}_j^{\left(l\right)}$ denotes the feature vector of $v_j$ at the $l$th GNN layer. ${\beta }_{ji}$ is the product of the square root of node degrees, i.e., ${\beta }_{ji}=\sqrt{\left|\tilde{N}\left(v_j\right)\right|}\sqrt{\left|\tilde{N}\left(v_i\right)\right|}$.

GAT learns edge weights that reflect to what extent different neighbors contribute to the new representation via a multi-head attention mechanism in an adaptive learning way when aggregating neighboring node features [30]. Correspondingly, ${\mathit{h}}_i^{\left(l+1\right)}$ in GAT is calculated as follows:

$${\mathit{h}}_i^{\left(l+1\right)}={\parallel }_{t=1}^T\sigma \left({\displaystyle \sum }_{v_j\in \tilde{N}\left(v_i\right)}{\alpha }_{ij}^{\left(t\right)}{W}^{\left(t\right)}{\mathit{h}}_j^{\left(l\right)}\right),$$

(5)

where $\parallel$ is the concatenation operation and $T$ is the number of attention heads. ${\alpha }_{ij}$ denotes the normalized weight coefficient between $v_i$ and $v_j$ and is calculated as:

$${\alpha }_{ij}=softma{x}_j\left(e_{ij}\right),$$

(6)

where $e_{ij}$ is the weight coefficient between $v_i$ and its neighbor $v_j$. $e_{ij}$ is calculated via a fully connected layer as:

$$e_{ij}=LeakyReLU\left(f_{2d^{\left(l+1\right)}}\left(\left[W{\mathit{h}}_i^{\left(l\right)}\parallel W{\mathit{h}}_i^{\left(l\right)}\right]\right)\right),$$

(7)

where $f_{2d^{\left(l+1\right)}}\left(·\right)$ is the fully connected layer with $2d^{\left(l+1\right)}$ output channels. By integrating Equations (6) and (7), ${\alpha }_{ij}$ is completely written as:

$${\alpha }_{ij}=\frac{\exp \left(LeakyReLU\left(f_{2d^{\left(l+1\right)}}\left(\left[W{\mathit{h}}_i\parallel W{\mathit{h}}_j\right]\right)\right)\right)}{{{\displaystyle \sum }}_{v_j\in \tilde{N}\left(v_i\right)}\exp \left(LeakyReLU\left(f_{2d^{\left(l+1\right)}}\left(\left[W{\mathit{h}}_i\parallel W{\mathit{h}}_j\right]\right)\right)\right)},$$

(8)

Global graph pooling is used to generate global graph representations for the scene graph after aggregating node features by graph convolutional layers or graph attention layers. A weight-and-sum pooling layer [36] is used to transform local node features into global graph features for the classification task. The global graph feature vector ${\mathit{h}}_{global}\in {ℝ}^{d^{\left(-1\right)}}$, where $d^{\left(-1\right)}$ is the number of output channels of the final GNN layer, is calculated as:

$${\mathit{h}}_{global}={\displaystyle \sum }_{i=1}^{n}sigmoid\left(f_1\left({\mathit{h}}_i^{\left(-1\right)}\right)\right){\mathit{h}}_i^{\left(-1\right)},$$

(9)

where $f_1\left(·\right)$ represents a linear transform layer with one output channel and ${\mathit{h}}_i^{\left(-1\right)}\in {ℝ}^{d^{\left(-1\right)}}$ represents the feature vector of $v_i$ output from the final GNN layer.

2.3.3. Multi-Output Logits of MopNet from Classifiers

The MopNet has two classifiers: a classifier with a multilayer perceptron (MLP) for the GNN output and a classifier with a fully connected layer for the CNN output. A concatenation operation is adopted to fuse the global graph feature representation $hg$ with the global visual feature vector generated from the intermediate feature map. The newly generated feature vector is fed into a two-layer MLP to produce the GNN output. The GNN output ${\mathit{o}}_{gnn}$ is calculated as:

$${\mathit{o}}_{gnn}=MLP\left(\left[{\mathit{x}}^{\left(mb\right)}\parallel {\mathit{h}}_{global}\right]\right),$$

(10)

where ${\mathit{x}}^{mb}$ is the global feature vector generated from the intermediate feature map selected for initializing the node feature matrix.

Additionally, the global feature vector ${\mathit{x}}^{\left(lb\right)}$ generated from the last feature map of CNN is fed into a fully connected layer to produce the logits as the CNN output ${\mathit{o}}_{cnn}$, which is calculated as:

$${\mathit{o}}_{cnn}=f_K\left({\mathit{x}}^{\left(lb\right)}\right),$$

(11)

where $K$ is the number of scene classes and $f_K\left(·\right)$ denotes a fully connected layer with $K$ output channels.

2.4. Loss Function of MopNet

The MopNet is trained with the joint loss $L_J$. $L_J$ is calculated by integrating the GNN loss $L_{gnn}$ and the CNN loss $L_{cnn}$ with a balance parameter $\lambda \in \left(0,1\right)$ as:

$$L_J\left(\theta \right)=\lambda L_{gnn}+\left(1-\lambda \right)L_{cnn},$$

(12)

where $\theta$ denotes model parameters, including GNN parameters ${\theta }_{gnn}$, CNN parameters ${\theta }_{cnn}$, and parameters of classifiers ${\theta }_{cl}$. $L_{gnn}$ and $L_{cnn}$ are separately calculated via a cross-entropy function. In this paper, label smoothing [37] is further adopted to reduce overfitting effects and improve prediction accuracy. The joint loss $L_J$ is rewritten as:

$$L_J\left(\theta \right)=\lambda L\left({\mathit{y}}^{\prime },{\mathit{o}}_{gnn}\right)+\left(1-\lambda \right)L\left(\mathit{y}’,{\mathit{o}}_{cnn}\right),$$

(13)

where $\mathit{y}’$ denotes the smoothed label, calculated from one-hot label $\mathit{y}$ according to [37]. $L\left({\mathit{y}}^{\prime },o_{gnn}\right)$ and $L\left(\mathit{y}’,{\mathit{o}}_{cnn}\right)$ are separately calculated as:

$$L\left(\mathit{y}’,{\mathit{o}}_{gnn}\right)={\displaystyle \sum }_{i=1}^K-\mathit{y}{’}_i\log \left(\frac{e^{{\mathit{o}}_{gnn}{}^T{w}_{gnn}^i}}{{{\displaystyle \sum }}_{j=1}^K{e}^{{\mathit{o}}_{gnn}{}^T{w}_{gnn}^j}}\right),$$

(14)

$$L\left(\mathit{y}’,{\mathit{o}}_{cnn}\right)={\displaystyle \sum }_{i=1}^K-\mathit{y}{’}_i\log \left(\frac{e^{{\mathit{o}}_{cnn}{}^T{w}_{cnn}^i}}{{{\displaystyle \sum }}_{j=1}^K{e}^{{\mathit{o}}_{cnn}{}^T{w}_{gnn}^j}}\right)$$

(15)

where $K$ denotes the number of scene classes and $w$ represents the weights and bias of the last layer.

In our MopNet, learnable parameters of the GNN and the CNN are simultaneously optimized by backward propagation during the training process of the model. The training process of MopNet is depicted in Algorithm 1. The adjacent matrix $A$ and the superpixel regions $R$ are obtained during graph structure construction, which is performed before the training process of the MopNet.

Algorithm 1. Training process of MopNet.

Input: RS images I, adjacent matrix A, segmented superpixel regions R and true labels y in training set. Output: GNN parameters ${\theta }_{gnn}$, CNN parameters ${\theta }_{cnn}$ and parameters of classifiers ${\theta }_{cl}$. Learning MopNet: for epoch = 1, 2, … do; Take I as input, calculate feature maps from the intermediate block mb, calculate feature maps from the last block lb; Take R as input, overlap R on the feature maps from the block mb, and initialize node feature matrix H according to Equation (2); Take A as input, update H with A using graph convolutional layers according to Equation (4) or using graph attention layers according to Equations (5)–(8); Produce global visual features ${\mathit{x}}^{\left(mb\right)}$ and $\mathrm{and}{\mathit{x}}^{\left(lb\right)}$ according to Equation (3); Produce global visual features ${\mathit{h}}_{global}$ according to Equation (9); Calculate GNN logits ${\mathit{o}}_{gnn}$ according to Equation (10); Calculate GNN logits ${\mathit{o}}_{cnn}$ according to Equation (11); Calculate the joint loss based on y according to Equations (12)–(15); Update ${\theta }_{gnn}$, ${\theta }_{cnn}$ and ${\theta }_{cl}$ via backward propagation; end for

3. Experiments and Results

The results of our experiments are presented and analyzed in this section. A description of the datasets used in this paper can be found in Section 3.1. The experimental settings are then described in Section 3.2. In Section 3.3, the experimental results on two public datasets are provided and compared with the results of the state-of-the-art methods.

3.1. Experimental Data Sets

Two RS image scene classification benchmark datasets were employed to test the effectiveness of the MopNet. The characteristics of the two used datasets are listed in

Table 1. Characteristics of the two RS image scene classification datasets used in this study.

Dataset	No. of Classes	Total Number of Images	No. of Images per Class	Image Size (in pixels)	Training Ratio
OPTIMAL-31 [38]	31	1860	60	$256\times$ 256	80%
Aerial Image Dataset (AID) [39]	30	1000	220–420	$600\times$ 600	20%, 50%

.

Some example images of the two datasets are shown in Figure 4 and Figure 5.

3.2. Experimental Settings

For MopNet, GAT and GCN were separately used as the GNN backbone, while ResNet50 was used as the CNN backbone. The MopNet without the CNN prediction, identified as SopNet, was also compared with the MopNet. Specifically, the following four experiments were conducted in our study: MopNet-GAT-ResNet50, MopNet-GCN-ResNet50, SopNet-GAT-ResNet50, and SopNet-GCN-ResNet50.

Some parameters in our method were set as follows. Felzenszwalb’s algorithm [40] was applied to generate superpixel regions from RS images in the graph structure construction, with the segmentation scale set as 50 and 300 for the OPTIMAL-31 and AID datasets, respectively. The GCN encoder used had two graph convolutional layers, while the GAT encoder used had two graph attention convolutional layers with eight attention heads. In these two GNN encoders, the hidden dimension and output dimension were both 512. The feature map from block “conv4_x” of ResNet50 was chosen for initializing node feature matrixes for scene graphs. All the experiments were trained for 100 epochs. Adam was used as the optimizer, and the learning rate was 1e-3. The mini-batch sizes were 16 and 8 for the OPTIMAL-31 and AID datasets, respectively.

Our model was implemented using PyTorch and the Deep Graph Library (DGL) [41]. All experiments were conducted on a personal computer equipped with an Intel core i9-10900K CPU and NVIDIA GeForce RTX 3090 GPU with 24 GB of memory.

3.3. Scene Classification Results

3.3.1. Results under Different Balance Parameters

To determine how the balance parameter $\lambda$ of the joint loss affects the classification results, different $\lambda$ values were set in MopNet-GAT-ResNet50 on the OPTIMAL-31 dataset.

Table 2. Overall accuracy on the OPTIMAL-31 dataset with different balance parameters of the joint loss by MopNet-GAT-ResNet50.

$\mathit{\lambda }$	Overall Accuracy (%)
0.1	95.16 ± 0.47
0.2	95.32 ± 0.46
0.3	95.34 ± 0.31
0.4	95.03 ± 0.57
0.5	95.16 ± 0.47
0.6	95.30 ± 0.55
0.7	96.06 ± 0.31
0.8	95.16 ± 0.38
0.9	94.62 ± 0.26

shows the results when $\lambda$ increased from 0.1 to 0.9. In general, as $\lambda$ increased, the accuracy first fluctuated and then decreased. The highest overall accuracy (OA) of 96.06% was obtained when $\lambda$ was 0.7. This indicates that both GNN-based and CNN-based branches contributed to improving the scene classification performance of MopNet but did not contribute equally. The GNN-based or CNN-based branch should not be given a fully dominant weight. In the following experiments, $\lambda$ was set as 0.7.

3.3.2. Results of MopNet on the OPTIMAL-31 Dataset

Quantitative results are presented for the OPTIMAL-31 dataset in

Table 3. Comparison of OAs (%) on the OPTIMAL-31 dataset.

Methods	OA (80% Training)
Fine-tuning ResNet50 [43]	90.46 ± 0.38
ARCNet-ResNet [38]	91.28 ± 0.45
MSNet [44]	93.92 ± 0.41
EfficientNet-B3-aux [45]	94.51 ± 0.75
ResNet_LGFFE [43]	94.55 ± 0.36
IDCCP with ResNet50-512 [46]	94.89 ± 0.22
Vision transformer [42]	95.56 ± 0.18
DM-GAF [23]	96.24 ± 1.10
SopNet-GCN-ResNet50	93.37 ± 0.68
MopNet-GCN-ResNet50 (Ours)	95.34 ± 0.31
SopNet-GAT-ResNet50	93.55 ± 0.71
MopNet-GAT-ResNet50 (Ours)	96.06 ± 0.31

. OA values of 96.06% and 95.34% were achieved by MopNet-GAT-ResNet50 and MopNet-GCN-ResNet50, outperforming the fine-tuned ResNet50 with increases of 5.60% and 4.88%, respectively. Generally, MopNet-GAT-ResNet50 had a higher OA and lower standard deviations than MopNet-GCN-ResNet50, indicating better performance and the robustness of using GAT as the GNN backbone on the OPTIMAL-31 dataset. Nevertheless, both MopNet-GAT-ResNet50 and MopNet-GCN-ResNet50 achieved better results than most of the compared methods in Table 3. Notably, our MopNet-GAT-ResNet50 obtained a higher accuracy than the method in [42] using the vision transformer, which has achieved remarkable performances on classification tasks in recent years.

The confusion matrix generated by MopNet-GAT-ResNet50 on the OPTIMAL-31 dataset is presented in Figure 6. It can be seen that 28 out of 31 categories had scores higher than 90%, with 21 categories having scores of 100%. MopNet-GAT-ResNet50 had a better ability to discriminate the classes that are easily misclassified by other methods, such as DM-GAF [23]. The commercial area, dense residential area, and industrial area were classified by our model with accuracies of 92%, 83%, and 100%, respectively, while these three classes only obtained accuracies of 83%, 67%, and 83%, respectively, by DM-GAF [23].

3.3.3. Results of MopNet on the AID Dataset

A comparison of the results on the AID dataset is presented in

Table 4. Comparison of OAs (%) of different methods on the AID dataset.

Methods	OA (20%)	OA (50%)
Fine-tuning ResNet50 [43]	86.48 ± 0.49	89.22 ± 0.34
ResNet_LGFFE [43]	90.83 ± 0.55	94.46 ± 0.48
ACNET [48]	93.33 ± 0.29	95.38 ± 0.29
DM-GAF [23]	94.05 ± 0.10	96.12 ± 0.14
EfficientNet-B3-aux [45]	94.19 ± 0.15	96.56 ± 0.14
SFCNN [47]	93.60 ± 0.12	96.66 ± 0.11
Combined CNN with GCN [32]	94.93 ± 0.31	96.70 ± 0.28
DCNN [10]	90.82 ± 0.16	96.89 ± 0.10
IDCCP with ResNet50-512 [46]	94.80 ± 0.18	96.95 ± 0.13
MSNet [44]	95.59 ± 0.15	96.97 ± 0.27
GLDBS [34]	95.45 ± 0.19	97.01 ± 0.22
SopNet-GCN-ResNet50	89.06 ± 0.39	93.56 ± 0.28
MopNet-GCN-ResNet50 (Ours)	95.53 ± 0.11	97.11 ± 0.07
SopNet-GAT-ResNet50	91.26 ± 0.43	95.16 ± 0.16
MopNet-GAT-ResNet50 (Ours)	95.16 ± 0.16	96.75 ± 0.11

. Our MopNet had a comparable or even better performance on the AID dataset than the methods presented in Table 4. On the AID dataset, MopNet-GCN-ResNet50 achieved a higher OA than MopNet-GAT-ResNet50 under training ratios of 20% and 50%, with OA values of 95.53% and 97.11%, respectively. Our MopNet obtained much better results than DM-GAF on the AID dataset, although our MopNet had a slightly lower OA than DM-GAF on the OPTIMAL dataset. In addition, our MopNet obtained greater improvements under a training ratio of 20% than under a training ratio of 50% compared with methods such as SFCNN [47], DCNN [10], and the method combining CNN with GCN [32]. This phenomenon indicates that our MopNet is robust for scene classification, especially when the training set is small.

Figure 7 and Figure 8 show the confusion matrixes generated by our MopNet-GCN-ResNet50 on the AID dataset under training ratios of 20% and 50%, respectively. The number of categories with classification accuracies higher than 95% was 22 and 24 under the training ratios of 20% and 50%, respectively. The most obvious misclassification occurred between the categories of resort and park, and between the categories of school and commercial area. Taking the results under the training ratio of 20% as an example, 4% of the images from the resort were misclassified as parks, and 3% of the images from the school were misclassified as commercial. Similar confusion also occurred for IDCCP [46] and the method combining GCN with CNN [32]. Nevertheless, our MopNet performed better in classifying these confusable categories than these two methods. Specifically, for IDCCP [46], 16% of images from the resort were classified as parks, and 6% of images from the school were classified as commercial. For the method combining a GCN with a CNN [32], 6% of the images from the resort were classified as parks, and 3% of the images from the school were classified as commercial.

3.3.4. Comparison Results of MopNet and SopNet

As seen from the experimental results, our MopNet obtained higher OAs with lower standard deviations than the corresponding SopNet. Specifically, as shown in Table 3, on the OPTIMAL-31 dataset, our MopNet with GCN and GAT as the GNN backbones obtained 1.97% and 2.69% improvements in OA compared to SopNet, respectively. According to Table 4, on the AID dataset, MopNet with GCN obtained 6.55% and 3.55% improvements in OA under the 20% and 50% training ratios compared to SopNet, respectively, while MopNet with GAT obtained 3.9% and 1.59% improvements. Moreover, MopNet obtained lower standard deviations of OAs than the corresponding SopNet.

The classification accuracies obtained by MopNet and SopNet for some typical categories are presented in Figure 9, Figure 10 and Figure 11. Some categories with complex spatial and topological relations among ground objects in the scene images were selected for comparison, such as church, commercial area, industrial area, center, school, square, and residential area. Some categories with no salient spatial topological dependences contained in the scene images, such as desert, lake, and bare land, were also selected for comparison. As seen in Figure 9, Figure 10 and Figure 11, MopNet usually obtained higher classification accuracies than the corresponding SopNet for these categories on the two datasets.

4. Discussion

4.1. Effectiveness of MopNet for Various Categories

The proposed MopNet had universally promising performance for various categories of RS scenes. As seen in Figure 6, Figure 7 and Figure 8, our MopNet had a robust capability in classifying various categories of scenes, although different categories of scenes usually have various geographical entities and spatial distributions among these entities. The number of graph nodes generated from an RS image is related to the number of geographical entities in these images. The average and the variance of graph node counts of various categories are presented in

Table 5. Node counts of each category in the OPTIMAL-31 dataset.

Class	Node Counts	Class	Node Counts	Class	Node Counts
airplane	131 ± 27	desert	133 ± 54	medium residential	210 ± 28
airport	194 ± 30	forest	197 ± 47	mobile home park	219 ± 26
baseball diamond	135 ± 33	freeway	162 ± 35	mountain	204 ± 49
basketball court	181 ± 38	golf course	135 ± 28	overpass	179 ± 33
beach	129 ± 38	ground track field	189 ± 37	parking lot	192 ± 23
bridge	120 ± 31	harbor	163 ± 24	railway	194 ± 32
chaparral	185 ± 37	industrial	226 ± 27	rectangular farmland	135 ± 47
church	198 ± 31	intersection	201 ± 22	roundabout	193 ± 26
circular farmland	154 ± 42	island	96 ± 30	runway	112 ± 28
commercial	212 ± 24	lake	147 ± 49
dense residential	243 ± 16	meadow	93 ± 33

and

Table 6. Node counts of each category in the AID dataset.

Class	Node Counts	Class	Node Counts	Class	Node Counts
airport	123 ± 36	farmland	70 ± 34	port	124 ± 53
bare land	46 ± 26	forest	74 ± 37	railway station	142 ± 50
baseball field	108 ± 29	industrial	175 ± 53	resort	153 ± 57
beach	55 ± 33	meadow	24 ± 15	river	72 ± 30
bridge	73 ± 31	medium residential	159 ± 45	school	190 ± 53
center	158 ± 51	mountain	105 ± 49	sparse residential	101 ± 21
church	204 ± 36	park	132 ± 42	square	155 ± 51
commercial	170 ± 52	parking	211 ± 41	stadium	163 ± 44
dense residential	228 ± 45	play ground	101 ± 33	storage	159 ± 47
desert	27 ± 21	pond	72 ± 30	viaduct	126 ± 43

. Complex RS scenes, such as dense residential scenes, usually had more graph nodes, while simple RS scenes, such as meadow scenes, usually had fewer graph nodes. For a specified category, the spatial and topological relationships imbedded in images of the category were imbedded in graph edges between corresponding nodes and were employed through the message propagation mechanism of MopNet. In this way, the MopNet was effective at scene classification for various categories.

As seen in Table 5, for the OPTIMAL-31 dataset, the categories of commercial, dense residential, industrial, and medium residential had a larger number of graph nodes. It was difficult for RS scene classification methods to distinguish these categories. Our MopNet obtained accuracies of 92%, 83%, 100%, and 100% for these categories, respectively, which are comparable to or higher than existing methods, such as DM-GAF [23]. In contrast, the category of island generated fewer graph nodes and was relatively easy to classify. For the island, our MopNet obtained a classification accuracy of 100%, while DM-GAF [23] obtained an accuracy of 83%.

As seen in Table 6, for the AID dataset, the commercial, dense residential, and industrial categories had abundant details in the scene images and thus had a large number of graph nodes. For these categories, our MopNet obtained accuracies of 98%, 97%, and 95%, respectively, under the training ratio of 20%, while the results generated by the method combining GCN with CNN [32] were 95%, 91%, and 94%, respectively. In contrast, categories such as desert and meadow had poor textures and thus had the fewest graph nodes. Our MopNet obtained accuracies of 97% and 99% separately for these two categories, while the method in [32] obtained accuracies of 92% and 98%, respectively.

4.2. Contribution of the GNN and CNN in MopNet to Scene Classification

Both the GNN-based and CNN-based branches of MopNet contribute to RS image scene classification but do not contribute equally. The contributions of the GNN and CNN are described as follows:

(1)

The GNN-based branch employs spatial and topological relationships imbedded in RS images, leading to an improved classification accuracy of MopNet. The GNN compensates for the shortcoming of the CNN by representing features in non-Euclidean space. Given that the balance parameter λ was suggested to be set as 0.7 by the comparison experiments, the GNN plays an important role in minimizing the joint loss of the MopNet. Moreover, the contribution of the GNN to MopNet can also be drawn from the comparison between our MopNet and GLDBS [34]. GLDBS [34] used two CNNs as the backbones of two branches, unlike our MopNet, which has a GNN branch beside a CNN branch. It is difficult for models such as GLDBS to effectively learn the spatial and topological information. The experimental results show that our MopNet obtained higher overall accuracies than GLDBS on the AID dataset.

(2)

The CNN-based branch helps to improve the performance and stability of MopNet. As shown in Table 3 and Table 4, MopNet obtained higher overall accuracies (OAs) and lower standard deviations of OAs in comparison to the corresponding SopNet, which did not involve the CNN prediction in the optimized objective of the model. As shown in Figure 9, Figure 10 and Figure 11, MopNet achieved obvious accuracy improvements in comparison to the corresponding SopNet for various categories, such as commercial and industrial in the OPTIMAL-31 dataset and center, resort, school, and square in the AID dataset. SopNet suffers from the effect of uncertainties caused by image segmentation and graph structure construction for images of these categories. For MopNet, the CNN-based branch helps to reduce the effect and thus improve the stability.

4.3. Differences between MopNets and Existing Methods Combining CNN and GNN

The proposed MopNet differs from the existing methods that also combine CNN and GNN [24,32] in terms of the training strategy and the prediction output.

(1)

Training strategy: A training strategy of jointly learning CNN and GNN was adopted in MopNet. The parameters of the CNN and GNN in MopNet are updated simultaneously with the guidance of a joint loss via the backpropagation mechanism. In contrast, the method in [24] adopts the step-by-step training strategy, training the CNN and the GNN separately. The parameters of the CNN and GNN in [24] were not simultaneously optimized as in our MopNet. The classification performance of the GNN in [24] may suffer from the effect of the representation ability of the CNN.

(2)

Prediction output: The experimental results indicate that the MopNet obtained higher classification accuracies with more robustness than the corresponding SopNet that did not involve the CNN output in the optimized objective. Moreover, our MopNet regarded both outputs of the GNN and CNN as the optimized objective, unlike the method in [32] using the GNN prediction alone. Our MopNet obtained better classification results than the method in [32]. In the future, MopNet will be investigated with more CNN backbones and various GNN backbones.

5. Conclusions

In this paper, a new multi-output network (MopNet) combining GNN and CNN is proposed for achieving high-accuracy RS image scene classification. The parameters of the CNN and GNN in MopNet are updated simultaneously with the guidance of a joint loss via the backpropagation mechanism. Experimental results on the OPTIMAL-31 and AID datasets show that the proposed MopNet combining GAT/GCN and ResNet50 has promising overall accuracies, outperforming the state-of-the-art methods. The proposed MopNet is able to achieve higher overall accuracies with lower variances than the corresponding single-output network (SopNet). The CNN-based branch of MopNet reduces the effect of uncertainties caused by image segmentation and graph structure construction for RS images, and thus helps to improve the performance and stability of MopNet. On the other hand, the GNN-based branch employs spatial and topological relationships imbedded in graph edges among geographical objects in RS images and helps to improve the classification accuracy of MopNet. Spatial and topological relationships imbedded in RS images are helpful for improving the performance of scene classification.