HyFormer: Hybrid Grouping-Aggregation Transformer and Wide-Spanning CNN for Hyperspectral Image Super-Resolution

Abstract

Hyperspectral image (HSI) super-resolution is a practical and challenging task as it requires the reconstruction of a large number of spectral bands. Achieving excellent reconstruction results can greatly benefit subsequent downstream tasks. The current mainstream hyperspectral super-resolution methods mainly utilize 3D convolutional neural networks (3D CNN) for design. However, the commonly used small kernel size in 3D CNN limits the model’s receptive field, preventing it from considering a wider range of contextual information. Though the receptive field could be expanded by enlarging the kernel size, it results in a dramatic increase in model parameters. Furthermore, the popular vision transformers designed for natural images are not suitable for processing HSI. This is because HSI exhibits sparsity in the spatial domain, which can lead to significant computational resource waste when using self-attention. In this paper, we design a hybrid architecture called HyFormer, which combines the strengths of CNN and transformer for hyperspectral super-resolution. The transformer branch enables intra-spectra interaction to capture fine-grained contextual details at each specific wavelength. Meanwhile, the CNN branch facilitates efficient inter-spectra feature extraction among different wavelengths while maintaining a large receptive field. Specifically, in the transformer branch, we propose a novel Grouping-Aggregation transformer (GAT), comprising grouping self-attention (GSA) and aggregation self-attention (ASA). The GSA is employed to extract diverse fine-grained features of targets, while the ASA facilitates interaction among heterogeneous textures allocated to different channels. In the CNN branch, we propose a Wide-Spanning Separable 3D Attention (WSSA) to enlarge the receptive field while keeping a low parameter number. Building upon WSSA, we construct a wide-spanning CNN module to efficiently extract inter-spectra features. Extensive experiments demonstrate the superior performance of our HyFormer.

1. Introduction

Hyperspectral images (HSIs) typically consist of tens or even hundreds of spectral bands, providing a richer source of spectral information. This abundance of information enables hyperspectral images to effectively distinguish different objects, materials, and land cover types. Thanks to these characteristics, hyperspectral images have been widely applied in numerous computer vision tasks, including medical image processing [1,2], object tracking [3], mineral exploration [4], hyperspectral anomaly detection [5], plant detection [6], etc. Hyperspectral imaging typically requires high spectral resolution, resulting in a large amount of data that needs to be collected. However, due to the hardware limitations of sensors and acquisition equipment, spatial resolution is often sacrificed to obtain higher spectral resolution. To better serve downstream visual tasks, enhancing the spatial resolution of hyperspectral images has become a crucial research topic.

Hyperspectral super-resolution is an effective technique for enhancing the spatial resolution of hyperspectral images, allowing for the reconstruction of low-resolution hyperspectral images into higher-resolution hyperspectral images. Hyperspectral super-resolution methods can be categorized into two types based on their use of auxiliary information: (1) fusion-based hyperspectral image super-resolution. (2) Single hyperspectral image super-resolution. The former requires additional auxiliary information, such as RGB, panoramic images (PAN) [7,8], multispectral images (MSI) [9,10], etc., to enhance spatial resolution, while the latter relies solely on a single low-resolution hyperspectral image to restore its corresponding high-resolution counterpart. While fusion-based hyperspectral image super-resolution can achieve good performance, it is limited by the requirement of acquiring auxiliary images in the same scene as the low-resolution image, which increases the complexity of the task. Therefore, research on single-image super-resolution is more aligned with the practical needs of real-world scenarios.

Over the past few decades, a multitude of remarkable single hyperspectral image super-resolution methods have been proposed. Akgun et al. [11] improved the spatial resolution of hyperspectral images by modeling the image acquisition process. Li et al. [12] considered the sparsity of spectral decomposition and the repetitiveness of spatial-spectral blocks while proposing a hyperspectral super-resolution architecture based on spectral mixture and spatial-spectral group sparsity. Wang et al. [13] proposed a tensor-based super-resolution method that models the intrinsic characteristics of hyperspectral images. However, these methods rely on handcrafted priors, which are difficult to optimize. Recently, with the rapid advancement of deep learning, deep neural networks have showcased remarkable non-linear fitting capabilities. Consequently, natural super-resolution methods [14,15,16,17,18,19,20,21] based on convolutional neural networks (CNNs) have garnered significant attention. Unlike natural images, hyperspectral images exhibit spectral continuity, where neighboring spectral bands are often correlated. Therefore, 3D convolution is frequently employed to extract features from hyperspectral images. Mei et al. [22] constructed a fully 3D convolutional neural network to perform hyperspectral super-resolution. Yang et al. [23] proposed a multi-scale wavelet 3D CNN by modulating wavelets with 3D CNN to improve the restoration of details. While 3D CNN has powerful representational capabilities, it is often accompanied by a substantial computational burden and a large number of parameters. Thus, Li et al. [24] proposed MCNet, which stacks hybrid modules modulating 2D and 3D convolutions to extract both spatial and spectral features. Furthermore, Li et al. proposed ERCSR to address the issue of parallel structure redundancy in MCNet [25]. Zhang et al. [26] proposed a multi-scale network that utilizes wavelet transform and multi-scale feature fusion to learn features across different spectral bands. In ref. [27], researchers designed a multi-domain feature learning strategy based on 2D/3D units to integrate information from different layers. Tang et al. [28] proposed FRLGN, which incorporates a feedback structure to propagate high-level information for guiding the generation of low-level features. Zhang et al. [29] explored the coupling relationship between the spectral and spatial domains, then utilized spectral high-frequency information to improve channel and spatial attention.

While CNN-based methods have achieved impressive results, they still face the following issue: the widespread use of convolution kernels with a size of 3 in these CNN-based methods severely limits the receptive field range, hindering the model from considering a wider range of contextual information. While it is theoretically expected that the receptive field increases as the network deepens, the effective receptive field is often much smaller than the anticipated result [30] in practice. Directly increasing the kernel size could improve the receptive field size. However, it also leads to a significant increase in parameters and computational complexity. This is especially impractical for 3D convolutions.

Recently, transformers [31] in the field of natural language processing have gained increasing popularity due to their powerful long-range modeling capabilities. Moreover, ViT [32] has successfully extended the application of transformers to the field of computer vision, leading to the emergence of numerous outstanding vision transformer models [33,34,35,36,37,38,39,40,41]. Vision transformers are primarily designed to capture long-range dependencies in the spatial domain. However, as shown in Figure 1, each spectral band of a hyperspectral image exhibits significant sparsity in the spatial domain, with many regions lacking meaningful information. Performing vanilla self-attention calculations on sparse spatial domains would result in significant wastage of computational resources. Consequently, vision transformers originally designed for natural images need to be further improved to adapt to hyperspectral images.

In this paper, we design a hybrid architecture named HyFormer to integrate the advantages of CNN and transformer for hyperspectral super-resolution tasks. CNN and transformer extract features from different perspectives. Specifically, the transformer branch could achieve intra-spectra interaction for fine-grained contextual details on each specific wavelength. The CNN branch helps to conduct efficient inter-spectra feature extraction among different wavelengths while maintaining a large receptive field. By effectively modulating the two types of features, their advantages can complement each other, thereby enhancing the modeling capability of the model.

In the transformer branch, we present a novel Grouping-Aggregation transformer (GAT) that aims to capture intra-spectra interactions of object-specific contextual information from a spectral perspective. By decomposing the spectra of each wavelength, diverse contextual details of objects are implicitly expressed in different channels. The proposed GAT considers each channel as an individual token and focuses on modeling long-range dependencies among different details within the spectra of the same wavelength. Moreover, self-attention can consider all positions in the spatial domain to further capture global dependencies. During the computation of self-attention, the GAT employs a novel grouping-aggregation self-attention, which consists of grouping self-attention and aggregation self-attention. Specifically, grouping self-attention is employed to extract features from rich details coming from different channels, while the grouping mechanism is utilized to maintain lower computational complexity. On the other hand, our aggregation self-attention aims to fuse features from different channels to facilitate the exchange of information across channels. The GAT modulates dual self-attention to adaptively model fine-grained contextual details.

In the CNN branch, we propose a Wide-Spanning Separable 3D Attention (WSSA) to enhance the receptive field of the model while keeping a low parameter number. This method stacks a set of specifically designed small-kernel 3D convolutions to simulate the same receptive field as a large-kernel 3D convolution. Specifically, this method consists of three steps to simulate a large-kernel 3D convolution. Firstly, we simulate a large-kernel 3D convolution using the concept of depthwise separable convolution to reduce the parameter number, which cascades a pointwise and a large-kernel depthwise 3D convolutions. Subsequently, the large-kernel depthwise 3D convolution is decomposed into a small-kernel 3D depthwise convolution and a small-kernel dilated depthwise 3D convolution. Finally, the aforementioned two types of small-kernel depthwise 3D convolutions are further separated in the spatial and spectral dimensions to reduce computational complexity. Unlike 2D pixel attention, which solely considers spatial dimensions, WSSA preserves the inherent spatial-spectral consistency information. Based on WSSA, we construct a wide-spanning CNN module to extract inter-spectra features among different wavelengths while achieving a large receptive field to consider a wider range of contextual information.

Our hybrid architecture enables adaptive feature interactions between the CNN and transformer modules at each layer to facilitate the fusion of various features.

In summary, the contributions of this paper can be summarized as follows:

• We propose a novel Grouping-Aggregation transformer (GAT) to capture intra-spectra interactions of object-specific contextual information from a spectral perspective. By modulating grouping self-attention and aggregation self-attention, GAT can adaptively model fine-grained contextual details.
• We introduce Wide-Spanning Separable 3D Attention (WSSA) to explore inter-spectra feature extraction. It significantly enhances the receptive field while only increasing minimal additional parameters.
• We designed a hybrid architecture named HyFormer that modulates the strengths of both CNN and transformer structures. HyFormer could adaptively fuse features extracted by CNN and transformer components, resulting in improved reconstruction outcomes. Abundant experimental results unequivocally demonstrate the substantial superiority of our HyFormer over state-of-the-art methods.

2. Proposed Methods

In this section, we will provide a detailed description of our method. The goal of hyperspectral image super-resolution is to restore a low-resolution hyperspectral image to a high-resolution hyperspectral image. Let ${\mathbf{I}}_{LR}\in {\mathbb{R}}^{B\times S\times h\times w}$ and ${\mathbf{I}}_{HR}\in {\mathbb{R}}^{B\times S\times H\times W}$ denote the low-resolution and high-resolution hyperspectral images, respectively. The S denotes the number of spectral bands. We use ${\mathbf{I}}_{SR}\in {\mathbb{R}}^{B\times S\times H\times W}$ to denote restored hyperspectral image. Therefore, the process of super-resolution can be represented as follows:

$${\mathbf{I}}_{SR}=\mathrm{Network}\left({\mathbf{I}}_{LR}\right).$$

(1)

2.1. Overall Network

The proposed HyFormer is illustrated in Figure 2. The top part represents the transformer branch, while the bottom part represents the CNN branch. There are interactions between the two branches at each layer to exchange information with different characteristics. At the ends of branches, the features extracted from both branches are adaptively fused to enhance the representation.

Initially, we employ separable 3D convolution [42] $f_{SF}$ to extract shallow-level features from the input low-resolution (LR) image. Separable 3D convolutions have been demonstrated to exhibit similar performance as conventional 3D convolutions while offering reduced computational complexity [42]. This process can be represented as follows:

$${\mathbf{F}}_{SF}=f_{3D}\left(\mathrm{Unsqueeze}\left({\mathbf{I}}_{LR}\right)\right),$$

(2)

where ${\mathbf{F}}_{SF}\in {\mathbb{R}}^{B\times C\times S\times h\times w}$ denotes extracted shallow-level features, and C denotes channel number. The $\mathrm{Unsqueeze}(·)$ denotes reshaping ${\mathbf{I}}_{LR}$ into the shape of ($B\times 1\times S\times h\times w$). Next, the feature ${\mathbf{F}}_{SF}$ is simultaneously fed into the transformer and CNN branches to extract different types of features. For transformer branch, the feature ${\mathbf{F}}_{SF}\in {\mathbb{R}}^{B\times C\times S\times h\times w}$ needs to be reshaped into ${\tilde{\mathbf{F}}}_{SF}\in {\mathbb{R}}^{(B\times S)\times C\times h\times w}$ by $f_{\mathrm{reshape}4\mathrm{D}}(·)$. The feature extraction process for both branches in the first layer can be represented as follows:

$$\begin{array}{c}{\mathbf{T}}_1=f_1^T\left(f_{\mathrm{reshape}4\mathrm{D}}\left({\mathbf{F}}_{SF}\right)\right)\hfill \\ {\mathbf{C}}_1=f_1^C\left({\mathbf{F}}_{SF}\right),\hfill \end{array}$$

(3)

where $f_1^T(·)$ and $f_1^C(·)$ respectively denote the first transformer and CNN modules. The ${\mathbf{T}}_1$ and ${\mathbf{C}}_1$ are extracted features. In the subsequent feature extraction layers, we perform adaptive fusion of the two types of features to complement each other. Mathematically, the subsequent feature extraction process can be represented as:

$$\left\{\begin{array}{c}{\mathbf{T}}_i=f_i^T\left({\mathrm{Conv}2\mathrm{D}}_{1\times 1}\left(\mathrm{Concat}(w_{i-1}^t{\mathbf{T}}_{i-1},w_{i-1}^c{f}_{\mathrm{reshape}4\mathrm{D}}\left({\mathbf{C}}_{i-1}\right))\right)\right)\hfill \\ {\mathbf{C}}_i=f_i^C\left({\mathrm{Conv}3\mathrm{D}}_{1\times 1\times 1}\left(\mathrm{Concat}({\widehat{w}}_{i-1}^t{f}_{\mathrm{reshape}5\mathrm{D}}\left({\mathbf{T}}_{i-1}\right),{\widehat{w}}_{i-1}^c{\mathbf{C}}_{i-1})\right)\right)\hfill \end{array}\right.(i=2,\dots ,L),$$

(4)

where $w_{i-1}^t,w_{i-1}^c$ and ${\widehat{w}}_{i-1}^t,{\widehat{w}}_{i-1}^c$ are learnable coefficients used to adaptively adjust the feature fusion ratio, with an initial value of 1. The fused features are then fed into convolution layers ${\mathrm{Conv}2\mathrm{D}}_{1\times 1}$ and ${\mathrm{Conv}3\mathrm{D}}_{1\times 1\times 1}$ to reduce the channel number. The $f_{\mathrm{reshape}5\mathrm{D}}(·)$ denotes the reshape operation from ($(B\times S)\times C\times h\times w$) to ($B\times C\times S\times h\times w$).

To enable the network to learn more informative representations, we perform feature fusion at the end of both the transformer and CNN branches. During this fusion process, learnable coefficients are employed to control the fusion ratio. Mathematically, this process can be represented as follows:

$$\left\{\begin{array}{c}\mathbf{T}={\mathrm{Conv}2\mathrm{D}}_{1\times 1}(\mathrm{Concat}({\epsilon }_1^t{\mathbf{T}}_1,\dots ,{\epsilon }_i^t{\mathbf{T}}_i)\hfill \\ \mathbf{C}={\mathrm{Conv}3\mathrm{D}}_{1\times 1\times 1}(\mathrm{Concat}({\widehat{\epsilon }}_1^c{\mathbf{C}}_1,\dots ,{\widehat{\epsilon }}_i^c{\mathbf{C}}_i)\hfill \end{array}\right.(i=1,\dots ,L),$$

(5)

where ${\epsilon }_i^t$ and ${\widehat{\epsilon }}_i^c$ denote learnable coefficients, initialized to 1.

The transformer branch focuses on capturing intra-spectra interactions at each specific wavelength, whereas the CNN branch facilitates efficient inter-spectra feature extraction among different wavelengths. Consequently, we fuse these two features to enhance the representation. Lastly, we employ deconvolution layers to perform upsampling, thereby increasing the spatial resolution. Due to the significant similarity between the LR input image and the SR output image, we incorporate the bicubic interpolated LR image into the output to guide the model to focus on high-frequency residual features. The final reconstruction process can be represented as follows:

$${\mathbf{I}}_{SR}=\mathrm{Bicubic}\left({\mathbf{I}}_{LR}\right)+f_{3D}\left(\mathrm{Up}({\mathbf{F}}_{SF}+{\mathrm{Conv}3\mathrm{D}}_{1\times 1\times 1}\left(\mathrm{Concat}({\gamma }^t{f}_{\mathrm{reshape}5\mathrm{D}}\left(\mathbf{T}\right),{\gamma }^c\mathbf{C})\right))\right),$$

(6)

where ${\gamma }^t$ and ${\gamma }^c$ are learnable coefficients to adjust the feature fusion ratio, with an initial value of 1. The $\mathrm{Up}(·)$ denotes the deconvolution to increase the spatial resolution. The $f_{3D}(·)$ is separable 3D convolution used to decrease the channel number.

2.2. Grouping-Aggregation Transformer Module

The transformer models have shown remarkable capability in capturing long-range dependencies in the spatial domain. However, these transformer models designed for natural images are not suitable for hyperspectral images due to their specific characteristics. Hyperspectral images exhibit sparsity in the spatial domain. Therefore, dense self-attention in the vanilla transformer would lead to numerous inefficient computations.

Based on the analysis mentioned above, we propose a grouping-aggregation transformer (GAT) to capture intra-spectra interactions of object-specific contextual information from a spectral perspective. During the extraction of shallow features, 3D convolution is applied to decompose each specific wavelength of the spectral data, thereby implicitly expressing diverse texture information across different channels. Our GAT treats each channel as a token and performs self-attention in the channel dimension to capture intra-spectra interactions for fine-grained contextual details on each specific wavelength. Meanwhile, self-attention allows for the consideration of all positions within the spatial domain, enabling the model to capture global dependencies more effectively.

We improve the standard of self-attention by introducing grouping-aggregation self-attention (GASA). The GASA is composed of grouping self-attention (GSA) and aggregation self-attention (ASA). To be more specific, we assign half of the channels to grouping self-attention, which is used to extract rich details from different channels. Meanwhile, the grouping mechanism ensures that the computation of self-attention is only performed within each group, reducing the computational cost of self-attention. The remaining half of the channels are dedicated to aggregation self-attention, which utilizes aggregation to merge features from different channels, allowing interaction among features of different textures. Subsequently, the extracted two features are concatenated to align the channel dimensions while achieving modulation of features from both self-attentions to adaptively model long-range channel dependencies from a spectral perspective.

Figure 3 illustrates the workflow of grouping-aggregation self-attention. Let $\mathbf{X}\in {\mathbb{R}}^{BS\times hw\times C}$ denote the input features of GAT. For the grouping of self-attention, we linearly map $\mathbf{X}$ to obtain $query$ ${\mathbf{Q}}^g$, $key$ ${\mathbf{K}}^g$, and $value$ ${\mathbf{V}}^g$:

$${\mathbf{Q}}^g=\mathbf{X}{\mathbf{W}}_g^Q,{\mathbf{K}}^g=\mathbf{X}{\mathbf{W}}_g^K,{\mathbf{V}}^g=\mathbf{X}{\mathbf{W}}_g^V,$$

(7)

where ${\mathbf{W}}_g^Q,{\mathbf{W}}_g^K,{\mathbf{W}}_g^V\in {\mathbb{R}}^{C\times \frac{1}{2}C}$ are learnable parameters. Subsequently, we equally divide the ${\mathbf{Q}}^g$, ${\mathbf{K}}^g$ and ${\mathbf{Q}}^g$ tensors by channels into k different groups, ensuring that the self-attention calculation is performed only within each group. The grouping self-attention can be formulated as:

$${\mathbf{A}}^g={\mathrm{Attention}}^g({\mathbf{Q}}^g,{\mathbf{K}}^g,{\mathbf{V}}^g)={\left\{{\mathbf{V}}_i^g\mathrm{SoftMax}({\left({\mathbf{K}}_i^g\right)}^T{\mathbf{Q}}_i^g/\sqrt{hw})\right\}}_{i=0}^k.$$

(8)

For aggregation self-attention, we map $\mathbf{X}$ to get $query$ ${\mathbf{Q}}^a$, $key$ ${\mathbf{K}}^a$, and $value$ ${\mathbf{V}}^a$:

$${\mathbf{Q}}^a=\mathbf{X}{\mathbf{W}}_a^Q,{\mathbf{K}}^a=\mathrm{Aggregation}\left(\mathbf{X}\right){\mathbf{W}}_a^K,{\mathbf{V}}^a=\mathrm{Aggregation}\left(\mathbf{X}\right){\mathbf{W}}_a^V,$$

(9)

where $\mathrm{Aggregation}(·)$ is $1\times 1$ convolution to aggregate diverse features from different channels, while decreasing channel number from C to $\xi C$, in which $\xi \in (0,1)$ is hyperparameter to control aggregation ratio. The ${\mathbf{W}}_a^Q\in {\mathbb{R}}^{C\times \frac{1}{2}C}$, ${\mathbf{W}}_a^K,{\mathbf{W}}_a^V\in {\mathbb{R}}^{\xi C\times \frac{1}{2}\xi C}$ are learnable parameters. The aggregation self-attention can be formulated as:

$${\mathbf{A}}^a={\mathrm{Attention}}^a({\mathbf{Q}}^a,{\mathbf{K}}^a,{\mathbf{V}}^a)={\mathbf{V}}^a\mathrm{SoftMax}({\left({\mathbf{K}}^a\right)}^T{\mathbf{Q}}^a/\sqrt{hw}).$$

(10)

Next, we concatenate ${\mathbf{A}}^g$ and ${\mathbf{A}}^a$ to incorporate features extracted by dual self-attention:

$$\mathbf{A}=\mathrm{Concat}({\mathbf{A}}^g,{\mathbf{A}}^a),$$

(11)

where $\mathbf{A}$ is the feature representation extracted by grouping-aggregation self-attention. As shown in Figure 4, after grouping-aggregation self-attention, we utilize a multi-layer perceptron (MLP) with the non-linear activation function GELU [43] to enhance the representations. Before the grouping-aggregation self-attention and MLP, we use the LayerNorm layer [44] to do normalization. The feature extraction process in the transformer module can be represented as follows:

$$\begin{array}{c}{\mathbf{X}}_T^{\prime }=\mathbf{X}+\mathrm{GASA}\left(\mathrm{LN}\left(\mathbf{X}\right)\right)\hfill \\ {\mathbf{X}}_T={\mathbf{X}}_T^{\prime }+\mathrm{MLP}\left(\mathrm{LN}\left({\mathbf{X}}_T^{\prime }\right)\right),\hfill \end{array}$$

(12)

where $\mathrm{LN}(·)$ denotes the LayerNorm and $\mathrm{MLP}(·)$ denotes multi-layer perceptron. The $\mathrm{GASA}(·)$ denotes the grouping-aggregation self-attention.

2.3. Wide-Spanning CNN Module

The 3D CNN has shown promising performance in hyperspectral image super-resolution, as it can extract features while preserving spatial-spectral consistency. It is indeed a common practice in previous works to utilize a $3\times 3\times 3$ convolution kernel in 3D convolution, which inherently restricts the receptive field of the model. This limitation becomes particularly problematic for hyperspectral images, as the sparse nature of the spatial domain necessitates a larger receptive field. A small receptive field hampers the model’s ability to comprehend context, leading to potential limitations in performance. Directly increasing the size of the convolution kernel can expand the receptive field, but it comes with a significant increase in parameters and computational complexity, making it an impractical approach. In this paper, we propose a novel approach called wide-spanning separable 3D attention to address the aforementioned issue.

The wide-spanning, separable 3D attention is shown in the bottom right corner of Figure 5. We stack a set of specifically designed small-kernel 3D convolutions to simulate the same receptive field as a large-kernel 3D convolution.

In this paper, we simulate the large-kernel 3D convolution through three steps. First, depthwise separable convolution can significantly reduce the parameters and computational complexity. We simulate a $17\times 17\times 17$ 3D convolution using the concept of depthwise separable convolution to reduce the parameter number, which cascades $1\times 1\times 1$ pointwise and $17\times 17\times 17$ depthwise 3D convolution. Second, we further decompose $17\times 17\times 17$ depthwise 3D convolution into a $5\times 5\times 5$ depthwise 3D convolution and a $5\times 5\times 5$ dilated depthwise 3D convolution with a dilation factor of $(3,3,3)$, which achieves the same receptive field as a $17\times 17\times 17$ depthwise 3D convolution. As shown in Figure 6, we demonstrate this process of simulating the large receptive field.

However, due to the inherent nature of 3D convolutions, they still have a relatively large number of parameters. Separable 3D convolution has been proven to have similar effects to vanilla 3D convolution [42]. Therefore, we adopt a similar concept to separable 3D convolution to further decompose the $5\times 5\times 5$ depthwise 3D convolution and the $5\times 5\times 5$ dilated depthwise 3D convolution mentioned earlier. Specifically, we use a $1\times 5\times 5$ depthwise 3D convolution and a $5\times 1\times 1$ depthwise 3D convolution to simulate a $5\times 5\times 5$ depthwise 3D convolution. Meanwhile, we use a $1\times 5\times 5$ dilated depthwise 3D convolution with a dilation factor of $(1,3,3)$ and a $5\times 1\times 1$ dilated depthwise 3D convolution with a dilation factor of $(3,1,1)$ to simulate a $5\times 5\times 5$ dilated depthwise 3D convolution. As a result, the parameter number is significantly reduced, allowing for more feasible usage.

To effectively extract inter-spectra features among different wavelengths while achieving a large receptive field to consider a wider range of contextual information, we designed a wide-spanning CNN module that is applied to the CNN branch. The structure of our CNN module is illustrated in Figure 5. We employ both 3D convolution and 2D convolution to extract features and apply the proposed wide-spanning separable 3D attention at the end of the module to expend the receptive field. Skip connections are employed between each component to enhance the flow of information.

As mentioned earlier, separable 3D convolution provides similar performance to regular 3D convolution. Thus, in our CNN module, we employ separable 3D convolution to extract local spatial and spectral features simultaneously. It preserves spatial-spectral consistency, which is beneficial for restoring physically meaningful spectral curves. Furthermore, to enhance the extraction of spatial information, we incorporate 2D CNN units following the 3D convolution to explore spatial features. Thanks to the inclusion of our wide-spanning separable 3D attention, the model has a wider receptive field, allowing it to extract spatial and spectral features within a larger spatial-spectral context. The feature extraction in the CNN module can be represented as follows:

$$\begin{array}{c}{\mathbf{X}}_{3D}=f_{3D}\left(\mathbf{X}\right)\hfill \\ {\mathbf{X}}_{2D}={\mathbf{X}}_{3D}+f_{2DUint}\left(f_{2DUint}\left({\mathbf{X}}_{3D}\right)\right)\hfill \\ {\mathbf{X}}_C={\mathbf{X}}_{2D}+{\mathbf{X}}_{2D}\odot f_{WSA}\left({\mathbf{X}}_{2D}\right),\hfill \end{array}$$

(13)

where $f_{3D}(·)$ denotes separable 3D convolution and $f_{2DUint}(·)$ denotes 2D convolution unit. The $f_{WSA}(·)$ is the proposed wide-spanning separable 3D attention.

3. Experiments

3.1. Datasets

In this section, we provide a comprehensive overview of the experimental results obtained using the proposed HyFormer on three publicly available hyperspectral datasets. We also present a detailed comparison with state-of-the-art (SOTA) methods to evaluate the performance of our approach. Additionally, we conduct ablation experiments for each module to validate their effectiveness and understand their individual contributions to the overall performance.

Our experiments are conducted on the following three hyperspectral datasets:

(1) CAVE dataset [45]: This dataset contains 31 scenes, which were gathered by a cooled CCD camera. The spectral wavelengths of the image range from 400 nm to 700 nm, with a step size of 10nm between bands. Therefore, each image has 31 spectral bands. Their spatial resolution is $512\times 512$ pixels.

(2) Harvard dataset [46]: This dataset consists of 77 hyperspectral images divided into indoor and outdoor scenes. They were captured by a Nuance FX, CRI Inc. camera with a wavelength range of 400 to 700 nm. Each hyperspectral image in the dataset has a size of $1040\times 1392\times 31$.

(3) Foster dataset [47]: This dataset consists of 30 images captured by a low-noise Peltier-cooled digital camera in the Minho region of Portugal. Each hyperspectral image contains 33 spectral bands, and they have a spatial resolution of $1204\times 1344$ pixels.

To ensure the fairness and reliability of our experimental evaluation, we train and test the models on each dataset separately. Because different datasets are collected using different hyperspectral cameras, which may have varying characteristics and imaging conditions. In our experiments, we adopt a random partitioning strategy where we allocate 80% of the samples from each dataset as the training set. The remaining 20% of samples are reserved exclusively for testing purposes. Due to the limited number of samples in the hyperspectral dataset, following [24,25], we employ data augmentation techniques during the training phase to augment the training samples. We randomly select 24 patches within each image and augment them through horizontal flipping, rotation (${90}^{\circ }$, ${180}^{\circ }$, and ${270}^{\circ }$), and scaling (1, 0.75, and 0.5) operations. We downsample each patch using bicubic interpolation with a scale factor r (r = 2, r = 4, and r = 8) to obtain a low-resolution hyperspectral image. During the testing phase, we assess the model’s performance by evaluating its results on the $512\times 512$ region located in the top left corner of each test image.

3.2. Evaluation Metrics

To quantitatively measure the performance of the model, we utilize metrics such as Peak Signal to Noise Ratio (PSNR), Structural Similarity (SSIM), and Spectral Angle Mapper (SAM) to evaluate results. They are defined as:

$$\begin{array}{c}\mathrm{PSNR}=\frac{1}{S}\sum _{s=1}^{S}10{log}_{10}(\frac{{\mathrm{MAX}}_s^2}{{\mathrm{MSE}}_s})\hfill \\ {\mathrm{MSE}}_s=\frac{1}{HW}\sum _{h=1}^H\sum _{w=1}^W{({\mathbf{I}}_{SR}(h,w,s)-{\mathbf{I}}_{HR}(h,w,s))}^2,\hfill \end{array}$$

(14)

where the ${\mathrm{MAX}}_s$ is maximal pixel value for sth band.

$$\begin{array}{c}\mathrm{SSIM}=\frac{1}{S}\sum _{s=1}^S\frac{(2{\mu }_{{\mathbf{I}}_{SR}}^s{\mu }_{{\mathbf{I}}_{HR}}^s+c_1)(2{\sigma }_{{\mathbf{I}}_{SR}{\mathbf{I}}_{HR}}^s+c_2)}{m\times n}\hfill \\ m={({\mu }_{{\mathbf{I}}_{SR}}^s)}^2+{({\mu }_{{\mathbf{I}}_{HR}}^s)}^2+c_1\hfill \\ n={({\sigma }_{{\mathbf{I}}_{SR}}^s)}^2+{({\sigma }_{{\mathbf{I}}_{HR}}^s)}^2+c_2,\hfill \end{array}$$

(15)

where ${\mu }_{{\mathbf{I}}_{SR}}^s$ and ${\mu }_{{\mathbf{I}}_{HR}}^s$ denote the mean of ${\mathbf{I}}_{SR}$ and ${\mathbf{I}}_{HR}$ for the sth band, respectively. The ${\sigma }_{{\mathbf{I}}_{SR}}^s$ and ${\sigma }_{{\mathbf{I}}_{HR}}^s$ denote the variance of ${\mathbf{I}}_{SR}$ and ${\mathbf{I}}_{HR}$ for the sth band. The ${\sigma }_{{\mathbf{I}}_{SR}{\mathbf{I}}_{HR}}^s$ denotes the covariance of ${\mathbf{I}}_{SR}$ and ${\mathbf{I}}_{HR}$ for the sth band. The $c_1$ and $c_2$ are two constants.

$$\mathrm{SAM}=\arccos (\frac{〈{\mathbf{I}}_{SR},{\mathbf{I}}_{HR}〉}{{∥{\mathbf{I}}_{SR}∥}_2{∥{\mathbf{I}}_{HR}∥}_2}),$$

(16)

where $\arccos (·)$ denotes the arccos function, and the $〈·,·〉$ denotes the dot product operation. The ${∥·∥}_2$ is $l_2$ norm.

In general, higher values of PSNR and SSIM indicate better reconstruction results, while a lower value of SAM indicates better reconstruction.

3.3. Implementation Details

Taking into account the trade-off between performance and complexity, we set the number of layers to 4 (L = 4) in the transformer and CNN branches and the channel number S for the extracted shallow features ${\mathbf{F}}_{SF}$ to 96. For transformer branch, each transformer module consists of two grouping-aggregation transformers. We set the number of heads for self-attention in each layer to 2. The number of groups k for grouping self-attention is set to 4, and the aggregation rate $\xi$ for aggregation self-attention is set to 0.25. For the CNN branch, we configure each CNN module to include one separable 3D convolution, two 2D convolution units, and a wide-spanning separable 3D attention block. In our experiments, we use $l_1$ loss as the loss function. To optimize our model, we employ the AdamW optimizer [48] with a weight decay of 0.02. During the training process, we set the batch size to 16 with an initial learning rate of 5 × ${10}^{-4}$. The learning rate is halved every 35 epochs, and the total number of epochs is set to 100. All experiments are conducted using PyTorch on Nvidia 3090 GPUs.

3.4. Comparisons with State-of-the-Arts

To validate the effectiveness of our method, we compare it with six state-of-the-art (SOTA) methods, which are SSPSR [49], MCNet [24], ERCSR [25], GELIN [50], and MDFF [27].

As shown in

Table 1. Quantitative evaluation of our HyFormer under different scale factor ($\times$2, $\times$4, and $\times$8) on CAVE, Harvard and Foster datasets. The bold indicates the best result.

Methods	Scale	CAVE			Harvard			Foster
		PSNR	SSIM	SAM	PSNR	SSIM	SAM	PSNR	SSIM	SAM
SSPSR [49]	×2	44.742	0.9916	2.925	45.221	0.9799	1.977	52.452	0.9960	4.517
MCNet [24]	×2	45.726	0.9926	2.707	45.224	0.9792	1.945	53.205	0.9962	4.415
ERCSR [25]	×2	45.806	0.9927	2.680	45.394	0.9797	1.916	52.966	0.9963	4.405
GELIN [50]	×2	45.502	0.9926	2.733	45.453	0.9802	1.975	52.616	0.9960	4.584
MDFF [27]	×2	45.854	0.9927	2.677	45.332	0.9796	1.920	53.300	0.9963	4.379
HyFormer (Ours)	×2	46.386	0.9936	2.558	45.804	0.9816	1.894	53.379	0.9964	4.369
SSPSR [49]	×4	37.927	0.9698	3.845	39.823	0.9363	2.463	44.814	0.9762	6.580
MCNet [24]	×4	38.687	0.9747	3.682	39.892	0.9362	2.495	45.259	0.9778	6.512
ERCSR [25]	×4	38.800	0.9751	3.670	40.056	0.9375	2.462	45.684	0.9797	6.517
GELIN [50]	×4	38.588	0.9739	3.772	40.071	0.9366	2.487	45.231	0.9783	6.617
MDFF [27]	×4	38.797	0.9753	3.662	40.049	0.9372	2.470	45.751	0.9796	6.505
HyFormer (Ours)	×4	39.356	0.9766	3.578	40.420	0.9388	2.445	46.174	0.9819	6.396
SSPSR [49]	×8	33.674	0.9280	5.205	35.479	0.8782	2.958	38.625	0.9217	8.436
MCNet [24]	×8	34.144	0.9316	5.133	35.896	0.8813	2.893	38.796	0.9246	8.425
ERCSR [25]	×8	34.227	0.9321	5.076	35.795	0.8813	2.883	38.827	0.9242	8.353
GELIN [50]	×8	34.213	0.9314	5.162	35.792	0.8802	2.864	38.958	0.9254	8.373
MDFF [27]	×8	34.307	0.9333	5.020	35.849	0.8811	2.916	38.920	0.9248	8.296
HyFormer (Ours)	×8	34.617	0.9360	4.830	36.251	0.8840	2.853	39.859	0.9358	8.112

, we present quantitative experiments with scale factors $\times$2, $\times$4, and $\times$8. Compared with state-of-the-art methods, we have significant advantages at each scale on all datasets. Specifically, for the CAVE dataset with a scale factor $\times$2, our method achieves a 0.532 dB higher PSNR than the second-best MDFF. For the CAVE dataset with scale factor $\times$4, our method also outperforms the second-best ERCSR by 0.556 dB. Meanwhile, our method outperforms the second-best method, MDFF, by 0.310 dB on the CAVE dataset with scale factor $\times$8. For the Harvard dataset, our method achieves PSNRs of 45.804 dB, 40.420 dB, and 36.251 dB, respectively, for scale factors $\times$2, $\times$4, and $\times$8, which are higher than other methods. For the Foster dataset, our method still shows advantages. Especially at a scale factor $\times$4, our PSNR is 0.423 dB higher than the second-best method. At scale factor $\times$8, our PSNR exceeds the second-best method by 0.901 dB, which represents a significant improvement. For the evaluation metric SAM, our method achieves the best results on CAVE, Harvard, and Foster datasets with scale factors $\times$2, $\times$4, and $\times$8. Taking the scale factor $\times$4 as an example, our SAM is 0.084 lower than the second-best method on the CAVE dataset. On the Harvard dataset, it is 0.017 lower than the second-best method. Additionally, on the Foster dataset, our SAM is 0.109 lower than the second-best method. These results demonstrate that our method can accurately recover spectral curves. For more quantitative results of the experiments, please refer to Table 1. In summary, from a quantitative perspective, our method possesses significant advantages. It outperforms the other comparative methods in various metrics, demonstrating clear superiority across all evaluation criteria.

As shown in Figure 7, Figure 8 and Figure 9, we present the mean absolute errors of the reconstructed images from the CAVE, Harvard, and Foster datasets. The mean error map provides a visual representation of the method’s performance. In the mean error map, darker colors indicate smaller errors, while brighter colors indicate larger errors. In Figure 7, Figure 8 and Figure 9, the mean error maps indicate that our proposed method achieves significantly smaller errors than other methods, particularly in high-frequency details where our method recovers more faithful textures. The bounding boxes highlight the regions where we have a clear advantage, with lower errors compared with other methods.

Specifically, Figure 7 consists of two peppers, and the challenge in its restoration lies in preserving the edges of the peppers and the color chart located above. From the mean error map, it is evident that our method exhibits the smallest error. The errors at the object edges are significantly smaller compared with other methods, indicating the superior performance of our approach. Furthermore, from the color chart above, it is evident that we have a clear advantage. The color chart restored by our method exhibits darker colors at the edges, while other methods tend to have brighter colors at the edges.

As shown in Figure 8, in our reconstruction result, there are more areas with lower brightness compared with other methods, indicating better performance. Additionally, our method demonstrates excellent reconstruction of the edge portions, indicating that our method excels at recovering high-frequency details.

Figure 9 illustrates the mean error map between the reconstructed result of hyperspectral image and its corresponding high-resolution image on the Foster dataset. We highlight the regions where we have significant advantages compared with other methods using bounding boxes. In these regions, our method exhibits lower brightness, indicating that we produce fewer errors in the reconstruction. Additionally, we achieve more faithful results at the edges of the objects, demonstrating that our method can better restore the details in the image.

Additionally, to demonstrate the precision of the spectral information restored by our method, we present the mean spectral difference curve. The average spectral difference curve vividly depicts the ability to recover spectral information. In Figure 10, we showcase the mean spectral difference curves obtained from the CAVE dataset. From Figure 10, it is evident that our curves remain in the lowest position consistently, indicating the smallest errors across all spectral bands in our reconstruction results. This advantage is obviously pronounced in Figure 10b.

In Figure 11, we present the mean spectral difference curves from the Harvard dataset. Our curves, once again, consistently remain in the lowest position, indicating that our method accurately recovers the spectral information of the hyperspectral images.

Floating-point operations (FLOPs) can accurately reflect the computational complexity of a model. In

Table 2. FLOPs and PSNR for all methods on Harvard ($\times$4).

	SSPSR	MCNet	ERCSR	GELIN	MDFF	HyFormer
FLOPs	6.025 T	2.232 T	2.182 T	5.343 T	2.517 T	5.103 T
PSNR	39.823	39.892	40.056	40.071	40.049	40.420

, we present a comparison of the FLOPs and PSNR for SSPSR, MCNet, ERCSR, GELIN, MDFF, and our HyFormer on the Harvard dataset with scale factor $\times$4. From the table, it can be seen that our HyFormer could achieve the highest PSNR with modest computational cost.

To further evaluate the performance on limited training samples, we conduct experiments on the CAVE ($\times$4) dataset by selecting 20% of the samples as our training set. As shown in

Table 3. Quantitative evaluation of our HyFormer under scale factor $\times$4 on CAVE dataset with limited training samples.

Method	PSNR	SSIM	SAM
SSPSR	37.468	0.9684	4.088
MCNet	37.768	0.9701	3.954
ERCSR	37.957	0.9711	3.828
GELIN	37.770	0.9696	3.988
MDFF	37.972	0.9711	3.855
HyFormer (Ours)	38.382	0.9733	3.705

, our method achieves the best performance in values of PSNR, SSIM, and SAM. Our PSNR is 0.41 dB higher than the second-best method, MDFF. This demonstrates that our method produces satisfactory performance even in the case of limited training samples.

3.5. Ablation Study

In this section, we conduct ablation studies to validate the effectiveness of our proposed HyFormer.

Effectiveness of the Grouping-Aggregation transformer Module. In our HyFormer, we devise a hybrid architecture incorporating a transformer branch and a CNN branch. The transformer branch employs a novel grouping-aggregation transformer (GAT), which effectively captures intra-spectra interactions of object-specific contextual information from a spectral perspective. In this part, we delve into an investigation of the transformer branch’s efficacy and evaluate its performance.

As shown in

Table 4. Comparison of PSNR, SSIM and SAM on CAVE ($\times$4) to validate the effectiveness of grouping-aggregation transformer (GAT) Module.

Structure	Params (M)	PSNR	SSIM	SAM
HyFormer w/o GAT	3.094	39.153	0.9758	3.629
HyFormer w/GAT	3.715	39.356	0.9766	3.578

, the model with GAT achieves a PSNR improvement of 0.203 dB compared with the model without GAT (i.e., only consisting of the CNN branch). This significant improvement indicates that incorporating GAT enhances the model’s ability to model spectral information. Additionally, from the parameters listed in the second column of Table 4, it can be observed that the inclusion of GAT does not significantly increase the number of parameters, indicating that the method is practical.

In the GAT, we employ grouping-aggregation self-attention, comprising grouping self-attention (GSA) and aggregation self-attention (ASA).

The GSA effectively captures fine-grained details from different channels, while the ASA facilitates the interaction of texture information across channels. They complement each other, enabling a more comprehensive modeling capability. To validate the effectiveness of grouping-aggregation self-attention, we substitute it with traditional channel self-attention (CSA) to evaluate the performance. As shown in

Table 5. Comparison of PSNR, SSIM and SAM on CAVE ($\times$4) to validate the effectiveness of grouping-aggregation self-attention (GASA).

Structure	Params (M)	PSNR	SSIM	SAM
GAT w/CSA	3.803	39.237	0.9763	3.606
GAT w/GASA	3.715	39.356	0.9766	3.578

, the model with CSA has higher parameters, but surprisingly, it yields a lower PSNR value. Therefore, this demonstrates the effectiveness of our grouping-aggregation self-attention.

Effectiveness of Wide-Spanning Separable 3D Attention. A larger receptive field allows the model to capture information from a broader region. This is crucial for hyperspectral super-resolution tasks, as the spatial domain of hyperspectral images is often sparse. The larger receptive field can encompass more useful information, aiding the model in understanding the context and improving its performance in capturing fine details. However, the current mainstream convolutions only employ a kernel size of 3, which limits the receptive field of the model. Thus, we propose wide-spanning separable 3D attention (WSSA) to address this issue. Our WSSA effectively increases the receptive field while keeping parameter growth relatively low. It achieves this by stacking a set of specifically designed small-kernel 3D convolutions to mimic the receptive field of a large-kernel 3D convolution. Furthermore, it utilizes separable 3D convolutions to approximate the small-kernel 3D convolutions, resulting in smaller parameters.

As evident from

Table 6. Comparison of PSNR, SSIM and SAM on CAVE ($\times$4) to validate the effectiveness of wide-spanning separable 3D attention (WSSA).

Structure	Params (M)	PSNR	SSIM	SAM
HyFormer w/o WSSA	3.651	39.216	0.9761	3.620
HyFormer w/WSSA	3.715	39.356	0.9766	3.578

, the inclusion of WSSA improves the PSNR value of the model by 0.14 dB, while the parameters only increase by 0.064 M. This demonstrates the effectiveness of our WSSA approach. It successfully improves the model’s performance while maintaining a minimal increase in the number of parameters.

Influence of Hyperparameters. We conduct experiments to test the influence of various important hyperparameters (number of layers L, number of groups k, and aggregation rate $\xi$) on the super-resolution performance. From

Table 7. Performance comparison of different parameter L on CAVE ($\times$4).

L	Params (M)	PSNR	SSIM	SAM
2	2.495	39.046	0.9761	3.608
4	3.715	39.356	0.9766	3.578
6	4.935	39.412	0.9767	3.564

, it can be observed that increasing the number of layers L improves the results in PSNR, SSIM, and SAM. However, an increase in the number of layers L also introduces a larger number of parameters. Therefore, to strike a balance between parameters and performance, we use L = 4 in our experiments. As shown in

Table 8. Performance comparison of different parameter k on CAVE ($\times$4).

k	PSNR	SSIM	SAM
2	39.264	0.9765	3.588
4	39.356	0.9766	3.578
6	39.232	0.9766	3.581

, the best performance is achieved when the number of groups $k=4$ in grouping self-attention. Thus, in our experiments, we configure the number of groups as $k=4$. The effect of different aggregation rates $\xi$ is shown in

Table 9. Performance comparison of different parameter $\xi$ on CAVE ($\times$4).

$\mathit{\xi }$	Params (M)	PSNR	SSIM	SAM
0.125	3.702	39.311	0.9769	3.580
0.25	3.715	39.356	0.9766	3.578
0.5	3.748	39.360	0.9770	3.595

. We set $\xi$ to 0.25 to strike a balance between the number of parameters and various performance metrics.

4. Conclusions

This paper introduces a method called HyFormer to improve the super-resolution of hyperspectral images, which incorporates three novel designs: (1) To capture intra-spectra interactions of object-specific contextual information, we propose a grouping-aggregation self-attention (GASA) mechanism for transformer. In GASA, grouping self-attention aims to extract diverse fine-grained features of targets, while aggregation self-attention enables information interaction among different channels. (2) To increase the receptive field to explore inter-spectra feature extraction, we introduce wide-spanning separable 3D Attention, which enlarges the receptive field with a low parameter number to better understand the contextual information in a hyperspectral image. (3) We design a hybrid architecture to adaptively fuse features extracted by transformer and CNN modules, leveraging their complementary strengths, thereby further enhancing the performance of hyperspectral super-resolution. Extensive experiments demonstrate the effectiveness of our proposed method.

There are also some barriers that need to be further improved. For example, it is worth mentioning that irregular shapes of objects in hyperspectral images often induce significant challenges to feature extraction with CNNs and transformers. The varying scales of targets also bring difficulties to the reconstruction. In our future research, we will focus on studying how to address the above issues to further improve the performance of hyperspectral image super-resolution. Additionally, considering the real-world application scenarios, we prepare to explore super-resolution methods that allow arbitrary scaling factors. The robustness of adapting to serious degradation is also a promising research direction in the future.