Spectral Image Reconstruction Using Recovered Basis Vector Coefficients

Abstract

Spectral imaging plays a crucial role in various fields, including remote sensing, medical imaging, and material analysis, but it often requires specialized and expensive equipment, making it inaccessible to many. Its application is also limited by the interdependent constraints of temporal, spatial, and spectral resolutions. In order to address these issues, and thus, obtain high-quality spectral images in a time-efficient and affordable manner, we proposed one two-step method for spectral image reconstruction from easily available RGB images under the down-sampling schemes. Specifically, we investigated how RGB values characterize spectral reflectance and found that, compared to the intuitive and straightforward RGB images themselves, their corresponding basis vector coefficients can represent the prior information of spectral images more explicitly and are better suited for spectral image reconstruction tasks. Thus, we derived one data-driven algebraic method to recover the corresponding basis vector coefficients from RGB images in an analytical form and then employed one CNN-based neural network to learn the patch-level mapping from the recovered basis vector coefficients to spectral images. To evaluate the effect of introducing the basis vector coefficient recovery step, several CNNs which typically perform well in spectral image reconstruction are chosen as benchmarks to compare the variation in reconstruction performance. Experimental results on a large public spectral image dataset and our real-world dataset demonstrate that compared to the unaltered version, those CNNs guided by the recovered basis vector coefficients can achieve significant performance improvement in the reconstruction accuracy. Furthermore, this method is plug-and-play, with very little computational performance consumption, thus maintaining a high speed of calculation.

1. Introduction

Spectral imaging technology is a non-invasive imaging modality that obtains simultaneous spectral reflectance and spatial information, providing the physical features inherent to the surfaces of objects composed of different materials and having vital significance in visual tasks. Spectral images encompass more detailed narrow-band spectral information compared to the standard RGB images, which essentially represent down-sampled versions of spectral images attained through integration with specific wide-band spectral sensitivity functions [1] (the details are provided in Equation (1)). Due to this unique property, spectral imaging technology has been widely used in remote sensing [2,3], camouflage target recognition [4,5,6], medical diagnostics [7,8,9,10], and many other fields [11]. Studies in spectral imaging systems are, therefore, extremely important and have been broadly conducted for decades.

Persistent challenges in the design of spectral imaging systems are mainly centered on improving the resolutions of different dimensions of the spectral data (e.g., the spectral, spatial, and temporal dimensions) [12]. According to the sampling strategy of hardware systems, methods in this area can be roughly divided into two groups: the conventional full-sampling schemes [13,14,15,16] and the recent down-sampling schemes [17,18,19].

Since realistic sensors can often only obtain data in up to two dimensions at once, the initial generation of spectral imaging systems is based on time-domain imaging, using mechanical path-length scanning to supplement the information of the missing dimension [20] (e.g., the whiskbroom, pushbroom [13,14,15], sequential filter wheels [16], and acousto-optic tunable filter (AOTF) [21,22] spectral imaging systems). Under the full-sampling schemes, these conventional scanning-based methods can gather all available three-dimensional data and then plausibly reconstruct the corresponding spectral images. These methods allow a high spatial or spectral resolution but also bring the drawbacks of poor time efficiency and restricted application scenarios.

To expand the applicability of spectral imaging systems in dynamic circumstances, several techniques using down-sampling schemes have been presented to produce spectral images in a time-efficient way. These methods all capture fewer measurements than full-sampling schemes and reconstruct spectral images from incomplete data, thereby enhancing the temporal resolution and preventing data overload in a short period [23]. For instance, by replacing the mechanical path-length scanning section with computational elements (e.g., spatial light modulators (SLM) and digital micromirror devices (DMD)), the acquisition of spectral images at video rate can be accomplished [12]. In addition to this, capitalizing on recent breakthroughs in compressive sensing theory [24], computed tomography imaging spectrometry (CTIS) [25] and coded aperture snapshot imaging (CASSI) have also been developed. Especially, the introduction of coded aperture enables CASSI-like methods [26,27,28,29,30] to implement snapshot imaging, which means the ability to capture spectral data within a single exposure. However, while the above systems can move beyond the problem of poor temporal resolution, they are hindered by diminished spectral and spatial resolutions instead. Other than this, the precision optical components required by some methods are not only prohibitively expensive but also reduce the luminous flux of the whole spectral image system. In such a system, when the light source is weak, the absolute number of collected photons can be quite modest, resulting in a low signal-to-noise ratio.

To obtain spectral images more conveniently and affordably, the popular recent focus is to reconstruct spectral images from readily available data without overly specialized equipment, such as the multi-channel images obtained by one or more than one commercial cameras [31,32], or under different shooting conditions (e.g., dynamic illumination or changing angles [33,34,35]). Spectral images capture light distribution across numerous narrow spectral bands within a scene, while multi-channel images are confined to several broad or narrow spectral bands. Consequently, the reconstruction of spectral images from multi-channel images can be perceived as achieving super-resolution within the spectral domain. Due to the lack of optical components to encode spectral and spatial information, such methods can readily achieve high luminous fluxes. Common reconstruction methods encompass pseudo-inverse methods [36], Wiener estimation methods [37], and kernel methods [38,39]. These methods heavily rely on certain prior knowledge (e.g., the sparsity, the similarity in spatial or spectral dimensions, and the statistical characteristics embedded in the prior datasets) or constraints imposed on the spectral reflectance distribution. Additionally, machine-learning techniques offer promising alternatives for integrating prior knowledge into the spectral reconstruction process [40]. A prevalent strategy involves addressing the mapping from a multi-channel image to its corresponding spectral image by training an end-to-end neural network, achieving implicit transformation. This phenomenon is evident in the recent new trends in image restoration and enhancement (NTIRE) challenges [41,42]. Under this paradigm, the acquisition of multi-channel images is a crucial step in accurately reconstructing full spectral information. While the plain three-channel RGB image is the most convenient, it is inadequate for accurate spectral reconstruction. Increasing the number of channels can improve accuracy, but compromises convenience. Thus, a balance must be struck between accuracy and convenience in multi-channel image acquisition. Conversely, the meticulous selection of camera response filters has the potential to elevate hyperspectral estimation accuracy by more than 33%, even when restricting the number of channels in the input image (e.g., in the context of RGB images) [43]. This revelation prompts us to explore the possibility of enhancing reconstruction performance without the need to increase the channel count. This can be achieved by transforming the RGB image to create inputs enriched with implicit information. While this perspective offers significant value, it has received comparatively less attention in current discussions.

In summary, there are currently mutual constraints between several important attributes of spectral imaging technology that limit its widespread application. In order to facilitate the application of spectral imaging technology in more general dynamic scenarios, we intend to develop a time-efficient algorithm that can simultaneously balance convenience, and affordability. Through the analysis of the above development status, our path to that goal was formulated as an improvement to the multi-channel-image-based down-sampling schemes. In this paper, on the foundation of those common end-to-end neural network methods with no hand-crafted features (i.e., the multi-channel images are directly used as input to the network), we propose an additional basis vector coefficient recovery step to exploit the prior knowledge embedded in spectral images explicitly. Specifically, through the pairs of RGB tri-stimulus values and related spectral reflectance in the prior training spectral datasets, we estimate the spectral weight function by one data-driven algebraic method and from this, the method for calculating the basis vector coefficients is derived. Then, we employed one convolutional neural network (CNN) to learn the patch-level mapping from recovered basis vector coefficients to spectral images, utilizing its excellent fitting ability and effective use of spatial contextual information.

The innovation and contributions of this work are summarized as follows:

• We investigated the characterization of spectral reflectance by RGB values and demonstrated the superiority of their corresponding basis vector coefficients for the reconstruction of spectral images.
• Accordingly, we developed one data-driven algebraic method for recovering these coefficients and used them as inputs for the employed CNN networks.
• To ensure the convenience of the spectral imaging systems, we validated the algorithm on a large spectral dataset and our real-world dataset with RGB images as input.
• To strike a balance between accuracy and convenience, we also conducted further research to investigate the effect of channels on the reconstruction performance and offered recommendations for optimal channel selection.

2. Methodology

2.1. RGB Digital Camera Imaging Principle

The RGB digital cameras sense the reflective scene through a tri-stimulus mechanism, causing three channels to respond differently to the same target [31]. Specifically, the tri-stimulus values are formed by conflating the spectral reflectance with the incoming illumination and spectral sensitivity function of the sensors [44]. Let $s\left(\lambda \right)$ denote the spectral reflectance of a specific scene with wavelength $\lambda$. The incoming illumination and the spectral sensitivity function of the $i$ color channel $(i=r,g,b)$ are indicated by $e\left(\lambda \right)$ and $q_i\left(\lambda \right)$. In this way, the response of each color channel $y_i$ can be presented as an integral over all the range of visible wavelengths as:

$$y_i={\int }_{\mathsf{\Omega }}^{}e\left(\lambda \right)q_i\left(\lambda \right)s\left(\lambda \right)d\lambda$$

(1)

where $\mathsf{\Omega }$ is the range of visible wavelengths.

For mathematical simplicity, we set $w_i\left(\lambda \right)=e\left(\lambda \right)q_i\left(\lambda \right)$ and sample the visible wavelength at $\Delta \lambda =10 \mathrm{n}\mathrm{m}$ intervals between 400~700 nm, resulting in the spectral image of 31 channels to be reconstructed. Equation (1) can, therefore, be discretized across wavelengths and represented in vector-matrix form as Equation (2):

$$\mathbf{y}={\mathbf{W}}^{⊺}\mathbf{s}$$

(2)

where $\mathbf{W}{\in \mathrm{R}}^{31\times 3}$ is written as

$$\mathbf{W}=\left[\begin{array}{ccc}{w}_r\left({\lambda }_1\right)& w_g\left({\lambda }_1\right)& w_b\left({\lambda }_1\right)\\ ⋮& ⋮& ⋮\\ w_r\left({\lambda }_{31}\right)& w_g\left({\lambda }_{31}\right)& w_b\left({\lambda }_{31}\right)\end{array}\right]$$

and $\mathbf{y}={\left[y_r,y_g,y_b\right]}^{⊺}{\in \mathrm{R}}^{3\times 1}$ and $\mathbf{s}={\left[s\left({\lambda }_1\right),\cdots ,s\left({\lambda }_{31}\right)\right]}^{⊺}{\in \mathrm{R}}^{31\times 1}$ are both column vectors.

According to Equation (2), we can consider that the essence of spectral reconstruction is to solve for the spectral reflectance of 31 channels when only the tri-stimulus values of the RGB camera are known, leading to the ill-posed inverse process of this equation. Directly solving such an ill-posed equation cannot lead to a unique solution.

2.2. Basis Vector Coefficients

Utilizing certain prior knowledge or constraints imposed on the distribution of spectral reflectance, Equation (2) can be solved more stably. As the spectral reflectance curve of a natural scene is smooth, the spectral information with similar wavelengths is correlated and can be represented in a lower-dimensional space [45]. In other words, the spectral reflectance can be adequately approximated by the linear combination of a small number of basis vectors:

$$\mathbf{s}\cong {\mathbf{U}}_N{\mathbf{c}}_N$$

(3)

where the columns in ${\mathbf{U}}_N{\in \mathrm{R}}^{31\times N}$ and ${\mathbf{c}}_N{\in \mathrm{R}}^{N\times 1}$, respectively, denote the basis vectors and their corresponding linear combination coefficients, and $N$ is the amount of employed basis vectors.

The decomposition process of basis vectors is a planning problem that allows basis vectors to be the best approximation to the target under the defined error function. Additionally, extracted from the prior training spectral datasets, those basis vectors implicitly characterize the spectral reflectance of natural scenes, especially the statistical distribution properties that have a greater impact on the reconstruction quality, adding a solid and practical constraint to this reconstruction. There are many well-established methods that can be used to perform the extraction of basis vectors, including singular value decomposition (SVD) [46], principal component analysis (PCA) [47], t-SNE [48], and autoencoder (AE) [49], to name a few. They can provide basis vectors that minimize reconstruction errors under different defined error function senses. For instance, the basis vectors produced by the SVD method are orthogonal, devoid of redundant information, and the error in the reconstructed result concerning the targets is minimized under the Euclidean distance definition which is consistent with the mean squared error (MSE) metric commonly used to evaluate the accuracy of spectral reconstruction. Current studies have shown that the spectral reflectance of a natural scene can be reconstructed with high accuracy using a linear combination of $5~7$ basis vectors [50]. That is, this constraint is indeed practical.

The numerous properties of the basis vectors mentioned above are also inherited by their coefficients. By transforming Equation (3) as follows, we can obtain some properties of the basis vector coefficients:

$${\mathbf{c}}_N={{\mathbf{U}}_N}^{+1}\mathbf{s}$$

(4)

where the symbol ${(\ast )}^{+1}$ represents the pseudo-inverse operation. Notice that the dimensions of the basis vector coefficients ${\mathbf{c}}_N$ and the spectral reflectance $\mathbf{s}$ are $(N\times 1)$ and $(31\times 1)$, respectively, so that in the case of $N<31$, the basis vector coefficients serve as a kind of reduced dimensional representation of the spectral reflectance.

It is not difficult to find that Equations (2) and (4) have the same form (i.e., both are the fork multiplication of one matrix to the spectral reflectance vector), which indicates that the tri-stimulus values and basis vector coefficients are formed by a similar computational process and are essentially a kind of encoding of spectral reflectance. The difference is that the former is encoded using the spectral sensitivity function of RGB cameras, while the latter is encoded using basis vectors. The spectral sensitivity function of RGB cameras is initially designed to mimic the visual characteristics of human eyes, showing that they are intrinsically unsuited for spectral reconstruction tasks. Conversely, the principle of basis vector extraction algorithms guarantees that those extracted basis vectors are the best approximation to the target under the defined error function. Furthermore, these defined error functions are generally consistent with metrics for evaluating the spectral reconstruction quality (e.g., the previously mentioned SVD method’s error function conforms to the MSE metric). That is to say, compared with the intuitive and straightforward RGB images, the recovered basis vector coefficients can represent the prior information of spectral images more explicitly and are more suitable for spectral reconstruction.

2.3. Spectral Reconstruction

The specific implementation of our proposed method consists of two main steps. Firstly, it is required to calculate the corresponding basis vector coefficients from the tri-stimulus values of input RGB images. After that, under the prior spectral datasets, we need to solve the mapping from the recovered basis vector coefficients to the related spectral reflectance. Once the computational methods for recovering the basis vector coefficients and the mapping function are derived and obtained, respectively, during the inference process, we can calculate the basis vector coefficients from the input RGB images, and then map them into the spectral images. In the following content, the specific details of these two steps will be elaborated. The flowchart of the entire calculation process is shown in Figure 1, and the symbols involved are described later.

Let us first identify a specific basis vector extraction algorithm and derive how to calculate its basis vector coefficients. The basis vectors extracted from the prior spectral datasets vary with the basis vector extraction algorithm, as does the basis vector coefficients recovery algorithm. In this article, the SVD algorithm, which performs a linear representation of the spectral reflectance, is employed to extract the orthogonal basis vectors without redundant information.

By combining Equations (2) and (3), we can obtain one general method for estimating the linear combination coefficients ${\widehat{\mathbf{c}}}_N$, which is also the $f(\ast )$ in Figure 1:

$${\widehat{\mathbf{c}}}_N={\left({\mathbf{W}}^{⊺}{\mathbf{U}}_N\right)}^{+1}\mathbf{y}$$

(5)

This method is straightforward and applicable to all linear basis vector extraction algorithms, but suffers from some systematic errors and shortcomings. On the one hand, from a mathematical point of view, the inverse matrix of this singular matrix will introduce unavoidable errors since we cannot obtain a unique solution from an ill-posed equation. To solve this problem, we can set the number of employed basis vectors and imaging sensors to be equal (for RGB cameras, this means setting $N$ to 3). Then, the matrix ${\mathbf{W}}^{⊺}{\mathbf{U}}_N$ would be singular, thus avoiding the systematic errors in the pseudo-inverse process. On the other hand, in general, the spectral weight function $\mathbf{W}$ requires some sophisticated and expensive instruments to measure, which is inconvenient and unacceptable for some application scenarios. However, for a defined scenario, the spectral weight function can be estimated using pairs of tri-stimulus values and related spectral reflectance in the prior training spectral datasets. In a spectral dataset with a huge number of spectral images, let there be a total of $M$ data pairs. Then, Equation (2) can be expanded as follows:

$$\mathbf{Y}={\mathbf{W}}^{⊺}\mathbf{S}$$

(6)

where $\mathbf{Y}{\in \mathrm{R}}^{3\times M}$ and $\mathbf{S}{\in \mathrm{R}}^{31\times M}$ represent the set of tri-stimulus values $\mathbf{y}$ and spectral reflectance $\mathbf{s}$ in the whole training spectral dataset, respectively. Since there are typically far more data pairs than 31 (i.e., $M\gg 31$), this equation is over-determined. Transformed into the following optimization problem of minimizing the sum of squared errors, such a system of those over-determined equations can be solved using the linear least squares method:

$$\begin{array}{c}\underset{\mathbf{W}}{\mathit{argmin}}‖\mathbf{Y}-\widehat{\mathbf{Y}}‖=\underset{\mathbf{W}}{\mathit{argmin}}‖\mathbf{Y}-{\mathbf{W}}^{⊺}\mathbf{S}‖\underset{\mathbf{W}}{=\mathit{argmin}}{‖\mathbf{Y}{-\mathbf{W}}^{⊺}\mathbf{S}‖}^2\end{array}$$

(7)

To solve this optimization problem, firstly, let $J\left(\mathbf{W}\right)$ denote the transposed objective function to be minimized:

$$J\left(\mathbf{W}\right)={‖{\mathbf{Y}}^{⊺}-{\mathbf{S}}^{⊺}\mathbf{W}‖}^2={\left({\mathbf{S}}^{⊺}\mathbf{W}-{\mathbf{Y}}^{⊺}\right)}^{⊺}\left({\mathbf{S}}^{⊺}\mathbf{W}-{\mathbf{Y}}^{⊺}\right)$$

(8)

Then, find the partial derivative of the objective function with respect to $\mathbf{W}$:

$$\frac{dJ\left(\mathbf{W}\right)}{d\mathbf{W}}=\frac{dJ\left(\mathbf{W}\right)}{d\left({\mathbf{S}}^{⊺}\mathbf{W}-{\mathbf{Y}}^{⊺}\right)}\frac{d\left({\mathbf{S}}^{⊺}\mathbf{W}-{\mathbf{Y}}^{⊺}\right)}{d\mathbf{W}}$$

(9)

Assign it to zero to minimize it:

$$2{\left({\mathbf{S}}^{⊺}\mathbf{W}-{\mathbf{Y}}^{⊺}\right)}^{⊺}{\mathbf{S}}^{⊺}=0⟹{\mathbf{S}\mathbf{S}}^{⊺}\mathbf{W}=\mathbf{S}{\mathbf{Y}}^{⊺}$$

(10)

Obviously, the ${\mathbf{S}\mathbf{S}}^{⊺}$ is a full-rank matrix, whose inverse matrix can be found directly. In this case, the spectral weight function $\mathbf{W}$ can be obtained:

$$\mathbf{W}={\left({\mathbf{S}\mathbf{S}}^{⊺}\right)}^{-1}\mathbf{S}{\mathbf{Y}}^{⊺}$$

(11)

Once the spectral weight function is solved, the estimated linear combination coefficients can be recovered according to Equation (5).

Then, although we can calculate the spectral reflectance of individual pixels using Equation (3) and then rearrange it to form the corresponding spectral images, this linear mathematical approach is straightforward and inaccurate because it does not take the optoelectronics of the device into account. When we consider a real digital image capture device, with noise, these estimations are unpredictable [51]. In addition, this pixel-based approach fails to utilize the spatial correlation information implied in the spectral images, which limits the reconstruction accuracy. Coincidentally, the neural networks, especially those based on patch-level mapping, not only have an excellent fitting ability but also can exploit spatial contextual information, which just forms a complement. Therefore, we can employ a neural network as the reconstruction algorithm module. What distinguishes different neural network methods is the network architecture. There are a number of traditional architectures, including ResNet [52,53], UNet [54], and DenseNet [55], etc. Usually, these architectures will be modified to suit the specific needs of different scenarios. In the following content, several neural network architectures that perform well in spectral reconstruction will be briefly introduced and used as benchmarks. The first benchmark we used in this work was a deep dense network, called the HSCNN-D model [56]. It consists of several dense blocks, and plays an essential role in the research of spectral image reconstruction, for it was the best performing model in the NTIRE 2018 challenge [41]. The second benchmark is a modified UNet proposed by Zhao [57] (hereinafter referred to as the ZYYNet model). In addition to down-sampling and up-sampling in the spectral dimension like the first model, the ZYYNet model also performs those sampling works in the spatial dimension to further exploit the correlation among pixels over receptive fields of different scales, precisely the characteristic of UNet.

3. Experiments on a Public Dataset

3.1. Settings

To demonstrate the effectiveness of our proposed method, we analyze the performance of those two benchmark networks before and after introducing the suggested additional basis vector recovery phase step. The specific settings of our experiments are described below.

Spectral dataset. The experiments of our proposed method and benchmarks are conducted on one new natural spectral image dataset provided by the NTIRE 2020 challenge [42]. Containing two tracks and consisting of 510 high-quality natural spectral images in total, this dataset is one of the most comprehensive datasets available. The “clean” track aims to recover spectral images from the noise-free RGB images, while the “real world” track requires participants to rebuild the spectral images from noisy JPEG-compression RGB images created by an unknown camera response function [58]. The original spectral images have a uniform spatial resolution of $482\times 512$, and the visible wavelength range of 400–700 nm is sampled at $\Delta \lambda =10 \mathrm{nm}$ intervals.

Evaluation metrics. In order to adequately perform the quantitative evaluation of our proposed method, several metrics are used, including the root mean squared error (RMSE), mean relative absolute error (MRAE), structural similarity index (SSIM) [59], and peak signal-to-noise ratio (PSNR). Among them, MRAE is the metric recommended by the NTIRE challenges for the ability to avoid over-weighting errors in higher luminance areas of the test images [58]. In addition, compared to the other metrics, that can only evaluate the spectral reflectance reconstruction accuracy of the individual pixels, the SSIM metric is used to assess the similarity of the whole spectral image to the ground truth. For the metrics mentioned above, including RMSE, MRAE, and SSIM, all values fall within the range of 0 to 1. Higher values of SSIM and PSNR indicate superior reconstruction performance, while the converse holds true for RMSE and MRAE.

Implementation details. With reference to the employed benchmarks in this work, the metric RMSE is used as the loss function and minimized with the well-known adaptive moment estimation method (Adam) [60] by setting ${\beta }_1=0.9$, ${\beta }_2=0.999$, and $ϵ={10}^{-8}$. All neural networks are trained for 35 epochs while the learning rate is initialized as ${10}^{-5}$ and scaled to one-tenth of the previous one every 10 epochs. During the training process, RGB patches of size $256\times 256$ and their corresponding spectral data cubes are fed into the models. Data augmentation is performed by means of random horizontal and vertical flipping. All networks are implemented within the PyTorch framework and trained on an NVIDIA RTX 2080Ti hardware platform.

3.2. Results

Experiments are conducted on the official validation set to assess the effectiveness of our proposed method. The average results of the employed benchmarks with vanilla input-end (i.e., the RGB image, denoted as “direct”) and modified input-end (i.e., the recovered basis vector coefficients, denoted as “coeff”) for each metric in the two tracks are presented in

Table 1. Quantitative results of spectral reconstruction on the public dataset. The better result is shown in bold.

Data Type	Metrics	ZYYNet		HSCNN-D
		Direct	Coeff	Direct	Coeff
“Real world” track	RMSE	0.03084	0.02371	0.04465	0.02988
	MRAE	0.16508	0.11817	0.23345	0.15860
	SSIM	0.88178	0.91654	0.86875	0.90669
	PSNR (dB)	29.883	32.378	27.102	30.001
“Clean” track	RMSE	0.03130	0.02301	0.04596	0.02772
	MRAE	0.15510	0.10936	0.22371	0.14196
	SSIM	0.88884	0.93197	0.88509	0.93387
	PSNR (dB)	29.705	32.736	26.943	30.935

. It can be clearly seen that after modifying the input-end of the networks to the recovered basis vector coefficients, the overall reconstruction accuracy is significantly improved with all the selected metrics.

As a further analysis, we also performed a statistical analysis based on the paired t-test using a series of boxplots (shown in Figure 2). The asterisk above each pair of data indicates that there is a statistically significant difference between these paired data (* p < 0.05). The spacings in each subsection of boxplots visually show the distribution of numerical data and skewness by displaying their data quartiles, which is also referred to as the five-number summary (i.e., the minimum, maximum, median, lower quartile, and upper quartile). In contrast to the mean values in Table 1, the results of the paired t-tests show that the proposed method not only improves the performance of the benchmarks on the validation set as a whole but also achieves a statistically significant improvement in the reconstruction accuracy of each sample. Except for the oddity of the HSCNN-D network on the MARE metric (Figure 2b, the part with a light-yellow background), the two benchmark networks with modified input-end demonstrated an improvement in each data quartile and in the median values on the various metrics and tracks. In addition, the decreased interquartile range (IQR) of almost all of the boxplots (except for the HSCNN-D network’s performance on the MRAE and SSIM metric under partial tracks), indicates that the variance of each metric has also decreased, thus further ensuring the reliability and consistency of the spectral image reconstruction.

3.3. Visualization

To evaluate the perceptual quality of the spectral reconstruction results, two metrics, RMSE and MRAE, are selected to draw the error maps between several recovered spectral images and their corresponding ground truths (see Figure 3 and Figure 4). The values of those two metrics all range from 0 to 1, and smaller values represent better reconstruction results (i.e., the blue parts in the error maps mean better reconstruction results). Obviously, the error maps of the “coeff” series are bluer than those of the “direct” series, no matter from the perspective of the overall or local texture. These figures demonstrate that our proposed method outperforms those benchmarks in terms of recovery results and reconstruction fidelity.

4. Experiments on the Real-World Dataset

4.1. Settings

In practice, we construct a hybrid-resolution imaging system for capturing real-world spectral images along with their corresponding RGB images. Figure 5 illustrates the configuration of the imaging spectrometer (Pika L, Resonon Inc., Bozeman, MT, USA), commercial color camera (EOS 6D Mark II, Canon Inc., Kyoto, Japan), and the arrangement of the light sources employed in this system. To account for the disparity in physical location and view field between the imaging spectrometer and the commercial color camera, the SIFI algorithm is utilized to detect and match feature points, thereby achieving image registration. An illustration set is utilized as the object to be captured, and some of the registered RGB images are shown in Figure 6.

Implementation details. In order to achieve improved reconstruction results, adjustments were made to various hyperparameters and learning rate strategies of the neural network, considering the differences in the intrinsic properties of the real-world dataset. Specifically, all neural networks were trained for 60 epochs while the learning rate was initialized as $4\times {10}^{-4}$ and the cosine annealing scheme was adopted.

4.2. Results

The average results of the employed benchmarks with vanilla input-end and modified input-end for each metric on the real-world dataset are presented in

Table 2. Quantitative results of spectral reconstruction on the real-world dataset. The better result is shown in bold.

Metrics	ZYYNet		HSCNN-D
	Direct	Coeff	Direct	Coeff
RMSE	0.08995	0.03487	0.09201	0.03894
MRAE	0.29733	0.14442	0.29992	0.14397
SSIM	0.78487	0.87190	0.78376	0.86707
PSNR (dB)	20.291	28.187	20.103	27.431

. Due to the degradation of the dataset quality caused by image registration, there has been an overall decline in the accuracy of the reconstruction. However, with the incorporation of our proposed basis vector coefficient recovery step, the neural network can still maintain a favorable level of reconstruction accuracy.

5. Discussion

5.1. Computational Efficiency and Flexibility

Essentially, our proposed method is formed by modifying the input-end of common end-to-end neural networks from RGB images to corresponding basis vector coefficients. Compared to the original benchmark, the difference is the addition of a module for recovering the basis vector coefficients. This module can be applied to any neural-network-based systems which reconstruct spectral images directly from RGB images, demonstrating the flexibility of our proposed method. In addition, this module is performed by a linear matrix operation with negligible computational effort, thus having the advantage of high computational efficiency.

5.2. The Effect of Channels

In this work, we set the quantity of employed basis vectors to three to match the number of channels of RGB cameras, for those conventional commercial cameras can only acquire up to three channels of images. However, for a broader range of cases, the multi-channel images used as model inputs may have a larger number of channels. In fixed industrial application scenarios, for example, a certain level of convenience can be sacrificed to obtain images with more channels. Intuitively, more channels of input multi-channel images can lead to a higher reconstruction accuracy. However, due to the high redundancy of spectral reflectance, the unlimited addition of channels does not effectively improve reconstruction accuracy. Instead, it increases the difficulty and cost of sampling. Here, we will discuss the effect of channels on reconstruction accuracy to guide the design of high-precision spectral imaging systems.

Firstly, the multi-channel images are synthesized as inputs to models by imitating the setup of the “clean” track. When the number of channels is greater than three, RGB cameras no longer apply. Thus, without using RGB cameras as the image capture device, there is no need to take the visual properties of the human eyes into consideration anymore, and there is more freedom in choosing the spectral sensitivity function of the imaging sensors. Filters, which have any required number of evenly spaced peaks, can be yielded by the concept of equivalent indices and the theory of the Fabry–Perot etalon [61]. Thus, Gaussian functions with evenly distributed peaks are used to synthesize the response of imaging sensors from the given spectral dataset in the hope of obtaining more fully unduplicated spectral information. In addition, a D65 light source was used for the incoming illumination, and the ZYYNet was employed as the reconstruction algorithm module with similar settings as described in Section 3.1.

The trend in spectral reconstruction accuracy under each metric as the number of channels grows is shown in Figure 7. Not surprisingly, as the number of channels grows, the performance of spectral reconstruction in each metric improves. This growth becomes slower when the number of channels is larger than six, which corresponds to the current studies [50].

In terms of the design of image capture devices, suppose a color camera is used to acquire the multi-channel image. In that case, the number of channels of the acquired images is a multiple of three, whether multiple cameras or dynamic illumination are used. In addition, modifying the Bayer filter in the camera to a $2\times 3$ or $3\times 3$ array also makes it convenient to obtain images with six or nine channels in one snapshot at an acceptable cost of spatial resolution. Considering the above reasons, in follow-up studies, we recommend that both the channels of multi-channel images and the number of employed basis vectors are set to six or nine to meet the physical constraint from basis vector theories and balance the convenience and reconstruction accuracy. However, in some application scenarios, where convenience is sought over accuracy, the direct use of RGB cameras to acquire images for spectral image reconstruction is still an excellent option.

6. Conclusions

In this work, we proposed an improved method for spectral image reconstruction from easily available RGB images. We observed that compared with the intuitive and straightforward RGB images, the recovered basis vector coefficients can represent the prior information of spectral images more explicitly and are more suitable for spectral reconstruction. Specifically, through pairs of tri-stimulus values and related spectral reflectance in the prior training spectral datasets, we can estimate the spectral weight function, and, from this the method for calculating the basis vector coefficients is derived. Then, to exploit the spatial contextual information of spectral images, and thus improve the reconstruction accuracy, we employed a CNN-based neural network to learn the patch-level mapping from the basis vector coefficients to the corresponding spectral image. As the experiment results demonstrated in one large dataset, modifying the input-end of common end-to-end neural networks from RGB images to corresponding basis vector coefficients can effectively improve the reconstruction performance.

Additionally, we have discussed the effect of channels on the reconstruction performance and recommend that both the channels of multi-channel images and the number of employed basis vectors should be set to six or nine to meet the physical constraint from basis vector theories and balance the convenience and reconstruction accuracy. Future work would be to develop a high-precision spectral imaging system based on multi-channel images and explore whether the overall performance can be further improved when other more sophisticated nonlinear methods are used to calculate the basis vector coefficients.