The Development of Snapshot Multispectral Imaging Technology Based on Artificial Compound Eyes

Abstract

In the present study, the advantages of multispectral imaging over hyperspectral imaging in real-time spectral imaging are briefly analyzed, and the advantages and disadvantages of snapshot spectral imaging and other spectral imaging technologies are briefly described. The technical characteristics of artificial compound eyes and multi-aperture imaging and the research significance of snapshot artificial compound eye multispectral imaging are also introduced. The classification and working principle of the snapshot artificial compound eye multispectral imaging system are briefly described. According to the realization method of the optical imaging system, the ACE snapshot multi-aperture multispectral imaging system is divided into plane and curved types. In the planar compound eye spectral imaging system, the technical progress of the multispectral imaging system based on the thin observation module by bound optics (TOMBO) architecture and the multispectral imaging system based on the linear variable spectral filter are introduced. At the same time, three curved multispectral imaging systems are introduced. Snapshot artificial compound eye multispectral imaging technology is also briefly analyzed and compared. The research results are helpful to comprehensively understand the research status of snapshot multispectral multi-aperture imaging technology based on artificial compound eyes and to lay the foundation for improving its comprehensive performance even further.

1. Introduction

In comparison to hyperspectral imaging (HSI), multispectral imaging (MSI) uses a lower number of bands in exchange for greater spatial resolution, which not only maintains the advantages of the high spectral resolution and strong recognition ability of the HSI system, but also has the characteristics of high spatial resolution and minor distortion of the single-band array imaging system. In practice, according to the specific application scenario, the spectrum characteristics, and actual target, choosing the corresponding band combination, low coupling between the band, and achieving a better balance for considerable amounts of information are important factors, to effectively overcome the traditional hyperspectral data dimension being high and the real-time data processing dealing with a large amount of data and significant uncertainty, as well as the difficulty of performing sample selections and other shortcomings. It has the advantages of being more economical and convenient, and having a high signal-to-noise ratio and simple data processing techniques. Therefore, MSI plays an increasingly important role in several areas of spectral imaging within specific environments with high real-time requirements [1,2].

The spectral imaging system obtains a three-dimensional (3D) dataset of the target scene by collecting two-dimensional (2D) spatial information (x, y) and one-dimensional spectral information (λ), which is called a data cube (x, y, λ). As is shown in Figure 1, spectral imaging can be divided into scanning and snapshot types according to the method of obtaining a complete data cube. Spectral scanning also includes three scanning categories: point, line, and wavelength [3,4]. Among them, point (also known as sweep) and line (also known as push sweep) spectral scanning belong to the spatial scanning type, which needs to complete the 1D or 2D slicing of the target scene according to the time series with the help of complex optical paths and scanning mechanisms. Therefore, the system has a complex structure and large volume, and it is easy to introduce motion and registration errors; the spectral scanning (also known as staring- or frame-imaging types) requires changing different filters or obtaining 2D images of one spectral segment at a time through an internal beam-splitting system. Therefore, the spectral scanning imaging system, in principle, is unable to perform spectral imaging on moving targets and rapidly changing scenes [5,6,7].

The snapshot MSI system, usually consisting of multiple independent imaging units of different spectral characteristics, using a single or multiple focal-plane detector (FPA), under the condition of no scanning mechanism, through a single exposure can obtain more than a 2D spectrum image at the same time, through the post-processing combination of 2D data for the 3D data cube. With the ability to capture dynamic or static objects located in dynamic environments, it is a powerful tool used to accurately obtain 3D data cubes. At the same time, the snapshot MSI system also possesses the characteristics of rapid acquisition speed, small size, compact structure, light weight, and portability, which has become the research hotspot at present, and a variety of snapshot MSI technologies have also been developed and are widely used in military reconnaissance, public security criminal investigations, biomedicine, life science, food safety, ecological monitoring, precision agriculture, and other military and civilian fields [3,4,7,8,9].

In nature, the compound eyes of insects are considered to be an ideal, miniaturized multi-aperture and large field-of-view (FOV) optical imaging system with a good, intelligent detection ability, including the high-sensitivity detection of moving targets and the high-resolution detection of light intensity, wavelength (color), or polarization [10,11,12,13]. The ACE is a new type of multi-aperture vision system that simulates the natural structure and function of insects’ compound eyes. It is the synthesis and development of photoelectric and micro-photoelectric technology to date, and it is a highly integrated miniaturized, lightweight, and intelligent imaging system. In comparison to the single-aperture vision system, artificial compound eyes (ACEs) break the constrained relationship between a large field of view and high resolution. It has optical characteristics, such as large FOV, low aberration, and infinite depth of field, as well as outstanding properties, such as better target-motion-detection ability, higher light-intensity sensitivity, and more compact structure size, and it also has good application potential. Thanks to the rapid development of micro–nanomaterials technology, processing technology, and image-sensor technology, ACE visual imaging technology has become a popular, innovative research field in the literature, is rapidly developing, and will soon be used in the field of compact multi-aperture MSI [14,15,16,17].

The snapshot compound eye MSI system, as a special compact multi-aperture photoelectric imaging system, has multiple independent microlens imaging units; it can, according to the specific tasks for different imaging units assigned to different optical systems and imaging conditions, obtain a high-frame frequency and multiple sets of spectra of the target image; it can rapidly access information to test the integrity of the data cube. It is characterized by its compact hardware, powerful function, flexible design, and diverse integration, and has good universality and scalability properties. Its working mode is similar to ordinary cameras, and it is easy to be implemented in engineering. In recent years, it has attracted considerable attention from researchers at home and abroad, and has become an important research topic in image-science studies [18,19,20].

In Section 1, the main classifications and working principles of snapshot ACE MSI technology are briefly described; Section 2 focuses on the progress of planar ACE multispectral snapshot imaging technology in the research; Section 3 introduces the progress of curved compound eye multispectral snapshot imaging technology; finally, snapshot compound eye MSI technology is briefly summarized in the Conclusion.

2. System Classification and Basic Principles

The compound eye of insects has attracted the attention of numerous experts and scholars at home and abroad, because of its compact structure, multimodal imaging, and unique optical properties. After 20 years of research, several achievements in the research have been attained on the planar ACE with a relatively simple structure, and polytype planar ACE MSI systems have also emerged. However, due to the nature of insect compound eyes being very complex, there are several constraints and limitations regarding the existing materials, processes, and technologies. At present, the research on ACEs remains in the simulation and realization levels of curved microlens arrays (MLAs). Several issues still exist in relation to the realization of the complete curved ACE system, and this system is also under investigation in the research, at present [21,22,23,24].

2.1. Planar ACE Multispectral Snapshot Imaging System

As a novel photoelectric imaging system integrated with ACE, the planar ACE multispectral snapshot imaging system consists of four parts: a planar MLA, spectral interference filter array, signal isolation structure, and image sensor. At the same time, the planar ACE multispectral snapshot imaging system is also a multichannel spectral imaging array. Each interference filter of the filter array corresponds to a microlens. The interference filter and microlens jointly form a spectral imaging unit, and a sub-image of the spectrum segment can be obtained on the corresponding band. Through the acquired multispectral image sequence, the data cube of the target scene is obtained, as is presented in Figure 2. It should be noted that there can be two installation positions for the interference filter array. One is in front of the MLA, as is presented in Figure 3a. The other is positioned after the MLA and before the FPA, as is presented in Figure 3b [3,4].

In general, the microlenses used in the planar ACE multispectral snapshot imaging system is exactly the same, and the FOV of the imaging unit is basically the same; therefore, it not only reduces the overall cost of the system, but is also relatively easy to implement in engineering, and the image reconstruction algorithm is relatively simple. In addition, due to the use of interference filter arrays and the relatively mature focal plane image sensors available on the market, when the resolution of the system is certain, more spectral channels are obtained and the spectral resolution is higher; however, the smaller the aperture of the microlens, the lower the optical flux and spatial resolution characteristics.

Due to the use of microlenses, the ACE usually operates at a relatively close distance. In order to achieve the MSI of distant targets, different methods of enlarging the focal length were proposed in the study. For example, the first method was to install a telescopic objective lens in front of the MLA. The system first images the target through the front telescopic objective lens, and then realizes a secondary image on the detector through the MLA, whose structure is presented in Figure 4a. Another method is to add a telescopic objective lens in front of the MLA to compress the FOV of the incident beam, and the rear MLA performs secondary imaging at the exit pupil of the telescope system, whose structure is presented in Figure 4b [8].

2.2. Curved Compound Eye Multispectral Snapshot Imaging System

The curved compound eye multispectral snapshot imaging system usually adopts a curved MLA integrated with an interference filter to simulate the multi-aperture spectral imaging function of a natural insect parallel compound eye. As presented in Figure 5, the parallel compound eye optical system consists of multiple independent sub-eye imaging units, the whole sub-eye is distributed on a curved surface, and the photosensitive units of all sub-eyes are also distributed on a curved surface. However, at present, all mature image sensors are planar, so it is difficult for artificial ACEs to adopt a curved MLA to be directly coupled with it, and it is difficult to obtain clear target images on planar detectors [15,24,25].

There are two schemes to solve the problem of image plane coupling that occurs between a curved MLA and FPA.

Scheme 1: Direct imaging method. This method requires each microlens to be imaged directly onto a focal plane image sensor. In order to adapt to the planar image sensor, it is necessary to design each imaging unit on the surface independently, and to create the designed MLA through special structures, technology, and flows to ensure that each microlens can focus on the FPA, as presented in Figure 6. The advantages of this method are that the structure of the imaging system is relatively simple and compact, but the design and processing features cause difficulties, and the image quality is not easy to guarantee. At the same time, the number of microlenses and the spectral resolution are low.

Scheme 2: Secondary imaging method based on relay image transfer. In this method, a relay image transfer system is introduced between the curved lens array and image sensor to convert the image information of the curved MLA to ensure that clear sub-images of each imaging unit can be obtained on the image sensor, as presented in Figure 7. The advantages of this method are that it reduces the difficulty of the design and fabrication of the MLA, the number of microlenses is high, and the system FOV is relatively large. However, due to the introduction of the relay image transfer system, the structure and volume of the system will increase [24].

3. Planar Compound Eye Multispectral Snapshot Imaging System

3.1. Compound Eye MSI System Based on TOMBO

In the year 2000, Tanida’s team from Osaka University in Japan pioneered the TOMBO (Thin Observation Module by Bound Optics), a new multi-aperture optoelectronic imaging system inspired by arthropod compound eyes that consists of multiple independent optoelectronic imaging units; a series of low-resolution sub-images can be obtained on the same CMOS photo-focal plane imaging structure, as presented in Figure 8. Through digital image-processing technology in the latter, high-resolution images can be reconstructed [26,27,28,29,30,31]. Indeed, TOMBO is not a simple imitation of the compound eye organ of insects in nature, but rather an efficient way to adapt to focal plane sensors at present. This method is also considered as an effective computational imaging platform based on multi-aperture imaging, and has been thoroughly studied by numerous scholars [19,32,33].

In the year 2003, based on this platform, Tanida’s team proposed the color TOMBO compound eye imaging system; the effect picture is shown in Figure 9. By integrating the RGB wideband filter array in the compound eye lens array, color images were reconstructed, which laid the technical foundation for the realization of the TOMBO multispectral compound eye imaging system [34,35,36,37,38].

3.1.1. Compound Eye MSI System Based on TOMBO Architecture

In the year 2003, Shogenji et al. from Osaka University in Japan proposed a compact compound eye MSI system based on TOMBO architecture, which is also the earliest compound eye snapshot multispectral camera created to date [8]. Compared to the traditional TOMBO system, the MSI system integrates an interference narrow-band filter for each lens element at the front end of the TOMBO MLA. Therefore, the system can generally be divided into four parts: the MLA, optical signal isolation device, color interference filter array, and image sensor, as is presented in Figure 10. The team constructed the system’s experimental prototype and conducted a feasibility experimental verification of the static scene [39,40].

The MLA used an orthogonal right-angle MLA (array positive orthogonal—APO) as the main imaging unit, model APO-Q-P500-AF1.3, composed of quartz glass with a lens spacing of 500 μm, 1.3 mm, and a lens diameter of 500 μm. Figure 11 presents the structural diagram of this series of MLA [41].

The CMOS image sensor’s pixel value was 1040 × 960. The pixel size was 6.25 μm and the bit depth was 12 bits. The number of pixels corresponding to a single imaging unit was 80 × 80; therefore, a maximum of 12 × 12 imaging units could be achieved. The actual effective pixels were 960 × 960 and there were 40 pixels on the left and right sides used for the alignment and mounting of the lens array, respectively.

It should be noted that the experimental validation did not use an interference filter array, but instead used narrow-pass interference filters with different wavelengths of seven FS40-VIS-2.00 obtained from CVI Laser Corp (New York, NY, USA), filter characteristics as shown in Figure 12. The central wavelengths of the interference filter were 400, 450, 500, 550, 600, 650, and 700 nm. By successively changing the different interference filters, array images of different wavelengths were obtained. Then, according to the designed spectral filter array template, the 960 × 960-pixel image was cropped and reconstructed, and the compound eye image using the spectral filter array template was synthesized. Although this method does not produce the filter array, it can achieve experimental verification. According to the obtained sub-image array, further processing was conducted to reconstruct images of different wavelengths with 480 × 480 pixels and 12 bits of bit depth.

The optical signal isolation devices used were 21 stainless-steel plates (50 μm thick). This consisted of a square-hole array fabricated by the etching process. The inner surface of the square-hole array was coated with an antireflective film. The wall thickness and height were approximately 50 and 1050 μm, respectively. The hole spacing was 500 μm, which matched the microlens spacing.

The experimental prototype of the compact compound eye MSI system is presented in Figure 13, and the main parameters are summarized in

Table 1. Characteristic parameters of the compact TOMBO multispectral compound eye imaging system.

System Unit	Key Parameters	Parameter Values
Microlens array	Model	APO-Q-P500-AF1.3
	Material	Quartz glass
	Lens-to-image distance (μm)	500
	Focal length of lens (mm)	1.3
	Lens diameter (μm)	500
Imaging sensor	Type	CMOS
	Pixel number	1040 × 960
	Size of pixel (μm)	6.25 × 6.25
	Number of cell pixels	80 × 80
	Imaging unit	12 × 12
	The bit depth of the image	12 bits
	Effective pixels	960 × 960
Interference filter	Central wavelength (μm)	400, 450, 500, 550, 600, 650, 700
	Spectral coverage	7
	Distance packing glass (μm)	0.3
Optical isolation device	Thickness (μm)	50
	Height (μm)	1050

.

The authors proposed an improved pixel rearrangement method that rearranged the pixels in the captured image geometrically onto a multi-channel virtual image plane. The processing flow is presented in Figure 14. Processing for the reconstruction consists of pre-processing, pixel-remapping, and post-processing stages.

To study the performance of the proposed method, a color image was captured using the prototype multispectral TOMBO system. Figure 15 presents some images of fruits and vegetables we obtained, as well as the reconstructed multispectral image results of different wavelengths. The image size was 480 × 480 pixels and the image bit depth was 12 bits. Figure 16 presents the RGB color image, which was obtained by applying a spectral-RGB conversion to the seven spectral images. This study only provides a brief introduction to the specific methods; please refer to the references for further information.

3.1.2. Compound Eye High-Speed Multispectral 3D-Imaging System Based on TOMBO

In the year 2010, Kagawa et al. from Osaka University in Japan proposed a compound eye high-speed multispectral 3D-imaging system based on TOMBO architecture [42]. The authors used a narrow-pass filter array and a rolling shutter mode of a CMOS image sensor to achieve a 2D decomposition of imaging wavelength and time, as presented in Figure 17. The imaging system consisted of three parts: an orthogonal array of N_x × N_y lenses, an optical crosstalk isolation array, and a CMOS image sensor. Among them, the CMOS image sensor functioned in the rolling shutter mode, and there were N_y imaging units with the same frequency bands in the vertical direction; these N_y imaging units were successively exposed in time. Therefore, N_y sub-images with the same frequency bands with different exposure times could be obtained within a frame period, and the effective frame rate was N_y times that of the original image sensor. At the same time, there were interference narrow-pass filters of different frequency bands of the N_x group in the horizontal direction to allow us to obtain the sub-images of different frequency bands of the N_x group.

Based on the abovementioned principles, the authors used an SXGA monochromatic CMOS image sensor, optical crosstalk isolation device, commercial 5 × 5 element MLA, and commercial color interference filter to build the experimental prototype and conduct the corresponding experimental verification. Figure 18 shows the workflow of the compound eye high-speed multispectral 3D-imaging system based on TOMBO architecture.

Experimental prototype parameters of the high-speed multispectral 3D-imaging system are shown in

Table 2. Specifications of the TOMBO prototype.

Imaging Unit	Lens Size (mm)	Lens Focal Length (mm)	Pixel Size (μm)	Number of Cell Pixels	Field Range	Central Wavelength (nm)
5 × 5	0.85 × 0.85	2.35	3.2 × 3.2	220 × 220	17° × 17°	440, 500, 550, 600, 640

. The filter array used Fujifilm gelatin band-pass filters to obtain wavelength values of 500, 550, and 600 nm, and Kodak deep tricolor filters and Fujifilm sharp-end filters to obtain wavelengths of 440 nm and longer than 640 nm, respectively.

Figure 19 presents a 5 × 5 array filter image.

Based on the experimental prototype, the authors conducted the multispectral snapshot imaging of the rotating fan and verified its MSI characteristics for a high-speed moving target. Figure 20a presents the fan target image, with white letters marked on blades of different colors; Figure 20b presents the captured compound eye multispectral image. The original frame rate of the CMOS image sensor was 20 fps, but the actual frame rate was approximately 100 fps, which was 5 times the original frame rate (N_y = 5).

Based on the experimental prototype, the authors conducted the multispectral snapshot imaging of the rotating fan and verified its MSI characteristics for a high-speed moving target.

3.1.3. Multispectral Motion Imaging System Based on TOMBO

In the year 2020, Nakanishi et al. from Osaka University in Japan studied the TOMBO architecture multispectral motion imaging system used in a field environment [43]. The authors developed a prototype by connecting an embedded computer to the multispectral TOMBO imaging system. The prototype was mounted on the UAV, and the airborne observation experiment was conducted to verify the feasibility and potential ability of the motional TOMBO MSI system in the field application.

Table 3. Specifications of multispectral TOMBO.

Number of imaging units	3 × 3
Lens type	Achromatic lens
Lens focal length	1.5 mm
Lens F number	6
Lens diameter	1 mm
Prototype quality	45 g
Image sensor	UI-1482LE-M
Size of pixel	2.2 μm × 2.2 μm
FOV of the prototype	50° × 50°
Sub-image pixel	550 × 550
Power consumption of the prototype	2.4 W
Spectral channels	8
Central wavelength of a narrow-pass filter	450, 520, 650, 740, 770, 850, 920, and 970 nm

summarizes the characteristic parameters of the experimental prototype. There are 9 channels and 8 spectral channels in total, and the spectral channel with a wavelength of 450 nm was used to obtain a stereo view.

Figure 21a shows the UAV used in the field flight experiment; Figure 21b presents the multispectral TOMBO load on the UAV.

Figure 22 presents the multispectral images of a ground oil furnace and vehicle target obtained from the field flight experiment of the TOMBO multispectral motion imaging system. The flight altitude was between 3 and 7 m.

Finally, the author used the normalized vegetation index (NDVI) to analyze and evaluate the image obtained, as presented in Figure 23. The experimental results show that the system can basically complete the target classification, which proves the effectiveness of TOMBO multispectral motion imaging when performing target-recognition activity.

3.1.4. Low-Cost MSI System Based on TOMBO

According to Scott A. Mathews from the Catholic University of America, MSI systems based on interferometric bandpass filters have great potential advantages in terms of cost, volume, and imaging form. With the rapid development of large-array image sensor cameras, the system cost will be increasingly reduced. The spectral imaging system based on TOMBO architecture can effectively solve the problem of motion targets, which is an effective and low-cost solution for obtaining snapshot spectral imaging. Based on the abovementioned comprehensive considerations, the author proposed a low-cost multi-aperture MSI system based on a commercial large-array image sensor CCD in the year 2008 and constructed a system prototype. The main parameters are presented in

Table 4. Specifications of the low-cost MSI system.

System Unit	Key Parameters	Parameter Values
Camera (LW11059)	CCD models	KAI-11002
	Target surface size	36.1 mm × 24 mm
	Picture element quantity	4008 × 2672
	Pixel size	9 μm × 9 μm
Lens	Quantity	18
	Focal length	5.9 mm
	F#	3
Filter array	Number of narrow-pass filters	17
	FWHM	8–10 nm
	Number of neutral filters	1 (OD3)

[44]. The system can be considered as a development of the TOMBO MSI system, with a total of 18 commercial lens imaging units (mounted on a lens array plate, as presented in Figure 24). Under the illumination condition of the active light source with the help of two 75 mm diameter, 15 mm focal-length flat convex lenses (as presented in Figure 25), the target scene was imaged on 18 sub-units at an appropriate rate, ensuring that the system could obtain 18 high-quality target sub-images in a short enough exposure time (as exhibited in Figure 26. The area marked in black in (a) is 400 × 400 pixels to reduce the artifacts and blur caused by motion, and the reconstructed data cube (x, y, λ) could attain dimensions of 400 × 400 × 17. The authors believe that the system can be used as a general platform for other forms of multi-aperture imaging.

3.1.5. Application of Compound Eye Multispectral Endoscopy Based on TOMBO

In the year 2006, Yamada et al. from the Hiroshima Institute of Technology in Japan first conducted a preliminary experimental verification of TOMBO for 3D endoscopy technology [45]. In the year 2011, Kagawa et al. from Shizuoka University in Japan applied TOMBO compound eye MSI technology to a 3D multifunctional compound eye endoscopic system, and combined it with wavefront coding technology, developing a principal prototype of the TOMBO endoscope based on a 3 × 2 aspheric lens array. The prototype presented in Figure 27a,b shows the image captured when observing the interior of a pipe with texture and a diameter of 20 mm. The main parameters are presented in

Table 5. Parameters of the TOMBO endoscope prototype.

System Unit	Key Parameters	Parameter Values
Image sensor	Pixel number	640 × 480
	Size of pixel	3.6 μm × 3.6 μm
Lens	Quantity	3 × 2
	Focal length of lens	1.1 mm
	Lens F#	3.5
	Lens diameter	0.8 mm
	Whole view	40°
	Image height	About 0.56 mm
	Distance between the lens	0.85 mm × 0.85 mm
Filter array	Color filter	RGB Bayer

[20,46].

In the year 2012, Kagawa et al. proposed TOMBO-based compound eye variable FOV visible-light and near-infrared polarization endoscopy technology. They introduced fixed and moving mirrors to control the FOV, polarization, and wavelength of the system, and realized a variety of observation modes, such as 3D-shape measurements, wide FOV, and the close observation of tissue structures under the skin [47]. Based on this form of technology, in the year 2014, Kagawa et al. proposed the TOMBO compound eye compact endoscope technology, which can obtain a wide FOV, close range, considerable depth of field, and 3D-snapshot MSI by adding a narrow-band filter to the MLA. Figure 28 presents the multispectral image obtained using the experimental prototype.

In the year 2013, Yoshimoto et al. from Japan estimated the stiffness of the target and measured the 3D deformation of the target by measuring the deformation of the transparent silicone rubber projection pattern using a TOMBO compound eye endoscope [48]. The endoscopic detection process is shown in Figure 29.

3.2. Multi-Aperture MSI System Based on LVF

3.2.1. Ultra-Compact Multi-Aperture Snapshot MSI System

In the year 2018, Hubold et al. from the Fraunhofer Institute of Applied Optics and Precision Engineering in Germany conducted a study on an ultra-compact multispectral snapshot imaging system [49,50,51,52] (see Figure 30). The authors designed and manufactured an MLA using up-to-date advanced micro-optical manufacturing technology combined with a conventional full-frame-format image sensor and a commercially linear variable spectral filter (LVF). A multispectral experimental prototype with a wide FOV and high spatial resolution was constructed, and a snapshot image acquisition of 11 × 6 spectral channels in the wavelength range of 450~850 nm was realized. The size of the prototype was only 60 × 60 × 28 mm³, the weight was 200 g, the maximum FOV could reach 68°, the linear spectral sampling was approximately 6 nm, and the single-channel spatial sampling was 400 × 400 pixels.

The imaging system mainly consisted of three micro-optical structure layers, as presented in Figure 31. The customized MLA was used as the main layer, which enabled the parallel imaging of the object using a single image sensor. In order to avoid optical crosstalk from occurring between adjacent imaging channels, a 3D aperture structure was designed as an optical isolation array as the second layer. A circular aperture was placed above the optical isolation structure, which matched the size and position of a single aperture of the MLA. The circular aperture below the optical isolation layer was required to maximize the single-path FOV and effectively utilize the available area of the sensor. The LVF was set as the third layer above the MLA, and the LVF was set at a certain inclination angle relative to the MLA, to ensure that each channel had a corresponding specific wavelength range. The micro-optical imaging system was integrated on a full-frame CCD image sensor of the same size as the LVF to realize the multi-aperture multispectral snapshot imaging. A total of 66 spectral sub-images were obtained via the detector.

The key to the system design was the relationship of the positions of the LVF, MLA, and FPA detectors. As presented in Figure 32, if there is no angle present between the LVF and MLA, then in the vertical direction, the spectral images obtained from the five imaging units (−2~3) are the same, and only the spectral images with a total of ten imaging units (−5~5) can be obtained in the horizontal direction. The number of effective spectral images is low, which also leads to the waste of imaging resources. When a certain angle is set between the LVF and MLA, the spectral images obtained by the five spectral units in the vertical direction are different, which considerably increases the number of spectral channels present in the system.

The total length of the system was 7.2 mm, the focal length was 3.65 mm, the F number was 7, the FOV was 68°, the spatial angular resolution was 0.12°, each channel was approximately 400 × 400 pixels, and the optical dispersion spot was the same size as a pixel. The main parameters of the system are summarized in

Table 6. Overview of the design parameters of the ultra-compact micro-optical system for MSI.

System Unit	Key Parameters	Parameter Values
LVF (LF103245)	Size (L × W × H)	50 × 25 × 1 mm3
	Mean transmittance	60–90%
	Central wavelength	λcenter: 450 nm–880 nm
	Bandwidth	2%*λcenter
	Distance to the MLA	0.3 mm
	Dip angle	9.5°
MLA	Aperture diameter (front)	0.51 mm
	Thickness of glass substrate	1.75 mm
	Diameter of microlens	1.6 mm/1.4 mm
	Radius of curvature of the microlens	1.94 mm
	Aperture diameter (rear)	1.6 mm
	Microlens spacing	3 mm
Filter array	Aperture diameter (front)	1.6 mm (round)
	Thickness	2.41 mm
	Maximum aperture (rear)	2.2 mm
	Distance from the enclosed glass	0.3 mm
Cover glass	Thickness	0.75 mm
	Distance to the image sensor	0.57 mm

.

Figure 33 presents the MTF curves of the optical system with three spectral channels and two fields of view. The pixel size of the image sensor was 7.4 μm and the corresponding cutoff frequency was 67 LP/mm. In the entire FOV and spectral range, the MTF value at the half-cutoff frequency was a little higher than the expected value of 0.5.

The LVF was placed close to the MLA aperture’s surface at an inclination of approximately 9.5° between the LVF and MLA, as presented in Figure 34.

The image sensor was a KAI-16000 full-frame CCD, the target surface size was 24 × 36 mm², the pixel size was 7.4 μm × 7.4 μm, the resolution was 4872 (H) × 3248 (V), and the number of pixels was 16 M. The image sensor (including mechanical structure), optical isolation array, and system prototype are presented in Figure 35.

Figure 36a shows the original spectral sub-images obtained using the prototype. Each image channel has a FOV of 68° and a spatial resolution of up to 400 × 400 pixels, spectral sampling at 6 nm steps, and a spectral range of 450 nm~850 nm. Figure 36b depicts the target scenario.

The experimental prototype and commercial spectrometer (1 nm spectral sampling) were used to image green leaves at different stages, and the obtained images were compared to verify that the system had both high spatial and spectral resolutions. Specifications of the system are shown in

Table 7. Specifications of the system.

Key Parameters	Parameter Values
Channel number	66
FPA type	A frame CCD
Dimensions of pixels	7.4 μm × 7.4 μm
Target surface size	24 mm × 36 mm
CCD pixels	4872 × 3248
Spectral region	450 nm~850 nm
Spectrum sampling	6 nm
Spatial resolution	400 × 400
Lens array	11×6
Lens F#	7
Focal length of lens	3.65 mm
Field range	68°
Optical structure length	7.2 mm
Distance between the lens	3 mm
Focal length of lens	12 mm
Weight	200 g
Prototype of volume	60 mm × 60 mm × 28 mm

.

The advantages of the M. BOLD method are as follows: by tilting the LVF, the number of spectral channels is considerably increased, and the difficulty to manufacture the LVF is considerably reduced when compared to that of the segmented filter. The commercial LVF greatly reduced the difficulty of manufacturing the spectrometer. The system was small and light in weight. The disadvantage was that, although the spectrum of the LVF variable spectral direction was assumed to be approximately unchanged within a short distance, when the LVF region covered by a single microlens was large, the spectral perturbation was obvious, and the larger the region covered, the larger the spectral perturbation, which was unfavorable for the spectral imaging of large apertures.

3.2.2. Compact, Miniature Snapshot Optically Replicating and Remapping Imaging Spectrometer

In the year 2019, Mu et al. from the Xi’an Jiaotong University of China, based on the ultra-compact multi-aperture snapshot MSI system proposed by Hubold et al. in Germany, proposed a compact, miniature snapshot optically replicating and remapping imaging spectrometer (ORRIS) [53,54,55]. Its principle is based on the shifting of sub-images replicated using a specially organized lenslet array and the filtering of each sub-image by a continuous variable filter (CVF). The 3D data cube was recovered just by using a simple image-remapping process. The use of the lenslet array and CVF caused the system to be very compact and miniature in size. It covers a wavelength region of 360 to 860 nm with 80 spectral channels with a spatial resolution of 400 × 400 pixels. The volume of the prototype is approximately 230 mm (length) × 70 mm (width) × 70 mm (height) and the weight is approximately 1.0 kg for finite imaging, and these values change to 50 mm (length) × 70 mm (width) × 70 mm (height) and 0.5 kg for infinite imaging. The prototype is verified by measuring outdoor static and dynamic scenes.

The compact ORRIS system is mainly composed of an M × N lenslet array, a rectangular CVF, and a FPA sensor, as presented in Figure 37a. In contrast to the ultra-compact multi-aperture snapshot MSI system of Hubold et al., the ORRIS system swaps the position of the lens array with the filter. The CVF stands on top of the FPA sensor, behind the lenslet array. The lenslet array includes a lateral-shift row and vertical-column lenslets. The shifting angle θ between the row direction of the lenslet array and the waveband direction of the CVF is presented in Figure 37b. As a result, the replicated row-direction sub-images are laterally shifted, and they cover gradually varied wavebands of the CVF. Different slices of the sub-images are continuously filtered by a narrow waveband with a central wavelength λ_i, as shown in Figure 37c. Then, the spatiospectral sub-images are acquired during a single exposure time on the FPA. The spectral image at each central wavelength λ_i is finally reconstructed with the remapping of the simple image. No complex reconstruction algorithm is required.

Assuming that the spectral interval covered by each sub-image is Δλ, the effective minimum bandwidth is δλ, and the number of lenses is N, then δλ = Δλ/N. Assume that Δx and Δy are the pitches of the sub-images along the x and y directions, respectively. Then, θ = tan − 1[Δy/(N × Δx)]. If Δx = Δy, θ = tan1(1/N). The lens array has M rows in total; therefore, the detectable spectral range is δλ’ = [−M − 1) × N] × δλ. This means that the number of spectral channels is C = [−M − 1) × N] with a bandwidth of δλ. The dimensions of the obtained 3D data cube depend on the size of the FPA and the available CVF. There is a tradeoff between the spatial and spectral resolutions. The spectral resolution increases with the number of lenslets, and the spatial resolution achieves the opposite result.

During the image-mapping process, the regions in the spectrum range of δλ of N sub-images are sequentially spliced to form the completed scene. Because δλ is very narrow and meets the assumption of a constant CVF spectrum, the spectral bandwidth of the reconstructed image following Mosaic is very narrow and the spectral disturbance is minor.

Figure 38a presents a schematic layout of the ORRIS system, and Figure 38b presents the proof-of-principle prototype. The combination of objective (EF 50 mm) and collimating (EF 50 mm) lenses forms an afocal telescope that is used for collimating the incident light originating from the objects. The lenslet array reimages the intermediate image onto the CVF and FPA. The size of the intermediate image is limited by a field stop located on the common focal plane of the objective and collimating lenses. The CVF is a commercial non-customed continuously variable bandpass filter with a wavelength range of 360 to 860 nm. The FPA is a CCD array with a pixel size of 7.4 µm × 7.4 µm. The sensor size of 24 × 36 mm can maximally cover the effective size of the CVF. The 9 × 10 lenslet array comprises 90 off-the-shelf plano-convex lenslets. They are mounted in a black, anodized, aluminum plate. Each lenslet has a focal length of 12 mm. The lenslet pitches are placed 3 mm along the row and column directions. The spatial resolution of each sub-image will be 400 × 400. An aperture array with a diameter of 1.5 mm is placed before the lenslet array to improve the imaging quality. Then, the F# of the lenslet becomes 8.

When the imaging target reaches infinity, the collimating telescope is no longer required; only the baffle array is inserted at the front so that the system will be compacted and miniaturized even further. When imaging a finite-distance object, adding a diffuse screen to the FOV aperture position can relax the imaging requirements. The main, characteristic parameters of the experimental prototype are presented in

Table 8. Specifications of the system.

Key Parameters	Parameter Values
Channel number	80
FPA type	CCD
Pixel size	7.4 μm × 7.4 μm
Target surface size	24 mm × 36 mm
Spectral region	360~860 nm
Spatial resolution	400 × 400
OL lens	EF50mm
CL lens	EF50mm
Lens array	9 × 10
Distance between the lens	3 mm
Focal length of lens	12 mm
Lens F#	8
	230 mm × 70 mm × 70 mm (limited object distance)
	50 mm × 70 mm × 70 mm (infinite distance)
Prototype weight	1.0 kg (limited object distance), 0.5 kg (infinite distance)

.

The static and dynamic scenes were verified by the principal prototype. Figure 39 depicts the 25-channel imaging results of the static scene. The original gray sub-image shown in Figure 39a presents a certain deviation in the row direction; using a simple image-remapping process, the continuously remapped spectral images are depicted in Figure 39b. Figure 39c shows the color-fusion images corresponding to each channel. Figure 39d presents the recovered 3D spectral image data cube. Figure 39e shows a single, gray sub-image following amplification. Figure 39f shows the RGB image of a similar scene captured using a cell phone. From the objects in the complex experimental scene, we can observe that the spatial resolution of the system is good enough to recognize them.

Figure 40 presents the partial-image results recovered from the dynamic-scene video obtained by the prototype. The experiment proves that automobiles moving on a crossroads are captured by the ORRIS via a video recording, which can be used for conducting surveillance in real time.

4. Curved Compound Eye Multispectral Snapshot Imaging System

4.1. Multi-Layer Compound Eye for MSI

In the year 2017, Chen et al. from the Chinese University of Hong Kong proposed the design, fabrication process, and verification results of a multi-layer artificial compound eye (ACE) for MSI [56]. A high-precision 3 × 4 MLA was designed and fabricated on the convex surface of a plano-convex lens with a diameter of 20 mm, focal length of 50 mm, and refractive index of 1.47 using vacuum hybrid imprinting technology. The curvature and diameter of each microlens were designed and optimized independently to ensure that each one could focus on the FPA. The thickness of the 3 × 4 microlenses from the surface of the large lens was 24 μm, the refractive index was 1.65, and the distance of the centers between neighboring microlenses was 2.54 mm. The diameters of each group were 900, 896, 891, and 887 μm. A light-absorption layer was coated on the upper surface of the large lens to suppress stray light in the imaging process. A multi-channel color filter array was fabricated by lithography at the bottom of the lens to perform spectral separation. A high-resolution CMOS sensor was coupled beneath a multilayer ACE. The overall structure of the multi-layer ACE is presented in Figure 41.

The multi-channel filters are blue (400–500 nm), green (500–600 nm), red (600–800 nm), and near-infrared (800–1200 nm), respectively. The spectral transmittance curves of the four filters are presented in Figure 42.

Figure 43 depicts the processing technology and flow of the multi-channel filter.

The minimum bandwidth of the color filter is 100 nm, which is relatively wide. In order to prevent the transmission of infrared light, an IR-CUT filter was added after the color filter. Figure 44 presents the working and field diagrams of the illumination source, target, and MSI system.

Several experiments were conducted to verify the spectral separation ability and application feasibility of the system. Figure 45 depicts the test image of the color blindness test card of the system. The number “86” can be observed via the red band and the number “9” can be observed via the green and blue bands, indicating that the frequency separation is good.

4.2. Multispectral Curved Compound Eye Camera

In the year 2020, Yu et al. (Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences) proposed a novel multispectral curved compound eye camera (MCCEC) [57]. The MCCEC mainly consists of a curved MLA integrated with selected narrow-band optical filters, an optical transmission subsystem, and a data-processing unit with an image sensor. The novel MCCEC system can achieve MSI at an ultra-large FOV and can obtain information for multiple spectrum segments in real-time. Moreover, the system has the advantages of its small size, light weight, and high sensitivity in comparison with conventional multispectral cameras.

The optical layout of the curved MLA, narrow-band filter array, and optical relay system for image plane transformations was carefully designed and optimized. The system structure is presented in Figure 46. The FOV of the MCCEC was determined by the front bionic curved compound eye. Narrow-band optical filters were fixed directly behind the MLA, and there were seven wavebands in total for MSI.

The main purpose of the introduction of the optical relay system is to solve the problem of the adaptability between curved MLA imaging and FPA and to improve the image resolution and quality of the system. The curved MLA has 117 microlenses with a focal length of 0.4 mm. MCCEC can achieve seven waveband MSIs with center wavelengths of 480, 550, 591, 676, 704, 740, and 767 nm in a large FOV of 120° and when the spectral bandwidth is 10 nm. The whole size of the optical system was 93 mm × 42 mm × 42 mm.

Figure 47 shows the overall layout of the multispectral channels of the MCCEC system. As is shown in Figure 47a, a cluster of seven microlenses with one microlens surrounded by six microlenses was utilized as a MSI unit, and these microlenses were attached with filters at different wavebands. Similarly, each microlens can be surrounded by the six microlenses so that a new cluster of seven microlenses can act as a new MSI unit. In this way, the whole working FOV of the MCCED can be covered by many MSI units. As a result, the images obtained by those microlenses with the same spectral filter can be stitched together to form an image with the same spectrum channel in the whole FOV, as shown in Figure 47b. The angle between adjacent lenses in the compound eye was designed to be 8°. By following the abovementioned design parameters, the curved compound eye can be designed to achieve an MSI function in the entire FOV. In this way, an MCCEC that can realize real-time MSIs in an ultra-wide FOV can be obtained.

The optical transmission subsystem has a FOV of 120°, an F-number of 4, and an effective focal length of 3 mm. The schematic diagram of the optical design of the MCCEC system is shown in Figure 48.

4.3. Biomimetic Multispectral Curved Compound Eye Camera

In 2021, Zhang et al., from the Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, proposed a novel biomimetic multispectral curved compound eye camera (BMCCEC) [58,59]. A BMCCEC has 104 effective multispectral ommatidia and a FOV of 98° × 98°, which is able to realize 7-band MSIs with center wavelengths of 500, 560, 600, 650, 700, 750, and 800 nm, and a spectral resolution of 10 nm. The principal prototype of a BMCCEC is shown in Figure 49.

The main performance parameters of the BMCCEC prototype are shown in

Table 9. Specifications of BMCCEC.

Key Parameters	Parameter Values
Number of ommatidia	127
Radius of the hemisphere	68 mm
Model of camera	Dalsa C5180M
Pixel size	4.5 μm × 4.5 μm
Image frame rate	30 fps
Spectral resolution	10 nm
Number of effective ommatidia	104
System focal length	5 mm
Sensor type	CMOS
Camera resolution	5120 × 5120
FOV	98° × 98°
Number of spectrum	7
The central wavelength of the bands	500, 560, 600, 650, 700, 750, and 800 nm

. Compared to the former MCCEC system, it improves the focal length of the system from 0.4 to 5 mm, in order to allow the system to work at long distances with a high spatial resolution, and the maximum FOV of the MCCEC was reduced from 120° to 98°.

Following spectral calibration, the authors tested the MSI capability of the BMCCEC prototype. A green-leaf plant and plant model were selected as the targets in the experiment. Following MSI, the image in the whole FOV containing all ommatidia units was reconstructed and is presented in Figure 50a. Based on the reconstructed image in the whole FOV, it is easy for one to locate the interesting targets. To retrieve a sampled spectral image for every spectrum channel, the following steps were taken. First, the reconstructed multispectral image in the whole FOV was differentiated and divided into seven spectral groups. Then, the multispectral images were reconstructed by using the projection method. Figure 50b shows the reconstructed images of all spectrum channels. In the seven spectral images, the target shows different gray levels.

In order to test the uniformity of the MSI in the whole FOV, the same object was placed at the center and edge of the FOV in the experiment. The target selected for the experiment was still green plants, which has a high absorption rate at around 600 nm and a high reflection rate at the band of 750~800 nm. Figure 51a,b show multispectral images obtained at different FOVs. Figure 51c shows the retrieved reflection spectrum curve. As can be observed, two curves coincide well with an average relative error of less than 10%. Therefore, the BMCCEC principal prototype performs well in the whole FOV of 98° × 98°.

5. Conclusions

As an application development direction of compound eye imaging technology, compound eye MSI technology has been widely considered by researchers at home and abroad in recent years and has become an important research topic in the field of image science. Numerous achievements have been made in the scientific field of planar compound eyes and their MSI systems. Among them, the compound eye MSI system based on TOMBO architecture has gradually developed its characteristics of achieving a high spatial resolution, low cost, high frame rate, and field-motion imaging. The technology tends to be well-developed and has been initially applied in the research. However, due to the limitations of the narrow-band interference filter array, the numbers of imaging channels and multi-spectra are relatively low at 18 and 17, respectively. Based on the LVF or CVF MSI technology, the numbers of imaging channels and multispectra significantly increase, and the number of imaging channels can reach 80 at present. As the detector used at present is mainly planar, most of the research conducted on curved-surface ACEs still simulates and implements characteristics of curved-surface MLAs. Several problems still exist in the implementation of curved-surface compound eye systems, and the curved-surface compound eye MSI system is still in the process of being explored and researched in the field. Although the results contribute to the advantages of the wide FOV attained by a curved compound eye and the number of imaging channels is significantly increased, several shortcomings still exist in terms of imaging quality and the number of spectra. In the future, image-processing studies will also be significantly increased and several problems in the application of real-time MSI will still exist.