A Hybrid Bionic Image Sensor Achieving FOV Extension and Foveated Imaging

Abstract

Based on bionic compound eye and human foveated imaging mechanisms, a hybrid bionic image sensor (HBIS) is proposed in this paper to extend the field of view (FOV) with high resolution. First, the hybrid bionic imaging model was developed and the structure parameters of the HBIS were deduced. Second, the properties of the HBIS were simulated, including FOV extension, super-resolution imaging, foveal ratio and so on. Third, a prototype of the HBIS was developed to validate the theory. Imaging experiments were carried out, and the results are in accordance with the simulations, proving the potential of the HBIS for large FOV and high-resolution imaging with low cost.

1. Introduction

Compound eyes and human eyes have been studied in regard to their remarkable properties for optical imaging [1,2,3,4,5]. Compound eye provides a large field of view (FOV), infinite depth of field (DOF), low aberrations and motion acuity, which can be used in many applications, such as high speed motion detection, large FOV surveillance and machine vision [6,7,8,9]. Artificial compound eyes can be divided into two types, namely planar artificial compound eyes (PACE) and curved artificial compound eyes (CACE). PACE [10,11,12] are mainly designed for super-resolution imaging, which use the sub-pixel shifts among the ommatidia to resample the scene with a relatively high spatial sampling frequency. For a typical PACE, the optical axes of ommatidia are parallel, and the FOVs of ommatidia are identical, so it is difficult to obtain large whole FOV with PACE. CACE [1,3,13] are mainly designed for FOV extension. Compared with PACE, the overlaps between FOVs of adjacent ommatidia of CACE are usually small enough to extend to the whole FOV while avoiding a blind zone. Therefore, the image resolution of CACE is lower than PACE under the same parameters of individual ommatidium. In order to obtain both large FOV and high resolution, the Aware-2 imaging system [14] utilizes a multi-scale lens and a micro-camera array achieving 120° and 38 μrad instantaneous FOV of a single pixel, but the system is bulky, and it is time-consuming due to the required calibration of 98 micro-cameras and the sub-image mosaic with iterative methods. In addition, large volume redundant data results in low efficiency for object detection or target tracking [15]. In recent years, the development of a flexible printed circuit board and liquid lens has motivated some remarkable CACE designs [1,16,17,18]. Each ommatidium on those sensors has one pixel in common, so they can achieve a large FOV while keeping relatively low spatial resolution. To improve the image quality, Woong-Bi Lee et al. [13] Utilized a digital signal processing algorithm to reconstruct images with high resolution, but the images including 16 × 16 pixels could not meet requirements. Also, some scanning methods have been proposed to enlarge the FOV of a single aperture or improve the image quality of CACE, but the imaging procedure is time-consuming [1,19,20], and the resolution of the image is still limited by the small volume of pixels.

Foveated vision is inspired by human eyes, and it provides a practical solution to achieve a large FOV (peripheral imaging) and high resolution (fovea imaging) [4,21,22,23,24] with low data redundancy. The foveal ratio is defined as the ratio between the spatial sampling frequencies of the fovea and the periphery. A large foveal ratio of foveated vision is attractive for many applications, such as medical facilities, surveillance, and robot navigation [4,21,25]. However, compared with CACE, the whole FOV is determined by peripheral imaging of a single image sensor, and the peripheral view also needs acceptable resolution for automatic visual tasks. Therefore, the tradeoff between FOV and the resolution of foveated vision needs to be further resolved.

Recently, some researchers have been studying the combination of a compound eye and foveated vision to improve the performance of imaging systems. Guillem Carles et al. [15] developed a multichannel imaging system which combines prism array and PACE to obtain FOV extension and foveated imaging. However, in order to achieve a high foveal ratio in the fovea, the FOV of each ommatidium overlaps in the fovea region, and the system only achieves two-fold FOV extension with a 5 × 5 camera array. Kang Wei et al. [26] designed a reconfigurable polymeric optofluidic device with an array of integrated millimeter-sized fluidic lenses, which combines the large FOV of CACE and the adaptive focusing capabilities of the human eye to achieve FOV and DOF extension. Xiongxiong Wu et al. [27] simulated a micro-camera array to achieve large FOV imaging like in a compound eye, with a foveated imager located behind each micro-camera to obtain high resolution in the regions of interest (ROI). However, compared with PACE, the spatial resolution is not improved. A combined method was proposed in our previous work [28], which possesses the large FOV characteristic of CACE and the retina-like feature of human vision. To the best of our knowledge, there is currently no practical imaging method that can obtain FOV extension and super-resolution simultaneously.

To solve the problems above, a hybrid bionic image sensor (HBIS) is proposed in this paper. Features of PACE, CACE and foveated vision are integrated to achieve FOV extension (by CACE), super-resolution (by PACE) of ROI and a large foveal ratio (foveated vision). The FOV is extended efficiently with a CACE structure. Foveated imaging by CACE and PACE allows a higher resolving power and larger FOV than traditional foveated vision. Risley prisms are widely used for accurate and fast beam scanning and pointing [29,30], and these are employed in front of the central ommatidium to imitate the movement of the fovea and generate sub-pixel shifts of sub-images for super-resolution reconstruction. A prototype is developed to test the features of HBIS by carrying out experiments with outdoor and indoor scenes. The experimental results show consistency with the theoretical analysis and its potential for a large FOV and foveal ratio at a low cost.

2. Methods

The proposed HBIS integrates features of PACE, CACE and foveated vision. Here, we only utilized a 3 × 3 ommatidia array to illustrate the method; a HBIS with more ommatidia and larger FOV also could be developed based on the proposed method. The schematic diagram of HBIS is shown in Figure 1, where C_ij is the sequence of each camera, and i and j denote the row and column numbers respectively. The red and yellow dot-dashed lines are the original optical axes of the cameras, and the blue dot-dashed line is the optical axis of the central ommatidium that is deviated by the Risley prisms, Φ_vmin and Φ_hmin denote the minimum angles between the edge of the FOV of central ommatidium (the edge closer to the red dot-dashed line) and the red-dashed line in the vertical and horizontal directions with the two prisms aligned, φ_v and φ_h denote inclined angles between the original optical axis of the central ommatidium and the optical axes of the peripheral ommatidia of C₁₂/C₃₂ and C₂₁/C₂₃, respectively. θ_v and θ_h denote half FOVs in the vertical and horizontal directions of the individual ommatidium. Risley prisms are composed of two identical prisms which can be rotated independently.

Firstly, HBIS is composed of a camera array distributed on a curved surface like CACE in an apposition compound eyes pattern [31]. Secondly, Risley prisms are controlled to be fast and accurate, to shift the images of the central ommatidium, that is, the fovea, in sub-pixel accuracy for super-resolution imaging of PACE. Thirdly, super-resolution imaging allows the fovea to have a higher spatial sampling frequency than the periphery, which achieves foveal imaging.

It should be noted that only the case with the original optical axis of the central ommatidium involved in the deviated FOV of the central ommatidium is studied here, and other cases are discussed in the discussion section.

2.1. FOV Extension

The HBIS works in two stages. In stage one, all of the ommatidia detect objects in their FOV independently, and the FOV of the fovea is fixed without scanning by Risley prisms. Once an object is detected by any ommatidia, the HBIS goes into stage two. In stage two, the FOV of the fovea is coarsely adjusted by rotating Risley prisms to stare at the object immediately. Then, the Risley prisms are precisely rotated over a small range to achieve imaging with sub-pixel shifts for super-resolution reconstruction. In the meantime, all the periphery ommatidia keep detecting objects in their own FOV to avoid missing objects. The HBIS will go back to stage one when the object moves out of the scan field of the fovea.

The same optical system and image sensor chip are employed for all of the ommatidia to reduce the cost and design complexity. The design of a CACE structure should ensure partial overlaps between adjacent ommatidia to avoid a blind zone wherever the fovea is “watching”. In Figure 1b, the phases of the two Risley prisms are inverse, which causes the optical axis of the fovea to be unchanged. Therefore, the condition of partial overlap is expressed as

$${\varphi }_{v\left(h\right)}\le 2{\theta }_{v\left(h\right)}.$$

(1)

In Figure 1c, the phases of the two Risley prisms are identical, and the deviation angles of the optical axis of the fovea achieve maximum values in the vertical and horizontal directions. To avoid a blind zone, the structure should satisfy the condition

$${\mathsf{\Phi }}_{v\left(h\right)\min }+{\theta }_{v\left(h\right)}\ge {\varphi }_{v\left(h\right)}.$$

(2)

It is obvious that Equation (2) is tougher than Equation (1) because the fixation points of the periphery ommatidia are fixed. When the fovea steers its FOV away from the center of the whole FOV of HBIS, as in Figure 1c, the FOVs of the periphery ommatidia need to be closer to the center of the whole FOV of HBIS than they need to be in Figure 1b. It can also be observed that the large scan field of the fovea makes the FOVs of the periphery ommatidia close to the center of HBIS, resulting in a small whole FOV of the HBIS. Hence, the value of φ_v(h) should be configured according to the values of Φ_vmin and Φ_hmin.

The imaging model with Risley prisms is illustrated in Figure 2a, where the dot-dashed line is the optical axis of the camera, the red dashed line is the deviated optical axis by the Risley prisms, Φ is the deviation angle of the optical axis, and Θ is the azimuth angle of the optical axis. To find the maximum values of Φ in the vertical and horizontal directions, the two prisms are aligned, and the phase angles are adjusted to 90° and 180°. Then non-paraxial ray tracing is utilized to calculate the deviation angles as follows.

Refractions occur on four surfaces of the Risley prisms. Figure 2b shows the case of refractions with the two prisms aligned, where I is the vector of incident light from one pixel, and R is the vector of emergent light. The two subscripts of I and R indicate the sequence of prisms and the sequence of surfaces of each prism, respectively.

Given a pixel (X, Y) located in the pixel array with M × N pixels, I₁₁ is calculated as

$${\mathit{I}}_{11}=\left(\frac{p\left(Y-n_0\right)}{\sqrt{p^2\left[{\left(Y-n_0\right)}^2+{\left(X-m_0\right)}^2\right]+f{\prime }^2}};\frac{p\left(X-m_0\right)}{\sqrt{p^2\left[{\left(Y-n_0\right)}^2+{\left(X-m_0\right)}^2\right]+f{\prime }^2}};\frac{f\prime }{\sqrt{p^2\left[{\left(Y-n_0\right)}^2+{\left(X-m_0\right)}^2\right]+f{\prime }^2}}\right),$$

(3)

where p is the pixel pitch, $f\prime$ is the focal length of the optical system, and (m₀, n₀) represents the center of the pixel array. The normal vectors of the four surfaces are calculated as:

$$\{\begin{array}{l}{\mathit{n}}_{11}=(0;0;1)\\ {\mathit{n}}_{12}=(-\sin \alpha \cos {\varphi }_1;-\sin \alpha \sin {\varphi }_1;\cos \alpha )\\ {\mathit{n}}_{21}=(\sin \alpha \cos {\varphi }_2;\sin \alpha \sin {\varphi }_2;\cos \alpha )\\ {\mathit{n}}_{22}=(0;0;1)\end{array},$$

(4)

where α is the wedge angle of the Risley prisms, ${\varphi }_1$ and ${\varphi }_2$ are phase angles of prism 1 and prism 2—that is, the angles from the x axis to the thin ends of the prisms in an anticlockwise direction. Following Snell’s law in vector form [29], the emergent light, R₂₂, through the Risley prisms is obtained. Then, the deviation angle, Φ, and the azimuth angle, Θ, as illustrated in Figure 2a, can be calculated with R₂₂.

Substituting X = 1:M/2, Y = 1 and ${\varphi }_1={\varphi }_2=180°$ into the non-paraxial ray tracing method above, the minimum inclined angle between the right edge of FOV and the negative direction of the z-axis is obtained as Φ_hmin. In the same way, the minimum inclined angle Φ_vmin between the lower edge of FOV and the negative direction of the z-axis can be obtained with X = 1, Y = 1:N/2 and ${\varphi }_1={\varphi }_2=270°$. Then, the conditions of Equation (2) can be fulfilled using the values of Φ_vmin and Φ_hmin.

The inclined angles between the optical axis of the fovea and the optical axes of the corner ommatidia are deduced with ${\phi }_c={\tan }^{-1}\sqrt{{\tan }^2{\phi }_v+{\tan }^2{\phi }_h}$.

The whole FOV of the HBIS in the vertical and horizontal directions is FOV_v(h) = 2(φ_v(h) + θ_v(h)), and the FOV is extended by 2φ_v(h).

2.2. Super-Resolution

To obtain super-resolution imaging of PACE, multiple images with sub-pixel shifts are sampled by the fovea. It has been proven that the pixels always shift along with the optical axis [29]. In addition, sub-pixel shifts of the images need only fine adjustment of Risley prisms over a small range. Therefore, we assume that the Risley prisms produce identical shifts for the optical axis and pixel array. For practical use, we only need to shift the optical axis of the fovea in sub-pixel accuracy. For a commercial, off-the-shelf camera whose resolution is limited by pixel size, super-resolution techniques can improve the spatial resolution to approximate the diffraction limit. The optical spatial cutoff frequency of a diffracted limitation is derived as v_o = D/1.22𝜆f’, where D is the entrance pupil [32] diameter, and λ is the optical wavelength. The Nyquist frequency of the pixel array is v_p = 1/2p. Assuming H = v_o/v_p, the spatial resolution of the camera can be improved up to H times with super-resolution techniques, theoretically. H is the ratio of v_o and v_p, and it is a constant for a given imaging system with a fixed focal length. The parameter, h (h ≤ H), is viewed as a resolution improvement factor. The step length of the sub-pixel shift in the image plane is sl = p/h, where h is an integer. Given the object’s distance, v, the step length of sub-pixel shifts in the object’s plane is SL = sl∙v/f’. Figure 3a,b shows the scan pattern of the optical axis with odd and even values of h, respectively. The numbers on the circle dots denote the sequences of sub-pixel points. The green circle dots represent the intersection of the object’s plane and the optical axis with the initial phase angle. The purple circle dots are arranged by referring to the green dots.

The deviation vectors of the two prisms, $\mathit{D}{\mathit{V}}_1^i$ and $\mathit{D}{\mathit{V}}_2^i$, are deduced with

$$\{\begin{array}{l}\mathit{D}{\mathit{V}}_1^i=(-k_1\cos {\varphi }_1^i;-k_1\sin {\varphi }_1^i)\\ \mathit{D}{\mathit{V}}_2^i=(-k_2\cos {\varphi }_2^i;-k_2\sin {\varphi }_2^i)\end{array},$$

(5)

where $k_1$ and $k_2$ are norms of the two prisms, and the superscript, I, refers to the index of the sub-pixel point in Figure 3a. The total deviation vector by the Risley prisms is

$$\mathit{D}{\mathit{V}}^i=\mathit{D}{\mathit{V}}_1^i+\mathit{D}{\mathit{V}}_2^i=(-k_1\cos {\varphi }_1^i-k_2\cos {\varphi }_2^i;-k_1\sin {\varphi }_1^i-k_2\sin {\varphi }_2^i).$$

(6)

$\mathit{D}{\mathit{V}}^i$ can also be deduced as

$$\mathit{D}{\mathit{V}}^i=\{\begin{array}{l}\mathit{D}{\mathit{V}}^0+\frac{sl\cdot v}{f\prime }\left(\mathrm{mod}\left(i,h\right)-\frac{h+1}{2};\mathrm{floor}\left(i/h\right)-\frac{h+1}{2}\right),\mathrm{when}h\mathrm{is}\mathrm{odd};\\ \mathit{D}{\mathit{V}}^0+\frac{sl\cdot v}{f\prime }\left(\mathrm{mod}\left(i,h\right)-\frac{h}{2};\mathrm{floor}\left(i/h\right)-\frac{h}{2}\right),\mathrm{when}h\mathrm{is}\mathrm{even}.\end{array},$$

(7)

where DV⁰ is the deviation vector of the initial optical axis before the sub-pixel shift. In the example shown in Figure 3a, DV⁰ = DV¹³.

Substituting I₁₁ = (0,0,−1) into the non-paraxial ray tracing method, the deviation angle, Φ, and the azimuth angle, Θ, of the optical axis can be obtained, and DV⁰ can be computed as (v∙tan Φ∙cos Θ; v∙tan Φ∙sin Θ). Then, k₁ and k₂ can be calculated as

$$\{\begin{array}{l}\left[\begin{array}{l}{k}_1\\ k_2\end{array}\right]={\left[\begin{array}{l}-\cos {\varphi }_1^0-\cos {\varphi }_2^0\\ -\sin {\varphi }_1^0-\sin {\varphi }_2^0\end{array}\right]}^{-1}\cdot \mathit{D}{\mathit{V}}^0,\mathrm{when}{\varphi }_1^0\ne {\varphi }_2^0\\ k_1=k_2=\frac{1}{2}\left|\mathit{D}{\mathit{V}}^0\right|,\mathrm{when}{\varphi }_1^0={\varphi }_2^0\end{array}.$$

(8)

According to the model illustrated in Figure 3c, and given the DVⁱ and norms (k₁ and k₂) of $\mathit{D}{\mathit{V}}_1^i$ and $\mathit{D}{\mathit{V}}_2^i$, the two phase angles ${\varphi }_1^i$ and ${\varphi }_2^i$ can be computed as

$$\{\begin{array}{l}{\varphi }_1^i={\varphi }_0^i\pm \arccos \left(\frac{k_0^2+k_1^2-k_2^2}{2k_0{k}_1}\right)\\ {\varphi }_2^i={\varphi }_0^i\mp \arccos \left(\frac{k_0^2+k_2^2-k_1^2}{2k_0{k}_2}\right)\end{array}.$$

(9)

Two sets of the inverse solutions are obtained, and the one closer to the initial phase angles, ${\varphi }_1^0$ and ${\varphi }_2^0$, is adopted.

Once the optical axis scans over one sub-pixel point, an image is sampled. After all the sub-pixel points have been scanned, a complete set of sub-images with sub-pixel shifts is obtained. Then, a feature-based image registration [12] and an interpolation method of scatter pixel points [15] are utilized for super-resolution reconstruction.

2.3. Foveated Imaging

Super-resolution imaging is only realized by the fovea, where spatial-variant resolution is formed over the entire FOV of HBIS, which resembles foveated vision. What is more, the foveal ratio of the proposed HBIS is adjustable over a large range, which is determined by the maximum resolution improvement factor, H. This means the resolution of the fovea is modulated by the needs for specific tasks, such as object detection, recognition or target tracking, etc.

In addition, the ROI imaged by the fovea can be redirected by adjusting the optical axis using Risley prisms. This ability is like the fovea movement of human eyes. In contrast to the accurate inverse solutions for super-resolution, ROI only needs an approximate inverse solution. Therefore, the paraxial model [33] is enough. The scan range of the fovea is defined by rotating the Risley prisms from 0° to 360° with the two prisms aligned.

3. Simulations and Analysis

3.1. FOV Extension

The FOV extension ratio (FER) is defined as the ratio between the whole FOV of the HBIS and the individual FOV of a single ommatidium. FER is viewed as the key indicator of HBIS, because we aim to demonstrate the capability of extending the FOV from one single aperture. Using given parameters of a pixel array and optical systems, the whole FOV can be determined by the parameters of Risley prisms, according to Section 2.1. We assume that the pixel pitch (p) is 3.75 μm, M × N is 960 × 1280, the focal length (f’) is 12 mm, the F-number (F) is 1.4, the object distance (v) is 50 mm and the resolution improvement factor (h) is 7. These parameters are unchanged for the rest of the simulations unless otherwise instructed. Based on the above methods, the results for the whole FOV and FER versus the refractive index, n, and the wedge angle, α, are shown in Figure 4. The red and blue shades represent the ranges of the whole FOV and FER. The upper limits of the shades correspond to Φ_v(h)min + θ_v(h) = φ_v(h) from Equation (2), and the lower limits correspond to φ_v(h) = θ_v(h).

In Figure 4, the trends of the curves (upper limits) are similar—that is, when α and n are small, the whole FOV and FER increase linearly with the growth of α and n until they reach the inflection points, and they are constants after the inflection points. When α is 1° (the smallest value employed), the optical axis of the fovea can only be slightly deviated, which means the scan field of the fovea is the smallest as well. So, the blind zone can be avoided without the FOVs of the periphery ommatidia designed getting too close to the center, leading to FER_v = 2.88 and FER_h = 2.9, maximumly. When α is 11°, the inflection point occurs, as shown in Figure 4b, which corresponds to the situation in which the right edge of FOV of the fovea is parallel to the initial optical axis before the Risley prisms when ${\varphi }_1={\varphi }_2=180°$. In this situation, the scan field of the fovea covers the whole FOV of the HBIS, and only the overlaps among the periphery ommatidia need to be ensured with φ_h ≤ θ_h. φ_h = θ_h corresponds to FER_h = 2. Regarding α = 9° in Figure 4a, n = 2.2 in Figure 4c and n = 2.4 in Figure 4d, the inflection points occur in the same reason, and they also have a two-fold FOV extension in respective directions.

In Figure 4, it is noted that the values of α and n at inflection points in horizontal direction are larger than that in vertical direction. Because the condition for inflection points in horizontal direction is φ_h = θ_h, and φ_v = θ_v is the condition for inflection points in vertical direction. As θ_h > θ_v, the inflection points in horizontal direction are with larger α and n than the inflection points in vertical direction.

3.2. Imaging with Sub-Pixel Shifts for Super-Resolution

To verify the proposed super-resolution imaging model, simulations of a sub-pixel scan of the optical axis are carried out. We assume that α is 4° and n is 1.5. The alignment error (AE) is defined as the deviation ratio between the simulated points and the ground truth, which is calculated as the ratio between the misalignment distance and the step length (SL) of the sub-pixel shifts. A large AE results in high redundancy of the multiple samplings in the scene. The maximum AE among the sub-pixel points is utilized to evaluate the data efficiency of the proposed method. The maximum AE varies with the initial phase angles, the refractive index (n) and the wedge angle (α), and the simulation results are shown in Figure 5.

In Figure 5a, simulations with ${\varphi }_1={\varphi }_2$ and $\left|{\varphi }_1-{\varphi }_2\right|=180°$ are avoided, points of which are marked by color circles in Figure 5a, because the cases with ${\varphi }_1={\varphi }_2$ cannot achieve the sub-pixel patterns of Figure 3, and the cases with $\left|{\varphi }_1-{\varphi }_2\right|=180°$ lead to control singularities which make it difficult to achieve a fast sub-pixel scan. From the curves, we can see that the maximum AE is below 0.01. The trends of the curves with ${\varphi }_1=5°$ and ${\varphi }_1=45°$ present approximate periodicity, because the scan pattern illustrates a square matrix, which is symmetrical about the directions 0°, 45° and 90°. Figure 5b,c present similar laws—that is, the maximum AE increases with the refractive index (n) and the wedge angle (α). The parameters are set as α = 4° in Figure 5b and n = 1.5 in Figure 5c. To sum up, the maximum AE always maintains relatively low values for the given parameters.

Taking the initial phase angles (5°,70°), (45°,130°) and (80°,20°) with α = 4° and n = 1.5 as examples, we further study performances of the HBIS for sub-pixel shifts. Based on the scanpath of Figure 3, the maximum ranges of phase angles for the three examples are 0.60°, 0.60° and 0.63°, respectively, which are very small and make fast sub-pixel scans possible. The maximum AEs of the pixel array of the three examples are 0.56, 0.53 and 0.49. The average AEs of the pixel array of the three examples are 0.071, 0.070 and 0.072, respectively. The maximum AEs of the optical axes of the three examples are 0.0085, 0.0061 and 0.0086, respectively. The maximum AEs of the pixel array are much larger than the average AEs of the pixel array, because a few pixels in the edge or corners of pixel array are deviated far more than other pixels closer to the center of pixel array. Even so, pixels with an AE of less than 0.6 only slightly decrease the capacity to super-resolve the scene [15].

In general, AEs of the whole pixels and the optical axis have relatively small values. In particular, the differences between the average AEs of the pixel array and the maximum AEs of the optical axis are no more than 0.07, verifying the consistency between the sub-pixel shifts of the optical axis and the pixel array and the effectiveness of the proposed super-resolution reconstruction method.

3.3. Foveated Imaging

Super-resolution imaging of the fovea and the original resolution imaging of the periphery ommatidia give rise to foveated imaging. The foveal ratio and the bandwidth saving ratio (BSR) are basic indicators of foveated vision. The resolution of the fovea is adjustable according to the needs of variable tasks and the specific circumstances. The foveal ratio is defined as the ratio between spatial sampling frequencies of the central and periphery ommatidia, which is equal to resolution improvement factor, h, of the fovea. BSR is calculated as

$$\mathrm{BSR}=\left(1-\frac{1}{h^2}\right)\left(1-\frac{1}{{\mathrm{FER}}_v{\mathrm{FER}}_h}\right).$$

(10)

From the definition of BSR, we can see that BSR is determined by h and FER. From the above analysis, we know that FER is mainly affected by n and α. Figure 6 shows the results of BSR versus the wedge angle, α, and the refractive index, n, respectively. The trends of the curves are similar to the curves of Figure 4, and the two inflection points correspond to the inflection points in Figure 4. BSR decreases, with the wedge angle, α, and the refractive index, n, growing until the last inflection points.

In addition, BSR with larger h are higher than those with lower h, but the amplification reduces gradually with the growth of h. It is worth noting that BSR only grows by 3.7% when h grows from 4 to 7. A larger h means more sub-pixel samplings of the scene and a larger volume of data than a lower h does. Therefore, h = 4 seems to be more effective and reasonable than the other values.

4. Experiments and Results

4.1. Prototype Parameters

A prototype was developed, and the main parameters are shown in

Table 1. The main parameters of the prototype.

Parameter Type	Abbreviation	Values
Pixel pitch	p	3.75 μm
Rows × columns of pixel array	M × N	960 × 1280
Focal length	f’	12 mm
F-number	F	1.4
Wedge angle	α	4°
Object distance	v	50 mm
Refractive index	n	1.5

. The FOV of the single ommatidium was 22.6° × 17.1°. Based on the analysis above, the key parameters of HBIS were φ_h ≤ 18.4°, φ_v ≤ 12.9° and 1 ≤ h ≤ 7. The prototype employed commercial off-the-shelf cameras, and mechanical assembly and 3D printing techniques were used for the frame, of which errors from size and assembly were inevitable. In order to avoid loss of scene, the parameters were set as φ_h = 15°, φ_v = 10° and φ_c = 17.8°. The prototype is shown in Figure 7. The two prisms were driven by two stepping motors, respectively, and the prisms and stepping motors were connected by conveyor belt. The stepping motors were controlled by a computer through serial ports.

4.2. Experimental Results

An outdoor scene was sampled to verify the proposed imaging method, and the resolution improvement factor was set as h = 4. Figure 8a shows the stitched image of the prototype. The irregular rectangles of the sub-images mapped by projection transformation were caused by errors in 3D printing and assembly. The whole FOV was 49° × 34°, and the FER_v and FER_h of the prototype were 2.0 and 2.2, respectively, which are verified by Figure 8a. The experimental results for the FOV extension were consistent with the theoretical values: FER_v = 2.2 and FER_h = 2.3. Figure 8b shows the super-resolution image in the fovea. The three pairs of local regions were located in the central and periphery regions of the super-resolution image. We can see that more details of the scene are restored, and less artifacts are retained after the super-resolution reconstruction.

To further explore the scene resolving capability of the fovea, an indoor experiment was carried out with h = 4, and the results are shown in Figure 9. From Figure 9b–f, we can see that the quality of the reconstructed image was greatly improved. In Figure 9b–d, the characters in the sub-image are hard to recognize, but they are legible enough to be distinguished after super-resolution reconstruction. Also, the super-resolution image has sharper edges than the sub-image, as shown in Figure 9e. In addition, from Figure 9f, we can see that the super-resolution method performs well in restoring the scene even in low contrast regions.

To summarize, we used a 3 × 3 camera array constituting a HBIS, to achieve FER > 2, an adjustable foveal ratio over a large range (from 1 to 7) and fovea moving capability. Its performances were in accordance with the schematic design and the simulations, including those for FOV extension, super-resolution and foveated imaging. We used less ommatidia to achieve comparable properties than the foveated imaging system with a 5 × 5 camera array [15].

5. Discussion

From the analysis above, we can deduce that the capacity of Risley prisms to deviate the FOV of the fovea is key for improving the performance of FOV extension, as shown in Figure 1c and Figure 4. Therefore, it is necessary to discuss the relationship between the deviated FOV of the fovea and FOV extension of the HBIS. For the situation in Figure 1c, the case has been studied preliminarily, in which the bottom margin of FOV of the fovea is below the original optical axis of the fovea. Thus, we further discuss different situations here.

For the situation in Figure 1c, the fovea contributes part of its FOV to the whole FOV of the HBIS. This situation corresponds to the curves before the inflection points in Figure 4. When there are no suspected objects detected, all the ommatidia, including the fovea, work on object detection, and only the original resolution images are sampled by the fovea without super-resolution reconstruction. Once suspected objects are found, the fovea are redirected to the objects by rotating the Risley prisms [33,34], and super-resolution imaging is then achieved.

The other situation shown in Figure 1c is that the lower edge of FOV of the fovea is beyond the initial optical axis of the central camera before the Risley prisms. This situation corresponds to the curves after the inflection points in Figure 4, in which a large wedge angle and refractive index would not change the whole FOV of HBIS. In this situation, the fovea only focuses on high level tasks such as salient object recognition and object tracking without contributing to FOV extension. As shown in Figure 5b,c we can see that larger wedge angle and refractive index values lead to larger image distortion than smaller values do. Therefore, there is no need to adopt larger values for the wedge angle and refractive index than the values at inflection points.

Taking α = 4° and the inflection point α = 11° of Figure 4b as two examples of different situations above, the FOV distributions of ommatidia were simulated and shown in Figure 10. φ_h and φ_v were set according to the upper limits (the red line) of Figure 4b. In Figure 10a, the HBIS achieves FER_v = 2.5 and FER_h = 2.6, while the fovea can only scan part of the whole FOV. In Figure 10b, the HBIS achieves FER_v = FER_h = 2 which is smaller than in Figure 10a, but the scan field of the fovea covers the whole FOV of the HBIS. It is verified that the larger the wedge angles are, the smaller the whole FOV is. The same law exists between the refractive index and the whole FOV. In addition, the fovea can move over the entire FOV of the HBIS for Figure 10b, and this makes the HBIS more outstanding than that presented in ref. [15] which uses a 5 × 5 camera array achieving twice the FOV extension and a foveal ratio of 5.9 without the capability of fovea movement.

6. Conclusions and Future Work

To achieve large FOV extension and high-resolution imaging simultaneously, we proposed an HBIS that combines features of CACE, PACE and foveated vision. An ommatidium array was arranged on a curved surface to achieve FOV extension of CACE. Risley prisms were used in front of the fovea for super-resolution imaging of PACE. The spatial variant imaging resolution achieved a large foveal ratio and BSR for foveated vision. In addition, the scan capability of the fovea imitates the fovea movement in human eyes. Simulations showed that by using a 3 × 3 camera array, the FOV can be extended by 2.9 times. Meanwhile, a foveal ratio of up to 7 was achieved, and the BSR was beyond 80%. A prototype was developed using commercial off-the shelf-products, and we obtained 2.3 times FOV extension and 4 times resolution improvement of the fovea using the prototype; these values are consistent with the simulations, proving the potential of HBIS to produce a large FOV and foveal ratio with low costs.

Our previous work designed a compound and human hybrid eye with a micro-lens array for 3D imaging [28,35]. Therefore, the proposed HBIS can also be used for 3D imaging with a large FOV and foveal ratio; this will be studied by our team in the future. In addition, the cellular neural/nonlinear network (CNN) paradigm proved to be an effective way of accelerating the image process for real-time applications [36,37]. In the next step, we will improve the HBIS by using CNN to achieve fast super-resolution reconstruction and object detection.