Research on Panorama Generation from a Multi-Camera System by Object-Distance Estimation

Abstract

Panoramic imagery from multi-camera systems often suffers the problem of geometric mosaicking errors due to eccentric errors between the optical centers of cameras and variations in object-distances within the panoramic environment. In this paper, an inverse rigorous panoramic imaging model was derived completely for a panoramic multi-camera system. Additionally, we present an estimation scheme aimed at extracting object-distance information to enhance the seamlessness of panoramic image stitching. The essence of the scheme centers around our proposed object-space-based image matching algorithm called the Panoramic Vertical Line Locus (PVLL). As a result, panoramas were generated using the proposed inverse multi-cylinder projection method, utilizing the estimated object-distance information. The experiments conducted on our developed multi-camera system demonstrate that the root mean square errors (RMSEs) in the overlapping areas of panoramic images are no more than 1.0 pixel. In contrast, the RMSEs of the conventional traditional methods are typically more than 6 pixels, and in some cases, even exceed 30 pixels. Moreover, the inverse imaging model has successfully addressed the issue of empty pixels. The proposed method can effectively meet the accurate panoramic imaging requirements for complex surroundings with varied object-distance information.

1. Introduction

In recent years, panoramic images have been extensively employed in ground remote sensing applications, including street image acquisition, traffic monitoring, virtual reality, robot navigation and mobile mapping applications, etc. [1,2,3,4,5]. At present, there are mainly two types of panoramic imaging methods on the market, namely catadioptric imaging and multi-camera imaging methods. The catadioptric panoramic method uses a plane mirror to refract the surrounding light to a single camera [6,7,8,9]. Due to their complex imaging mechanisms and manufacturing processes, catadioptric panoramic systems have been less commonly used compared to the second methods. With the development of sensors and computing abilities, many panoramic camera systems have been developed by a combination of multiple cameras [10,11,12,13,14,15,16,17,18,19,20,21,22,23,24], which can be divided into two categories: monocular panorama systems and binocular panorama systems. Although a binocular panorama system is able to provide the 3D scene information needed for seamless panorama generation and 3D reconstruction, it requires higher complexity of system design and processing than that of the monocular panorama system [25]. Therefore, the development of monocular panorama systems using multiple cameras and the generation of accurate panoramic images remain crucial in the fields of remote sensing and computer vision applications.

In recent years, several monocular panoramic imaging systems based on multiple cameras have been developed. Among these, professional panoramic systems often incorporate multiple cameras, such as Point Grey’s Ladybug series cameras [10,11], Google’s vehicle NCTech [12], Leica’s mobile panoramic cameras [13], and more. To achieve precise panorama generation with these integrated systems, relevant research can be categorized into two main areas. The first approach is feature point-based algorithms [19,20,21,22,23]. Various tools have been developed in the multimedia industry for panorama creation, such as Kolor Autopano and Autostitch [19], among others.

Panoramas are generated based on the second method, which relies on a panoramic imaging model derived from the pre-calibrated geometric relationships between cameras. Both the ideal and rigorous panoramic spherical and cylindrical imaging models have been proposed [24,25,26,27,28] and successfully applied for vehicle-based remote sensing mapping and panoramic SLAM 3D reconstruction, particularly with the professional panoramic camera LadyBug3 [25]. However, it is important to mention that both the ideal and rigorous panoramic imaging models are developed under the assumption of constant object-distance information in the surroundings.

In order to meet the requirements of accurate panoramic mapping and 3D reconstruction applications, the accuracy and robustness of virtual panorama still need to be improved by embodying the varied object-distance information into a mosaicking imaging scheme. In this case, it is a vital challenge to accurately estimate the object-distance information of the surrounding scene. In addition, problems such as empty pixels and pixel aggregation in the virtual panoramas generated by the direct panoramic imaging models may still exist.

This paper presents a panorama generation scheme and the advanced technologies used for a monocular panoramic system. The primary contributions of this work lie in three sides:

(1)

In contrast to expensive panoramic camera systems that rely on professional cameras, a low-cost panoramic camera system is developed by a combination of eight low-cost web cameras in this paper.

(2)

An inverse rigorous panoramic imaging model is derived completely based on the pre-calibration of the inner orientation elements, distortion parameters of each camera, and the relative orientation relationships between cameras.

(3)

A scheme is proposed to estimate object-distance information within a surrounding scene. By expending the traditional Vertical Line Locus (VLL) object-based image matching algorithm [29], a panoramic Vertical Line Locus algorithm called PVLL is proposed to derive the optimized object-distance grids.

The rest of the paper is organized as follows: Section 2 introduces a panoramic camera system integrated with multiple cameras, developed by our team. Moreover, the inverse rigorous panoramic imaging model is derived completely based on pre-calibration of the panoramic camera system. In Section 3, we propose the panoramic generation method, with a specific focus on the object-distance estimation algorithm. The detailed experiment and analysis are provided in Section 4. Section 5 summarizes this research and provides recommendations related to future research works.

2. Related Work

2.1. Feature Point-Based Panoramic Imaging Method

For a set of overlapping images acquired with a multi-camera panoramic system, it is possible to generate a panoramic image relying on feature point-based methods. In the approach, the SIFT feature points are detected and matched on the overlapping regions of each pair of images from adjacent cameras [19,20,21]. Assuming that the scene is flat, the homographic estimations [22,23] are performed based on corresponding feature-points for each two adjacent cameras. At this point, all the transformations are modified relative to the master image, which is required for the generation of the panoramic images. The panoramic imaging approach is an extension of feature-based image registration and stitching of a single pair of images, which can be applied in scenes with rich texture features. However, this method heavily depends on achieving a sufficient distribution and accurate matching of feature points, as well as requiring substantial overlap between adjacent cameras. Additionally, it requires less variation in object-distance information from the surroundings, as the homographic transformation only exists between two planes.

2.2. Traditional Panoramic Imaging Model Based Method

The panoramic imaging model-based method, which departs from the feature-based method, requires accurate inner orientation elements of each camera and the relative orientation parameters between cameras to develop an accurate projection transformation relationship from the coordinate system of a perspective camera to that of the virtual panoramic camera.

Traditionally, the ideal panoramic spherical and cylindrical imaging model are formulated by disregarding the eccentric error between the center of the virtual panorama camera and that of an actual camera [26,27]. The rigorous panoramic imaging model is further proposed by incorporating the eccentric vectors among the optical centers of the cameras into the panoramic imaging model. The rigorous model is theoretically more accurate than the ideal model [25]. However, both the ideal and rigorous panoramic imaging models are developed without consideration of the varied object-distance information in a scene. Lin [30] tries to estimate a unified cylinder projection radius based on a loss function to minimize the projection error of the whole scene. Due to difficulties in fitting a regular cylinder surface for a real environment, stitching errors may still appear where the projection radius departs from the object-distance information. In addition, traditional panoramic models describe the transformation relationship from perspective imaging to panoramic imaging, which can easily lead to pixel aggregation or empty pixels in panoramic images.

Aiming at accurate and seamless panoramic imaging with a multi-camera system, an inverse panoramic imaging method is proposed by estimation of object-distance information of the surrounding scene, construction of the inverse transformation model and introduction of the eccentric errors.

3. Proposed Method

3.1. Overview of the Panoramic System

In this paper, we have designed a panoramic camera system that incorporates eight low-cost, high-definition web cameras mounted on an octagonal platform, as illustrated in Figure 1a. The panoramic camera system has a ${360}^{°}$ view field in the horizontal direction and ${45}^{°}$ view field in the vertical direction. Each camera has 2952 × 1944 pixels and is equipped with a lens of 5 mm focal length, with ${60}^{°}$ view field in the horizontal direction and ${45}^{°}$ view field in the vertical direction.

3.2. Proposed Inverse Panoramic Imaging Model

3.2.1. Relative Orientation between a Camera and the Panoramic Camera

According to the structure of our designed system in Figure 1, the relative orientation relationships are derived in this section. Let $C_i$ be the optical central point of the $i^{th}$ camera $(i=1,2,\dots ,8)$, and $C_i-xyz$ be the image space coordinate system. The panoramic cylindrical coordinate system is defined with its origin, set as O, located at the optical center shared by all cameras. The X, Y, and Z axes of this coordinate system are aligned parallel to the y, z, and x axes, respectively, of the image space coordinate system of the first camera, as illustrated in Figure 2. Thus, the orientation rotation matrix of the $i^{th}$ camera can be derived by Equation (1).

$$R_i=R_i^{\prime }\Lambda$$

(1)

where $R_i$ is the rotation matrix from the panoramic cylindrical coordinate system to the camera coordinate system of the $i^{th}$ camera. $R_i^{\prime }$ is the rotation matrix from the image space coordinate system of the first camera to that of the $i^{th}$ camera, and $\Lambda$ is a matrix, $\Lambda =\left(\begin{array}{ccc}1& 0& 0\\ 0& 0& -1\\ 0& 1& 0\end{array}\right)$.

In addition, it is required to calculate the relative translation vector, set as $t_i$, between point O and point $C_i$ by Equation (2).

$$t_i=\frac{\sum _{i=1}^8{\left(R_i\right)}^T{t}_i^{\prime }}{8}-{\left(R_i\right)}^T{t}_i^{\prime }$$

(2)

where $t_i^{\prime }$ is the pre-calibrated relative translation vector between the optical centre point of the first camera and that of the $i^{th}$ camera.

Aiming at removing empty pixels and pixel aggregation in virtual panoramas, an inverse panoramic imaging model is further derived in this paper.

3.2.2. Inverse Panoramic Imaging Model

Let point $P$ be projected on the cylinder surface, which is captured from the $i^{th}$ camera and mapped to point $p_i$ in the real image $I_i$. The ideal image model can be derived based on a traditional collinear equation [11], as shown in Equation (3).

$$\left[\begin{array}{c}{u}_{i,p}\\ v_{i,p}\\ 1\end{array}\right]=\lambda K_i{R}_i\left[\begin{array}{c}{X}_P-X_i\\ Y_P-Y_i\\ Z_P-Z_i\end{array}\right]$$

(3)

where $\lambda$ is the scale factor, $K_i$ is the inner orientation matrix of the $i^{th}$ camera, $\left(u_{i,p},v_{i,p}\right)$ is the undistorted coordinate of $p_i$. ${\left[\begin{array}{ccc}{X}_P& Y_P& Z_P\end{array}\right]}^T$ and ${\left[\begin{array}{ccc}{X}_i& Y_i& Z_i\end{array}\right]}^T$ are the cylindrical coordinates of points $P$ and $C_i$, respectively. In addition, the cylindrical projection point $P$ satisfies Equation (4):

$$X_P^2+Y_P^2=r^2$$

(4)

where $r$ represents the cylindrical radius.

The panoramic cylindrical surface is unwrapped onto a panoramic plane, with $o-xy$ being taken as its panoramic planar coordinate system, as illustrated in Figure 2. Point P is mapped to point $p$ based on Equation (5), which is derived from Equations (3) and (4).

$$\left[\begin{array}{c}{u}_{i,p}\\ v_{i,p}\\ 1\end{array}\right]=\lambda K_i{R}_i\left[\begin{array}{c}rsin\frac{\pi x_p}{L}-X_i\\ rcos\frac{\pi x_p}{L}-Y_i\\ 2\pi r/L\left(W/2-y_p\right)-Z_i\end{array}\right]$$

(5)

where $x_p,y_p$ stand for the coordinate of point $p$ in the virtual image in the x- and y- directions, respectively.

3.3. Object-Distance Estimation Algorithm

A panoramic image generated from the multi-camera system can be divided into the overlapping areas projected from two adjacent cameras and non-overlapping areas projected from only one camera, as shown in Figure 3. Traditionally, the panoramic images can be generated with a single projection radius $r$ under the assumption of a constant object-distance in the surroundings. While projection errors may occur due to variations in object-distance information, these errors are typically not easily discernible in the non-overlapping areas of a virtual panoramic image. However, stitching seams and geometric inconsistencies may become apparent and visible in the overlapping regions of a panoramic image. For example, the object points $Q_1$, $Q_2$, and $Q_3$ are respectively projected onto distinct overlapping regions of a virtual cylinder surface with the radius $r$ by individual cameras, as shown in Figure 4. In this case, the displacements between corresponding projection points $Q_{k,L}$ and $Q_{k,R}$ (k = 1, 2, 3) from the same object point are contingent upon the difference between the object-distance $d_k$ and the radius $r.$

In this case, a crucial challenge lies in estimating the object-distance information of the surrounding scene captured within the overlapping regions of each pair of adjacent cameras. To solve the problem, this paper presents a three-step algorithm for determining object-distance information in the surroundings to create seamless panoramic images.

In the first step, we estimate the scope of overlapping regions in a panoramic image and generate pyramid grids. In the second step, object-distance information is determined using an improved VLL (Vertical Locus Line Method for the surroundings captured in the overlapping regions of the panoramic image). Finally, we employ an interpolation method to estimate object-distance information for each non-overlapping region.

3.3.1. Generation Algorithm of Pyramid Grids in Overlapping Regions

In this paper, each overlapping region of a panoramic image is first generated around the central line, which is defined as the angular bisector vector between two adjacent cameras.

As illustrated in Figure 3, let $Q_{ij}$ be a point on the cylinder surface, and $\overrightarrow{{OQ}_{ij}}$ be the vector in the same direction as the angular bisector vector between $\overrightarrow{{OC}_i}$ and $\overrightarrow{{OQ}_j}$. The vector $\overrightarrow{{OQ}_{ij}}$ can be derived by Equation (6). In addition, $Q_{ij}$ is then projected onto the point $q_{ij}$ within the panoramic image. The $x$-coordinate ${gap}_{i,j}$ of $q_{ij}$ can be computed by Equation (7). Further, the overlapping region $A_{i,j}$, with a width of $w$ and ${gap}_{i,j}$ setting as the central line, is generated, expanding $x_{i,j}$ pixels away from the y-axis.

$$\overrightarrow{{OQ}_{i,j}}=\lambda (\frac{\overrightarrow{{OC}_i}}{‖{OC}_i‖}+\frac{\overrightarrow{{OC}_j}}{‖{OC}_j‖})$$

(6)

$$x_{i,j}=\frac{W}{2\pi }\arctan (\frac{Y_{i,j}}{X_{i,j}})$$

(7)

In Equations (6) and (7), $\overrightarrow{{OQ}_{ij}}$ is set as ${\left[X_{i,j}{Y}_{i,j}{Z}_{i,j }\right]}^T,$ where $X_{i,j},Y_{i,j},Z_{i,j }$ represent the $X$, $Y$ and $Z$ cylindrical coordinates of point ${\mathrm{Q}}_{\mathrm{i}\mathrm{j}}$, respectively. $\lambda$ is the scale factor, and $x_{i,j}$ is the x-coordinate of $q_{ij}$ in the panoramic image coordinate system. $W$ is the width of the virtual panorama image. The parameters $i$ and $j$ take on the following values: $i=1,2,3,\dots ,8, j=1,2,3,4,\dots ,8$.

By adopting a strategy of progressive refinement from coarse to fine, eight object-distance maps (ODMs) are constructed for each overlapping region of a virtual panoramic image. Assuming that the resolution of $A_{i,j}$ is $W\prime$ pixels by $H\prime$ pixels, a pyramid grid is established with $N$ levels for each overlapping region as follows:

First, the $N$ th level of the pyramid grid is segmented into cells with dimensions $\frac{W\prime }{2^N}$ by $\frac{H\prime }{2^N}$. Each cell at this level has a resolution of $2^N$ pixels by $2^N$ pixels, and the pixels within the cell share the same object-distance.Second, the pyramid grids from the $({N-1)}^{th}$ to the top layer of the $t^{th}$ level can be generated in a coarse-to-fine manner. At the $l^{th}$ level, the grid has dimensions of $\frac{W\prime }{2^l}$ by $\frac{H\prime }{2^l}$ cells, and each cell has a resolution of (of $2^l\frac{W\prime }{2^N}$ pixels by $2^l\frac{W\prime }{2^N}$ pixels), where $l$ takes values from N−1 down to $t$.

Considering the balance of efficiency and accuracy, the parameters $W\prime$, $H\prime$, $N$ and $t$ are given empirical values, namely 256, 2560, 8, and 4, respectively.

3.3.2. Estimation of Object-Distance Pyramid Maps in Overlapping Regions

The traditional object space-based matching method, which was called the Vertical Locus Line Method (VLL) [29], is often used to estimate depth information by accurately matching a pair of image patches without directly calculating the three-dimensional computation for object points. Previous approaches of the VLL matching algorithm are mainly constrained to perspective projection images and require priori depth information. As a result, the VLL matching algorithm may exhibit reduced efficiency or failure due to the absence of prior object-space information. This paper proposes the Panoramic Vertical Locus Line (PVLL) matching algorithm to estimate the object-distance of panoramic surroundings based on pyramid grids ranging from coarse to fine.

(1)

Estimation of Candidate Object-Distance

An object-distance set is estimated priorly to increase the efficiency of the search-space size for PVLL matching. Supposing that the minimum and maximum object-distances of surroundings are defined as $r_{min}$ and $\mathrm{\infty }$, the reprojected locations of the object point $Q_{ij}$ (where $r_{min}\le r\le \mathrm{\infty }$) will be restricted between point $p_{i,min}$ to point $p_{i,max}$ in image $I_i$, and between point $p_{j,min}$ and point $p_{j,min}$ in $I_j$, respectively, as shown in Figure 5.

Given the object-distance of point $Q_{ij}$ is $r$, the horizontal displacement $h\left(r\right)$ between the projection location and point $p_{i,max}$ in image $I_i$ can be approximately computed by Equation (8).

$$h\left(r\right)\approx \frac{L}{\alpha }\arccos \left(\frac{r-d\mathit{cos}\left(\frac{\pi }{2n}\right)}{\sqrt{d^2+r^2-2dr\mathit{cos}\left(\frac{\pi }{2n}\right)}}\right)$$

(8)

where $L$ is the width of the image of the sub-camera, $\alpha$ is the horizontal field of view of the sub-camera, and $d$ is the displacement between the optical center $C_{\mathrm{i}}$ and the origin $O$ of the cylindrical projection coordinate system.

Further, the list of candidate object-distances, set as $r_{list}=\left\{r_0,r_1,r_2,\cdots ,r_{T-1}\right\}$, can be acquired using Algorithm 1 based on Equation (8), where $\epsilon$ is the displacement steps in pixels.

Algorithm 1: Solution of Projected Radius List

Input: $L$, $\alpha$, $d$, $n$, $r_{min}$, $\epsilon$ Output: $r_{list}=\{r_0,r_1,r_2,\cdots ,r_{T-1}\}$ 1: $r\leftarrow r_{min}+1$, $r_0\leftarrow r_{min}$, $m\leftarrow 1$ 2: $h\leftarrow \frac{L}{\alpha }\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{c}\mathrm{o}\mathrm{s}(\frac{r-d\cos \left(\frac{\pi }{2n}\right)}{\sqrt{d^2+r^2-2dr\cos \left(\frac{\pi }{2n}\right)}})$ 3: repeat 4:  $h_b\leftarrow \frac{L}{\alpha }\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{c}\mathrm{o}\mathrm{s}(\frac{r-d\cos \left(\frac{\pi }{2n}\right)}{\sqrt{d^2+r^2-2dr\cos \left(\frac{\pi }{2n}\right)}})$ 5:  if $abs\left(h-h_b\right)>\epsilon$ then 6:    $r_m\leftarrow r,m\leftarrow m+1$, $h\leftarrow h_b$ 7:  end if 8:  $r\leftarrow r+1$ 9: until $h<\epsilon$ Output: $\{r_0,r_1,r_2,\cdots ,r_{T-1}\}$

As the object-distance changes continuously from $r_0$ to $r_{T-1}$, the back-projection location of an object point imaged in an overlapping region approximately moves along a straight line by ε pixels in a real image, as shown in Figure 6. Here, $\epsilon$ has been set to 1 pixel to provide an optimal balance between the object-distance resolution and computational efficiency.

(2)

Panoramic Vertical Line Locus Algorithm

Definition 1. 

The corresponding image patches are represented as $I_i\left(G_{i,j}^l\left(\phi ,\psi \right),R_{i,j}^l\left(\phi ,\psi \right)\right)$ and $I_j\left(G_{i,j}^l(\phi ,\psi ),R_{i,j}^l\left(\phi ,\psi \right)\right)$ at the $l$th pyramid level, respectively, where $G_{i,j}^l\left(\phi ,\psi \right)$ be the grid cell located at the ${\phi }^{th}$ row and ${\psi }^{th}$ column of the pyramid grid in $A_{i,j}$, $r_{R_{i,j}^l\left(\phi ,\psi \right)}$ be the object-distance in this cell, $R_{i,j}^l\left(\phi ,\psi \right)$ denotes the index of $r_{R_{i,j}^l\left(\phi ,\psi \right)}$ in $r_{list}$, and here $l$ takes values from N down to t.

We can assume that the pixel point $a$ in the grid cell is reprojected to locations of $(u_{a,i},v_{a,i})$ in image $I_i$ and $(u_{a,j},v_{a,j})$ in image $I_j$ using $r_{R_{i,j}^l\left(\phi ,\psi \right)}$ as the cylinder radius based on Equation (9) derived from Equation (5), respectively. Thus, a pair of grey values can be acquired based on the grey bilinear interpolation algorithm using four-pixel points around the points $(u_{a,i},v_{a,i})$ and $(u_{a,j},v_{a,j})$, respectively. In this way, $I_i\left(G_{i,j}^l\left(\phi ,\psi \right),R_{i,j}^l\left(\phi ,\psi \right)\right)$ and $I_j\left(G_{i,j}^l(\phi ,\psi ),R_{i,j}^l\left(\phi ,\psi \right)\right)$ can be generated with a resolution of $2^l$ pixels by $2^l$ pixels, respectively.

$$\left[\begin{array}{c}{u}_{a,i}\\ v_{a,i}\\ 1\end{array}\right]=s_i{K}_i{R}_i\left[\begin{array}{c}{r}_{R_{i,j}^l\left(\phi ,\psi \right)}sin\frac{\pi x_a}{L}-X_i\\ r_{R_{i,j}^l\left(\phi ,\psi \right)}cos\frac{\pi x_a}{L}-Y_i\\ \frac{2\pi r_{R_{i,j}^l\left(\phi ,\psi \right)}}{L}\left(\frac{W}{2}-y_a\right)-Z_i\end{array}\right], \left[\begin{array}{c}{u}_{a,j}\\ v_{a,j}\\ 1\end{array}\right]=s_j{K}_j{R}_j\left[\begin{array}{c}{r}_{R_{i,j}^l\left(\phi ,\psi \right)}sin\frac{\pi x_a}{L}-X_j\\ r_{R_{i,j}^l\left(\phi ,\psi \right)}cos\frac{\pi x_a}{L}-Y_j\\ \frac{2\pi r_{R_{i,j}^l\left(\phi ,\psi \right)}}{L}\left(\frac{W}{2}-y_a\right)-Z_j\end{array}\right]$$

(9)

Definition 2. 

Let  $NCC(G_{i,j}^l\left(\phi ,\psi \right),R_{i,j}^l\left(\phi ,\psi \right))$ be the Normalized Cross-Correlation image matching parameter between image blocks $I_i\left(G_{i,j}^l\left(\phi ,\psi \right),R_{i,j}^l\left(\phi ,\psi \right)\right)$ and $I_j\left(G_{i,j}^l\left(\phi ,\psi \right),R_{i,j}^l\left(\phi ,\psi \right)\right).$

The value of $NCC(G_{i,j}^l\left(\phi ,\psi \right),R_{i,j}^l\left(\phi ,\psi \right))$ can be computed by Equation (10).

$$NCC(G_{i,j}^l\left(\phi ,\psi \right),R_{i,j}^l\left(\phi ,\psi \right))=\frac{\sum _{x=0}^s\sum _{y=0}^s\left(g_i\left(x,y\right)-\overline{g_i}\right){(g}_j(x,y)-\overline{g_j})}{\sqrt{\sum _{x=0}^s\sum _{y=0}^s{\left(g_i\left(x,y\right)-\overline{g_i}\right)}^2}\sqrt{\sum _{x=0}^s\sum _{y=0}^s{{(g}_j(x,y)-\overline{g_j})}^2}}$$

(10)

where $g_i\left(x,y\right)$ and $g_j\left(x,y\right)$ are the greyscales located at $\left(x,y\right)$ in the image block $I_i\left(G_{i,j}^l\left(\phi ,\psi \right),R_{i,j}^l\left(\phi ,\psi \right)\right)$ and $I_j\left(G_{i,j}^l\left(\phi ,\psi \right),R_{i,j}^l\left(\phi ,\psi \right)\right)$, respectively. $\overline{g_i}$, $\overline{g_j}$ are the average grey values of the two image patches, respectively, where $s=2^l$.

Object-distance estimation based on the above definitions: Different from previous VLL-based methods that predict object-distance information directly, the proposed PVLL object-space matching method computes the index of the required object-distance in the estimated list $r_{list}$ from coarse to fine.

First, an objective function $E_{i,j}$ is generated at the $N^{th}$ pyramid layer in the overlapping region $A_{i,j}$ to optimize the object-distance information of each grid cell. $E_{i,j}$ is proposed based on the similarity of the image intensity between a pair of corresponding image patches and the smoothness of object-distance between adjacent grid cells, as depicted in Equation (11).

$$E_{i,j}={\rho }_1\sum _{\phi =0}^{\mathsf{\Phi }-1}\frac{NCC(G_{i,j}^N\left(\phi ,0\right),R_{i,j}^N\left(\phi ,0\right))}{\mathsf{\Phi }}+{\rho }_2\sum _{\phi =1}^{\mathsf{\Phi }-1}\frac{-\left|R_{i,j}^N\left(\phi ,0\right)-R_{i,j}^N\left(\phi -1,0\right))\right|}{T}$$

(11)

where ${\rho }_1$ and ${\rho }_2$ are the weight coefficients of similarity and smoothness, respectively. $E_{i,j}$ is the objective function; $\mathsf{\Phi }-1$ is the count of rows for the $N^{th}$ pyramid gird. Further, a group of optimum object-distance indices, denoted by ${\left(R_{i,j}^N\left(\mathrm{0,0}\right),R_{i,j}^N\left(\mathrm{1,0}\right),\cdots ,R_{i,j}^N\left(\mathsf{\Phi }-\mathrm{1,0}\right)\right)}^{\ast }$, can be estimated by iteratively solving Equation (12) using a dynamic programming algorithm [31]. Then, an object-distance map (ODM) is generated for the overlapping region $A_{i,j}$ at a resolution of $\frac{w}{2^N}$ by $\frac{H}{2^N}$ in the $N^{th}$ level, based on the optimal index parameters. Additionally, a corresponding index map is also created.

$${\left(R_{i,j}^N\left(\mathrm{0,0}\right),R_{i,j}^N\left(\mathrm{1,0}\right),\cdots ,R_{i,j}^N\left(\mathsf{\Phi }-\mathrm{1,0}\right)\right)}^{\ast }=\underset{R_{i,j}^N\left(\mathrm{0,0}\right),R_{i,j}^N\left(\mathrm{1,0}\right),\cdots ,R_{i,j}^N\left(\mathsf{\Phi }-\mathrm{1,0}\right)}{\mathit{argmax}}{E}_{i,j}$$

(12)

Second, from the ${(N-1)}^{th}$ to the top pyramid layer $t^{th}$, we employ an iterative image matching procedure as follows:

For each grid cell $G_{i,j}^l\left(\phi ,\psi \right)$, a group of candidate object-distances $\left\{r_{R_{i,j}^l\left(\phi ,\psi \right)+k\zeta }|k=0,\pm 1,\dots ,\pm K;\zeta =\frac{1}{K}\right\}$ are determined by Equation (13).

$$r_{R_{i,j}^l\left(\phi ,\psi \right)+k\zeta }=r_{⌊R_{i,j}^l\left(\phi ,\psi \right)+k\zeta ⌋}+k\zeta \left(r_{⌈R_{i,j}^l\left(\phi ,\psi \right)+k\zeta ⌉}-r_{⌊R_{i,j}^l\left(\phi ,\psi \right)+k\zeta ⌋}\right)$$

(13)

where $R_{i,j}^l\left(\phi ,\psi \right)$ are set as the optimum estimation of the object-distance index of the grid cell at the $\frac{\phi }{2}$ row and $\frac{\psi }{2}$ column in the previous layer, where $l$ takes values from N − 1 down to t.

Then, these candidates are chosen with regular index increments ζ, changing between the minimum object-distance $r_{R_{i,j}^l\left(\phi ,\psi \right)-1}$ and the maximum object-distance $r_{R_{i,j}^l\left(\phi ,\psi \right)+1}$. The value of K has been determined through experimentation to achieve the optimal balance between candidate performance and computational load. The optimum value of the parameter k, represented by $k^{\ast }$, is therefore extracted by searching the maximum NCC value using Equation (14), which is derived from Equations (9) and (10).

$$k^{\ast }=\underset{k}{\mathrm{argmax}}\left\{NCC\left(G_{i,j}^N\left(\phi ,\psi \right),R_{i,j}^l\left(\phi ,\psi \right)+k\zeta \right)|k=0,\pm 1,\dots ,\pm K\right\}$$

(14)

where $NCC\left(G_{i,j}^l\left(\phi ,\psi \right),R_{i,j}^l\left(\phi ,\psi \right)+k\zeta \right)$ is computed by a pair of corresponding image patches based on Equation (10), and the image patches are generated by reprojection of each pixel in $G_{i,j}^l\left(\phi ,\psi \right)$ onto $I_i$ and $I_j$ based on Equation (9), respectively.

Therefore, the optimal object-distance is set as $r_{R_{i,j}^l\left(\phi ,\psi \right)+k^{\ast }\zeta }$ for the grid cell $G_{i,j}^l\left(\phi ,\psi \right)$.

In this way, the resolution and structure of the object-distance maps, along with the index maps, are updated with the optimum object-distance for each grid cell, until reaching the top pyramid level.

3.4. Object-Distance Interpolation Method in the Non-Overlapping Regions

An interpolation method is adopted to derive object-distance information in the non-overlapping regions. For a pixel point $p\left(x_p,y_p\right)$ located in the non-overlapping region between two overlapping regions $A_{i,j}$ and $A_{j,\mathrm{k}}$, its object-distance can be interpolated from the known object-distance information of two points $p_l\left(x_{p_l},y_p\right)$ and $p_r\left(x_{p_r},y_p\right)$ using Equation(15), as shown in Figure 7.

$$r^t\left(x_p,y_p\right)=\left(\begin{array}{cc}\frac{x_{p_r}-x_p}{x_{p_r}-x_{p_l}-w}& \frac{x_p-x_{p_l}}{x_{p_r}-x_{p_l}-w}\end{array}\right)\left(\begin{array}{c}{r}^t\left(x_{p_l},y_p\right)\\ r^t\left(x_{p_r},y_p\right)\end{array}\right)$$

(15)

where $r^t\left(x_p,y_p\right),{ r}^t\left(x_{p_l},y_p\right)$, and $r^t\left(x_{p_r},y_p\right)$ are the object-distances of pixel point $p$, $p_l$ and $p_r$, respectively. The object-distance information can be calculated using the object-distance maps in the top-level of the aforementioned pyramid grids.

The values of $x_{p_l}$ and $x_{p_r}$ are determined by $x_{j,k}^L-\frac{w}{2}$ and $x_{i,j}^L+\frac{w}{2}$, respectively, where $w$ represents the width of an overlapping region. In this context, $i$ takes values from 1 to 7, j takes values from 2 to 7, and k takes values from 3 to 8, including 0. The parameters of $x_{i,j}^L$ and $x_{j,k}^L$ represent the displacements of the centre lines ${gap}_{i,j}$ and ${gap}_{j,k}$ from the y-axis of the panoramic image system, respectively. In the same way, the object-distances can be estimated for all the pixel points in the non-overlapping regions.

Finally, a panoramic image can be generated based on Equation (9) with the estimated object-distance information on the overlapping regions and non-overlapping regions.

4. Experimental Results

In this section, we conducted detailed experiments and carried out a comprehensive analysis for generating panoramic images using the panoramic camera system developed in this paper. The algorithm for the panorama generation was implemented in C++ using the OpenCV 4.3 development package on a computer equipped with an AMD Ryzen 7 5800H, 3.2 GHz processor, and 16 GB main memory.

4.1. Calibration of the Panoramic Camera

In this paper, the multi-camera system was fixed on an octagonal structure to ensure the constant relationships among cameras. A two-step method is conducted to calibrate the internal parameters of each camera and ROPs (relative orientation parameters). Firstly, more than 30 images of a high-precision chessboard were taken with each camera at different poses and positions, as shown in Figure 8. The inner orientation elements and the distortion coefficients of each camera were calibrated by Zhang’s calibration method [32]. The interior orientation elements for each camera are shown in

Table 1. Inner orientation elements of each camera.

Cam	${\mathit{f}}_{\mathit{x}}$	${\mathit{f}}_{\mathit{y}}$	${\mathit{c}}_{\mathit{x}}$	${\mathit{c}}_{\mathit{y}}$	${\mathit{k}}_1$	${\mathit{k}}_2$
1	2.2782	2.4294	1.3259	0.9473	0.0281	0.0098
2	2.2684	2.4194	1.1535	1.0435	0.0255	0.0036
3	2.2729	2.4238	1.3974	1.0620	0.0257	0.0045
4	2.2759	2.4256	1.2523	1.0469	0.0285	−0.0028
5	2.2601	2.4104	1.2447	9.4271	0.02718	0.0095
6	2.2744	2.4254	1.1629	1.0865	0.0253	0.0013
7	2.2761	2.4268	1.2567	1.0027	0.0253	0.0089
8	2.2731	2.4244	1.5161	1.0635	0.0306	−0.0094

.

Secondly, a professional three-dimensional calibrate field was used to calibrate the ROPs. Ten groups of images were taken at different poses and positions, where each group of images are captured at the same time, as shown in Figure 9.

The external parameters of the $i^{th}$ camera at the $k^{th}$ exposure epoch were calibrated based on the traditional PnP method [33]. Let the first camera be the master camera, while the other seven cameras are slave cameras. Let $R_{i,k}$ and $t_{i,k}$ represent the rotation matrix and position of the $i^{th}$ camera relative to the world coordinate system at the kth exposure epoch, where k = 1, 2, …, K. In this experiment, the value of K is 10. The ROPs between the master camera and any slave camera can be computed by Equations (16) and (17). The optimum relative orientation parameters can be seen in

Table 2. Relative orientation parameters of the combined cameras.

Cam	α/rad	β/rad	γ/rad	X/cm	Y/cm	Z/cm
1–2	0.0174	0.7814	−0.0029	4.8081	0.0295	−1.9565
1–3	0.0091	1.5850	0.0057	6.5616	−0.0294	6.6810
1–4	0.0046	2.3532	0.0038	4.8170	0.0683	−11.5291
1–5	−0.0205	−3.1253	−0.0108	−0.2132	0.1015	−13.3953
1–6	−0.0140	−2.3357	−0.0212	−4.7553	0.0871	−11.2954
1–7	−0.0011	−1.5729	−0.0238	−6.8198	0.0953	−6.7039
1–8	0.0105	−0.7723	−0.0197	−4.6815	0.0800	−1.8884

.

$$R_i^{\prime }=\underset{R_i^{\prime }\in S{O}_3}{\mathrm{argmin}}\sum _{k=0}^K‖R_i^{\prime }-R_{1,k}{\left(R_{i,k}\right)}^T‖$$

(16)

$$t_i^{\prime }=\frac{1}{K}\sum _{k=0}^K(t_{i,k}-R_{1,k}{t}_{i,k})$$

(17)

4.2. Visualized Analysis

We conducted a performance comparison of our proposed method (Method C) with AutoStitch [19] (Method A) and the traditional cylindrical panoramic stitching method [32] (Method B) using both indoor and outdoor scenes, as depicted in Figure 10.

Figure 11 displayed the panoramas for an outdoor scene. In Figure 11a, stitching truncation (resulting in structural discontinuity) can be observed in the roadway, highlighted with an orange box. In Figure 11b, the repetition of road lamps is evident, highlighted with the blue box. In contrast, the panoramic image generated by our algorithm (Method C) demonstrates visual consistency within the highlighted overlapping regions, as depicted in Figure 11c.

Figure 12 displays the panoramas for an indoor scene. In Figure 12a, more truncation and ghosting of the ceiling can be noticed, as highlighted with the orange box. Furthermore, the panoramic image shows significant gaps, particularly in areas with a low-texture environment lacking distinct feature points, which are highlighted with red boxes in the overlapping regions. In Figure 12b, you can observe that the ceilings are repeatedly generated, highlighted with an orange box. In contrast, the panoramic image generated by our algorithm (Method C) is complete and seamless in the area highlighted within the same orange box, as depicted in Figure 12c.

Therefore, it becomes evident that when compared to Method A and Method B, our approach (Method C) maintains visual consistency in both indoor and outdoor scenes and effectively eliminates stitching gaps, especially in scenes with low-texture regions.

4.3. Quantitative Analysis

Our quantitative analysis was conducted based on the following steps:

(1)

Each pixel within every overlapping region of a panoramic image was individually projected onto two overlapping original images captured by adjacent cameras, and the grayscale value of the projection points was obtained. In this way, two panoramic image patches were generated for each overlapping region, as shown in Figure 13.

(2)

The SIFT [34] feature points were then extracted and matched for each pair of panoramic image patches. The average Euclidean distance was then computed between corresponding feature points in every pair of panoramic image patches. In addition, the parameters of the Structural Similarity Index (SSIM) [31], Peak Signal-to-Noise Ratio (PSNR) [35], Normalized Cross-Correlation (NCC) coefficients, and Stitched Image Quality Evaluator (SIQE) [36] were determined to quantify the dissimilarities between each pair of panoramic image patches. The Root Mean Square Error (RMSE), average SSIM, average PSNR, and average NCC are further computed based on the average Euclidean distances, SSIM values, PSNR values, NCC coefficients, and SIQE respectively, obtained from the dataset of 160 panoramas.

(3)

Moreover, using the open-source library functions [37], we calculated the Stitched Panoramic Image Quality Assessment (SPIQA) [38] parameter for each group of images from eight cameras and the corresponding generated panoramic images. Then, we calculated the average of 160 SPIQAs.

Figure 13. An image pair generated from adjacent cameras.

Figure 13. An image pair generated from adjacent cameras.

This comprehensive analysis allowed us to evaluate the quality and dissimilarities based on various metrics and parameters, encompassing both structural and pixel-wise differences, as depicted in

Table 3. Stitching accuracy and efficiency.

Scene	Method	RMSE	SSIM	PSNR	NCC	SPIQA	SIQE	Time(s)
	Method A	6.3341	0.5530	16.2321	0.7076	0.6131	60.0379	8.9112
Outdoor	Method B	8.2422	0.5575	15.5963	0.5650	0.5839	54.7586	0.1251
	Method C	0.7498	0.6003	20.6502	0.8157	0.8586	68.4837	0.7539
	Method A	16.0997	0.7237	20.0779	0.8399	0.7582	51.6073	8.7745
Indoor	Method B	34.9983	0.7579	17.4994	0.3803	0.7938	58.7872	0.1268
	Method C	0.9837	0.8197	27.0731	0.8593	0.9047	63.0523	0.7473

.

Because the object distance in the outdoor scene is significantly larger than the eccentric distance between the optical center of a sub-camera and that of the virtual panoramic camera, the stitching accuracy of a panorama is less affected by inconsistencies in object-distance information within the surrounding scene. Thus, the proposed method only has a small improvement compared to Methods A and B. However, due to the restricted range of object-distances in indoor scenes, the eccentricity errors between cameras cannot be disregarded and may lead to substantial stitching errors when employing both Method A and Method B, especially when there are variations in the object-distance. Our proposed algorithm generates a panorama by utilizing the multiple projection radiuses from the estimation of object-distance information, which demonstrates far higher accuracy than Method A and B. In addition, it can be seen clearly that the stitching time of Method C is much higher than that of Method A and B.

5. Conclusions and Future Work

Aiming at remote sensing and mapping applications, we have designed a low-cost ${360}^{°}$ image acquisition system through the integration of eight web cameras, which were securely mounted on a circular rig. To solve the mosaicking problems caused by inconstant and inconsistent object-distance information within a scene, a panoramic image generation algorithm was proposed to improve the existing direct mono-cylinder imaging model. The key aspect of the algorithm is the Panoramic Vertical Line Locus matching algorithm called the PVLL algorithm, which is proposed by extending the traditional VLL algorithm. Moreover, the object-distance grids can be generated based on the PVLL algorithm in the overlapping regions of a panorama. Thus, the panorama can be generated by utilizing the multiple projection radiuses from the estimation of object-distance information. The experiments show that the proposed imaging method successfully improves the stitching accuracy and visual effects of virtual panoramas.

In summary, this paper discusses the problems of geometric mosaicking errors in multi-camera panoramic systems and conducts research on the combination of accurate calibration, depth estimation, and post-processing techniques to ensure a seamless and accurate panoramic view, despite variations in object distances and eccentric errors between camera optical centers. The method described in this paper is suitable for a multi-camera system with an annular structure, and we plan to extend it to a panoramic system with a 720-degree spherical design in our future work for ground remote sensing applications in both outdoor and indoor environments.