Viewport Rendering Algorithm with a Curved Surface for a Wide FOV in 360° Images

Abstract

Omnidirectional visual content has attracted the attention of customers in a variety of applications because it provides viewers with a realistic experience. Among the techniques that are used to construct omnidirectional media, a viewport rendering technique is one of the most important modules. That is because users can perceptually evaluate the quality of the omnidirectional service based on the quality of the viewport picture. To increase the perceptual quality of an omnidirectional service, we propose an efficient algorithm to render the viewport. In this study, we analyzed the distortions in the viewport picture to be the result of sampling. This is because the image data for the unit sphere in the omnidirectional visual system are non-uniformly sampled to provide the viewport picture in the conventional algorithms. We propose an advanced algorithm to construct the viewport picture, in which the viewport plane is constructed using a curved surface, whereas conventional methods use flat square surfaces. The curved surface makes the sampling interval more equally spaced than conventional techniques. The simulation results show that the proposed technique outperforms the conventional algorithms with respect to the objective quality based on the criteria of straightness, conformality, and subjective perceptual quality.

1. Introduction

Omnidirectional visual content has attracted more attention in the fields of gaming devices, remote lecturing systems, broadcasting, movies, and streaming services because it provides users with a more realistic experience than conventional techniques [1,2,3,4]. Many companies have developed a variety of devices that are related to omnidirectional visual technology. To meet the demands of the industry, the Joint Video Exploration Team (JVET) of ITU-T VCEG (Q6/16) and ISO/IEC MPEG (JTC 1/SC 29/WG 11) has been working on 360° video coding as part of the explorations that are being conducted for developing coding technologies for future video coding standards since 2015 [5,6,7].

An omnidirectional visual system consists of a variety of core technologies, including those for picture capturing, stitching [8,9,10,11], bundle adjustment [12,13], blending [14,15], 360° image formatting [16,17], 360° image encoding [18,19], streaming 360° data [20], and viewport rendering [7]. Recently, stereo omnidirectional visual systems have been introduced to increase the sense of immersion for users. The stereo omnidirectional visual content gives viewers more realistic images and videos [21,22]. Among these essential techniques, viewport rendering is one of the most important modules. Viewport rendering provides users with the final image frames and the users can evaluate the perceptual quality of the omnidirectional visual system based on the quality of the viewport picture (VP) [20,23,24].

During the last decade, many experts have discussed the distortions that have resulted from the various methods to render the VP. Some algorithms bend the straight lines into curved lines in the VP, whereas other methods differently stretch objects in the vertical and horizontal directions. This is because these methods cannot overcome the limitations that result from the geometrical properties of the 360° VR coordinate system.

To solve these problems, many researchers have developed advanced algorithms [25,26,27,28,29,30,31,32]. In a study by Sharpless et al. [25], they modified the cylindrical projection [26] to maintain the shape of the straight lines. An automatic content-aware projection method was proposed by Kim et al. [27]. Kim et al. developed a saliency map to classify the objects in the image according to their importance levels and they combined multiple images to minimize the distortion. In [28], Kopf et al. introduced the locally-adapted projections, where a simple and intuitive user interface allowed the specification of regions of interest to be mapped to the near-planar parts, thereby reducing the bending artifacts. In [29], Tehrani et al. presented the unified solution to correct the distortion in one or more non-occluded foreground objects by applying object-specific segmentation and affine transformation of the segmented camera image plane, where the algorithm was assisted by a simple and intuitive user interface. The algorithm proposed by Jabar et al. [30] moved the center of the projection to minimize the stretching error and bending degradation. Because the algorithms proposed in [27,30] created a saliency map [31] to determine the content of the 360° image, the additional process increased the complexities of these methods. Kopf et al. [32] defined the projected surface of the viewport as the surface of the cylinder, where the size of the cylinder is modified according to the field of view (FOV).

In this study, we consider the distortions in the VP to be the result of sampling because the image data on the unit sphere in the omnidirectional visual system are non-uniformly sampled to provide the VP for conventional algorithms. We propose an advanced algorithm to render the VP, where the viewport plane is constructed using a curved surface, whereas the conventional methods have used flat square surfaces. The curved surface makes the sampling interval more equally spaced than the conventional techniques.

This study is organized as follows. In Section 2, we explain the basic procedure to render the VP and two conventional algorithms: perspective projection [33] and stereographic projection [33]. Some processes are modeled, and we derive some equations to calculate the core parameters. In Section 3, we discuss the limitations of the conventional algorithms and the reasons why the pixels in the VP are distorted. We propose an advanced technique to render the VP in Section 4, where a curved surface is used for the projection plane. In Section 5, we demonstrate the performance of the proposed algorithm and compare it with the various conventional techniques. Finally, the conclusion is presented in Section 6.

2. Conventional Methods to Render the Viewport

2.1. Basic Procedure to Render the Viewport Picture

Figure 1 shows the 3D XYZ coordinate system that is used to represent the 3D geometry of a 360° image, where the $(\mathrm{X},\mathrm{Y},\mathrm{Z})$ coordinates are based on a right-hand coordinate system [7].

The sphere can be sampled with the longitude $\left(\mathsf{\varphi }\right)$ and latitude $\left(\mathsf{\theta }\right)$, where longitude $\mathsf{\varphi }$ is in the range of [−π, π], and latitude $\mathsf{\theta }$ is in the range of [−π/2, π/2]. In Figure 1, the longitude $\mathsf{\varphi }$ and latitude $\mathsf{\theta }$ are defined by the angle starting from the X-axis and they move in a counter-clockwise direction and by the angle from the equator toward the Y-axis, respectively. The relationships between the $(\mathrm{X},\mathrm{Y},\mathrm{Z})$ coordinates and $(\mathsf{\varphi },\mathsf{\theta })$ are described as follows [7].

$$\mathrm{X}=\cos (\mathsf{\theta })\cos (\mathsf{\varphi })$$

(1)

$$\mathrm{Y}=\sin \left(\mathsf{\theta }\right)$$

(2)

$$\mathrm{Z}=-\cos \left(\mathsf{\theta }\right)\sin (\mathsf{\varphi })$$

(3)

$$\mathsf{\varphi }{=\tan }^{-1}\left(-\frac{\mathrm{Z}}{\mathrm{X}}\right)$$

(4)

$${\mathsf{\theta }=\sin }^{-1}\left(\frac{\mathrm{Y}}{\sqrt{{\mathrm{X}}^2{+\mathrm{Y}}^2{+\mathrm{Z}}^2}}\right)$$

(5)

Figure 2 shows the viewport rendering procedure for the given 360° image, where the coordinates of the pixels in the viewport picture (VP) and UV plane (a rectangular $\mathrm{ABCD}$) are denoted by $(m,n)$ and $(u,v)$, respectively. The pixel values in the VP depend on several variables, including the FOV, the horizontal and vertical resolutions of the VP, and the projection technique. The procedure to calculate a pixel value of $(m,n)$ in the VP is equivalent to getting the value of pixel $P^{\prime }$ at the corresponding location $(\mathrm{X},\mathrm{Y},\mathrm{Z})$ in the 3D sphere image. Therefore, this procedure consists of three steps: (step 1) the derivation of the coordinate $(u,v)$, which is mapped to $(m,n)$; (step 2) calculation of the corresponding location $(\mathrm{X},\mathrm{Y},\mathrm{Z})$ of $P^{\prime }$ on the unit sphere for the $(u,v)$ that was calculated in (step 1); (step 3) obtaining the pixel value at $(\mathrm{X},\mathrm{Y},\mathrm{Z})$. The first and second steps are represented by two functions $f_M(m,n)$ and $f_p(u,v)$, respectively, as follows:

$${(u,v)=f}_M(m,n)$$

(6)

$${(\mathrm{X},\mathrm{Y},\mathrm{Z})=f}_p(u,v)$$

(7)

The detailed processes of $f_M(m,n)$ and $f_p(u,v)$ vary according to the projection method, which could be the perspective projection [33], stereographic projection [33], and so on. Note that the UV plane is an intermediate surface on which the sample data is projected. The viewers can watch the screen of the viewport plane, not the UV plane. In the following subsections, we explain two typical projections in detail.

In Figure 2, we assume that the viewing angle is along the Z-axis. If the VP selected by the user is not along the Z-axis, $(\mathrm{X},\mathrm{Y},\mathrm{Z})$ is rotated to ${(\mathrm{X}}^{\prime }{,\mathrm{Y}}^{\prime }{,\mathrm{Z}}^{\prime })$ by using a rotation equation. The pixel value at ${(\mathrm{X}}^{\prime }{,\mathrm{Y}}^{\prime }{,\mathrm{Z}}^{\prime })$ is used to calculate the pixel value of $(m,n)$ in the VP.

$$\left[\begin{array}{c}{\mathrm{X}}^{\prime }\\ {\mathrm{Y}}^{\prime }\\ {\mathrm{Z}}^{\prime }\end{array}\right]=\mathrm{R}\left[\begin{array}{c}\mathrm{X}\\ \mathrm{Y}\\ \mathrm{Z}\end{array}\right]$$

(8)

The rotation matrix R is defined as follows:

$$\mathrm{R}=R_{\mathsf{\varphi }}{R}_{\mathsf{\theta }},$$

(9)

where

$$R_{\mathsf{\varphi }}=\left[\begin{array}{ccc}\cos \left({\mathsf{\varphi }}_c+\pi /2\right)& 0& \sin \left({\mathsf{\varphi }}_c+\pi /2\right)\\ 0& 1& 0\\ -\sin \left({\mathsf{\varphi }}_c+\pi /2\right)& 0& \cos \left({\mathsf{\varphi }}_c+\pi /2\right)\end{array}\right]$$

(10)

$$R_{\mathsf{\theta }}=\left[\begin{array}{ccc}1& 0& 0\\ 0& \cos \left({\mathsf{\theta }}_c\right)& \sin \left({\mathsf{\theta }}_c\right)\\ 0& -\sin \left({\mathsf{\theta }}_c\right)& \cos \left({\mathsf{\theta }}_c\right)\end{array}\right]$$

(11)

In (10) and (11), we define $\left({\mathsf{\varphi }}_c{,\mathsf{\theta }}_c\right)$ as the angles between the view axis and the Z-axis along the axes of longitude $\mathsf{\varphi }$ and latitude $\mathsf{\theta }$, respectively. Note that the method of conversion from spherical coordinates to Cartesian coordinates is unstable around the zenith axis.

2.2. Perspective Projection

When perspective (i.e., rectilinear) projection [33] is used in the procedure to render a VP, the cross-section of the UV plane and the sphere can be represented as in Figure 3, where point $P^{\prime }$ is projected onto $P$ along the straight line between $O$ and $P$. In this case, the width $W_{uv}$ and height $H_{uv}$ of the UV plane depend on the horizontal and vertical FOVs, respectively, as follows:

$$W_{uv}=2\tan \left(\frac{F_h}{2}\right)$$

(12)

$$H_{uv}=2\tan \left(\frac{F_v}{2}\right)$$

(13)

where $F_h$ and $F_v$ denote the FOVs in the horizontal and vertical directions, respectively. By applying (12) and (13), the function $f_M(m,n)$ of (6) is represented by the following equations for the perspective projection [33].

$$u=(m+0.5)\frac{W_{uv}}{W_{VP}},0\le {m<W}_{VP}$$

(14)

$$v=(n+0.5)\frac{H_{uv}}{H_{VP}},0\le {n<H}_{VP}$$

(15)

where $W_{VP}$ and $H_{VP}$ are the width and height of the VP, respectively. The position $($u,v$)$ of $P$ in Figure 3 is represented with 3D coordinates $(x,y,z)$ using the geometric structure as follows:

$$\left[\begin{array}{c}x\\ y\\ z\end{array}\right]=\left[\begin{array}{cc}1& 0\\ 0& -1\\ 0& 0\end{array}\right]\left[\begin{array}{c}u\\ v\end{array}\right]+\left[\begin{array}{c}-W_{uv}/2\\ H_{uv}/2\\ 1\end{array}\right]$$

(16)

Point $P$ is projected onto point $P^{\prime }$ on the unit sphere, where the coordinates $(\mathrm{X},\mathrm{Y},\mathrm{Z})$ of $P^{\prime }$ can be determined as follows:

$$\mathrm{X}=\frac{x}{\sqrt{x^2{+y}^2{+z}^2}}$$

(17)

$$\mathrm{Y}=\frac{y}{\sqrt{x^2{+y}^2{+z}^2}}$$

(18)

$$\mathrm{Z}=\frac{z}{\sqrt{x^2{+y}^2{+z}^2}}$$

(19)

Equations (16)–(19) explain the detailed procedure of $f_p(u,v)$ in (7) when the perspective projection [33] is used.

2.3. Stereographic Projection

Figure 4 shows the UV plane and sphere when the stereographic projection [33] is used, where point $P$ is projected onto $P^{\prime }$ along the straight line between $O^{\prime }$ and $P$, whereas the line between $O$ and $P^{\prime }$ is used in the perspective projection [33].

The cross-section of Figure 4 is shown in Figure 5, where the width $W_{uv}$ and height $H_{uv}$ of the UV plane can be determined as follows:

$$W_{uv}=\frac{4\sin \left(\frac{F_h}{2}\right)}{1+\cos \left(\frac{F_h}{2}\right)}$$

(20)

$$H_{uv}=\frac{4\sin \left(\frac{F_v}{2}\right)}{1+\cos \left(\frac{F_v}{2}\right)}$$

(21)

In the stereographic projection, $f_M(m,n)$ from (6) is implemented by substituting (20) and (21) into (14) and (15). After $(u,v)$ has been derived by using $f_M(m,n)$, it can be represented with 3D coordinates $(x,y,z)$ by applying (16). When the stereographic projection is used, the 3D coordinates of $P^{\prime }$ are as follows:

$$\mathrm{X}=\frac{4x}{x^2{+y}^2+4}$$

(22)

$$\mathrm{Y}=\frac{4y}{x^2{+y}^2+4}$$

(23)

$$\mathrm{Z}=\frac{8}{x^2{+y}^2+4}+1$$

(24)

3. Limitation of the Conventional Methods

According to Jabar et al. [24], the conventional algorithms that are used to render the viewport produce a variety of distortions when the FOV is wide, although these distortions may not be noticeable for narrow FOVs. In this section, we discuss the reasons why the conventional techniques have limitations for wide FOVs followed by the analysis of the degradation tendency according to the degree of the FOV.

3.1. Distortions in Conventional Methods

The subjective qualities of the viewports that are generated with various FOVs are compared in Figure 6, where (a–c) and (d–f) show the perspective [33] and stereographic [33] projections, respectively. As observed in Figure 6, when the perspective projection is used, the viewport shows a distorted perspective for a wide FOV. On the other hand, when the stereographic projection is used, straight lines are severely bent as the FOV increases.

3.2. Analysis of the Distortion Resulting from Conventional Methods

As illustrated in Figure 2, the pixel values in the viewport are set to those with the corresponding $P^{\prime }$ points on the unit sphere. When the pixel value at $(m,n)$ in the VP is denoted by $I(m,n)$, the corresponding $P^{\prime }$ point on the unit sphere is represented by $P_{(m,n)}^{\prime }$, where the coordinate of $P_{(m,n)}^{\prime }$ is denoted with the form ${(\mathrm{X}}_{(m,n)}{,\mathrm{Y}}_{(m,n)}{,\mathrm{Z}}_{(m,n)})$. To analyze the distance between the positions of the corresponding $P_{(m,n)}^{\prime }$ points, we calculated the vertical length of the arc between $P_{(m,n)}^{\prime }$ and $P_{(m,n+1)}^{\prime }$ and the horizontal length between $P_{(m,n)}^{\prime }$ and $P_{(m+1,n)}^{\prime }$, as demonstrated in Figure 7. Note that the length is the sampling interval that is used to generate a VP from a 360° image.

In Figure 7, $d^h(m,n)=\left|\overline{P_{(m,n)}^{\prime }{P}_{(m+1,n)}^{\prime }}\right|$ and $l^h(m,n)=\left|\overbrace{P_{(m,n)}^{\prime }{P}_{(m+1,n)}^{\prime }}\right|$ are the shortest direct distance and the length of the arc between the horizontally nearest $P_{(m,n)}^{\prime }$ neighbors. Similarly,$d^v(m,n)$ and $l^v(m,n)$ are the values for the vertically nearest neighbors, respectively, as follows:

$$d^h(m,n*)=\left|\overline{P_{(m,n*)}^{\prime }{P}_{(m+1,n*)}^{\prime }}\right|=\sqrt{{{(\mathrm{X}}_{(m,n*)}-{\mathrm{X}}_{(m+1,n*)})}^2{+(\mathrm{Y}}_{(m,n*)}-{\mathrm{Y}}_{(m+1,n*)}{)}^2{+(\mathrm{Z}}_{(m,n*)}-{\mathrm{Z}}_{(m+1,n*)}{)}^2}$$

(25)

$$d^v(m*,n)=\left|\overline{P_{(m*,n)}^{\prime }{P}_{(m*,n+1)}^{\prime }}\right|=\sqrt{{{(\mathrm{X}}_{(m*,n)}-{\mathrm{X}}_{(m*,n+1)})}^2{+(\mathrm{Y}}_{(m*,n)}-{\mathrm{Y}}_{(m*,n+1)}{)}^2{+(\mathrm{Z}}_{(m*,n)}-{\mathrm{Z}}_{(m*,n+1)}{)}^2}$$

(26)

$$l^h(m,n*)=\left|\overbrace{P_{(m,n*)}^{\prime }{P}_{(m+1,n*)}^{\prime }}\right|{=2\sin }^{-1}\left(\frac{d^h(m,n*)}{2}\right)$$

(27)

$$l^v(m*,n)=\left|\overbrace{P_{(m*,n)}^{\prime }{P}_{(m*,n+1)}^{\prime }}\right|{=2\sin }^{-1}\left(\frac{d^v(m*,n)}{2}\right)$$

(28)

where $0\le {m<W}_{VP}-1$, $0\le {n<H}_{VP}-1$, $n^{*}$ and $m^{*}$ are the specific values that are used to calculate the lengths of the arcs between the horizontal and vertical positions. Figure 8 presents the normalized values of $l^h(m,n*)$ according to the horizontal FOV ($F_h$) when the perspective and stereographic projections are used. In this figure, we can observe that the length between the locations of the consecutive sampling points on the sphere decreases as $m$ moves away from the center (${m=(W}_{VP}/2)-1$). This means that the pixels in the boundary region of a VP are made by sampling the $P^{\prime }$ points with reduced sampling intervals on the sphere. This makes the pixels in the VP look like a stretched form of $P^{\prime }$ points on the sphere. This tendency is more significant as $F_h$ increases. As observed in Figure 8a,b, the amount of stretching in the perspective projection is greater than that in the stereographic projection. Note that the analysis for $l^v{(m}^{*},n)$ is similar to those in Figure 8, although it is not presented in this paper.

4. Algorithm Proposed to Render the Viewport

4.1. Curved UV Surface

By analyzing Figure 8 in the previous section, when the VP is rendered with the wide FOV, the degradation in the VP is due to the non-uniformed sampling space. In this section, we propose a model for the UV surface, which makes the sampling space uniform. Figure 9 depicts a model to represent the proposed UV surface, which is tangent to the unit sphere. The proposed UV surface is a part of the outer sphere whose center and radius are $O^{\prime }$ and $r$, respectively. The outer sphere intersects with the unit sphere at the center of the UV surface. The outer sphere is represented by the following equation.

$$x^2{+y}^2{+(z+r-1)}^2{=r}^2$$

(29)

In Figure 10, the red curve represents the cross-section of the proposed UV surface, where $F_v$ is the vertical FOV and $F_v^{OUT}$ is the corresponding vertical angle at the center $O^{\prime }$ of the outer sphere to cover the surface that is generated by applying $F_v$ at the center $O$ of the unit sphere. The relationship between $F_v$ and $F_v^{OUT}$ is as follows:

$$\tan \left(\frac{F_v}{2}\right)=\frac{r \sin \left(\frac{F_v^{OUT}}{2}\right)}{r \cos \left(\frac{F_v^{OUT}}{2}\right)-(r-1)}$$

(30)

Equation (30) can be rewritten as follows:

$$F_v^{OUT}{=4\tan }^{-1}\left(\frac{\sqrt{r^2{+(2r-1)\tan }^2\left(\frac{F_v}{2}\right)}-r}{(2r-1)\tan \left(\frac{F_v}{2}\right)}\right)$$

(31)

$$u=\frac{m+0.5}{W_{VP}},0\le {m<W}_{VP}$$

(32)

$$v=\frac{n+0.5}{H_{VP}},0\le {n<H}_{VP}$$

(33)

Note that Equation (31) is numerically stable only for $r>0.5$. $F_v^{OUT}$has a complex number for $0<r<0.5$. When $r=0.5$, $F_v^{OUT}$ is not defined. However, because the curved UV surface is a part of the outer sphere as shown in Figure 10, $r$ would be larger than 1. Thus, in the proposed algorithm, $F_v^{OUT}$is always numerically stable.

Equations (32) and (33) explain the detailed process for $f_M(m,n)$ from (6) when the proposed UV surface is used. Based on the geometry that is related to the outer sphere, the coordinates $(x,y,z)$ of $P$ are derived as follows:

$${\mathsf{\varphi }}^{OUT}{=F}_h^{OUT}\text{×}\left(0.5-u\right)$$

(34)

$${\mathsf{\theta }}^{OUT}{=F}_v^{OUT}\text{×}\left(0.5-v\right)$$

(35)

$${x=r\cos (\mathsf{\theta }}^{OUT}\left)\sin \right({\mathsf{\varphi }}^{OUT})$$

(36)

$${\mathrm{y}=r \sin (\mathsf{\theta }}^{OUT})$$

(37)

$${\mathrm{z}=r \cos (\mathsf{\theta }}^{OUT}\left)\cos \right({\mathsf{\varphi }}^{OUT})-(r-1)$$

(38)

Point $P$ is projected onto point $P^{\prime }$ on the unit sphere, where the coordinates of $P^{\prime }$ are $(\mathrm{X},\mathrm{Y},\mathrm{Z})$, which are calculated by using (17)–(19). When the proposed UV surface is used, $f_p(u,v)$ from (7) is implemented by applying (34)–(38) and (17)–(19).

Figure 11 displays the VPs that occur from applying the curved UV surface for the FOV = {75, 120} and $r$ = {2, 5, 20, 0.3}, where a larger outer sphere produces flatter straight lines. When $r$ is set to 0.3, the generated viewports degrade significantly, because $r$ = 0.3 is included in the range of instability in Equation (31).

4.2. Adjusted Sampling Intervals

In this section, we adjust the sampling intervals that are made by using the curved UV surface so that the sampling positions are more evenly spaced. Figure 12 presents a mapping function between the original index $u$ and the modified index $u^{\prime }$ on the UV plane, where the function passes through {S(1), S(3), S(5)} on the diagonal line and {S(2), S(4)}, which are not on the diagonal line. {S(1), S(2), S(3), S(4), S(5)} are ancillary parameters that are used to explain the mapping function. The quantity of the deviation of {S(2), S(4)} from the diagonal line is set to $w$, which is defined as follows:

$$w=\frac{{\mathrm{FOV}-\mathrm{thr}}_{\mathrm{FOV}}}{\mathsf{\alpha }},\mathrm{FOV}\ge {\mathrm{thr}}_{\mathrm{FOV}}$$

(39)

where ${\mathrm{thr}}_{\mathrm{FOV}}$ is a threshold for the FOV. When the FOV is larger than ${\mathrm{thr}}_{\mathrm{FOV}}$, i.e., when the FOV is wide, the sampling intervals are modified. In (39), the parameter $\mathsf{\alpha }$ is set based on the empirical data. Using this mapping function, the sampling points $P^{\prime }$ on the unit sphere are more uniformly sampled horizontally. This reduces the degradation (e.g., stretching) in the boundary regions of the viewport.

The mapping function in Figure 12 can be modeled with the following equation:

$$u\prime =g\left(u\right)=c_0+c_{1}u+c_2{u}^2+c_3{u}^3$$

(40)

where the variables $\left\{c_0{,c}_1{,c}_2{,c}_3\right\}$ are calculated by applying a polynomial regression algorithm [34] as follows:

$$\left[\begin{array}{c}{u}_1^{\prime }\\ u_2^{\prime }\\ u_3^{\prime }\\ u_4^{\prime }\\ u_5^{\prime }\end{array}\right]=\left[\begin{array}{cccc}1& u_1& u_1^2& u_1^3\\ 1& u_2& u_2^2& u_2^3\\ 1& u_3& u_3^2& u_3^3\\ 1& u_4& u_4^2& u_4^3\\ 1& u_5& u_5^2& u_5^3\end{array}\right]\left[\begin{array}{c}{c}_0\\ c_1\\ c_2\\ c_3\end{array}\right]$$

(41)

$$\overrightarrow{u}=\mathbf{U}\overrightarrow{c}$$

(42)

$$\overrightarrow{c}={\left({\mathbf{U}}^{\mathrm{T}}\mathbf{U}\right)}^{-1}{\mathbf{U}}^{\mathrm{T}}\overrightarrow{u}$$

(43)

Note that all of the techniques that are explained in this subsection can be applied for $v$.

In Figure 8a,b, if the sampling positions are ideal, then the normalized $l^h(m,n*)$ is equal to 1 for all $\mathrm{m}$ values. Thus, the degradations in these figures can be evaluated by using the following equation.

$${\mathrm{L}}_{\mathrm{n}*}^{\mathrm{h}}=\frac{1}{W_{VP}-1}{\displaystyle \sum }_{m=0}^{W_{VP}-2}{\left\{1-\left(\frac{l^h(m,n*)}{\mathrm{Max}\left(l^h(m,n*)\right)}\right)\right\}}^2$$

(44)

In order to analyze the effect of the parameter $r$ of equation (29) in the proposed algorithm, the values of ${\mathrm{L}}_{\mathrm{n}*}^{\mathrm{h}}$ of (44) for various$r$ are shown in Figure 13. In the figure, as the value of ${\mathrm{L}}_{\mathrm{n}*}^{\mathrm{h}}$ decreases, the quality of the viewport increases because it means that the data on the sphere are sampled uniformly. As shown in Figure 13, the optimal values of $r$ differ based on ${\mathrm{F}}_{\mathrm{h}}$, where the optimal values of $r$ for ${\mathrm{F}}_{\mathrm{h}}$ = {75, 90, 120, 150} are {3.2061, 3.4041, 3.1588, 2.9040}, respectively. The optimal values for $r$ were obtained by Quasi-Newton Method [35].

To demonstrate the effect of the proposed UV surface, Figure 14 displays the normalized $l^h(m,n*)$ when $(u,v)$ of the UV plane is modified by applying the mapping technique of Figure 12 for $u$ and $v$ independently after the proposed UV surface is employed with the optimized values of $r$.

Based on (44), the degradations of the various methods in Figure 8 and Figure 14 are summarized in

Table 1. Degradation evaluated with (44) for the various techniques.

${\mathbf{F}}_{\mathbf{h}}.$	Perspective Projection	Stereographic Projection	Curved UV $\mathbf{Surface}\mathbf{with}\mathbf{the}\mathbf{Optimized}\mathit{r}$	Curved UV Surface with $\mathit{r}=0.3$
75	0.039604	0.002697	0.000010	0.173255
90	0.078799	0.005504	0.000051	0.335975
120	0.229415	0.016779	0.000289	0.559911
150	0.517675	0.039523	0.000701	0. 597737

. As observed in Table 1, the degradation of the proposed algorithm is much smaller than those for conventional algorithms. The degradation values in the $r$ = 0.3 column are larger than those in other columns because $r$ = 0.3 is one instance of invalid data. The analysis for $l^v(m,n*)$ is similar to those in Figure 8 and Figure 14, although it is not provided in this manuscript.

5. Simulation Results

To demonstrate the performance of the proposed algorithm, we compared it with a variety of conventional techniques, including the perspective projection [33], stereographic projection [33], Pannini projection [25], automatic content-aware (ACA) projection [27], and adaptive cylindrical (AC) projection [32].

5.1. Quantitative Evaluation

In this section, an objective evaluation is performed for the various conventional techniques and the proposed algorithm, where straightness and conformality are used as criteria, as demonstrated by Kim et al. [27]. Figure 15 explains how to calculate the straightness and conformality as criteria in the objective evaluation. The evaluation equations for these are as follows:

$$\mathrm{Straightness}=\frac{{\mathsf{\alpha }}_2}{{\mathsf{\alpha }}_1{+\mathsf{\alpha }}_2}.$$

(45)

$$\mathrm{Conformality}=\frac{{\min (\mathsf{\beta }}_1{,\mathsf{\beta }}_2{,\mathsf{\beta }}_3{,\mathsf{\beta }}_4)}{{\max (\mathsf{\beta }}_1{,\mathsf{\beta }}_2{,\mathsf{\beta }}_3{,\mathsf{\beta }}_4)}$$

(46)

The maximum values of straightness and conformality are 1. As these values get closer to 1, the degradation in the straightness and conformality decreases.

Figure 16 displays the patterns that occur using the various algorithms when nested squares are used as a test pattern to evaluate the straightness and conformality. It can be observed that some lines are curved and some red circles have been deformed. The bent lines imply that the applied methods result in poor performance with respect to the straightness. In a case where the circle is badly deformed, we know that the conformality of the algorithm is not good.

Based on (45) and (46), the evaluation results for the straightness and conformality are summarized in

Table 2. Quantitative evaluation of the projection results.

Projection Method	Straightness	Conformality
Perspective [33]	0.999999	0.719107
Stereographic [33]	0.973522	0.878603
Pannini [25]	0.97737	0.788848
ACA [27]	0.978032	0.753724
AC [32]	0.978640	0.616209
Proposed Algorithm with the optimized r	0.980597	0.826667

, where the best results and the second-best results are represented with red and blue numbers, respectively.

As observed in Figure 16 and Table 2, the perspective method shows the best performance in the straightness, whereas it provides one of the worse results for the conformality. In contrast, the stereographic method shows the smallest value for the straightness and the largest value for the conformality. This implies that the stereographic algorithm outperforms the other techniques in maintaining the shape of the circles in the viewport, but it bends the lines more than the other schemes. Although the proposed algorithm does not provide the best performance for the straightness and conformality, the overall performance of the proposed algorithm is in the upper ranks. In particular, Figure 16 shows that the overall subjective deformation is much less than the other techniques.

5.2. Qualitative Evaluation

In this section, we evaluate the subjective performances of the various algorithms that are used to render a viewport picture. Figure 17 presents the six test images that are used for the subjective evaluation, which show the equirectangular images whose horizontal and vertical FOVs are 360° × 180°, respectively.

Figure 18 displays the viewport images that are rendered by the various projection methods. The viewport images rendered by the equirectangular pictures are shown in Figure 18 in which the horizontal and vertical FOVs of the viewport are set to 150° and 120°, respectively. As demonstrated in Figure 18, the subjective qualities of the proposed method are much higher than those of the others. This is because the objects are deformed and the lines are bent in the pictures that are produced by the perspective [33], stereographic [33], Pannini [25], ACA [27] and, AC [32] methods.

5.3. Perceptual Evaluation

To provide simulation results for the perceptual evaluation, the single stimulus method of BT.500 [36] is used for the images in Figure 19. The mean opinion score (MOS) is a simple measurement to obtain the viewers’ opinions. The MOS provides a numerical indication of the quality that is perceived by the viewers. The MOS is expressed as a single number in the range of 1–5, where 1 means bad and 5 means excellent. In this test, the viewport images with a wide FOV were evaluated and they were scored by 20 viewers who were graduate students. Figure 19 shows the MOS values. The MOS values of the proposed method are always much higher than those of the conventional methods [25,27,31,33].

5.4. Complexity Comparision

To compare the computational complexities of the algorithms, we evaluated the central processing unit (CPU) times that were required by the various algorithms, where a personal computer incorporating an AMD Ryzen 5 2600 six-core Processor @3.40 GHz and DDR4 32 GB was used. The CPU times are summarized in

Table 3. Central processing unit (CPU) time consumed by the various algorithms.

Projection Method	Consumed CPU Time (Second)
Perspective [33]	0.201
Stereographic [33]	0.241
Pannini [25]	0.270
ACA [27]	1.066
AC [32]	0.253
Proposed Algorithm with the optimized r	0.340

, where the proposed algorithm is much simpler than ACA [27]. This is because ACA [27] required additional processes to detect the line component and to optimize the related parameter, whereas the proposed algorithm does not need these. The complexities of the other conventional algorithms, such as the perspective [33], stereographic [33], Pannini [25], and AC [32] algorithms are approximately equal to that of the proposed method.

6. Conclusions

We discussed the limitations of the conventional algorithms that are used to render the VP and we have proposed an algorithm to overcome these problems. A curved surface was used to efficiently project the pixel data on the unit sphere of an omnidirectional visual system onto the VP. Even though conventional techniques use a flat square plane to project the pixel data with non-uniform sampling intervals, the proposed curved surface reduced the non-uniformity and this resulted in a perceptually enhanced VP.

Using rendering engines can be considered to implement the proposed algorithm in real-time applications. First, the rasterization-based pipeline in the GPU is a unit designed to increase the speed of filling the pixel values in the triangle meshes on the screen. However, the proposed algorithm projects a value on the sphere to the pixel in the viewport. Consequently, the proposed algorithm is not suitable for implementation using the rasterization-based pipeline in the GPU. Second, the ray tracing-based pipeline in the GPU can be considered to enhance the algorithm’s performance. In this scenario, we need to access the data pixel-by-pixel in the module of the ray-tracing pipeline. However, most commercial GPU products do not provide the accessibility. Third, we can consider Intel Embree, which is a collection of high-performance ray tracing kernels. Since the Embree does not have the limitations of GPU-based systems, it best matches the proposed algorithm.