Trajectory and Phase Shift Optimization for RIS-Equipped UAV in FSO Communications with Atmospheric and Pointing Error Loss

Abstract

This paper proposes a new framework for reconfigurable intelligent surface (RIS)-equipped unmanned aerial vehicles (UAVs) in free-space optical (FSO) communication. To ensure practicality, we consider atmospheric loss caused by fog, which leads to an inhomogeneous medium for laser propagation. In addition, we incorporate the pointing error loss caused by the power fraction on the photodetector (PD) into the system and derive a closed-form expression for the elliptical beam footprint in the pointing error loss. We then propose a leading angle assisted particle swarm optimization (PSO) method to efficiently optimize the numerical results of pointing error loss. Furthermore, after obtaining these numerical results as a precondition, the UAV trajectory is optimized using the proximal policy optimization (PPO) method to achieve the maximum average capacity. Numerical simulations demonstrate that the proposed optimization method achieves greater efficiency and accuracy compared to the decode-and-forward (DF) relay and deep Q-learning (DQN) methods.

1. Introduction

Free space optical (FSO) communications were developed and researched extensively as a potential technology for sixth generation (6G) wireless communication networks [1]. Different from radio frequency (RF) communications, the main feature of FSO communications is to use the laser beam to carry information, which brings various benefits, including high transmission rate, free of eavesdroppers by transmission directivity, and free of frequency spectrum crowding [2]. However, there are two problems that affect the performance of FSO communications. The first one is that FSO communication requires line-of-sight (LOS) links between the light source (LS) and the photodetector (PD); however, in practical circumstances, the connection between LS and PD could be blocked by obstacles [3]. The second problem is the susceptibility to the varying propagation environment caused by rain, fog, and other low-visibility atmospheric phenomena [4].

With the intention of dealing with the non-LOS requirement and maximizing the throughput, the authors in [5] utilized a fixed position relay. Usually, this type of relay is deployed on a building to assist non-LOS transmission. Following that, inter-relay cooperation was proposed in FSO communication to reduce the outage probability in [6]. However, the relay is fixed in a position, so that the communication between the transmitter and receiver lacks flexible connectivity capability. To tackle this issue, a UAV was introduced as an operating platform to assist communication and improve system performance flexibility and efficiency. For example, in [7], the authors considered a UAV decode-and-forward relay in a downlink maritime communication scenario, solving the average achievable rate maximization problem. Furthermore, the authors in [8] proposed a multi-UAV assisted solution to achieve the capacity maximization criterion assuming a cognitive network. Motivated by using a UAV in RF communications, there are existing works that consider a UAV relay for improving FSO communication. For example, in [9], the authors introduced a dual-hop space-air-ground integrated RF/FSO network; with the assistance of a UAV, the satellite could link the terrestrial users with low outage probability. In [10], a similar scenario was proposed for the multi-users on the ground using quadrature amplitude modulation. Furthermore the UAV facilitating the link between space and air, in [11], was utilized to assist the FSO communication between ground non-LOS transmitter and receiver deployed in an urban area. Therefore, the deployment of the UAVs for FSO communication is a valid solution for high data rate purposes, especially under complex circumstances. However, the power consumption and additional signal processing are significant problems for a UAV equipped-relay mobile platform [12], which is a limitation for performance and UAV operating endurance.

For the purpose of reducing power consumption and processing time delay, reflecting intelligent surface (RIS) technology was widely utilized in wireless networks. Thanks to the feature of RIS that can control wireless signal transmission through passive phase adjustment with low cost [13]. In [14], the feasibility was described for RIS application on new generation RF communication, where the channel has been turned into a controllable system that can apply the optimization. Furthermore, in paper [15], the phase shifts of RIS can be designed and improved by the joint transmit beamforming method. Not only in RF communication but the work [16] also indicated that RIS and its feature of freedom rotations could be utilized in FSO communication. What is more, the work in [17] showed that RIS could be regarded as a mirror with a phase shift function in FSO communication, which means eliminating the time delay and power consumption caused by signal processing. The works in [16,17] showed the fixed position mounted RIS could solve the LOS requirement of FSO communication. However, the flexibility and capability coverage for fixed mounted RIS is limited. Therefore, in [18], a UAV was deployed to improve the flexibility and capability of coverage for FSO communication. The work [19] indicates that the RIS can be assembled in a UAV platform and capture the advantages from both to maximize the system throughput. However, the UAV was set to hovering in a fixed position, which lacks the consideration of dynamic trajectory. To further highlight and clarify the novelty of this work, the main difference between the proposed work and existing works is given in

Table 1. Comparisons of some existing works, the ✗ stands for not included, the ✓ stands for included, and the N/A stands for not applicable.

Ref.	Channel Medium	Fog Impact Modelling	No Transmission Delay	RIS Optimization	UAV Trajectory Design
[5]	FSO/RF	✗	✗	N/A	N/A
[6]	FSO	✗	✗	N/A	N/A
[7]	RF	N/A	✗	N/A	✗
[8]	RF	N/A	✗	N/A	✗
[9]	FSO/RF	✗	✗	N/A	✗
[10]	FSO/RF	✗	✗	N/A	✗
[11]	FSO	✗	✗	N/A	✗
[12]	RF	N/A	✓	✓	✗
[13]	RF	N/A	✓	✓	✗
[16]	FSO	✗	✓	✓	N/A
[17]	FSO	✗	✓	✓	N/A
[18]	FSO	✗	✓	✓	✗
This paper	FSO	✓	✓	✓	✓

.

In order to mitigate atmospheric loss and pointing error loss, UAV trajectory optimization is considered for UAV-assisted FSO communication. The paper [20] shows that machine learning can be utilized in communication scenarios. In [21], a traditional UAV trajectory optimization method based on geometry analysis was proposed to find the shortest path and best cellular network coverage in a connectivity-constrained system. Furthermore, in [22], a 2D UAV trajectory was designated by the conventional non-convex optimization under different optimal conditions such as speed limitation, hovering duration requirement, and coverage requirement. As the learning method identifies the trends and patterns of data much more easily, the efficiency and accuracy can be improved continuously by gaining experience [23]. Therefore, when the optimal problem is non-convex or NP-hard, an AI or learning-powered method is a potential option for UAV trajectory optimization [24]. For example, in [25], trajectory design and power allocation based on typical machine learning optimization were formulated for maximizing the instantaneous sum rate of multi-UAVs. Moreover, conventional machine learning was limited when processing the raw data such as UAV flight status [26]. The deep reinforcement learning (DRL) method constructed by multiple processing layers to learn representations of data is, therefore, more efficient for trajectory optimization in wireless communication scenarios. The DRL algorithm is proposed to solve the instability problem and to improve the data inefficiency. In [27], by proposing the policy-based learning method, where an estimator of the policy gradient is been added. Also, a stochastic gradient ascend is exploited in order to achieve the maximum rewards. These features showed us more clear options for our optimization method. In the work [28], the DRL learning method was proposed as an online altitude control and scheduling policy for UAVs to obtain instantaneous channel state information (CSI) in real time along with any adjustment to its deployment altitude. What is more, in [29], a multi-UAV network was constructed, where the problem is non-convex with sophisticated states, and the individual UAV may not know the reward functions of other UAVs, the DRL was utilized as an improved clip and count-based algorithm for the multi-agent deep reinforcement learning scenario, which enables each UAV to select its policy in a distributed manner. Not only the altitude control and multi-agent optimization, in [30], the DRL method was proposed to optimize the new generation terahertz communication throughput, including UAVs to ground stations association, transmit power, and trajectory optimization problems. In [31], the DRL shows its high applicability for the complicated scenario where a multi-UAV trajectory planning task is required. Both the geographical fairness of user equipment and overall energy consumption are needed to optimize.

Motivated by these works, the RIS-equipped UAV for FSO communication under the influence of atmospheric and pointing error loss will be investigated in this paper. To overcome the physical impacts and improve performance, the optimization proposed for pointing error loss uses a novel leading angle assisted particle swarm optimization (PSO) method, which is competent in efficiently finding the optimal continuous phase shifts. Furthermore, we consider a PPO optimization for determining the UAV trajectory in FSO communication. The main contributions of this work are as follows:

• We propose a new RIS-equipped UAV FSO communication technique in a non-LOS scenario with atmospheric and pointing error loss and derive a closed-form expression with the laser beam incident upon the PD having non-orthogonality status. An ellipse beam footprint geometrical model is considered to express the power density on the PD.
• Based on the proposed framework, we derived the closed-form expression for physical impacts, a novel leading angle assisted PSO optimization method is proposed to optimize the numerical results of pointing error loss as a precondition. Then, the PPO method is introduced to solve the UAV trajectory optimization to reduce the complicated physical impacts and optimize the average capacity.
• Simulation results verify that the leading angle assisted PSO and PPO methods are efficient and indicate that the UAV tends to fly to the area with the maximum value of average capacity and avoids the fog. Furthermore, the proposed RIS-assisted strategy improves the average capacity significantly compared to the conventional decode-and-forward (DF) relay-assisted UAV networks.

The remainder of this paper is organized as follows. In Section 2, the system and problem formulation are described, where we clear our aim and model the FSO channel coefficients. After confirming the target of optimization, Section 3 presents the first optimization with the PSO method for optimizing the phase shifts of the RIS, which minimizes the impact of pointing error loss. Then, Section 4 operates the second optimization for the UAV trajectory, based on the PPO learning method. The details of the UAV operation under these two optimizations with analysis and discussion are shown in Section 5. Finally, the conclusion of the work is given in Section 6.

Notations: We use lower-case, boldface lower-case letters to represent scalars and vectors, respectively. Furthermore, $\left|\right|·\left|\right|$ is the norm of a vector; $\mathrm{erf}(·)$ denotes the error function and $\exp (·)$ denotes the exponential function; $“·”$ and $“\times ”$ denote the dot and cross product between vectors, respectively.

2. System and Problem Formulation

In this paper, we consider a 3D Cartesian coordinate system for FSO communication networks, as shown in Figure 1. A laser source (LS) as a transmitter aims to communicate with a photodetector (PD) receiver through a Gaussian laser beam. Furthermore, a UAV carries the RIS as a reflecting platform. We assume that the UAV could acquire the needed information in our scenario, such as the position of and PD [32]. We also assume that there is no direct link between the LS and PD, and the LS is located at the origin point at $O(0,0,0)$, and consider a PD has a circular detection aperture with radius a and the center of the PD is located at $S(x_s,y_s,0)$. For practical consideration, we assume the beam is affected by fog, which means the atmospheric conditions vary during the beam propagation. We model a fog that can severely affect the system’s Channel State Information (CSI), and treat fog as a spherical entity with radius $r_f$ for the sake of simplicity and initial simulations [33] (Note: In the ’Optimization Results’ of this paper, we address the fog in irregular shape for practical thinking and present corresponding results), located at $F(x_f,y_f,z_f)$ (Note: It should be noted that the UAV’s ability to ascertain the location of fog is facilitated by certain technologies and methods. These can include meteorological data services, onboard atmospheric sensors, or advanced imaging techniques [34,35]). To satisfy the aiming conditions from LS to PD via RIS, we assume that the rotations of the LS can keep tracking the UAV (Note: The details of the laser tracking and aiming method can be achieved by using precise pointing, acquisition and tracking technology [36], we assume the UAV is deployed and control by other ground station to operate the communication task in this area, This paper does not consider the orientation of the UAV because the phase shift change caused by the orientation of the UAV can be added to the optimal phase shifts of the RIS once we know the flight orientation of the UAV). Moreover, we assume the area of the RIS is big enough to cover the beam footprint from the LS [37]. Due to the laser beam following the specular reflection with the RIS, this perspective is central to our methodology as the reflective properties of the RIS allow it to direct the path of light, we consider the status of RIS instead of UAV and introduce the concept of leading angles, which stand for the rotation status of RIS. As shown in Figure 2, the leading angles can be expressed by ${\theta }_1$ and ${\theta }_2$, which stand for the rotations along the x-axis and y-axis with counterclockwise direction. This approach integrates the UAV status information such as the Angle of Arrival (AoA) of the UAV into the leading angles of the RIS. Therefore, the phase shifts of the RIS at the nth time slot can be set as

$$\mathrm{\Theta }\left[n\right]({\theta }_1\left[n\right],{\theta }_2\left[n\right])\in (-\pi ,\pi ).$$

(1)

We consider the 3D flight operation for the UAV with the nth time slot, $n\in \left\{1,2,\dots ,N\right\}$. The constraints of phase shifts of RIS can be denoted as $\mathrm{\Phi }=\left\{\mathrm{\Theta }\right[n],n=1,2,\dots ,N\}$. The coordinates of the UAV with time slots are also set as following the positive integer waypoints in the flight zone, which are denoted as $G\left[n\right](x_g\left[n\right],y_g\left[n\right],z_g\left[n\right])$, where $G\left[1\right]$ and $G\left[N\right]$ denote the initial and final positions of the UAV. We set the velocity of the UAV as $V\left[n\right]$ at the nth time slot, and the maximum and minimum velocity of the UAV are set as $V_{max}$ and $V_{min}$, the operation height of the UAV is limited from $H_l$ to $H_h$. According to the above description, we have the following constraints:

$$\begin{array}{cc}\hfill & \mathrm{\Phi }=\left\{\mathrm{\Theta }\right[n],n=1,2,\dots ,N\}\hfill \\ \hfill & -\pi \le \mathrm{\Theta }\left[n\right]\le \pi \hfill \\ \hfill & G\left[1\right]=(x_g\left[1\right],y_g\left[1\right],z_g\left[1\right])\hfill \\ \hfill & G\left[N\right]=(x_g\left[N\right],y_g\left[N\right],z_g\left[N\right])\hfill \\ \hfill & V_{min}\le V\left[n\right]\le V_{max},n=1,2,\dots ,N\hfill \\ \hfill & H_l\le z_g\left[n\right]\le H_h.\hfill \end{array}$$

(2)

The channel modeling of the system is measured by the channel coefficients at the nth time slot from the LS to PD via the RIS as $h\left[n\right]$, which are represented by (Note: Here, due to the turbulence of atmosphere, is a random variable that only relies on the constant from circumstances, most of the papers engage the UAV and FSO communication optimization and do not take it into consideration, such as [38,39])

$$h\left[n\right]=h_t{h}_a\left[n\right]h_p\left[n\right],$$

(3)

where $h_t$ is the turbulence of the atmosphere, this parameter is related to the wind speed in refractive index and the laser wavelength [40], where the probability density function (PDF) of $h_t$ can be denoted as [41]

$$f\left(h_t\right)=\frac{2{\left(\alpha \beta \right)}^{\frac{\alpha +\beta }{2}}{h}_t^{\frac{\alpha +\beta }{2}-1}{K}_{\alpha -\beta }\left(2\sqrt{\alpha \beta h_t}\right)}{\mathrm{\Gamma }\left(\alpha \right)\mathrm{\Gamma }\left(\beta \right)},$$

(4)

where $K_{\alpha -\beta }(.)$ is the modified Bessel function of the second kind, the $\mathrm{\Gamma }(.)$ is the Gamma function, and the $\alpha$ and $\beta$ denote the numbers of small and large turbulence cells as follows,

$$\begin{array}{cc}\hfill & \alpha ={\left(exp\left(\frac{0.49{\sigma }_R^2}{{\left(1+1.11{\sigma }_R^{\frac{12}{5}}\right)}^{\frac{7}{6}}}\right)-1\right)}^{-1},\hfill \\ \hfill & \beta ={\left(exp\left(\frac{0.51{\sigma }_R^2}{{\left(1+0.69{\sigma }_R^{\frac{12}{5}}\right)}^{\frac{5}{6}}}\right)-1\right)}^{-1},\hfill \end{array}$$

(5)

respectively, where Rytov variance ${\sigma }_R$, which can be calculated as ${\sigma }_R^2=0.5k^{\frac{7}{6}}{C}_N^2{L}^{\frac{11}{6}}$, and $C_N^2$ is the index of refraction structure parameter, $k=\frac{2\pi }{{\lambda }_f},$ can be obtained by optical wavelength ${\lambda }_f$.

The $h_a\left[n\right]$ is the atmospheric loss caused by fog, which can be given by

$$h_a\left[n\right]=exp(-{\sigma }_c{d}_c\left[n\right]-{\sigma }_f{d}_f\left[n\right]),$$

(6)

where ${\sigma }_c$ and ${\sigma }_f$ denote the measurement of atmospheric attenuation conditions under clean air and fog [42], respectively, which can be given as ${\sigma }_c=\frac{3.91}{L_v}{\left(\frac{\lambda }{550}\right)}^{-p_c}$ and ${\sigma }_f=\frac{3.91}{L_v}{\left(\frac{\lambda }{550}\right)}^{-p_f}$ where

$$p_c=\left\{\begin{array}{c}\begin{array}{cc}\hfill 1.6& L_v>50\\ \hfill 1.3& 6<L_v<50,\end{array}\hfill \end{array}\right.$$

(7)

and

$$p_f=\left\{\begin{array}{c}\begin{array}{cc}\hfill 0.16L_v+0.34& 1<L_v<6\\ \hfill L_v-0.5& 0.5<L_v<1\\ \hfill 0& L_v<0.5,\end{array}\hfill \end{array}\right.$$

(8)

where $L_v$ denotes the visibility range in air. Furthermore, $d_c\left[n\right]$ and $d_f\left[n\right]$ denote the laser propagation distances in clear air and fog at the nth time slot, respectively, which can be obtained as in Appendix A.

Due to the UAV operating with a 3D flight, the incident laser beam is not perpendicular to the PD, and the Gaussian profile beam footprint is an ellipse on the PD plane as shown in Figure 3. The pointing error loss $h_p\left[n\right]$ is caused by laser power fraction on the PD and can be the integral results from the beam power intensity, denoted as

$$I(\mathit{\rho };\left[n\right])=\frac{2}{\pi W_x\left[n\right]W_y\left[n\right]}exp\left(-\frac{{2\left|\right|\mathit{\rho }\left|\right|}^2}{W_x\left[n\right]W_y\left[n\right]}\right),$$

(9)

then consider it under the polar coordinate system of the detector plane with radial a and angular $2\pi$, we can express the pointing error loss as

$$\begin{array}{cc}\hfill h_p\left[n\right]=& {\int }_{A}I\left(\right||\mathit{\rho }\left[n\right]-\mathit{r}\left[n\right]\left|\right|)\mathrm{d}\mathit{\rho }\left[n\right]\hfill \\ \hfill =& {\int }_0^a{\int }_0^{2\pi }\frac{2}{\pi W_y\left[n\right]W_x\left[n\right]}exp\frac{-2{(\mathit{\rho }\left[n\right]-\mathit{r}\left[n\right])}^2}{W_y\left[n\right]W_x\left[n\right]}\mathit{\rho }\left[n\right]\mathrm{d}\varphi \left[n\right]\mathrm{d}\mathit{\rho }\left[n\right]\hfill \end{array}$$

(10)

where A is the detector area, I is the normalized spatial distribution of the transmitted intensity, $\mathit{\rho }\left[n\right]$ is the radial vector which starts from the beam center and ends at any point on the PD, $\mathit{r}\left[n\right]$ is the pointing error vector starting from the center of the PD and ending at the center of the beam footprint with norm $r\left[n\right]$.

After some derivations, we can obtain the closed-form expression of pointing error loss as follows:

$$\begin{array}{cc}\hfill h_p\left[n\right]=& \left(r\left[n\right]\sqrt{2\pi W_y\left[n\right]W_x\left[n\right]}\right(\mathrm{erf}\left(\frac{\sqrt{2}r\left[n\right]}{\sqrt{W_y\left[n\right]W_x\left[n\right]}}\right)\hfill \\ \hfill & +\mathrm{erf}\left(\frac{\sqrt{2}(a-r\left[n\right])}{\sqrt{W_y\left[n\right]W_x\left[n\right]}}\right))+W_y\left[n\right]W_x\left[n\right]\hfill \\ \hfill & \left(exp\left(\frac{-2r^2\left[n\right]}{W_y\left[n\right]W_x\left[n\right]}\right)-exp\left(\frac{-2{(a-r\left[n\right])}^2}{W_y\left[n\right]W_x\left[n\right]}\right)\right))\hfill \\ \hfill & \frac{1}{W_y\left[n\right]W_x\left[n\right]}.\hfill \end{array}$$

(11)

where $W_y\left[n\right]$ and $W_x\left[n\right]$ denote two different beamwidths of the beam footprint, which can be calculated by using a rotation matrix and the law of specular reflection as in Appendix B.

According to the intensity modulation and direct detection (IM/DD) method as in [43], the average capacity between the LS and PD can be obtained by representing the probability density function (PDF) of the received electrical signal-to-noise ratio (SNR) $f\left(\gamma \right[n\left]\right)$

$$c\left[n\right]={log}_2(1+ϵ\gamma \left[n\right]),$$

(12)

where $ϵ=\frac{e}{2\pi }$ [44], and the received SNR $\gamma \left[n\right]$ can be obtained by

$$\gamma \left[n\right]=\frac{2P_t^2{\mu }^2{h}_a^2\left[n\right]h_p^2\left[n\right]h_t^2}{{\sigma }^2},$$

(13)

where $P_t$ is the transmitting power from the LS, and ${\sigma }^2$ is the Gaussian noise variance, and $\mu$ is the detector responsivity.

Then, to overcome the atmospheric and pointing error loss, and maximize the average capacity by optimizing the phase shifts of the RIS $\mathrm{\Phi }=\left\{\mathrm{\Theta }\right[n],n=1,2,\dots ,N\}$, and the trajectory of the UAV $\mathit{G}=\left\{G\right[n],n=1,2,\dots ,N\}$, we formulate a problem as follows:

$$\begin{array}{cc}\hfill \left(\mathbf{P}1\right):& \underset{\mathrm{\Phi },\mathit{G}}{max}\frac{1}{N}\sum _{n=1}^{N}c\left[n\right]\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\left(1\right)\left(2\right).\hfill \end{array}$$

(14)

According to the formulated optimal problem, it can be seen that the constraint of phase shifts problem (1) and the UAV trajectory constraint (2) are independent; the optimal problem can be divided into two independent optimal problems, one is to optimize the phase shift, and the other is optimizing the UAV trajectory to maximize the average capacity. In the following section, the phase shift optimization of the RIS is proposed.

3. PSO-Based Optimization of RIS Phase Shifts

On the purpose of fulfilling the target to achieve the maximum average capacity, where the maximum SNR $\gamma \left[n\right]$ should be reached. Then, the target translates to maximize the pointing error and atmospheric loss coefficients $h_a\left[n\right]$ and $h_p\left[n\right]$. Therefore, we exploit Particle Swarm Optimization (PSO) to optimize the pointing error loss within each position point of the UAV during its operation. Due to the UAV operating the 3D flight, the incident laser beam is not perpendicular to the PD, and the Gaussian profile beam footprint is an ellipse on the PD plane. Furthermore, we considered the pointing error loss $h_p\left[n\right]$ in this work to model the effect of the fluctuation, which is equivalent to considering imperfect CSI at the PD. By iteratively adjusting leading angles of RIS in each iteration $m\in \left\{1,2,\dots ,M\right\}$ of PSO, the number of $h_p\left[n\right]$ will be maximized as a precondition in each UAV position, denoted as

$$\begin{array}{cc}\hfill \left(\mathbf{P}2\right):& \underset{\mathrm{\Phi },\mathit{G}}{max}{h}_p\left[n\right]\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\left(1\right)\left(2\right).\hfill \end{array}$$

(15)

This method allows for the decoupling of the phase shifts optimization problem and the UAV trajectory optimization problem, which simplifies our optimization challenge into two independent yet separated problems. In this paper, the PSO algorithm was proposed for the purpose of obtaining high-precision continuous phase shifts leading angles for the most optimized RIS status. The steps of PSO are listed as:

3.1. Initialization

Set the initial conditions of the PSO, which starts from generating the population $p\in \{1,2,\dots ,P\}$ of particle swarms. The following description of the algorithm is for each swarm, as we set the loop iterations in the PSO algorithm as $m\in \left\{1,2,\dots ,M\right\}$. Then the initial velocity of each swarm can be set as $v_1$[1] and $v_2$[1]. After that, the corresponding updating parameter including local and global acceleration $c_1$ and $c_2$ are also defined.

3.2. Calculate the Leading Angles

The leading angles ${\theta }_1^{*}$ and ${\theta }_2^{*}$ are RIS phase shifts, which are proposed to accelerate the optimization, calculated as the center of the beam aligned with the center of the PD, and the calculation method can be found in Appendix C.

3.3. Set Personal and Global Best

After calculating the leading angles of phase shifts, we set the initial personal best ${\theta }_{P_1}={\theta }_1^{*}$ and ${\theta }_{P_2}={\theta }_2^{*}$, as the same as the global best ${\theta }_{G_1}$ and ${\theta }_{G_2}$.

3.4. PSO Main Loop

At the beginning of the main loop of the PSO algorithm, the swarm status is updated by the initial settings, denoted as

$$\begin{array}{cc}\hfill & {\theta }_1[m+1]={\theta }_1\left[m\right]+v_1\hfill \\ \hfill & {\theta }_2[m+1]={\theta }_2\left[m\right]+v_2.\hfill \end{array}$$

(16)

After updating the swarm status at the beginning of each loop, the swarm velocity in each loop can be denoted using random parameter $s=\mathrm{rand}(·)$ between 0 and 1, and constants $c_1$ and $c_2$

$$\begin{array}{cc}\hfill v_1[m+1]=v_1\left[m\right]+& c_{1}s({\theta }_{P_1}\left[m\right]-{\theta }_1\left[m\right])+\hfill \\ & c_{2}s({\theta }_{G_1}\left[m\right]-{\theta }_1\left[m\right])\hfill \\ \\ \hfill v_2[m+1]=v_2\left[m\right]+& c_{1}s({\theta }_{P_2}\left[m\right]-{\theta }_2\left[m\right])+\hfill \\ & c_{2}s({\theta }_{G_2}\left[m\right]-{\theta }_2\left[m\right]).\hfill \end{array}$$

(17)

After the swarm position and velocity are all updated, the current pointing error loss can be gained for further comparison. Furthermore, the cross boundary treatment [45] is included as a limitation for the searching swarm could not move outside the search zone, denoted as

$${\theta }_1\left[m\right]=\left\{\begin{array}{c}\begin{array}{c}min({\theta }_{max},2{\theta }_{min}-{\theta }_1\left[m\right]),{\theta }_1\left[m\right]<{\theta }_{min}\hfill \\ max({\theta }_{min},2{\theta }_{max}-{\theta }_1\left[m\right]),{\theta }_1\left[m\right]>{\theta }_{max},\hfill \end{array}\hfill \end{array}\right.$$

(18)

$${\theta }_2\left[m\right]=\left\{\begin{array}{c}\begin{array}{c}min({\theta }_{max},2{\theta }_{min}-{\theta }_2\left[m\right]),{\theta }_2\left[m\right]<{\theta }_{min}\hfill \\ max({\theta }_{min},2{\theta }_{max}-{\theta }_2\left[m\right]),{\theta }_2\left[m\right]>{\theta }_{max},\hfill \end{array}\hfill \end{array}\right.$$

(19)

where the search boundary is $\mathrm{\Theta }\left[n\right]({\theta }_1\left[n\right],{\theta }_2\left[n\right])\in (-\pi ,\pi )$, and normally the search zone is set to include all the phase shifts, which make the beam footprint overlap with the PD. After confirming that the swarm will not be outside the search boundary, the local best position of the swarm should be updated. The update method uses a comparison of the target function output between the current swarm and the local best swarm position, then updates the local best pointing error loss if it’s less than the current output. Similar to updating the global best, by comparing the global best and local best pointing error loss output, the most optimized phase shifts and their corresponding pointing error loss can be found. The whole algorithm is written as pseudocode in Algorithm 1.

Algorithm 1: PSO

1: input the parameter set of the PSO $P,M,c_1,c_2,v_1,v_2$    2: derive the rotation quaternion and its rotation matrix.    3: extract the leading angles of phase shifts.    4: input the initial local and global best positions ${\theta }_{P_1},{\theta }_{P_2},{\theta }_{G_1},{\theta }_{G_2}$.    5: for $m=1,2,\dots ,M$    6: for $p=1,2,\dots ,P$      7: update the swarm status and swarm velocity as (16) and (17).      8: output the current pointing error loss $h_p\left[m\right]$      9: implement the cross boundary treatment as (18) and (19).     10: compare the current swarm output and the local best swarm output.   11:  if $h_p({\theta }_1\left[m\right],{\theta }_2\left[m\right])>h_p({\theta }_{P_1}\left[m\right],{\theta }_{P_2}\left[m\right])$   12:  update the current angles and output as personal best angles and output   13:  end if     14: compare the personal best swarm output and the global best swarm output.   15:  if $h_p({\theta }_{P_1}\left[m\right],{\theta }_{P_1}\left[m\right])>h_p({\theta }_{G_1}\left[m\right],{\theta }_{G_2}\left[m\right])$   16:  update the personal angles and output as global best angles and output   17:  end if   18: end for   19: end for   20: output the global best outcome as the maximum pointing error.

4. PPO-Based Optimization of the UAV Trajectory

After solving the formulated problem $\mathbf{P}2$ to gain the most optimized pointing error based on leading angles, the constraints of the optimization problem were only based on the trajectory of the UAV. Then, when the optimal phase shift is given, problem $\mathbf{P}1$ can be rewritten as

$$\begin{array}{cc}\hfill \left(\mathbf{P}3\right):& \underset{\mathit{G}}{max}\frac{1}{N}\sum _{n=1}^{N}c\left[n\right]\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\left(2\right).\hfill \end{array}$$

(20)

To solve $\mathbf{P}3$ as maximizing the average capacity based on the maximum $h_p\left[n\right]$ and UAV flight constraints, a PPO algorithm is introduced for planning the operating trajectory of the UAV. As shown in Figure 4, the proposed PPO-based trajectory optimization algorithm is based on the Markov decision process (MDP) and improved by the prioritized experience replay (PER) method, which is introduced in the following.

• MDP formulation

The deep reinforcement learning (DRL) method is based on the MDP, which includes the state space, action space, and reward design. The MDP of this system considers time steps with the index of time slots n with the upper limitation N, where $n\in \left\{1,2,\dots ,N\right\}$.

• State space

The state-space S includes the UAV’s operating zone limitations as the input of the PPO, consisting of all accessible UAV positions in the whole terrain, and also includes the remaining operation time of the UAV.

• Action Space

The action space A includes the current direction, velocity, and coordinates of the UAV.

• Reward Design

The reward function is formulated as the average capacity of the FSO communication. As the UAV can be at any 3D waypoint $G\left[n\right]$, there is a corresponding average capacity $c\left[n\right]$.

4.1. PPO Algorithm

Learning Algorithm

In this section, we present the proposed PPO learning algorithm for UAV trajectory design, and the steps are written as pseudocode in Algorithm 2. PPO is an on-policy, model-free reinforcement learning algorithm in which decision-making is based on data collected from the most up-to-date policy. The goal of the PPO model is to enable the agent to execute optimal actions that maximize long-term cumulative rewards. Consequently, in the PPO model, the selected action may not be the optimal choice for the current time slot, but it aims to be the optimal choice for pursuing long-term benefits. The central controller, which controls the UAV, acts as the agent in this process.

At each time slot, the agent observes a state from the state space S, which consists of the coordinates of the RIS-equipped UAV. The action space A represents the set of available actions, the agent based on the current state and decided by stochastic policy ${\pi }_{{\upsilon }_k}$, where the actions decided by policy rules can be denoted as $a_t$. After executing the actions, the agent receives a reward determined by the average capacity under the current FSO connectivity condition.

In order to adjust the surrogate objective problem where the huge ratio can be denoted as

$$r_t\left({\upsilon }_k\right)=\frac{{\pi }_{{\upsilon }_{k+1}}\left(a_t|s_t\right)}{{\pi }_{{\upsilon }_k}\left(a_t|s_t\right)},$$

(21)

where ${\upsilon }_k$ is the policy parameter that includes the operation movement rules established for the UAV. Between the current policy ${\pi }_{\upsilon }$ and new policy ${\pi }_{{\upsilon }_{k+1}}$, via the PPO-clip update policy

$${\upsilon }_{k+1}=\arg \max L^{clip}\left({\upsilon }_k\right),$$

(22)

to maximize the clipped surrogate objective $L^{clip}\left({\upsilon }_k\right)$, which can be denoted as

$$\begin{array}{c}\hfill L^{clip}\left({\upsilon }_k\right)={\widehat{E}}_t[min(r_t\left({\upsilon }_k\right){\widehat{D}}_t,\\ \hfill \mathrm{clip}(r_t\left({\upsilon }_k\right),1-\varrho ,1+\varrho ){\widehat{D}}_t\left)\right],\end{array}$$

(23)

where the clip function can be denoted as

$$\mathrm{clip}(\varrho ,{\widehat{D}}_t)=\left\{\begin{array}{c}\begin{array}{c}1+\varrho {\widehat{D}}_t\ge 0\hfill \\ 1-\varrho {\widehat{D}}_t<0\hfill \end{array}\hfill \end{array}\right.$$

(24)

and $\varrho$ is a hyperparameter, ${\widehat{E}}_t$ is the estimator of the advantage function ${\widehat{D}}_t$. The conventional update policy between two iterations relies on the Monte Carlo approximation, where the surrogate objective is maximized, denoted as

$${\upsilon }_{k+1}=\arg \max L\left({\upsilon }_k\right).$$

(25)

When the value of advantage ${\widehat{D}}_t$ for the state-action pair is positive, the objective reduces to

$$L^{pos}\left({\upsilon }_k\right)={\widehat{E}}_t\left[\min \left(\frac{{\pi }_{{\upsilon }_{k+1}}\left(a_t|s_t\right)}{{\pi }_{{\upsilon }_k}\left(a_t|s_t\right)},(1+\varrho )\right){\widehat{D}}_t\right],$$

(26)

when the value of advantage ${\widehat{D}}_t$ for the state-action pair is negative, the objective reduces to

$$L^{neg}\left({\upsilon }_k\right)={\widehat{E}}_t\left[\max \left(\frac{{\pi }_{{\upsilon }_{k+1}}\left(a_t|s_t\right)}{{\pi }_{{\upsilon }_k}\left(a_t|s_t\right)},(1-\varrho )\right){\widehat{D}}_t\right].$$

(27)

After updating the policy by maximizing the PPO-clip objective, the value function needs to be fitted by regression on mean-squared error.   

Algorithm 2: PPO

1: initialize the environment   2: initialize the policy parameter ${\upsilon }_0$ and value function parameters ${\eta }_0$   3: for $k=0,1,2,\dots ,K$     4: collect set of trajectories $P_k={\upsilon }_i$ by running policy ${\pi }_k=\pi \left({\upsilon }_k\right)$ in the environment.   5: compute rewards ${\widehat{R}}_t=c\left[n\right]$   6: compute advantage estimates ${\widehat{D}}_t$ based on the current value function $V_{{\eta }_k}$   7: update the policy by maximizing the PPO-clip objective ${\upsilon }_{k+1}=\arg \max L^{clip\left({\upsilon }_k\right)}$.   8: fit value function by regression on mean-squared error ${\eta }_{k+1}$ via gradient descent.   9: end for

5. Optimization Results

This section shows the UAV trajectory after optimization under different conditions. The parameters used in these numerical results and their corresponding values are listed in

Table 2. Simulation Parameters.

Parameter	Value
Visibility range clean/fog	200/0.3$\mathrm{km}$
Transmitting power	45$\mathrm{dBm}$
LS coordinate	(0, 0, 0)$\mathrm{m}$
PD coordinate	(0, 100, 0)$\mathrm{m}$
Laser wavelength	1550$\mathrm{nm}$
Receiver radius	0.05$\mathrm{m}$
Detector responsivity	0.5
Atmospheric turbulence	0.91
UAV height limitation	60–90$\mathrm{m}$
UAV velocity limitation	1.5$\mathrm{m}/\mathrm{s}$

. The parameter settings follow the practical scenario. For example, the visibility range in fog follows the research on weather effects on FSO communication [46]. The selected laser type is 1550$\mathrm{nm}$ wavelength that most FSO scenarios use [47], we take $h_t=0.91$ for weak turbulence when the wind is not strong for this type of laser [48,49]. The UAV settings follow the civil aviation law [50] of height limitation and flight performance of consumer drones [51]. Based on [36,37], we also set the minimum and maximum height of the UAV as $H_l=60\mathrm{m}$ and $H_h=90\mathrm{m}$, and the maximum speed of the UAV as $1.5\mathrm{m}/\mathrm{s}$ (Note: To simplify the optimization problem, as in [21,22,25,52], the acceleration of the UAV is not considered in this paper, we note that it is affected by the minimum and maximum velocity and total flight times).

We use a decode-and-forward (DF) relay-assisted FSO system as a comparison benchmark to show the performance gain of the proposed scheme. We assume that the UAV has both FSO PD and transmitter function, each hop’s pointing error is modeled with a circular beam footprint, and the misalignment is set as zero between footprint and PD, which will provide the best case for the benchmark scenario, which also shows the comparison if the results of PSO is not obtained precondition for trajectory optimization. Moreover, the deep Q-network (DQN) is utilized as the trajectory optimization benchmark in this paper.

Figure 5 compares the trajectories under RIS and DF scenarios without fog effect with the different flight times. It is shown that the UAV operating the flight with maximum speed reaches the area with maximum capacity under both RIS and DF scenarios. The average capacity of PPO-based trajectory in the 150-time slots is 4.05 $\mathrm{bits}/\mathrm{s}/\mathrm{Hz}$. Due to the reduction in flight times, the UAV could not fly to the optimal location to achieve the maximum capacity. Therefore the average capacity with 100-time slots is 3.91 $\mathrm{bits}/\mathrm{s}/\mathrm{Hz}$ for PPO. Furthermore, we calculated the average capacity of the DF relay trajectories under zero misalignment pointing error in the 150 and 100-time slots, which are 2.64 $\mathrm{bits}/\mathrm{s}/\mathrm{Hz}$ and 2.52 $\mathrm{bits}/\mathrm{s}/\mathrm{Hz}$, respectively. In

Table 3. Comparison of 150 and 100 flight times without fog.

Flight Time	RIS (bits/s/Hz)	DF (bits/s/Hz)
PPO $N=150$	4.05	2.64
PPO $N=100$	3.91	2.52
DQN $N=150$	3.87	2.49
DQN $N=100$	3.77	2.46

, the comparison results demonstrate the average capacity of the proposed RIS-assisted FSO system outperforms that of the DF relay-assisted system significantly. Moreover, compared with the benchmark DQN, the proposed PPO-based trajectory optimization achieves a higher transmission rate. This result verifies that PPO could avoid overestimation for efficient convergence to obtain better performance.

Figure 6 shows the trajectories when fog with radius $r_f=15\mathrm{m}$ with 150, 100, and 50 time slots flight time. The UAV is operating the flight tendency to avoid the fog, which also passes close to the zone and dives to the lowest altitude to obtain the maximum value of average capacity. Therefore, the avoidance and diving action of the PPO-based optimization brings the average capacity to 3.92 $\mathrm{bits}/\mathrm{s}/\mathrm{Hz}$ and 3.77 $\mathrm{bits}/\mathrm{s}/\mathrm{Hz}$ for 150 and 100 time slots, respectively, while DQN only achieves 3.77 $\mathrm{bits}/\mathrm{s}/\mathrm{Hz}$ and 3.64 $\mathrm{bits}/\mathrm{s}/\mathrm{Hz}$, respectively. However, the lack of flight time causes the UAV to lose the tendency to dive for 50 time slots, which only has 1.37 $\mathrm{bits}/\mathrm{s}/\mathrm{Hz}$ average capacity, and the average capacity comparison is demonstrated in

Table 4. Comparison of 150, 100, and 50 flight times with $r_f=15\mathrm{m}$ fog.

Flight Time	Average Capacity (bits/s/Hz)
PPO $N=150$	3.92
PPO $N=100$	3.77
PPO $N=50$	1.37
DQN $N=150$	3.77
DQN $N=100$	3.64
DQN $N=50$	1.17

.

Figure 7 demonstrates the comparison of the results with 150 time slots between the different fog conditions, whose radii are $15\mathrm{m}$ and $20\mathrm{m}$. Compared to DQN, PPO achieves better performance due to avoiding overestimation in Q-networks. Under the same total flight time, all of the trajectories show the tendency to avoid to entering the fog even when the radius of fog is changed and also show the tendency to decrease to the lowest altitude limitation.

Table 5. Comparison for different fog conditions under 150 time slots.

Fog Radii (m)	Average Capacity (bits/s/Hz)
PPO No fog	4.04
PPO $r_f=15$	3.92
PPO $r_f=20$	3.19
DQN No fog	3.87
DQN $r_f=15$	3.77
DQN $r_f=20$	2.97

shows the comparison of average capacity among the trajectories when different fog conditions are assumed. We can see that due to avoiding the fog, the UAV moves far from the area with maximum average capacity, the trajectory with the highest average capacity is reached when the fog does not exist, and the fog with the largest radius causes the lowest average capacity of trajectory.

Figure 8 exhibits the trajectories comparison with altered flight time as 100 and 60 in the presence of fog with $r_f=15\mathrm{m}$ located at $(0,20,65)\mathrm{m}$. The starting and ending points of the UAV are also adjusted to (−$20,30,80)\mathrm{m}$ and $(20,30,80)\mathrm{m}$, respectively. As shown in

Table 6. Comparison of 100 and 60 flight times without fog.

Flight Time	Average Capacity (bits/s/Hz)
PPO $N=100$	3.91
PPO $N=60$	3.09
DQN $N=100$	3.48
DQN $N=60$	3.02

, the PPO-based optimization method maintains its ability to make UAV trajectory operating avoiding and diving, yielding the average capacity of 3.91 $\mathrm{bits}/\mathrm{s}/\mathrm{Hz}$ and 3.09 $\mathrm{bits}/\mathrm{s}/\mathrm{Hz}$ for 100 and 60 time slots, respectively. On the other hand, the DQN strategy, while effective, demonstrates a lower performance with average capacities of 3.48 $\mathrm{bits}/\mathrm{s}/\mathrm{Hz}$ and 3.02 $\mathrm{bits}/\mathrm{s}/\mathrm{Hz}$ for 100 and 60 time slots, respectively. The results underline the efficiency of the proposed PPO-based trajectory optimization, validating its ability to better avoid overestimation for improved performance.

As shown in Figure 9 and detailed in

Table 7. Comparison of 150 and 100 flight times with irregular fog.

Flight Time	Average Capacity (bits/s/Hz)
PPO $N=150$	4.31
PPO $N=100$	3.73
DQN $N=150$	3.61
DQN $N=100$	3.51

, the PPO optimization method still outperforms DQN under the conditions of irregular fog, achieving an average capacity of 4.31 $\mathrm{bits}/\mathrm{s}/\mathrm{Hz}$ and 3.73 $\mathrm{bits}/\mathrm{s}/\mathrm{Hz}$ for 150 and 100 flight time respectively. As comparison, the DQN method still achieves lower average capacities, reaching only 3.61 $\mathrm{bits}/\mathrm{s}/\mathrm{Hz}$ and 3.51 $\mathrm{bits}/\mathrm{s}/\mathrm{Hz}$ for the same flight time. These results reaffirm the superiority of the PPO method in efficiently learning and converging to an optimal UAV flight strategy. Moreover, these findings underscore the fact that a larger number of flight times (150 compared to 100) provides the UAV with more options for path selection, thereby leading to better performance in both PPO and DQN cases.

For all results, it is noticed that under the same conditions, if the UAV’s flight extends and the average capacity increases due to the emergence of a more extensive scope for the UAV to tactically maneuver to the positions optimal for mitigating the pointing error loss. Within these specific air spaces, given the spatial 3D space relationships amongst the LS, UAV, and PD, the laser’s beam footprint on the PD tends to approximate a more circular shape, thereby elevating the upper constraint of the pointing error loss coefficient. Thus, when the UAV is close to these optimal spots during its flight, there’s a marked enhancement in the FSO link’s average capacity.

6. Conclusions

In this paper, we proposed an RIS-equipped UAV deployed in a non-LOS FSO communication scenario. We considered a practical scenario with atmospheric and pointing error loss. Our method combined the use of PSO for RIS phase shift optimization with PPO for UAV trajectory. Specifically, with PSO method, the pointing error loss was efficiently addressed by optimizing the phase shifts of RIS. This assures the FSO communication quality for all UAV flight times. Then, the UAV trajectory with optimized average capacity under determinate flight times was found. The PPO optimization was proposed because of its stable training compared to the other optimization learning methods as it limits the policy update at each step, reducing the possibility of policy divergence. Our results showed that our approach was effective and performed better than other optimization learning methods we looked into. However, there are more aspects and future work that can be explored. The PSO and PPO optimization methods have the potential to look into scenarios with multiple UAVs or multiple links, while other communication impacts could be considered as well. There is also potential to explore other algorithms or combine methods for even better results.