Energy-Efficient Device-to-Device Communications for Green Internet of Things Using Unmanned Aerial Vehicle-Mounted Intelligent Reflecting Surface

Abstract

The Internet of Things (IoT) serves as a crucial element in interconnecting diverse devices within the realm of smart technology. However, the energy consumption of IoT technology has become a notable challenge and an area of interest for researchers. With the aim of achieving an IoT with low power consumption, green IoT has been introduced. The use of unmanned aerial vehicles (UAVs) represents a highly innovative approach for creating a sustainable green IoT network. UAVs offer advantages in terms of flexibility, mobility, and cost. Moreover, device-to-device (D2D) communication is essential in emergency communications, due to its ability to support direct communication between devices. The intelligent reflecting surface (IRS) is also a hopeful technology which reconstructs the radio propagation environment and provides a possible solution to reduce co-channel interference resulting from spectrum sharing for D2D communications. The investigation in this paper hence focuses on energy-efficient UAV-IRS-assisted D2D communications for green IoT. In particular, a problem of optimization aimed at maximizing the system’s average energy efficiency (EE) is formulated, firstly, by simultaneously optimizing the power coefficients of all D2D transmitters, the UAV’s trajectory, and the base station (BS)’s active beamforming, along with the IRS’s phase shifts. Second, to address the problem, we develop a multi-agent twin delayed deep deterministic policy gradient (MATD3)-based scheme to find a near-optimal solution, where D2D transmitters, the BS, and the UAV cooperatively learn to improve EE and suppress the interference. To conclude, numerical simulations are conducted to assess the availability of the proposed scheme, and the simulation results demonstrate that the proposed scheme surpasses the baseline approaches in both convergence speed and EE performance.

1. Introduction

The Internet of Things (IoT) as an important information network has been widely used in many areas of everyday life such as smart home, smart transportation, smart city, etc. [1,2]. Nevertheless, there are a vast quantity of IoT devices worldwide contributing to a significant increase in energy usage, placing additional strain on the global electric grid and exacerbating environmental changes. Efforts have been made to extensively explore technical solutions that promote efficient consumption in order to achieve the objective of green communications and enhance the utilization efficiency of transmission energy [3]. In recent years, with the increase in IoT devices, device-to-device (D2D) technology has been developed as an emerging technology to enhance spectrum efficiency (SE) and address the growing lack of spectrum resource scarcity [4]. The D2D technology can be widely used in various scenarios, including IoT [5], emergency communications [6], and social networks [7]. Recent research has shown that D2D technology has remarkable advantages in improving SE and enhancing the user experience [8].

Green IoT, a novel paradigm emphasizing energy efficiency (EE) to address the environmental impact of IoT technologies for achieving sustainable and environmentally friendly development, is regarded as a prospective research avenue in the IoT domain [9]. However, as a crucial technology in IoT, the constrained onboard battery capacity hinders the realization of the full potential of D2D communication. Enhancing the system EE of D2D communication through effective resource management remains an ongoing research focus. Furthermore, in some scenarios with long distances or numerous obstacles, the communication between D2D devices can be severely affected. For instance, the authors of [10] focused on the unmanned aerial vehicle (UAV) location and resource allocation in UAV-assisted D2D communication for industrial IoT, and confirmed that the proposed decomposition-based algorithm can improve the system EE when compared with other benchmarks. The use of intelligent reflecting surfaces (IRSs) offers a promising option to address this issue, which can eliminate interference between D2D pairs and thus enhance EE by reconstructing the wireless communication environment [11,12,13]. Specifically, the IRS consists of multiple electromagnetic reflecting elements that are passive in nature. By altering the amplitude and phase of the incoming signal, it achieves a beamforming gain through reflection [14]. The deployment of IRS on urban building surfaces presents a significant improvement opportunity for guaranteeing quality of service in base station (BS) coverage blind spots. This improvement is achieved by establishing reflective line-of-sight (LoS) links. Furthermore, the IRS-assisted system offers several advantages over traditional relay systems, including low cost and low power consumption [15]. Moreover, D2D communications in cellular networks is subject to mutual interference from cellular users and other D2D devices. Using IRS to effectively regulate each reflection element’s amplitude or phase shift coefficient, and resource allocation of the D2D communication network can effectively mitigate the interference [16]. Unlike traditional approaches like alternating optimization, deep reinforcement learning (DRL) does not require prior knowledge and exhibits lower complexity and better performance. Recently, some research has already applied it to IRS-assisted wireless communication systems. For example, in [17], the authors introduce the single-agent deep deterministic policy gradient (DDPG) to optimize the passive beamforming of IRS in MISO systems and the authors of [18] extend this method to IRS-assisted MIMO systems. In [19], a multi-agent reinforcement learning (MARL)-based scheme was proposed for the joint design of passive beamforming and power control. Furthermore, to address the optimization problem of the mixed action space in IRS-assisted D2D communication networks, the authors of [20] proposed a novel multi-agent multi-pass deep Q-networks algorithm using centralized training and a decentralized execution (CTDE) framework.

Additionally, considering the UAV’s high mobility, it can act as a relay and be deployed at high altitudes to provide LoS links between ground terminals. The advantages of both IRS and UAV in improving communications and networks have been demonstrated [21]; many works have started to investigate the potentials of IRS mounted on a UAV, termed aerial UAV-IRS [22,23,24,25,26]. Because of the high mobility of the UAV, UAV-assisted communication systems often exhibited time-varying and dynamic characteristics, thereby makes the trajectory optimization and resource allocation for such systems intricate. The rapid advancement in machine learning has brought reinforcement learning to the forefront as a promising solution for tackling these challenges. Consequently, several studies have been dedicated to exploring DRL in UAV-IRS-assisted communication systems. The authors of [23] employed DDPG and double deep Q-learning network (DDQN) algorithms to tackle the challenge of trajectories and phase shift optimization in an IRS-assisted UAV network, and the numerical results demonstrated that DDPG-based continuous control achieves a better performance. The researchers in [24] introduced an innovative SWIPT method that involved optimizing resource allocation and a twin delayed DDPG (TD3)-based algorithm was used to obtain the solution to the problem. The authors of [25] considered the outdated channel state information (CSI) and developed a novel twin-agent DDPG algorithm to optimize radio resource allocation. However, the schemes based on single-agent DRL left out of consideration the interaction between multiple communicating nodes, which leads to poor performance in cooperative tasks such as D2D communication and vehicle communication. In addition, single-agent DRL requires a centralized data center to collect the status information of all agents and carries out centralized training, which results in a significant communication overhead. In contrast, a localized observation-based MARL algorithm has been proposed for communication systems [26,27]. It was demonstrated in [28] that the MARL algorithm can achieve better performance and robustness in UAV-IRS-assisted systems compared to single-agent reinforcement learning.

Furthermore, there are few works that focus on the system EE maximization for UAV-IRS-assisted D2D networks in the downlink scenario. For example, Ref. [29] considered a multiple UAV-IRS-enhanced anti-jamming D2D communication network and maximized the achievable sum rate by optimizing the IRS mode selection and phase shift, where each UAV is equipped with an IRS as an aerial relay to forward signals from multiple D2D users. In [30], the authors considered a scenario in which a UAV was equipped with an active IRS-assisted terahertz band D2D communication; the maximum system sum rate was achieved through reasonable power control and beamforming. In [31], the author proposed a distributed deep learning algorithm for realizing power control and channel selection. However, the previous studies did not take into account the impact caused by the movement of the UAV. Additionally, in [32], the researchers investigated a D2D communication system in the uplink scenario that utilized UAV-IRS assistance; they further employed a DDQN-based algorithm that optimized both the UAV’s flight trajectory and the IRS’s phase shift. But, in practical applications, D2D communication is usually considered as a distributed cooperative scenario. Using traditional centralized algorithms requires real-time access to global information, inevitably resulting in a significant amount of communication overhead [33]. In general, there has not been sufficient investigation into the integration of UAV-IRSs into D2D communications for the downlink scenario.

Motivated by the potential advantages and features of UAV-IRS, this paper explores energy-efficient UAV-IRS-assisted D2D systems, where the IRS is mounted on a rotary-wing UAV to serve as an aerial relay. In this paper, we aim to maximize the system’s average EE by jointly optimizing the considered system’s resource management along with the movement of the UAV-IRS. In order to obtain a near-optimal solution, we come up with a multi-agent DRL (MADRL)-based approach. The primary contributions of this paper are the following:

• We investigate the downlink of D2D communications assisted by UAV-IRS, in which the BS, the UAV-IRS, and all D2D pairs collaborate to achieve improved EE performance. Specifically, the UAV carries the IRS to establish LoS links between various communication nodes. To maximizing the system EE over time in a changing environment, we formulate an optimization problem that involves optimizing the UAV’s trajectory, the BS’s active beamforming, the IRS’s passive beamforming, and the D2D transmitters’ power control.
• To solve the proposed EE maximization problem, we use a Markov Game to model the cooperative task considering each D2D pair, BS, and UAV-IRS. Consequently, the resource allocation and trajectory optimization problem is addressed using multi-agent twin delayed DDPG (MATD3) [34]. To enhance its learning efficiency, prioritized experience replay (PER) is employed. In addition, the algorithm’s complexity is thoroughly examined.
• The availability of the proposed algorithm is validated through simulations, with numerical results demonstrating that the proposed scheme outperforms benchmark schemes in terms of convergence speed and EE performance.

The remaining part of this paper is organized as follows. The system model is presented in Section 2. The EE maximization optimization problem is formulated in Section 3. In addition, Section 4 introduces the MATD3-based algorithm. Section 5 provides the numerical results of the simulation, while Section 6 concludes the paper.

2. System Model

Figure 1 depicts our considered UAV-IRS-assisted D2D communication underlying cellular network in the downlink scenario, including an UAV-IRS, a BS, L cellular users (CUs), and K D2D pairs. Specifically, a rotary-wing UAV carries a passive IRS as an aerial relay to forward communication between D2D devices. In addition, we denote the sets of all CUs and D2D pairs as $\mathcal{L}=\{1,\cdots ,l,\cdots ,L\}$ and $\mathcal{K}=\{1,\cdots ,k,\cdots ,K\}$, respectively. Each D2D communication pair includes a transmitter with only one antenna and a receiver with only one antenna, and it is assumed that all D2D communication pairs and CUs share the same spectrum resources. Let $D_k^t$ and $D_k^r$ represent the k-th D2D transmitter and the k-th receiver, respectively. The UAV-IRS has a uniform rectangular array (URA) comprising $M=M_y\times M_z$ reflection elements, while the BS is equipped with a uniform linear array (ULA) consisting of $N_t$ antennas.

Let ${\mathit{u}}_{k,t}={[x_{k,t},y_{k,t},0]}^T$ and ${\mathit{u}}_{k,r}={[x_{k,r},y_{k,r},0]}^T$ represent the coordinates of $D_k^t$ and $D_k^r$, respectively. The BS is located at the coordinates ${\mathit{u}}_B={[x_B,y_B,0]}^T$. Furthermore, we make the assumption that the rotary-wing UAV maintains a constant altitude characterized by $H_u$. The total time period T is partitioned into N equal time slots, represented by ${\delta }_n$ for each slot. Therefore, let $\mathit{q}\left[n\right]={[x_u\left[n\right],y_u\left[n\right],H_u]}^T$ represent the UAV location information at the n-th time slot. The movement of the UAV must comply with the following restrictions:

$$\parallel \mathit{q}[n+1]-\mathit{q}\left[n\right]\parallel \le v_{max}{\delta }_n,n=0,\cdots ,N,$$

(1a)

$$x_{min}\le x_u\left[n\right]\le x_{max},y_{min}\le y_u\left[n\right]\le y_{max},n=0,\cdots ,N,$$

(1b)

$$\mathit{q}\left[0\right]={[0,0,H_u]}^T,$$

(1c)

where (1a) stands for the mobility constraints of the UAV including start point and end point, $v_{max}$ represents the drone’s maximum flying velocity, (1b) sets the constraint for the flying range of the UAV, and (1c) specifies the initial location.

In this paper, it is assumed that the presence of obstacles results in the absence of direct links between any two nodes. The channel coefficients from transmitter $D_k^t$ to the UAV-IRS, from the UAV-IRS to receiver $D_k^r$, from the BS to the UAV-IRS, and from the UAV-IRS to the CU l are denoted by ${\mathit{h}}_{k,TA}\left[n\right]\in {\mathbb{C}}^{M\times 1}$, ${\mathit{h}}_{k,AR}\left[n\right]\in {\mathbb{C}}^{M\times 1}$, ${\mathit{H}}_{BA}\left[n\right]\in {\mathbb{C}}^{M\times N_t}$, and ${\mathit{h}}_{l,AU}\left[n\right]\in {\mathbb{C}}^{M\times 1}$, respectively. It is presumed that the channels between any ground devices and the UAV-IRS are regarded as LoS links.

Considering that the UAV-IRS-D2D link and the UAV-IRS-CU link may be blocked by obstacles, the path loss between the UAV-IRS and other communication nodes can be modeled as a probabilistic LoS path model [35]. It can be described as

$$P_{LoS}\left(\theta \right)=\frac{1}{1+a_{1}exp(-b_1[\theta -a_1])},$$

(2)

where $a_1$ and $b_1$ are fixed values that vary based on the specific conditions or circumstances, and $\theta =\arctan \left(\frac{h}{d}\right)$ is the elevation angle, where h and d are the altitude intercept and the projector range between the UAV-IRS and the BS/users. Then, the path loss can be described as

$$PL=({\eta }_{LoS}{P}_{LoS}\left(\theta \right)+P_{NLoS}\left(\theta \right){\eta }_{NLoS})\times {\beta }_0{d}^{-{\alpha }_0},$$

(3)

where ${\beta }_0={\left(\frac{4\pi f}{c}\right)}^{-2}$ is the constant coefficient related to the antenna gain and frequency.

$\Theta =\mathrm{diag}{\left[{\beta }_1{e}^{j{\theta }_1}\cdots {\beta }_M{e}^{j{\theta }_M}\right]}^H$ represents the passive beamforming of the IRS, in which ${\beta }_m,{\theta }_m,\forall m\in \{1,2,\cdots M\}$ are the amplitude and phase shift coefficients of the m-th reflection element, respectively. In this paper, the primary focus of the optimization adjustment lies in the phase shift. Therefore, the amplitude coefficient is fixed to a value of one, i.e., ${\beta }_m=1$. The reflected interference channel from other transmitters can be represented as $h_{j,k,TA}\in {\mathbb{C}}^{M\times 1},j\ne k$. Subsequently, the received signal at receiver $D_k^r$ is

$$\begin{array}{cc}\hfill y_{D,k}\left[n\right]& =\underset{\mathrm{The}\mathrm{desired}\mathrm{signal}}{\underbrace{{\mathit{h}}_{k,AR}^H\left[n\right]\Theta \left[n\right]{\mathit{h}}_{k,TA}\left[n\right]\sqrt{p_k\left[n\right]}{s}_k}}+n_d\hfill \\ & +\underset{\mathrm{Interference}\mathrm{from}\mathrm{other}\mathrm{D}2\mathrm{D}\mathrm{transmitters}}{\underbrace{\sum _{j\ne k}^K{\mathit{h}}_{k,AR}^H\left[n\right]\Theta \left[n\right]{\mathit{h}}_{j,TA}\left[n\right]\sqrt{p_j\left[n\right]}{s}_j}}\hfill \\ & +\underset{\mathrm{Interference}\mathrm{from}\mathrm{BS}}{\underbrace{\sum _{l=1}^L({\mathit{h}}_{l,AU}^H\left[n\right]\Theta \left[n\right]{\mathit{H}}_{BA}\left[n\right]){\mathit{\omega }}_l\left[n\right]s_l}},\hfill \end{array}$$

(4)

where ${\mathit{\omega }}_l\left[n\right]\in {\mathbb{C}}^{N_t\times 1},\forall l\in \mathcal{L}$, and $p_k$, $s_k$ represent the active beamforming vector at the BS, the power coefficient of the transmitter $D_k^t$, and the transmit data from the $D_k^t$ to $D_k^r$, respectively. $n_d\sim \mathcal{N}(0,{\sigma }_d^2)$ denotes the AWGN noise. The corresponding SINR at $D_k^r$ is

$$\begin{array}{cc}\hfill {\gamma }_{D,k}\left[n\right]& =\frac{{\left|({\mathit{h}}_{k,AR}^H\left[n\right]\Theta \left[n\right]{\mathit{h}}_{k,TA}\left[n\right])\right|}^2{p}_k\left[n\right]}{{\sum }_{j\ne k}^K{\left|({\mathit{h}}_{k,AR}^H\left[n\right]\Theta \left[n\right]h_{j,TA}\left[n\right])\right|}^2{p}_j\left[n\right]+I_{\mathrm{C}}}\hfill \end{array},$$

(5)

where $I_{\mathrm{C}}={\sum }_{l=1}^L{|{\mathit{h}}_{l,AU}^H\left[n\right]\Theta \left[n\right]{\mathit{H}}_{BA}\left[n\right]{\mathit{\omega }}_l\left[n\right]|}^2+{\sigma }_d^2$.

In addition, the signal received at $\mathrm{CU}l$ can be denoted as

$$\begin{array}{cc}\hfill y_{C,l}\left[n\right]& =\underset{\mathrm{The}\mathrm{desired}\mathrm{signal}}{\underbrace{{\mathit{h}}_{l,AU}^H\left[n\right]\Theta \left[n\right]{\mathit{H}}_{BA}\left[n\right]{\mathit{\omega }}_l\left[n\right]s_l}}+n_0\hfill \\ & +\underset{\mathrm{Intereference}\mathrm{from}\mathrm{D}2\mathrm{D}\mathrm{transmitters}}{\underbrace{\sum _{k=1}^K{\mathit{h}}_{l,AR}^H\left[n\right]\Theta \left[n\right]{\mathit{h}}_{k,TA}\left[n\right]\sqrt{p_k\left[n\right]}{s}_k}}\hfill \\ & +\underset{\mathrm{Intereference}\mathrm{from}\mathrm{other}\mathrm{CUs}}{\underbrace{\sum _{j\ne l}^L({\mathit{h}}_{l,AU}^H\left[n\right]\Theta \left[n\right]{\mathit{H}}_{BA}\left[n\right]){\mathit{\omega }}_j\left[n\right]s_j}},\hfill \end{array}$$

(6)

where $n_0\sim \mathcal{N}(0,{\sigma }_0^2)$ denotes the AWGN noise at $\mathrm{CU}l$. Accordingly, the corresponding SINR at $\mathrm{CU}l$ is

$$\begin{array}{c}\hfill {\gamma }_{C,l}\left[n\right]=\frac{{\left|({\mathit{h}}_{l,AU}^H\left[n\right]\Theta \left[n\right]{\mathit{H}}_{BA}\left[n\right]){\mathit{\omega }}_l\left[n\right]\right|}^2}{{\sum }_{k=1}^K{\left|({\mathit{h}}_{l,AR}^H\left[n\right]\Theta \left[n\right]{\mathit{h}}_{k,TA}\left[n\right])\right|}^2{p}_k\left[n\right]+I_{\mathrm{C}l}}\end{array},$$

(7)

where $I_{\mathrm{C}l}={\sum }_{j\ne l}^L{\left|({\mathit{h}}_{l,AU}^H\left[n\right]\Theta \left[n\right]{\mathit{H}}_{BA}\left[n\right]){\mathit{\omega }}_j\left[n\right]\right|}^2+{\sigma }_0^2$. Thus, the achievable rate of receiver $D_k^r$ and $\mathrm{CU}l$ in the n-th time slot can be described as

$$R_{D,k}\left[n\right]={log}_2(1+{\gamma }_{D,k}\left[n\right]),$$

(8a)

$$R_{C,l}\left[n\right]={log}_2(1+{\gamma }_{C,l}\left[n\right]),$$

(8b)

respectively.

The propulsion power consumption of a rotor-craft UAV is modeled as [36]

$$\begin{array}{cc}\hfill E_{\mathrm{UAV}}\left[n\right]& ={\delta }_n[P_0\left(1+\frac{3{\left(V\left[n\right]\right)}^2}{U_{\mathrm{tip}}^2}\right)+\frac{1}{2}{d}_0\rho sA{\left(V\left[n\right]\right)}^3\hfill \\ & +P_i{\left(\sqrt{1+\frac{{\left(V\left[n\right]\right)}^4}{4v_0^4}}-\frac{{\left(V\left[n\right]\right)}^2}{2v_0^2}\right)}^{\frac{1}{2}}],\hfill \end{array}$$

(9)

where $V\left[n\right]=\frac{\sqrt{{\parallel \mathit{q}\left[n\right]-\mathit{q}[n-1]\parallel }^2}}{{\delta }_n}$ stands for the UAV’s instantaneous speed in the n-th time slot; $P_0$ and $P_i$ are the constant blade profile power and induced power, respectively; $U_{tip}$ and $v_0$ are the constant blade tip speed and mean rotor-induced velocity during hover, respectively; $\rho$, s, and A are the air density, rotor solidity, and rotor disc area, respectively; and $d_0$ stands for the fuselage drag ratio.

Considering the energy expenditure of the system circuit and IRS elements, the system EE is expressed as

$$\begin{array}{cc}\hfill EE\left[n\right]=& \frac{{\sum }_{k=1}^K{R}_{D,k}\left[n\right]+{\sum }_{l=1}^L{R}_{C,l}\left[n\right]}{{\sum }_{l=1}^L{\parallel {\mathit{\omega }}_l\left[n\right]\parallel }^2+{\sum }_{k=1}^K{p}_k\left[n\right]+E_{\mathrm{CR}}\left[n\right]}\hfill \end{array},$$

(10)

where $E_{\mathrm{CR}}\left[n\right]=E_{\mathrm{CIR}}+E_{\mathrm{IRS}}+E_{\mathrm{UAV}}\left[n\right]$, with $E_{\mathrm{CIR}}$ and $E_{\mathrm{IRS}}$ being the power consumption of the circuits and the IRS, respectively.

3. Problem Formulation

The aim of this paper is to enhance the average system EE by simultaneously optimizing the flight path of the UAV, $Q=\left\{\mathit{q}\right[n],\forall n\}$, the passive beamforming matrix at the IRS, $\Theta =\{\Theta [n],\forall n\}$, the power allocation coefficients for D2D transmitters, $P=\{p_k\left[n\right],\forall k,n\}$, and the BS’s active beamforming matrix, $\Omega =\{{\mathit{\omega }}_l\left[n\right],\forall l,n\}$. The optimization problem can be represented as

$$\begin{array}{cc}\hfill P1:\underset{Q,P,\Omega }{max}& \frac{1}{N}\sum _{n=1}^{N}EE\left[n\right],\hfill \end{array}$$

(11a)

$$\begin{array}{cc}\hfill s.t.& \left(1a\right),\left(1b\right),\left(1c\right),\hfill \end{array}$$

(11b)

$$\begin{array}{cc}\hfill & \sum _{l=1}^L{\parallel {\mathit{\omega }}_l\left[n\right]\parallel }^2\le P_{\mathrm{BS}},\hfill \end{array}$$

(11c)

$$\begin{array}{cc}\hfill & p_k\left[n\right]\le P_{\mathrm{D}2\mathrm{D}},\forall k,\hfill \end{array}$$

(11d)

$$\begin{array}{cc}\hfill & {\theta }_m\in (0,2\pi ],{\beta }_m=1,\forall m\hfill \end{array}$$

(11e)

$$\begin{array}{cc}\hfill & R_{D,k}\left[n\right]\ge R_{th,k},\forall k,\hfill \end{array}$$

(11f)

$$\begin{array}{cc}\hfill & R_{C,l}\left[n\right]\ge R_{th,l},\forall l,\hfill \end{array}$$

(11g)

where (11b) is the UAV’s movement constraint; (11c) and (11d) denote the power constraints at D2D transmitters and the BS, with $P_{D2D}$ and $P_{BS}$ being the maximal transmit power of each transmitter and the BS, respectively; constraints (11f) and (11g) are the QoS constraints of each D2D pair and CU, respectively. The non-convex constraints (11b) and (11e) make the problem intractable to solve. To tackle this challenge, the following section will utilize a MADRL algorithm.

4. The Proposed Solution

In this section, we begin by employing the Markov Game framework to model the optimization problem $P1$. Then, we will introduce the various elements of the multi-agent environment. Since a fully decentralized MARL-based algorithm faces the problem of a non-stationary environment and is difficult to converge, a MATD3 approach based on the CTDE framework is adopted.

4.1. Markov Game Formulation

Since the transmitter in each D2D communication pair cannot directly communicate with other transmitters, the formulated problem $P1$ can be regarded as a Markov Game. In this setting, each communicating node serves as an agent and aims to optimize the long-term cumulative reward by utilizing observations and selecting actions based on its individual policy. Given the non-stationary nature of the environment, it is necessary for all agents to work cooperatively in order to maximize the shared reward. To ensure each communication node works cooperatively, the UAV-IRS, the BS, and each transmitter are considered as agents. Hence, there are $K+2$ agents in the system. Let $I_k$, $I_b$, and $I_u$ represent the agents of each transmitter, the BS, and the UAV-IRS, respectively. Thus, the set of all agents can be defined as $I=\{I_1,\cdots ,I_K,I_b,I_u\}$. The Markov Game for the considered UAV-IRS-assisted D2D communication scenario can be viewed as a tuple $({\left\{o_i\right\}}_{i\in I},{\left\{a_i\right\}}_{i\in I},\mathcal{P},r,\gamma )$, where the set of the observation space and action space of $K+2$ agents are denoted as $O=\left\{o_1,\cdots ,o_K,o_b,o_u\right\}$ and $A=\left\{a_1,\cdots ,a_K,a_b,a_u\right\}$, respectively. $\mathcal{P}$ refers to the probability of all agents performing actions by exploiting the current state and transitioning to the subsequent state, r is the reward function, and $\gamma$ is the reward discount factor. In a Markov Game, each agent aim to maximize its own total expected reward $R_i={\sum }_{n=0}^{n=N}\gamma r_i,0<\gamma \le 1$. In order to solve the problem of non-stationarity in a multi-agent environment, it is assumed that the policies of all other agents are known. The specific design of observations, actions, and rewards are as follows.

4.1.1. Observation

The observations of $I_k$, $I_b$, and $I_u$ are denoted as $o_k$, $o_b$, and $o_u$, respectively. Since each D2D transmitter only knows its local observations and partial interference information, to simplify the analysis process, it is assume that the CSI can be obtained by adopting the channel estimation method that is used in [37,38]. The observation $o_k$ contains the CSI between $D_k^t$ and $D_k^r$, and the interference information from other D2D transmitters and the BS at the $(n-1)$-th time slot, which can be represented as

$$\begin{array}{c}\hfill o_k\left[n\right]=\{{\mathit{h}}_{k,AR}^H[n-1]\Theta [n-1]{\mathit{h}}_{k,TA}[n-1],\\ \hfill {\mathit{h}}_{k,AR}^H[n-1]\Theta [n-1]{\mathit{h}}_{j,TA}[n-1],\\ \hfill {\mathit{h}}_{l,AU}^H[n-1]\Theta [n-1]{\mathit{H}}_{BA}[n-1]\},\forall k\in \mathcal{K}.\end{array}$$

(12)

Similarly, the observation of the BS contains the CSI between the BS and CUs, and the interference channel information from D2D transmitters and the BS at the $(n-1)$-th time slot, and can be expressed as

$$\begin{array}{c}\hfill o_b\left[n\right]=\{{\mathit{h}}_{l,AU}^H[n-1]\Theta [n-1]{\mathit{H}}_{BA}[n-1],\\ \hfill {\mathit{h}}_{l,AR}^H[n-1]\Theta [n-1]{\mathit{h}}_{k,TA}[n-1]\},\forall l\in \mathcal{L}.\end{array}$$

(13)

Additionally, the observation $o_u$ contains the UAV trajectory and the CSI from the UAV-IRS to other devices at the $n-1$th time slot, and is given as

$$\begin{array}{cc}\hfill o_u\left[n\right]=& \{q[n-1],{\mathit{h}}_{k,AR}^H[n-1]\Theta [n-1]{\mathit{h}}_{k,TA}[n-1],\hfill \\ & {\mathit{h}}_{l,AU}^H[n-1]\Theta [n-1]{\mathit{H}}_{BA}[n-1],\hfill \\ \hfill & {\mathit{h}}_{l,AR}^H[n-1]\Theta [n-1]{\mathit{h}}_{k,TA}[n-1]\}.\hfill \end{array}$$

(14)

4.1.2. Action Space

For agent $I_k$, the action $a_k\left[n\right]=\left\{p_k\left[n\right]\right\},\forall k\in \mathcal{K}$ includes the power allocation coefficient of the k-th transmitter. For agent $I_b$, the action $a_b\left[n\right]=\{{\mathit{\omega }}_1\left[n\right],\dots ,{\mathit{\omega }}_l\left[n\right],\dots ,{\mathit{\omega }}_L\left[n\right]\}$ includes the active beamforming vector for all CUs. For agent $I_u$, the action $a_u\left[n\right]=\{\Theta \left[n\right],q\left[n\right]\}$ contains the passive phase shift matrix and the UAV’s trajectory.

4.1.3. Reward Function

Considering our objective of improving the average EE, the reward function can be formulated as follows:

$$\begin{array}{cc}\hfill r\left(a\right[n],s[n\left]\right)& =EE\left[n\right]+{\eta }_{r,k}\sum _{k=1}^K{p}_{r,k}\left[n\right]+{\eta }_{r,l}\sum _{l=1}^L{p}_{r,l}\left[n\right]\hfill \\ \hfill +& {\eta }_b{p}_b\left[n\right]+{\eta }_g\sum _{k=1}^K{p}_{g,k}\left[n\right]+{\eta }_u{p}_u\left[n\right],\hfill \end{array}$$

(15)

where $p_u,p_{g,k},p_b,p_{r,k},p_{r,l}$ are the penalties when the constraints are not satisfied. Let $p_{r,k}={[R_{th,k}-R_{D,k}\left[n\right]]}^{+}$ and $p_{r,l}={[R_{th,l}-R_{D,l}\left[n\right]]}^{+}$ denote the QoS constraint penalty; Let $p_{g,k}={[P_{BS}-{\sum }_{l=1}^L{\parallel {\mathit{\omega }}_k\left[n\right]\parallel }^2]}^{+}$ and $p_b={[P_{D2D}-p_k\left[n\right]]}^{+}$ denote the maximum transmit power constraint penalty, respectively, in which ${\left[x\right]}^{+}=\max \{0,x\}$; the UAV’s trajectory constraint penalty is defined as $p_u$. The non-negative constants ${\eta }_{r,k},{\eta }_{r,l},{\eta }_b,{\eta }_g,{\eta }_u$ represent the weight coefficients used to balance the different penalty functions.

4.2. MATD3 Approach

MATD3 is a multi-agent extended version of the single-agent TD3, which adopts the mode of the CTDE framework in the training process. As demonstrated in Figure 2, each agent first obtains the local observation $o_i\left[n\right]$ and feeds it into the actor network. Then, each agent obtains the action $a_i\left[n\right]$ to execute according to its own policy ${\pi }_i\left[n\right]$ at the time slot n, and interacts with the environment to obtain a new observation $o_i[n+1]$ and store $(o_i\left[n\right],a_i\left[n\right],r\left[n\right],o_i[n+1])$ in the experience buffer pool D. Subsequently, the critic network of each agent incorporates the global state, which encompasses the observations and actions of all other agents.

Unlike multi-agent DDPG (MADDPG), MATD3 integrates two techniques: clipped double Q-learning (CDQ) and target policy smoothing. In the MATD3 algorithm, the training of the critic network is performed centrally, and input to the critic network includes both the observed states and the actions taken by other agents. The centralized training process assumes training taking place in the UAV-IRS, during which all communication nodes upload channel state information while forwarding data through UAV-IRS. The CDQ-learning technique is utilized to mitigate the issue of Q-value overestimation. Specifically, each agent comprises an evaluated actor network, two evaluated critic networks, a target actor network, and two target critic networks. Each evaluated actor network outputs action $a_i\left[n\right]={\pi }_i\left(o_i\left[n\right]\right|{\theta }_i^{\mu })+n_i\left[n\right]$ with the local observation $o_i\left[n\right]$ and the network parameter ${\theta }_i^{\mu }$. The evaluated critic networks output the Q values $Q_{i,1}(o_i\left[n\right],a_i\left[n\right]|{\theta }_{i,1}^q)$ and $Q_{i,2}(o_i\left[n\right],a_i\left[n\right]|{\theta }_{i,1}^q)$ to evaluate the action of the actor network with the local observation $o_i\left[n\right]$ and the action $a_i\left[n\right]$. Specifically, the centralized Q value is the minimum value between $Q_{i,1}$ and $Q_{i,2}$. During the training phase, a mini-batch of data is utilized to update the critic network parameter ${\theta }_i^q$ by minimizing the temporal difference (TD) error. Differently from the single-agent TD3, the critic input of each agent on MATD3 has additional information, such as the actions and observations of other agents, in addition to its own state–action information. TD3 adopts the target network approach to fix the Q-network in the TD target.

The target value y can be computed through

$$\begin{array}{c}\hfill y=r_i+\gamma \min \{Q_{i,1}(o^{\prime },a_1^{\prime },\cdots ,a_I^{\prime }|{\theta }_{i,1}^{q\prime }),\\ \hfill Q_{i,2}(o^{\prime },a_1^{\prime },\cdots ,a_I^{\prime }|{\theta }_{i,2}^{q\prime })\}{\mid }_{{\mathit{a}}^{\mathbf{\prime }}=\{{\pi }_i\left(o_i^{\prime }\right|{\theta }_i^{\mu \prime })+ϵ,i\in \mathcal{I}\}}\end{array},$$

(16)

where $ϵ=clip\left(\mathcal{N}\right(0,\sigma ),-c,c)$ is the clipped Gaussian noise and c is a variable parameter.

The evaluated critic network is updated through minimizing the loss function:

$$minL\left({\theta }_{i,j}^q\right)={\mathbb{E}}_{\mathit{o},a,r,{\mathit{o}}^{\prime }}\left[{(Q_i(\mathit{o},a_1,\cdots ,a_I|{\theta }_{i,j}^q)-y)}^2\right],i\in \mathcal{I},j\in \{1,2\}.$$

(17)

The evaluated actor network of each agent is updated through gradient descent:

$$\begin{array}{cc}\hfill {\nabla }_{{\theta }_i^{\mu }}{J}_i\left({\pi }_i\right)& ={\mathbb{E}}_{o,a\sim D}[\nabla {\theta }_i^{\mu }{\pi }_i\left(o_i|{\theta }_i^{\mu }\right)\times \hfill \\ & {\nabla }_{a_i}{Q}_i\left(\mathit{o},\mathit{a}|{\theta }_{i,1}^q\right){|}_{\mathit{a}=\{{\pi }_i\left(o_i\right|{\theta }_i^{\mu }),i\in \mathcal{I}\}}].\hfill \end{array}$$

(18)

The target network parameters ${\theta }_{i,1}^{q\prime }$, ${\theta }_{i,2}^{q\prime }$ and ${\theta }_i^{\mu \prime }$ are updated by

$$\begin{array}{c}{\theta }_{i,1}^{q^{\prime }}\leftarrow {\theta }_{i,1}^q+(1-\tau ){\theta }_{i,1}^{q^{\prime }},\\ {\theta }_{i,2}^{q^{\prime }}\leftarrow {\theta }_{i,2}^q+(1-\tau ){\theta }_{i,2}^{q^{\prime }},\\ {\theta }_i^{{\mu }^{\prime }}\leftarrow {\theta }_i^{\mu }+(1-\tau ){\theta }_i^{{\mu }^{\prime }}.\end{array}$$

(19)

4.3. Prioritized Experience Replay

The technique of classical experience replay involves randomly selecting samples from the experience replay buffer in a uniform manner. This is carried out to reduce the correlation between the samples. However, the importance of experience is ignored. In the case of sparse rewards, agents receive a reward only after executing multiple correct actions, resulting in limited transitions to encourage proper learning. Therefore, using random sampling of experiences in this scenario can result in reduced learning efficiency. Prioritized experience replay is adopted to solve such problems [39], which is an improved method of the experience replay buffer. It determines the order in which experience samples are extracted by introducing a priority and replay probability. Based on priority, high-priority experience samples are more likely to be extracted and used for training the agent.

In this paper, proportional prioritization is adopted to restore the priority of a transition, which can be expressed as follows:

$$\begin{array}{c}\hfill p_i^t=\left|{\delta }_i^t\right|+ϵ,\end{array}$$

(20)

where ${\delta }_i$ is the TD error and $ϵ$ is a small fixed value to avoid the probability of zero. The TD error represents the difference between the current Q value and the Q value that should be pursued in the next step, and higher TD errors will be assigned a higher priority. Thus, the sampling probability can be defined as follows:

$$\begin{array}{c}\hfill P_i^t=\frac{{\left(p_i^t\right)}^{\alpha }}{{\sum }_k{\left(p_i^k\right)}^{\alpha }},\end{array}$$

(21)

where $\alpha$ is the hyperparameter that regulates the degree of priority.

Due to the introduction of bias through PER, which alters the data distribution, it becomes necessary to employ importance sampling to mitigate the impact. The weights for importance sampling can be defined as follows:

$$\begin{array}{c}\hfill w_i^t={\left(\frac{1}{N_0}·\frac{1}{P_i^t}\right)}^{\beta },\end{array}$$

(22)

where $\beta$ is the hyperparameter that regulates the degree of bias introduced by PER and $N_0$ corresponds to the number of existing transitions in the experience replay pool. Consequently, Equation (16) can be reconstructed as the loss function:

$$\begin{array}{c}\hfill L\left({\theta }_i^q\right)={\mathbb{E}}_{\mathit{o},a,r,{\mathit{o}}^{\prime }}\left[{\mathit{\omega }}_i{(Q_i(\mathit{o},a_1,\cdots ,a_I|{\theta }^q)-y)}^2\right].\end{array}$$

(23)

Algorithm 1 outlines the training process of the optimization for resource allocation and trajectory using MATD3.

Algorithm 1: Joint resource allocation and trajectory using MATD3

4.4. Complexity Analysis

The time computation complexity depends on the network operations between two layers. Since our proposed algorithm includes two actor networks and four critic networks, its complexity is given by $O(2\times {\sum }_{l=1}^{L_a}{n}_l^a·n_{l+1}^a+4\times {\sum }_{u=1}^{L_c}{n}_u^c·n_{u+1}^c)$, in which the $n_l$-th and the $n_u$-th are the number of operations of the l-th actor and the u-th critic network layers, respectively; $L_a$ represents the number of layers in the actor network, while $L_c$ represents the number of layers in the critic network. For the online execution phase, the time complexity of each actor network is $O(2\times {\sum }_{l=1}^{L_a}{n}_l^a·n_{l+1}^a)$.

5. Simulation Results

In this section, we validate the efficacy of the proposed algorithm in optimizing resource allocation and designing the UAV trajectory. It is assumed that the UAV maintains a constant altitude of $H_u=25\mathrm{m}$ throughout the flight. For comparison, we compare the system EE performance under fixed position and random trajectory. The UAV-IRS’s initial position is configured as (25 m, 25 m, 25 m), while the location of the BS is fixed at (0 m, 30 m, 5 m). Moreover, the position of the UAV in the fixed position scheme is consistent with the initial position of the proposed scheme. All CUs and D2D pairs are located around the initial UAV position. The simulation platform is based on AMD Ryzen 7735H, NVIDIA GeForce RTX4060, python3.7.4 and Torch-1.12.1. Each agent’s actor and critic networks are built with three fully connected layers, comprising 512, 256, and 128 neural units, respectively. The energy consumption model parameters of the rotary-wing UAV are set based on [36]. Formal verification is crucial in IoT systems to ensure safety, security, and reliability by detecting errors, verifying complex interactions, and enhancing trust in the system’s performance, scalability, and compliance with specifications [40,41]. In order to reduce the complexity of the experiment and considering that we currently only have a single drone, we have not yet addressed the issues of safe drone operation and carried out formal verification. For the relevant parameters of the probabilistic path loss model, we set $a=9.61,n=0.16$ [35]. According to [42], the total energy consumption of the system’s circuit power and the IRS is set as $E_{\mathrm{IRS}}+E_{\mathrm{CIR}}=4w$. For the hidden layer, the Relu activation function is applied and, for the output layer of actor networks, the Tanh activation function is employed. The number of training episodes is set as 5000 and each episode has 200 time steps. In this paper, each time step is treated as a time slot. Additional simulation parameters can be found in

Table 1. Simulation parameters.

Parameters	Description	Value
${\sigma }_0$, ${\sigma }_d$	Noise power	−169 dBm
f	Carrier frequency	2.8 GHz
$P_{BS}$	The BS’s maximum transmit power	30 dBm
$P_{D2D}$	Transmitter’s maximum transmit power	23 dBm
${\alpha }_0$	Path loss exponent	2.2
K	Number of D2D pairs	2
L	Number of CUs	2
$U_{tip}$	Blade tip speed	200 m/s
$d_0$	Drag coefficient of the UAV fuselage	0.3
$\rho$	The air density	1.225 kg/m3
s	The solidity of the UAV’s rotor	0.05
A	Rotor disc area of the UAV	0.79
$P_0$	The blade profile power of the UAV	580.65
$P_i$	The induced power of the UAV	790.6715
$R_{th,l}$	Minimum rate requirements of CU l	0.25 bps/Hz
$R_{th,k}$	Minimum rate requirements of D2D k	0.25 bps/Hz
$v_{max}$	UAV’s max velocity	25 m/s
${\delta }_n$	Time slot length	0.1 s
${\eta }_{NLoS}$	Additional attenuation coefficient of LoS	−20 dB
${\eta }_{LoS}$	Additional attenuation coefficient of LoS	−1 dB
$\gamma$	Discount factor for rewards	0.95
$\tau$	Soft update rate	0.01
$\mathcal{B}$	Batch size	256
$\mathcal{N}$	Experience buffer size	50,000
$\mathcal{M}$	Maximum training steps per episode	200
$\mathcal{E}$	Maximum training episodes	5000
$\alpha$	The hyperparameter of PER	0.6
$\beta$	The hyperparameter of PER	0.4

[36,43].

Figure 3 illustrates the trajectory of the UAV after optimization. Obviously, the UAV will fly to a fixed area and fluctuate in a small range after the optimization. When it flies to the center of the map, the UAV hovers at the center of all users to enhance the average system EE while meeting the QoS constraints of all users.

Figure 4a shows the episode return variation during training with different learning rate settings. Observing the learning process under different learning rate settings, it becomes apparent that the MATD3-based algorithm gradually converges. Obviously, the learning curve tends to converge after 1000 episodes. Specifically, the proposed algorithm at the learning rate of 0.0001 converges faster and obtains a better performance than the proposed algorithm at the learning rate of 0.001 and 0.00001. The reason behind this is that the excessively large learning rate may result in excessively large weight updates, thereby causing the loss to become too small and missing the optimal solution. Furthermore, a too small learning rate also leads to poor performance, since it results in tiny updates to the parameter, making it difficult for the algorithm to effectively learn the characteristics of the environment and the reward function. Figure 4b illustrates the average EE versus episodes with the same parameter setting. This proves that the reward setting is consistent with our optimization goal. Finally, the proposed algorithm is superior when examining the converged average EE with the learning rate being set to 0.0001.

Figure 5 demonstrates the average EE versus the number of episodes of different schemes with $M=25$. The benchmark schemes include DDPG, TD3, SD3 [44], MADDPG, MATD3 with fixed UAV position, and MATD3 with fixed phase shifts. The state input of all single-agent algorithms (DDPG, TD3, SD3) is set to global state information. The EE improvement achieved by simultaneously optimizing the resource allocation and the UAV trajectory is clearly superior to that of the fixed position and the fixed phase shift schemes. The main reason is that the reflective surface in the fixed position is difficult to meet the rate constraint of all users, and another reason is that the UAV-IRS requires greater energy consumption to sustain a hover state and maintain a fixed position, compared to maintaining a flight state. It can also be observed that the MATD3-based scheme surpasses the MADDPG-based scheme and all single-agent-based schemes; the DDPG-based scheme performs the worst due to the presence of high input dimensions. Furthermore, since MATD3 introduces dual Q-value networks, it increases the complexity of the training process and then results in a slower convergence speed compared to the MADDPG-based scheme.

Furthermore, Figure 6 illustrates the correlation between the number of elements (M) in the IRS and the average EE. It can be observed that increasing M can enhance the system EE when it is small; however, the EE performance gain becomes smaller and smaller when M is very large. This is due to the increased dimensionality of the action space, which requires more time for the algorithm to achieve convergence. In the comparison, it is evident that the proposed MATD3-based approach obtains a higher EE than the other two benchmarks in the case of more IRS elements.

Figure 7 illustrates the relationship between the system’s average EE and the maximum transmit power constraint, $P_{\mathrm{D}2\mathrm{D}}$, under various schemes. Specifically, in the fixed position method, without optimizing the movement, the UAV is fixed at a fixed location and during the entire time period. In the random trajectory method, without optimizing the trajectory, the UAV moves randomly. The average EE of the proposed scheme gradually improves with an increase in the maximum transmit power budget, $P_{\mathrm{D}2\mathrm{D}}$, as illustrated in Figure 7. This improvement can be attributed to the increase in the system sum rate achieved by allocating more transmit power. It can also observed that the random trajectory scheme and fixed position scheme cannot gain improvement by increasing the transmit power. Furthermore, the figure demonstrates that, by deploying the IRS in a flexible manner, the performance of the system EE can be greatly enhanced.

6. Conclusions

This study investigates the application of UAV-IRS-assisted D2D communications to enhance system EE by jointly optimizing the UAV trajectory, D2D transmitter power coefficients, BS’s beamforming vector, and IRS passive beamforming matrix, while the rate requirements of all cellular users and D2D users are simultaneously satisfied. Considering the distributed nature of D2D communication, we use MADRL to train each communication node and propose a training framework based on the CTDE framework. To address the formulated problem, we initially formulate it as a Markov Game and subsequently introduce a solution approach utilizing MATD3. The numerical results indicate that our proposed scheme greatly improves the EE. However, the scalability of MATD3 in real-world scenarios is a potential challenge. As the number of agents increases, the complexity of the interaction among them grows exponentially. This can lead to issues such as communication overhead, increased computational resources, and difficulties in maintaining a stable training process. Another point to note is that this work does not consider the onboard energy constraints of the UAV and the channel allocation for D2D communications, which will be addressed in future works. In the current experiments, we have overlooked the complexity of the real environment. In future work, we will consider further designing path planning for security issues and conducting formal verification using real-time data.