A Power Allocation Scheme for MIMO-NOMA and D2D Vehicular Edge Computing Based on Decentralized DRL

Abstract

In vehicular edge computing (VEC), some tasks can be processed either locally or on the mobile edge computing (MEC) server at a base station (BS) or a nearby vehicle. In fact, tasks are offloaded or not, based on the status of vehicle-to-infrastructure (V2I) and vehicle-to-vehicle (V2V) communication. In this paper, device-to-device (D2D)-based V2V communication and multiple-input multiple-output and nonorthogonal multiple access (MIMO-NOMA)-based V2I communication are considered. In actual communication scenarios, the channel conditions for MIMO-NOMA-based V2I communication are uncertain, and the task arrival is random, leading to a highly complex environment for VEC systems. To solve this problem, we propose a power allocation scheme based on decentralized deep reinforcement learning (DRL). Since the action space is continuous, we employ the deep deterministic policy gradient (DDPG) algorithm to obtain the optimal policy. Extensive experiments demonstrate that our proposed approach with DRL and DDPG outperforms existing greedy strategies in terms of power consumption and reward.

1. Introduction

With the development of the Internet of vehicles (IoV), autonomous vehicles are becoming increasingly popular. At the same time, a series of smart vehicular user (SVU) devices and applications installed on autonomous vehicles have emerged. Communication among these SVU devices and applications is extremely popular [1,2,3,4,5]. Therefore, to reduce the burden of SVUs, vehicular edge computing (VEC) has been introduced to process tasks offloaded by SVUs, while ensuring low processing latency for these tasks [6,7,8]. When SVUs have tasks to process, they can choose to process these tasks locally or offload them to a mobile edge computing (MEC) server on a VEC vehicle or at a base station (BS) [9,10]. Such task offloading incurs multiple kinds of power consumption. To be clear, in this paper, the power consumed for offloading to the BS is defined as vehicle-to-infrastructure (V2I) processing power, and the power consumed for offloading to a VEC vehicle is defined as vehicle-to-vehicle (V2V) processing power. In addition, SVUs also process tasks on their local CPUs; the task processing power consumed by the local CPU of an SVU is defined as local processing power in this paper.

In this paper, V2I communication is assumed to be based on the multiple-input multiple-output and nonorthogonal multiple access (MIMO-NOMA) technology, due to high spectrum utilization and channel capacity. However, in the MIMO-NOMA system, the interference of SVUs with each other and the mobility of SVUs lead to uncertain channel conditions [11].

V2V communication is assumed to be based on device-to-device (D2D) technology, which is a core technology for smart cities, high-quality video streaming, and disaster-relief networks. It offers various advantages in terms of fairness, energy efficiency, and spectral efficiency [12,13,14,15]. In the D2D communication scenario considered in this paper, each SVU communicates only with a specific VEC vehicle, and the communication distance remains constant. Therefore, V2V communication is working on the interference-free channel conditions [16].

In a real scenario, task arrival will be random. In this case, the latency and power consumption of the SVUs for processing the tasks will also be uncertain [11]. For example, when the V2I channel conditions are relatively worse, SVUs should choose to process their tasks either locally or through V2V communication to reduce latency and power consumption. Considering the randomness of task arrival and the uncertainty of the V2I channel conditions, it is necessary to design an optimal power-allocation scheme with the aim of minimizing latency and power consumption.

This paper employs the deep reinforcement learning (DRL) framework to design such a scheme. Since the deep deterministic policy gradient (DDPG) algorithm is suitable for solving problems in the continuous action space, we will employ this characteristic of the algorithm to solve the problem. In most previous work, the BS was responsible for collecting global information, including the SVU state, and then determining the action of each SVU; however, the centralized approach incurs higher power consumption and larger latency [17,18,19,20,21,22,23,24,25,26,27,28,29,30]. Only a few works have adopted decentralized DRL frameworks to design related schemes in which each SVU observes its own surrounding environment to determine its action. In this way, it can effectively reduce the overall latency and power consumption [31,32]. However, to the best of our knowledge, the coexistence of MIMO-NOMA-based V2I communication and D2D-based V2V communication was not considered in the VEC system based on the decentralized DRL.

In this paper, we propose a power-allocation model in VEC based on decentralized DRL to improve power consumption and latency, considering the coexistence of MIMO-NOMA-based V2I communication and D2D-based V2V communication, as well as the randomness of task arrival, the channels interference in MIMO-NOMA and the mobility of SVUs. (The source code has been released on https://github.com/qiongwu86/MIMO-D2D (accessed on 19 February 2023). The main contributions of this article are summarized as follows.

(1)

We propose a power allocation model in VEC based on the decentralized DRL, defining the action function, state function, and reward function. The DDPG algorithm is employed to deal with the continuous action space problem and to guide the model to learn the optimal policy.

(2)

Performance testing of the trained model in a large number of experiments shows that the proposed approach outperforms other existing ones.

The rest of this paper is organized as follows. Section 2 reviews some related work. Section 3 describes the system model. Section 4 presents the design of the DRL framework and the related functions. Section 5 describes the DDPG algorithm used for model training. We present some simulation results to demonstrate that our proposed approach outperforms other existing ones in Section 6. Finally, Section 7 concludes the paper.

2. Related Work

In this section, we review some work on D2D and MIMO-NOMA technology in MEC and VEC based on DRL.

2.1. D2D and MIMO-NOMA Technology in MEC and VEC

Many works have considered the application of MIMO-NOMA and D2D technology in MEC and VEC systems. In [33], Ding et al. proposed a multiuser MIMO (MU-MIMO) MEC system with the goal of optimizing the system cost, power consumption and latency. In [11], Zhu et al. constructed a VEC system based on MIMO-NOMA technology, in which vehicles can choose to process tasks either locally or offload them to the BS via the MIMO-NOMA channels. In this system, the DDPG algorithm was employed to optimize latency and power consumption. In [34], Liu et al. designed a millimeter-wave D2D MEC model as the basis of an optimal algorithm for task assignment. In [35], Li et al. proposed a MEC system supporting D2D. It formulated a two-stage optimization algorithm with the goal of improving resource utilization and network capacity. However, these works did not consider the coexistence of MIMO-NOMA and D2D technology in MEC and VEC.

2.2. V2V and V2I Communication in VEC

Some research works have also employed V2V and V2I communication in VEC systems. In [11], Zhu et al. considered V2I communication based on MIMO-NOMA in a VEC system. In [1], Raza et al. proposed a 5G-based VEC system in which the V2I communication was based on the millimeter-wave mode. In [36], Zhang et al. constructed a software-defined networking (SDN)-assisted VEC system with the goal of optimizing the system overhead while considering 802.11p-based V2I communication and V2V communication. In [37], Bai et al. designed a VEC system containing vehicular cloudlets and considered D2D-based V2V communication. However, these works did not consider the coexistence of MIMO-NOMA-based V2I communication and D2D-based V2V communication in VEC systems.

2.3. DRL-Based Resource Allocation in VEC

There have been many studies on DRL-based resource allocation in VEC. In [38], Ning et al. proposed a VEC system consisting of multiple roadside units (RSUs), SVUs and a single BS. With the aims of optimizing the system cost, the DDPG algorithm was employed to obtain the optimal resource allocation scheme. In [39], Ren et al. proposed a VEC system consisting of multiple RSUs, BSs, and SVUs. A centralized resource allocation mechanism based on DRL was designed to obtain an optimal network resource-allocation scheme. In [23], Liu et al. designed a semi-Markov process-based scheme for maximizing VEC network utility and employed the deep Q-learning network (DQN) algorithm to achieve optimal resource allocation. However, none of the above works considered decentralized DRL-based VEC systems.

Some works have also considered decentralized DRL-based VEC systems. In [31], Ye et al. constructed a VEC system with both V2I communication and V2V communication, in which SVUs used the DQN algorithm to select the transmission band for their tasks, thus optimizing the system capacity and latency. In [11], Zhu et al. designed a VEC system consisting of a BS and SVUs, considering the mobility of SVUs and MIMO-NOMA technology. The SVUs employed the DDPG algorithm to allocate processing power, thus optimizing the latency and power consumption. However, [11] did not consider the presence of V2V communication, while [31] did not consider the mobility of SVUs, MIMO-NOMA, and D2D technology.

As seen from the above review, no previous works have considered the coexistence of MIMO-NOMA-based V2I communication and D2D-based V2V communication in the power allocation problem for decentralized DRL-based VEC systems. This motivates us to start this work. We shall show it in detail.

3. System Model

The system model is shown in Figure 1. A MEC server is placed on each VEC vehicle and at the BS with multiple antennae. Based on the axial distance to the BS, the VEC server coverage is divided into J lanes, where SVUs may move at different speeds in different lanes. The time duration for which SVUs in lane j remain within the communication range of the BS is divided into $N_{j,m}$ slots, and the length of each slot is ${\tau }_0$. In each slot, tasks arrive randomly in the buffers of the SVUs. At the same time, each SVU allocates local processing power, V2I processing power, and V2V processing power to process tasks either locally or offload them to the MEC server. In addition, the V2I channel conditions continuously change due to the mobility of SVUs and the channel’s interference in MIMO-NOMA. In contrast, V2V communication is based on D2D technology, and it is assumed that each SVU communicates only with a specific VEC vehicle in the same lane, and each VEC vehicle processes tasks for only one specific SVU; thus, the V2V channel conditions can be treated as time invariant. In V2I communication, each SVU first transmits tasks to the BS; the BS processes the tasks and employs the zero-forcing (ZF) technique to detect the signal and noise associated with each SVU from all SVU signals and then obtains the signal-to-interference-plus-noise ratio (SINR) of each SVU for V2I communication. In the next slot, the BS transmits the SINR of each SVU to that SVU. In contrast to the traditional scheme of centralized DRL, in this work, each SVU can determine its own power allocation according to its own observations of the environment. It is a decentralized mode. In the following, we shall introduce the system’s mobility model, task-computation model, and communication model and describe the relevant environmental information, such as the buffer capacity of SVU k, the SINRs of V2I communication and V2V communication, and the position of SVU k. The notations used in this article can be seen in

Table 1. Notations used in this article.

Notation	Description	Notation	Description
$a_{k,n}$	Action of SVU k in slot n.	$a_k\left(n\right)$	Number of task bits of SVU k arriving in slot n.
$a_k$	Abbreviation for $a_{k,n}$.	$a_k^i$	Action of the ith tuple.
$a_k^{\prime }$	Abbreviation for $a_{k,n+1}$.	${a^{\prime }}_k^i$	Next action of the ith tuple.
${\beta }_0$	Channel power gain at the reference distance.	$B_k\left(n\right)$	Buffer capacity of SVU k in slot n.
D	Diameter of the BS’s range.	$e\left(n\right)$	Error vector.
$d_{k,L}\left(n\right)$	Number of task bits processed locally by SVU k in slot n.	$d_{k,I}\left(n\right)$	Number of task bits processed for SVU k performing V2I communication in slot n.
$d_{k,j}\left(n\right)$	Distance between SVU k and the BS along the x-axis in slot n.	$d_{k,V}\left(n\right)$	Number of task bits processed for SVU k performing V2V communication in slot n.
$f_i$	An exponentially distributed random variable with unit mean.	$f_k\left(n\right)$	CPU frequency of SVU k in slot n.
${\mathit{g}}_k^s\left(n\right)$	Small-scale fading channel gain of SVU k in slot n.	${\mathit{g}}_{k,I}\left(n\right)$	Channel vector of SVU k for V2V communication in slot n.
$g_k^p\left(n\right)$	Large-scale fading coefficient for V2I communication at slot n.	$g_r$	Channel power gain at 1 m.
$\mathit{G}\left(n\right)$	MIMO-NOMA channel martix.	$h_{k,V}$	Channel gain of SVU k for V2V communication in slot n.
$J\left({\mu }_k\right)$	Objective function.	$K_{max}$	Maximum number of episodes in the training stage.
$L\left({\zeta }^k\right)$	Loss function.	L	Number of CPU cycles for processing one bit.
${\mathit{N}}_s\left(n\right)$	Noise received by the BS.	$N_{j,m}$	Last slot in lane j.
$N_{max,j}$	Maximum number of SVUs in lane j.	$N_a$	Number of antennae.
${\mathit{P}}_{k,j}\left(n\right)$	Location of SVU k in slot n.	${\mathit{P}}_B$	Position of the BS antenna.
$p_{k,I}\left(n\right)$	Processing power of SVU k for V2I communication in slot n.	$p_{k,L}\left(n\right)$	Local processing power of SVU k in slot n.
$p_{k,V}\left(n\right)$	Processing power of SVU k for V2V communication in slot n.	$P_{max,V}$	Maximum V2I processing power.
$P_{max,I}$	Maximum V2I processing power.	$P_{max,L}$	Maximum local processing power.
$Q^{{\mu }_{{\theta }^k}}(s_{k,n},a_{k,n})$	Action value function of SVU k.	$Q^{{\zeta }^k}(s_{k,n},a_{k,n})$	Action value function output from the critic network.
$Q^{{\zeta }^{k^{\prime }}}(s_{k,n},a_{k,n})$	Action value function output from the target critic network.	$r_{k,n}$	Reward of SVU k in slot n.
$r_k$	Abbreviation for $r_{k,n}$.	$r_k^i$	Reward of the ith tuple.
$\Gamma$	Experience buffer.	$s_{k,n}$	State space of SVU k in slot n.
$s_k$	Abbreviation for $s_{k,n}$.	$s_k^i$	State of the ith tuple.
$s_k^{\prime }$	Abbreviation for $s_{k,n+1}$.	${s^{\prime }}_k^i$	Next state of the ith tuple.
T	Maximum number of tuples in a minibatch.	$v_k$	Velocity of SVU k.
$v_j$	Velocity of lane j.	$w_1$	Distance from lane 1 to the BS.
$W_d$	Bandwidth.	$w_{k,j}$	Distance between SVU k driving in lane j and antennas along the y-axis.
$w_d$	Lane width.	$y_k^i$	Target value of the ith tuple.
$\mathit{y}\left(n\right)$	Signal received by the BS.	${\alpha }_k^C$	Learning rate of the critic network.
${\alpha }_k^A$	Learning rate of the actor network.	${\alpha }_h$	Path loss exponent for V2V communication.
${\rho }_k$	Normalized channel correlation coefficient of SVU k.	${\lambda }_k$	Mean rate of task arrival for SVU k.
${\zeta }^k$	Parameter of the critic network.	${\zeta }^{k^{\prime }}$	Parameter of the target critic network.
$\kappa$	Effective converted capacitance of SVU k.	$R_k$	Distance between SVU k and the corresponding VEC vehicle.
${\theta }^k$	Parameter of the actor network.	${\theta }^{k^{\prime }}$	Parameter of the target actor network.
$\gamma$	Discount factor.	${\Delta }_n$	Exploration noise in slot n.
${\mu }_{{\theta }^k}$	Policy of SVU k approximated by the actor network.	${\mu }_k^{*}$	Optimal policy of SVU k.
${\sigma }_R^2$	Variance of the Gaussian noise in communication.	$\tau$	Update degree parameter for the target networks.
${\tau }_0$	Slot duration.	${\omega }_1,{\omega }_2$	Reward weight factors.
${\gamma }_{k,I}\left(n\right)$	SINR of SVU k for V2I communication in slot n.	$\eta$	Path loss exponent.
${\gamma }_{k,V}\left(n\right)$	SINR of SVU k for V2V communication in slot n.

.

3.1. Mobility Model

Let ${\mathit{P}}_{k,j}\left(n\right)$ denote the position of SVU k in lane j in slot n. We establish a three-dimensional right-angle coordinate system as shown in Figure 1, where the BS is the origin point, the x-axis direction is the travel direction of SVU k, the y-axis represents the vertical distance to the BS, and the z-axis represents the height difference to the BS. Let $d_{k,j}\left(n\right)$ and $w_{k,j}$ be the distances between SVU k in slot n and the BS along the x-axis and y-axis, respectively. Therefore, ${\mathit{P}}_{k,j}\left(n\right)$ can be expressed as $(d_{k,j}\left(n\right),w_{k,j},0)$, where $w_{k,j}$ is equal to the vertical distance of lane j from the BS, which can be calculated as

$$w_{k,j}=(j-1)·w_d+w_1,$$

(1)

where $w_d$ is the lane width and $w_1$ is the distance between the BS and lane 1.

Similar to [40], for simplification, we shall employ the discrete approximation model and assume that the position of SVU k is constant within each individual slot because the time duration ${\tau }_0$ of each slot is small. Since the velocity $v_j$ of SVU k in lane j is constant, $d_{k,j}\left(n\right)$ can be expressed as

$$d_{k,j}\left(n\right)=d_{k,j}(n-1)+v_j{\tau }_0,$$

(2)

where $d_{k,j}\left(n\right)\in \left[-\frac{D}{2},\frac{D}{2}\right]$ and $d_{k,j}\left(1\right)=-\frac{D}{2}$. SVU k can determine whether it is within the coverage area of the BS based on its own current position $d_{k,j}\left(n\right)$, which reflects the mobility of SVU k.

3.2. Communication Model

3.2.1. V2I Communication

The channel matrix between the BS and the SVUs in slot n can be expressed as $\mathit{G}\left(n\right)=[{\mathit{g}}_{1,I}\left(n\right),\cdots ,{\mathit{g}}_{k,I}\left(n\right),\cdots ,{\mathit{g}}_{M,I}\left(n\right)]\in {\mathbb{C}}^{N_a\times M}$, where $N_a$ is the number of BS antennae and ${\mathit{g}}_{k,I}\left(n\right)\in {\mathbb{C}}^{N_a\times 1}$ is the channel vector between SVU k and the BS. In the MIMO-NOMA channels, the signals received by the BS in slot n from all SVUs can be expressed as

$$\begin{array}{c}\hfill \mathit{y}\left(n\right)=\sum _{k\in \mathcal{M}}{\mathit{g}}_{k,I}\left(n\right)\sqrt{p_{k,I}\left(n\right)}{s}_k\left(n\right)+{\mathit{N}}_s\left(n\right),\\ \hfill p_{k,I}\left(n\right)\in [0,P_{max,I}],\end{array}$$

(3)

where ${\mathit{N}}_s\left(n\right)$ is Gaussian white noise and $s_k\left(n\right)$ is complex data symbol with unit variance. Here, $p_{k,I}\left(n\right)$ is the V2I processing power of SVU k in slot n, with $P_{max,I}$ being the maximum V2I processing power of SVU k. In addition, ${\mathit{g}}_{k,I}\left(n\right)$ reflects the path loss of SVU k for V2I communication, which can be expressed as [41]

$${\mathit{g}}_{k,I}\left(n\right)={\mathit{g}}_k^s\left(n\right)\sqrt{g_k^p\left(n\right)},$$

(4)

where ${\mathit{g}}_k^s\left(n\right)$ is the small-scale fading channel gain and $g_k^p\left(n\right)$ is the large-scale fading coefficient. $g_k^p\left(n\right)$ reflects the mobility of SVU k and is calculated as

$$g_k^p\left(n\right)=\frac{g_r}{{∥{\mathit{P}}_{k,j}\left(n\right)-{\mathit{P}}_B∥}^{\eta }},$$

(5)

where $g_r$ is the channel gain at 1 m and $\eta$ is the path loss exponent. Meanwhile, ${\mathit{P}}_{k,j}\left(n\right)=(d_{k,j}\left(n\right),w_{k,j},0)$ is the position of SVU k in slot n, and ${\mathit{P}}_B=(0,0,H_a)$, where $H_a$ is the height of the BS antenna. Note that ${\mathit{P}}_{k,j}\left(n\right)$ can be calculated from Equations (1) and (2).

The small-scale fading channel gain is initialized as ${\mathit{g}}_k^s\left(0\right)\sim \mathcal{CN}(\mathbf{0},{\mathit{I}}_K)$, where ${\mathit{I}}_K$ is an $N_a\times N_a$ identity vector.

The relationship between ${\mathit{g}}_k^s\left(n\right)$ and ${\mathit{g}}_k^s(n-1)$ can be expressed as [42]

$${\mathit{g}}_k^s\left(n\right)={\rho }_k{\mathit{g}}_k^s(n-1)+\sqrt{1-{\rho }_k^2}\mathit{e}\left(n\right),$$

(6)

where $\mathit{e}\left(n\right)$ is an error vector and we can obtain ${\rho }_m$ as described in [43]. ${\rho }_k$ is the normalized channel correlation coefficient and is correlated with $\theta$, which is the angle between the movement direction of SVU k, i.e., the x-axis, and the direction of communication, i.e., ${\mathit{P}}_B-{\mathit{P}}_{k,j}\left(n\right)$. $\theta$ is calculated as

$$\theta =arccos\left(\frac{{\mathit{x}}_0·({\mathit{P}}_B-{\mathit{P}}_{k,j}\left(n\right))}{∥{\mathit{P}}_B-{\mathit{P}}_{k,j}\left(n\right)∥}\right),$$

(7)

where ${\mathit{x}}_0=(1,0,0)$.

By using Equations (5)–(7), SVU k can obtain the channel vector ${\mathit{g}}_{k,I}\left(n\right)$.

Subsequently, the BS employs the ZF technique [42] to obtain the SINR of SVU k for V2I communication, denoted by ${\gamma }_{k,I}\left(n\right)$, which is calculated as

$${\gamma }_{k,I}\left(n\right)=\frac{p_{k,I}\left(n\right)}{{∥{\mathit{g}}_k^G\left(n\right)∥}^2{\sigma }_R^2},$$

(8)

where $p_{k,I}\left(n\right)$ is the V2I processing power of SVU k in slot n, ${\sigma }_R^2$ is the noise power, and ${\mathit{g}}_k^G\left(n\right)$ is the kth row of the pseudoinverse of $\mathit{G}\left(n\right)$. Therefore, the relationship between ${\mathit{g}}_k^G\left(n\right)$ and ${\mathit{g}}_{k,I}\left(n\right)$ is

$${\mathit{g}}_k^G\left(n\right){\mathit{g}}_{i,I}\left(n\right)=\left\{\begin{array}{c}1,i=k,\hfill \\ 0,i\ne k.\hfill \end{array}\right.$$

(9)

Based on Equations (3)–(9), the BS can obtain ${\gamma }_{k,I}\left(n\right)$ and transmit it to SVU k in the next slot. Thus, SVU k is able to observe ${\gamma }_{k,I}(n-1)$ in the local environment in slot n, which reflects the uncertain channel conditions of SVU k for V2I communication caused by mobility of SVU k.

3.2.2. V2V Communication

Similar to [16], V2V communication is based on D2D technology. Since the channel conditions between SVU k and its corresponding VEC vehicle are time invariant, the channel gain between them can be expressed as

$$h_{k,V}={\beta }_0{f_i}^2{R}_k^{-{\alpha }_h},$$

(10)

where $f_i$ is an exponentially distributed random variable with unit mean. ${\alpha }_h$ and ${\beta }_0$ are the path loss exponent and the channel power gain at the reference distance, respectively, for V2V communication. $R_k$ is the distance between the communicating vehicles. Since each SVU communicates only with a specific VEC vehicle in the same lane, $R_k$ is a constant.

Therefore, the SINR of SVU k for V2V communication, denoted by ${\gamma }_{k,V}\left(n\right)$, is calculated as

$${\gamma }_{k,V}\left(n\right)=\frac{p_{k,V}\left(n\right)h_{k,V}}{{\sigma }_R^2},$$

(11)

where $p_{k,V}\left(n\right)\in [0,P_{max,V}]$ is the V2V processing power of SVU k in slot n.

By using Equations (11) and (12), SVU k can obtain the SINR for V2V communication in slot n, where ${\gamma }_{k,V}\left(n\right)$ is related only to $p_{k,V}\left(n\right)$, which reflects the fact that the channel conditions of SVU k for V2V communication are time invariant.

3.3. Task-Computation Model

The relationship between the buffer capacity $B_k\left(n\right)$ and $B_k(n-1)$ of SVU k in slot n is calculated as

$$B_k\left(n\right)={[B_k(n-1)-(d_{k,L}(n-1)+d_{k,I}(n-1)+d_{k,V}(n-1))]}^{+}+a_k(n-1),$$

(12)

where ${[·]}^{+}=max(0,·)$ and $a_k(n-1)$ is the number of tasks arriving in slot n. $d_{k,L}(n-1)$, $d_{k,I}(n-1)$, and $d_{k,V}(n-1)$ are the numbers of tasks processed in slot $n-1$ for local processing, V2I processing, and V2V processing, respectively. The descriptions of how to calculate $d_{k,L}(n-1)$, $d_{k,I}(n-1)$ and $d_{k,V}(n-1)$ are given below.

3.3.1. Local Processing

Let L be the computational intensity of the tasks, i.e., the number of cycles required for the CPU to process one bit. Let $f_k(n-1)$ be the CPU processing frequency of SVU k in slot $n-1$. Therefore, $d_{k,L}(n-1)$ is calculated as

$$d_{k,L}(n-1)={\tau }_0{f}_k(n-1)/L,$$

(13)

where $f_k(n-1)$ is calculated as

$$\begin{array}{c}\hfill f_k(n-1)=\sqrt[3]{p_{k,L}(n-1)/\kappa },\\ \hfill p_{k,L}\in [0,P_{max,L}],\end{array}$$

(14)

where $p_{k,L}(n-1)$ is the local processing power of SVU k in slot $n-1$ and $\kappa$ is a constant that reflects the effective converted capacitance.

3.3.2. V2I and V2V Processing

Since the computational resource of a MEC server is assumed to be sufficient, the latency of a MEC server in processing tasks is negligible. Moreover, the size of the computation result is very small, so the feedback latency is also negligible. Therefore, according to Shannon’s theorem, $d_{k,I}(n-1)$ and $d_{k,V}(n-1)$ are calculated as

$$d_{k,I}(n-1)={\tau }_0{W}_d{log}_2(1+{\gamma }_{k,I}(n-1)),$$

(15)

$$d_{k,V}(n-1)={\tau }_0{W}_d{log}_2(1+{\gamma }_{k,V}(n-1)),$$

(16)

where $W_d$ is the bandwidth and ${\gamma }_{k,I}(n-1)$ and ${\gamma }_{k,V}(n-1)$ are the SINRs of SVU k at slot $n-1$ for V2I communication and V2V communication, respectively.

The buffer capacity $B_k\left(n\right)$ of SVU k can be calculated from $B_k(n-1)$ based on Equations (12)–(16). Since $B_k\left(n\right)$ depends on $a_k(n-1)$, $d_{k,L}(n-1)$, $d_{k,I}(n-1)$ and $d_{k,V}(n-1)$, it reflects the randomness of task arrival and the uncertainty of the channel conditions for V2I communication.

4. Problem Formulation

In this section, we describe the DRL-based framework, which consists of state, action, and reward functions. The state is defined based on the environment of each SVU in slot n; the action corresponds to the power allocation of each SVU, which is based on a policy $\mu$ and the reward is the benefit earned as a result of the action, which is related to the power consumption and latency.

4.1. State

In this paper, each SVU observes its surrounding environment to determine its power allocation. The V2I channel conditions for each SVU are uncertain because of the channel’s interference in MIMO-NOMA and the mobility of each SVU. Moreover, the task arrival is random. Based on joint consideration of these two issues, the state is formulated to reflect the uncertainty of V2I channel conditions and the randomness of task arrival.

In the system model, the distance of SVU k from the BS along the x-axis $d_{k,j}\left(n\right)$ reflects its mobility. In addition, according to Equations (8) and (9), the SINR ${\gamma }_{k,I}\left(n\right)$ of SVU k for V2I communication depends on ${\mathit{g}}_k^G\left(n\right)$, which in turn depends on ${\mathit{g}}_{k,I}\left(n\right)$; thus, we find that ${\gamma }_{k,I}\left(n\right)$ depends on ${\mathit{g}}_{k,I}\left(n\right)$. Therefore, ${\gamma }_{k,I}\left(n\right)$ reflects the uncertainty of the V2I channel conditions. Moreover, according to Equations (12)–(16), the buffer capacity $B_k\left(n\right)$ of SVU k in slot n is a function of $a_k(n-1)$ and ${\gamma }_{k,I}(n-1)$, where $a_k(n-1)$ reflects the randomness of the task-arrival rate and ${\gamma }_{k,I}(n-1)$ reflects the uncertainty of the V2I channel conditions. Therefore, $B_k\left(n\right)$ reflects both the randomness of task arrival and the uncertainty of the V2I channel conditions. Since SVU k can observe $d_{k,j}\left(n\right)$, ${\gamma }_{k,I}(n-1)$ and $B_k\left(n\right)$ in the local environment, the state of SVU k in slot n can be expressed as

$$s_{k,n}=[B_k\left(n\right),{\gamma }_{k,I}(n-1),d_{k,j}\left(n\right)],$$

(17)

where ${\gamma }_{k,I}(n-1)$ depends on ${\mathit{g}}_{k,I}\left(n\right)$ and the buffer capacity $B_k\left(n\right)$ is related to ${\gamma }_{k,I}(n-1)$ and $a_k(n-1)$. Since ${\mathit{g}}_{k,I}\left(n\right)$ and $a_k(n-1)$ are continuous values, the state space of SVU k is continuous.

4.2. Action

SVU k allocates its local processing power $p_{k,L}\left(n\right)$, V2I processing power $p_{k,L}\left(n\right)$, and V2V processing power $p_{k,L}\left(n\right)$ in accordance with the current state observed in slot n. Thus, the action of SVU k in slot n can be expressed as

$$a_{k,n}=[p_{k,L}\left(n\right),p_{k,I}\left(n\right),p_{k,V}\left(n\right)].$$

(18)

note that similar to [44], we consider the action space of SVU k to be continuous.

4.3. Reward

The reward is an evaluation based on the previous action. In this paper, we aim to minimize power consumption and latency. As mentioned in Section 3, the latency of task offloading is a constant. Thus, based on Little’s theorem [45], the reward of SVU k is defined as

$$r_{k,n}=-\left[{\omega }_1\left(p_{k,L}\left(n\right)+p_{k,I}\left(n\right)+p_{k,V}\left(n\right)\right)+{\omega }_2{B}_k\left(n\right)\right],$$

(19)

where ${\omega }_1$ and ${\omega }_2$ are nonnegative weight factors and ${\omega }_1+{\omega }_2=1$.

Accordingly, the cumulative discount reward of SVU k can be calculated as

$$J\left({\mu }_k\right){\mathbb{E}}_{{\mu }_k}\left[\sum _{n=1}^{N_{j,m}}{\gamma }^{n-1}{r}_{k,n}\right],$$

(20)

where $\gamma$ is a constant that reflects the degree of discount applied to the long-term reward.

5. Solution

In this section, we first introduce the training process, which is based on the DDPG algorithm. Then, we describe how the performance of the trained model is tested in the testing stage.

5.1. Training Stage

Since the DDPG algorithm is capable of solving problems with the continuous action space, we employ the DDPG algorithm to obtain the optimal policy.

The DDPG algorithm combines the deterministic policy gradient (DPG) approach with the actor–critic framework; it is a modification of the DQN algorithm and can solve problems with the continuous action space. The DDPG algorithm is composed of four neural networks: an actor network, a target actor network, a critic network, and a target network. Here, the actor network and target actor network are employed to update the policy ${\mu }_{{\theta }^k}$, thus obtaining the optimal policy. The critic network and target critic network are employed to evaluate the policy.

The flow of the training stage is summarized in Algorithm 1. Note that ${\theta }^k$ and ${\theta }^{k^{\prime }}$ denote the parameters of the actor network and target actor network, respectively. ${\zeta }^k$ and ${\zeta }^{k^{\prime }}$ denote the parameters of the critic network and target critic network, respectively. ${\Delta }_n$ is the noise parameter in slot n.

Algorithm 1: Model training stage based on the DDPG algorithm

In the training stage, we randomly initialize ${\theta }^k$ and ${\zeta }^k$, while ${\theta }^{k^{\prime }}$ and ${\zeta }^{k^{\prime }}$ have the same initial values as ${\theta }^k$ and ${\zeta }^k$, respectively. Then, we define an experience buffer $\Gamma$ with sufficient space to store the transitions for each slot (lines 1–2).

Without loss of generality, let us consider that model training starts for SVU k. In the first episode, the position of SVU k is first reset to within the range of the BS antennae. $d_{k,j}\left(1\right)$ is initialized as $-\frac{D}{2}$, and $B_k\left(1\right)$ is initialized as half of the buffer capacity. Then, ${\mathit{g}}_{\mathit{k}}^s\left(0\right)$ is randomly initialized and ${\mathit{g}}_k^G\left(0\right)$ is calculated by using Equation (9). Meanwhile, the SINR ${\gamma }_{k,I}\left(0\right)$ of SVU k for V2I communication is calculated by using Equation (8). Thus, the state of SVU k in slot 1 is obtained, i.e., $s_{k,1}=[B_k\left(1\right),{\gamma }_{k,I}\left(0\right),d_{k,j}\left(1\right)]$ (lines 3–5).

Subsequently, given the initial input $s_{k,1}$ to the actor network in slot 1, the corresponding policy ${\mu }_{{\theta }^k}\left(s_{k,1}\right|{\theta }^k)$ is obtained. The noise ${\Delta }_1$ is randomly initialized and then SVU k performs an action $a_{k,1}$ based on the current policy and the noise, ${\mu }_{{\theta }^k}\left(s_{k,1}\right|{\theta }^k)+{\Delta }_1$. With the performed action, the local processing power $p_{k,L}\left(1\right)$, the V2I processing power $p_{k,I}\left(1\right)$, and the V2V processing power $p_{k,V}\left(1\right)$ are determined. Then, SVU k obtains a reward $r_k\left(1\right)$ in accordance with Equation (19). The SINR ${\gamma }_{k,I}\left(0\right)$ for V2I communication is obtained in accordance with Equations (3)–(9), and $B_k\left(2\right)$ is obtained from Equations (12)–(16), where $d_{k,L}\left(1\right)$, $d_{k,I}\left(1\right)$, and $d_{k,V}\left(1\right)$ are obtained from Equation (13), (15) and (16), respectively. Additionally, $d_{k,j}\left(2\right)$ is obtained from Equation (2). Thus, the state of SVU k in slot 2 is obtained, i.e., $s_{k,2}=[B_k\left(2\right),{\gamma }_{k,I}\left(1\right),d_{k,j}\left(2\right)]$. Then, the tuple $(s_{k,1},a_{k,1},r_{k,1},s_{k,2})$ is stored in the experience buffer $\Gamma$. If the number of tuples in $\Gamma$ does not exceed T, SVU k proceeds to the next slot and repeats the above process (lines 6–10).

Once the number of tuples reaches T, the parameters ${\theta }^k$, ${\zeta }^k$, ${\theta }^{k^{\prime }}$, and ${\zeta }^{k^{\prime }}$ are updated toward maximizing $J\left({\mu }_{{\theta }^k}\right)$. The parameter ${\theta }^k$ is updated in accordance with the gradient of the policy, i.e., the gradient in the direction of $J\left({\mu }_{{\theta }^k}\right)$, which is denoted by ${\nabla }_{{\theta }^k}J\left({\mu }_{{\theta }^k}\right)$. Let $Q^{{\zeta }^k}(s_{k,n},a_{k,n})$ be the action value function of SVU k, which is the output of the critic network. According to [46], the task of solving for ${\nabla }_{{\theta }^k}J\left({\mu }_{{\theta }^k}\right)$ can be replaced by solving for the gradient of $Q^{{\zeta }^k}(s_{k,n},a_{k,n})$.

Now, we further describe how the parameters are updated, as follows. Figure 2 illustrates the parameter update process of the DDPG algorithm. First, SVU k randomly selects T tuples from the experience buffer to form a minibatch. For convenience, we use $r_k$, $s_k$, $a_k$, $s_k^{\prime }$, and $a_k^{\prime }$ to denote $r_{k,n}$, $s_{k,n}$, $a_{k,n}$, $s_{k,n+1}$, and $a_{k,n+1}$, respectively. Then, let $(s_k^i,a_k^i,r_k^i,{s^{\prime }}_k^i)$ denote the ith tuple in the minibatch. For the ith tuple, SVU k inputs ${s^{\prime }}_k^i$ into the target actor network, which outputs ${a^{\prime }}_k^i$ based on ${s^{\prime }}_k^i$. Then SVU k inputs ${s^{\prime }}_k^i$ and ${a^{\prime }}_k^i$ into the target critic network, which outputs the action value function $Q^{{\zeta }^{k^{\prime }}}({s^{\prime }}_k^i,{a^{\prime }}_k^i)$. Thus, the target value of tuple i can be calculated as (line 11)

$$y_k^i=r_k^i+\gamma Q^{{\zeta }^{k^{\prime }}}({s^{\prime }}_k^i,{a^{\prime }}_k^i){|}_{{a^{\prime }}_k^i={\mu }_{{\theta }^{k^{\prime }}}\left({s^{\prime }}_k^i\right|{\theta }^{k^{\prime }})}.$$

(21)

then, $s_k^i$ and $a_k^i$ are used as inputs to the critic network, which outputs the action value function $Q^{{\zeta }^k}(s_k^i,a_k^i)$. Thus, the loss function for tuple i can be expressed as

$$L_i={\left[y_k^i-Q^{{\zeta }^k}(s_k^i,a_k^i)\right]}^2.$$

(22)

accordingly, the loss function for all tuples can be expressed as

$$L\left({\zeta }^k\right)=\frac{1}{T}\sum _{i=1}^T{L}_i.$$

(23)

Then, SVU k updates ${\zeta }^k$ based on Equations (21)–(23) [47] (line 12). Afterward, ${\nabla }_{{\theta }^k}J\left({\mu }_{{\theta }^k}\right)$ can be obtained from $Q^{{\zeta }^k}(s_k^i,a_k^{\mu })$, which is the output of the critic network. We have

$$\begin{array}{cc}\hfill & {\nabla }_{{\theta }^k}J\left({\mu }_{{\theta }^k}\right)\hfill \\ \hfill & \approx \frac{1}{T}\sum _{i=1}^T{\nabla }_{{\theta }^k}{Q}^{{\zeta }^k}(s_k^i,a_k^{\mu }){|}_{a_k^{\mu }={\mu }_{{\theta }^k}\left(s_k^i\right|{\theta }^k)}\hfill \\ \hfill & =\frac{1}{T}\sum _{i=1}^I{\nabla }_{a_k^{\mu }}{Q}^{{\zeta }^k}(s_k^i,a_k^{\mu }){|}_{a_k^{\mu }={\mu }_{{\theta }^k}\left(s_k^i\right|{\theta }^k)}\hfill \\ & ·{\nabla }_{{\theta }^k}{\mu }_{{\theta }^k}\left(s_k^i\right|{\theta }^k).\hfill \end{array}$$

(24)

note that the chain rule is utilized here, since $a_k^{\mu }={\mu }_{{\theta }^k}\left(s_k^i\right|{\theta }^k)$ is an input for $Q^{{\zeta }^k}(s_k^i,a_k^{\mu })$.

Similarly, SVU k updates the parameter of the actor network in accordance with Equation (24) [47] (line 13).

In slot $N_{j,m}$, SVU k updates ${\zeta }^{k^{\prime }}$ and ${\theta }^{k^{\prime }}$, i.e., (line 14)

$${\zeta }^{k^{\prime }}\leftarrow \tau {\zeta }^k+(1-\tau ){\zeta }^{k^{\prime }},$$

(25)

$${\theta }^{k^{\prime }}\leftarrow \tau {\theta }^k+(1-\tau ){\theta }^{k^{\prime }},$$

(26)

where $\tau \ll$ 1 is a constant.

Finally, SVU k proceeds to the next slot and uses $s_k^{\prime }$ as the input to the actor network. This current episode continues until slot $N_{j,m}$ is reached. When the number of episodes reaches $K_{max}$, training of the system model is complete.

5.2. Testing Stage

In the testing stage, we test the performance of the trained system model. Algorithm 2 shows the flow of the testing process.

Algorithm 2:Testing stage for the trained model

6. Simulation Results and Analysis

In this section, we demonstrate through simulation experiments that our proposed DDPG algorithm can obtain the optimal policy. The simulation experiments are divided into a training stage and a testing stage. The simulation tool is Python 3.7.

The key parameters of the experiments are listed in

Table 2. Key parameters in simulations.

Parameters of the System Model
Parameter	Value	Parameter	Value
${\sigma }_R^2$	${10}^{-9}$ W	$h_r$	$-30$ dB
${\beta }_0$	$-30$ dB	$W_d$	1 MHZ
${\tau }_0$	20 ms	$\kappa$	${10}^{-28}$
$v_1$	20 m/s	$v_2$	25 m/s
$v_3$	30 m/s	$w_d$	10 m
L	500 cycles/bit	${\lambda }_k$	3 Mbps
H	10 m	$N_a$	4
D	500 m	$P_{max,V}$	1 W
$P_{max,I}$	1 W	$P_{max,L}$	1 W
$R_k$	65 m	$w_1$	10 m
${\alpha }_h$	2
Parameters of the Training Process
Parameter	Value	Parameter	Value
${\alpha }_k^C$	$0.001$	${\alpha }_k^A$	$0.0001$
${\omega }_1$	$0.9$	${\omega }_2$	$0.1$
$\gamma$	$0.99$	$\tau$	$0.001$
$K_{max}$	1000	T	64
$K_{max}^{\prime }$	10	$\left|\Gamma \right|$	$2.5\times {10}^5$

. The learning rates for the actor network and critic network are ${10}^{-3}$ and ${10}^{-4}$, respectively [47]. The size of the experience replay buffer is $\left|\Gamma \right|$. Task arrival follows a Poisson process, where the mean arrival rate is ${\lambda }_k$. SVU k is driving in lane 2, and its V2I communication will be interfered with three other vehicles when $d_{k,j}\left(n\right)=0$.

6.1. Training Stage

Figure 3 shows the learning curve in the training stage, where the reward is the average value in each episode. One can see that the average reward rises quickly from episode 0 to episode 12. Then, the curve declines from episode 12 to episode 400. This reflects that SVU k is adjusting its policy toward the optimal reward. From episode 400 to episode 1000, the rewards become stable with little jitter. The reason for the existence of jitter is the presence of exploration noise, which prevents SVU k from falling into a local optimum.

6.2. Testing Stage

We present performance tests performed on the trained model in the testing stage. Figure 4 and Figure 5 show the test performance under the DDPG algorithm and three other greedy (GD) policies in terms of the power consumption, buffer capacity, and reward, where the performance is recorded as the average value over 10 episodes. In the following, we introduce the three policies.

• GD-Local policy: SVU k first maximally allocates the local processing in each slot. The remaining tasks are equally allocated to V2I processing and V2V processing.
• GD-V2I policy: SVU k first maximally allocates the V2I processing power in each slot. The remaining tasks are equally allocated to local processing and V2V processing.
• GD-V2V policy: SVU k first maximally allocates the V2V processing power in each slot. The remaining tasks are equally allocated to local processing and V2I processing.

Figure 4a shows the power allocation under the DDPG algorithm at different distances, and Figure 4b compares the power consumption under all four policies at different distances. Figure 4a shows the local-and-V2V processing power and the V2I processing power at different distances under the DDPG algorithm. When $d_{k,j}\left(n\right)<0$, the V2I processing power increases, and the local-and-V2V processing power decreases significantly. This is because the path loss decreases; thus, the channel conditions for V2I communication improve as SVU k approaches the BS. Therefore, as SVU k approaches the BS, SVU k allocates more power to V2I processing. When SVU k moves away from the BS, SVU k allocates more power to local-and-V2V processing. When $d_{k,j}\left(n\right)>0$, local-and-V2V processing power increases rapidly, while the V2I processing power decreases. This is because when $d_{k,j}\left(n\right)=0$, the other three vehicles impose interference on SVU k, causing the V2I channel conditions to deteriorate. As the V2I channel conditions become worse, more power needs to be allocated to local-and-V2V processing. From Figure 4b, it can be seen that the power consumption under the GD-V2I policy gradually decreases when $d_{k,j}\left(n\right)<0$ and rapidly increases when $d_{k,j}\left(n\right)>0$, which clearly matches the changing V2I channel conditions.

Figure 5a,b compares the buffer capacity and reward under the four policies at different distances. From Figure 5a, it can be seen that the buffer capacity increases when $d_{k,j}\left(n\right)=0$ under the GD-V2I policy. This is because the V2I communication is interfered with by the other three vehicles when $d_{k,j}\left(n\right)=0$. In contrast, the buffer capacity under the DDPG algorithm remains stable. This means that SVU k is able to process tasks in a timely manner even though the channel conditions are degraded. As shown in Figure 5a, the reward under the DDPG algorithm is better than the rewards under the other three policies most of the time.

Figure 6a compares the average buffer capacity under the four policies. There is no significant difference in buffer capacity among the four policies. Nevertheless, the GD-V2I policy has the largest buffer capacity because of variation of V2I channel conditions. Meanwhile, as seen from Figure 6b, the average power consumption under the DDPG algorithm is significantly superior to the other three policies. Compared to that under the GD-Local policy, the average power consumption under the DDPG algorithm is reduced by 24.4%. Similarly, the average power reduction under the DDPG algorithm is 51.3% compared to the GD-V2I policy and 23.1% compared to the GD-V2V policy.

Figure 6c shows the cumulative discount reward under the four policies. We can see that the cumulative discount reward under the DDPG algorithm is better than those under the other three policies. This is because of the adaptability of the DDPG algorithm, which allows the model to rapidly adjust the power allocation.

Figure 7a,c shows the cumulative discount reward, power consumption, and buffer capacity under the four policies at different task-arrival rates. As the task-arrival rate increases, the cumulative discount reward decreases, the power consumption increases, and the buffer capacity becomes larger for all four policies. It can also be seen that although the DDPG algorithm is superior to the other three policies in terms of the power consumption and cumulative discount reward, its buffer capacity is slightly higher than those under the GD-Local and GD-V2V policies. This is because the DDPG algorithm aims to obtain the maximum cumulative discount reward. Due to its focus on power consumption, and the fact that its buffer capacity performance was not given more attention, the result is slightly worse.

Table 3. Performance comparison under the four policies.

Policies	Average Power Consumption	Average Buffer Capacity	Cumulative Discount Reward
DDPG	A	B	A
GD-Local	B	B	B
GD-V2I	D	C	D
GD-V2V	B	B	B

compares the performance in terms of average power consumption, average buffer capacity and cumulative discount reward under the four policies, where A, B, C, and D stand for different performance levels in a descending order.

7. Conclusions

In this paper, we have proposed a decentralized DRL-based VEC power allocation model that considers not only the coexistence of D2D-based V2V communication and MIMO-NOMA-based V2I communication, but also the mobility of SVUs, the randomness of task arrival, and the channels’ interference in MIMO-NOMA. Extensive simulations demonstrate that the average power consumption and reward under the DDPG algorithm is superior to those of other policies. Meanwhile, since the proposed DDPG algorithm focuses on power consumption, it may incur a compromise for buffer capacity. For future work, we will consider the coexistence of many-to-many D2D and MIMO-NOMA technology in VEC systems.