Multi-Agent-Deep-Reinforcement-Learning-Enabled Offloading Scheme for Energy Minimization in Vehicle-to-Everything Communication Systems

Abstract

Offloading computation-intensive tasks to mobile edge computing (MEC) servers, such as road-side units (RSUs) and a base station (BS), can enhance the computation capacities of the vehicle-to-everything (V2X) communication system. In this work, we study an MEC-assisted multi-vehicle V2X communication system in which multi-antenna RSUs with liner receivers and a multi-antenna BS with a zero-forcing (ZF) receiver work as MEC servers jointly to offload the tasks of the vehicles. To control the energy consumption and ensure the delay requirement of the V2X communication system, an energy consumption minimization problem under a delay constraint is formulated. The multi-agent deep reinforcement learning (MADRL) algorithm is proposed to solve the non-convex energy optimization problem, which can train vehicles to select the beneficial server association, transmit power and offloading ratio intelligently according to the reward function related to the delay and energy consumption. The improved K-nearest neighbors (KNN) algorithm is proposed to assign vehicles to the specific RSU, which can reduce the action space dimensions and the complexity of the MADRL algorithm. Numerical simulation results show that the proposed scheme can decrease energy consumption while satisfying the delay constraint. When the RSUs adopt the indirect transmission mode and are equipped with matched-filter (MF) receivers, the proposed joint optimization scheme can decrease the energy consumption by 56.90% and 65.52% compared to the maximum transmit power and full offloading schemes, respectively. When the RSUs are equipped with ZF receivers, the proposed scheme can decrease the energy consumption by 36.8% compared to the MF receivers.

1. Introduction

The sixth generation (6G) is expected to enhance energy efficiency and the intelligence level to meet mass connectivity and low latency requirements of wireless communication network [1]. Supported by 6G, vehicle-to-everything (V2X) with the requirements of low communication delay is an instrumental element of future connected autonomous vehicles [2]. This promising technology can obtain the location and speed of vehicles and road hazards, reduce traffic accidents and improve road safety [3,4], and has gained significant interest from vehicle manufacturers, researchers and scientific communities [5]. However, as intelligent applications, i.e., in-vehicle Internet access and cooperative collision warning [6], high mobility of the device [7] and low-latency requirements [8], using V2X technology to handle computation-intensive tasks that require a large amount of data computation and communication resources is a challenge [9]. By offloading tasks to mobile edge computing (MEC) servers, i.e., base stations (BSs) [7], road-side units (RSUs) [10] and unmanned aerial vehicles (UAVs) [11,12,13], MEC technology can meet the demand for computation resources in V2X networks [14]. In addition, offloading tasks to the MEC servers that are located close to the vehicles can effectively reduce the computation delays, meeting real-time communication requirements [15,16].

In [17], the authors developed a three-tier offloading architecture where vehicles can offload tasks to the RSUs and BS, and the average latency can be effectively reduced. In [18], vehicles offload tasks to the infrastructures and other vehicles to minimize the communication delay. In [19], the weighted sum of delay and energy consumption is minimized while considering the impact of interference on MEC-assisted V2X systems. In order to reduce the power consumption, the joint connection modes, uplink paths and task assignment optimization are studied in [20]. The communication and computation resource allocation, optimized jointly, is studied in [21] to minimize the cost of the V2X system under delay and energy constraints. In [22], the authors optimized user association and resource allocation jointly to maximize the overall data rate of both cellular users and device-to-device pairs. In [23], the end-to-end latency is minimized by optimizing the user association and resource allocation jointly. But, due to the coupling of many parameters, these optimization problems are highly non-convex and difficult to solve.

Deep reinforcement learning (DRL) is one of the leading research fields of artificial intelligence [24]. It can find the globally optimal or near-optimal solutions for more complex optimization problems even under an unknown environment [25,26,27], and has been adopted in numerous applications [28]. In [29], the authors present a DRL algorithm to maximize the long-term cache hit rate of the system, which consists of a single BS and several users. In a given time slot, the number of contents that users can request from the BS is fixed, and the BS acts as an agent and makes a content store decision. The simulation results show that the DRL algorithm can improve the cache hit rates and provide significant savings in runtime. In [30], the authors studied the UAV-assisted V2X system, which consists of several BSs and vehicles. In order to minimize the age of information, the DRL algorithm is proposed to jointly optimize the transmit power and computation offloading. The simulation results show that the DRL algorithm can significant reduce the age of information.

This paper studies an MEC-assisted multi-vehicle V2X system, where the BS and RSUs work as MEC servers jointly to offload the tasks of vehicles. In order to reduce energy consumption while ensuring the requirement of V2X communication delay, the energy consumption minimization problem is formulated under delay and power constraints, and the DRL algorithm is proposed to solve the non-convex energy consumption optimization problem. The abbreviations and acronyms used in this paper are summarized in

Table 1. List of acronyms in alphabetical order.

Acronym	Description	Acronym	Description
BS	Base station	RSU	Road-side unit
DQN	Deep Q-network	SINR	Signal-to-interference-plus-noise-ratio
DRL	Deep reinforcement learning	6G	Sixth generation
I2B	Infrastructure-to-base-station	UAV	Unmanned aerial vehicle
KNN	K-nearest neighbors	V2B	Vehicle-to-base-station
MF	Matched filter	V2X	Vehicle-to-everything
MEC	Mobile edge computing	V2I	Vehicle-to-infrastructure
MADRL	Multi-agent deep reinforcement learning	ZF	Zero forcing
MIMO	Multi-input multi-output

and the main contributions of this work can be summarized as follows:

(1)

To compute the computation-intensive tasks for the computation-limited vehicles, we study an MEC-assisted multi-vehicle V2X communication system, where multi-antenna RSUs with linear receivers and a BS with a zero-forcing (ZF) receiver offload the tasks of vehicles jointly. In order to control the energy consumption, we formulate the energy consumption minimization problem and transform the non-convex optimization problem into a multi-agent decision process. Thus, each vehicle is capable of making intelligent decisions in its own communication environment.

(2)

In order to solve the non-convex optimization problem while satisfying the delay and power constraints, the multi-agent deep reinforcement learning (MADRL)-enabled server association, transmit power and offloading ratio joint optimization scheme is proposed. According to the reward function, which is related to the delay and energy consumption, vehicles are trained to select the beneficial transmit power, server association and offloading ratio. In order to reduce the action space dimensions and complexity of the MADRL algorithm, the improved K-nearest neighbors (KNN) algorithm is used to assign the vehicles that are located within the coverage area of several RSUs to the specific RSUs. Vehicles that are allocated to the same RSU as a group, and vehicles in the l-th group, can only offload tasks to the RSU l or BS.

(3)

Numerical results show that the proposed MADRL scheme can reduce more energy consumption compared with the full offloading and maximum transmit power schemes, and RSUs equipped with ZF receivers can decrease more energy consumption compared to the matched-filtering (MF) receivers. In addition, the proposed DRL scheme has a stable convergence under the different number of vehicles and data packet sizes.

2. System Model

As shown in Figure 1, we consider an uplink MEC-assisted multi-vehicle V2X communication system that consists of a BS equipped with $N^{\mathrm{B}}$ antennas, L RSUs equipped with $N^{\mathrm{R}}$ antennas and K single-antenna vehicles. In addition, the number of RSUs $L=4$ and number of vehicles $K=6$, and the radii of the BS and RSUs are 100 m and 200 m, respectively. In order to ensure real-time communication when the number of vehicles increases or vehicles move at a high speed, the RSUs and BS all work as the MEC servers jointly to offload the computation-intensive task of the vehicles. Thus, the vehicles can process the information locally or offload it to the RSUs via a vehicle-to-infrastructure (V2I) communication link or the BS via a vehicle-to-BS (V2B) communication link for edge computing. Meanwhile, each RSU has two transmission modes: $\mathbf{direct}$ $\mathbf{mode}$, which transmits the processed task to corresponding vehicles directly, and $\mathbf{indirect}$ $\mathbf{mode}$, which transmits the task to the wide-coverage BS, which transmits the processed information to the vehicles. For simplicity, the RSUs adopt the indirect transmission mode.

Significant interference caused by the multi-antenna BS and RSUs will increase the computation complexity of the multi-input multi-output (MIMO) system [31]. Thus, we adopt the linear receivers for the BS and RSUs to reduce the computation complexity [32]. Specifically, the BS is equipped with a ZF receiver and RSUs with linear receivers, respectively.

2.1. Channel Model of the Multi-Antenna RSUs

In our vehicular system, the RSUs are equipped with $N^{\mathrm{R}}$ antennas to serve many vehicles at the same time. Assuming that the number of vehicles in the system $K=K_1+K_2+\cdots +K_L$, with $K_l$ denoting the number of vehicles in RSU l, the received signal at the l-th RSU can be defined as

$${\mathit{Y}}_l^{\mathrm{R}}={\left({\mathit{P}}_l^{\mathrm{R}}\right)}^{1/2}{\mathit{G}}_l^{\mathrm{R}}{\mathit{X}}_l^{\mathrm{R}}+{\mathit{Z}}_l^{\mathrm{R}},$$

(1)

where the diagonal matrix ${\mathit{P}}_l^{\mathrm{R}}$ is the transmission power matrix of the vehicles in RSU l, and ${\mathit{P}}_l^{\mathrm{R}}=\mathrm{diag}[p_1^{\mathrm{R}},p_2^{\mathrm{R}},\dots ,p_{K_l}^{\mathrm{R}}]$. $p_k^{\mathrm{R}}$ are the transmit power of the k-th vehicle in the l-th RSU, with indexes k denoting the k-th vehicle in the l-th RSU. ${\mathit{G}}_l^{\mathrm{R}}$ and ${\mathit{X}}_l^{\mathrm{R}}$ denote the channel matrix and the transmitted signal matrix between the vehicles and the l-th RSU, respectively. ${\mathit{Z}}_l^{\mathrm{R}}$ denotes the additive complex Gaussian white noise at the l-th RSU. The channel matrix between the vehicles and the l-th RSU can be defined as

$${\mathit{G}}_l^{\mathrm{R}}=[{\mathit{g}}_{1,l}^{\mathrm{R}},{\mathit{g}}_{2,l}^{\mathrm{R}},\dots ,{\mathit{g}}_{K_l,l}^{\mathrm{R}}]={\mathit{H}}_l^{\mathrm{R}}{\left({{\mathit{D}}_l}^{\mathrm{R}}\right)}^{1/2},$$

(2)

where ${\mathit{H}}_l^{\mathrm{R}}=[{\mathit{h}}_{1,l}^{\mathrm{R}},{\mathit{h}}_{2,l}^{\mathrm{R}},\dots ,{\mathit{h}}_{K_l,l}^{\mathrm{R}}]$ and diagonal matrix ${\mathit{D}}_l^{\mathrm{R}}=\mathrm{diag}[{\beta }_{1,l}^{\mathrm{R}},{\beta }_{2,l}^{\mathrm{R}},\dots ,{\beta }_{K_l,l}^{\mathrm{R}}]$ are the small-scale and large-scale fading matrix between the vehicles and the l-th RSU, respectively. ${\mathit{h}}_{k,l}^{\mathrm{R}}$ and ${\beta }_{k,l}^{\mathrm{R}}$ denote the small-scale and large-scale fading between the k-th vehicle and the l-th RSU, respectively. Thus, the channel vector between the vehicle k and the l-th RSU is given by ${\mathit{g}}_{k,l}^{\mathrm{R}}=\sqrt{{\beta }_{k,l}^{\mathrm{R}}}{\mathit{h}}_{k,l}^{\mathrm{R}}$.

Interference in the MIMO system will increase the analysis complexity of the multi-vehicle V2X system [33]. Thus, the liner equalizers such as the MF and the ZF receivers are applied to reduce the analysis complexity of the system [32]. The MF receiving matrix is represented by ${\mathit{A}}_l^{\mathrm{MF}}={\left({\mathit{G}}_l^{\mathrm{R}}\right)}^{\mathrm{H}}$. Thus, the received signal at the l-th RSU which is equipped with an MF receiver, is given by

$${\mathit{Y}}_l^{\mathrm{MF}}={\mathit{A}}_l^{\mathrm{MF}}{\mathit{X}}_l^{\mathrm{R}}={\left({\mathit{G}}_l^{\mathrm{R}}\right)}^{\mathrm{H}}{\mathit{X}}_l^{\mathrm{R}}.$$

(3)

The signal-to-interference-plus-noise ratio (SINR) for vehicle k at the l-th RSU can be defined as

$${\gamma }_{k,l}^{\mathrm{MF}}={\displaystyle \frac{p_k^{\mathrm{R}}\left|\right({\mathit{g}}_{k,l}^{\mathrm{R}}{)}^{\mathrm{H}}{\mathit{g}}_{k,l}^{\mathrm{R}}{|}^2}{I_{k,l}^{\mathrm{MF}}}},$$

(4)

where $I_{k,l}^{\mathrm{MF}}={\displaystyle \sum _{i=1,i\ne k}}{p}_i^{\mathrm{R}}\left|\right({\mathit{g}}_{k,l}^{\mathrm{R}}{)}^{\mathrm{H}}{\mathit{g}}_{i,l}^{\mathrm{R}}{|}^2+{\sigma }^2\left|\right|{\mathit{g}}_{k,l}^{\mathrm{R}}{\left|\right|}^2$ is the interference between the k-th vehicle and the l-th RSU equipped with the MF receiver.

The ZF receiving matrix is represented by ${\mathit{A}}_l^{\mathrm{ZF}}={\left({\left({\mathit{G}}_l^{\mathrm{R}}\right)}^{\mathrm{H}}{\mathit{G}}_l^{\mathrm{R}}\right)}^{-1}{\left({\mathit{G}}_l^{\mathrm{R}}\right)}^{\mathrm{H}}$. Thus, the received signal at the l-th RSU, which is equipped with a ZF receiver, can be defined as

$${\mathit{Y}}_l^{\mathrm{ZF}}={\mathit{A}}_l^{\mathrm{ZF}}{\mathit{X}}_l^{\mathrm{R}}={\left({\left({\mathit{G}}_l^{\mathrm{R}}\right)}^{\mathrm{H}}{\mathit{G}}_l^{\mathrm{R}}\right)}^{-1}{\left({\mathit{G}}_l^{\mathrm{R}}\right)}^{\mathrm{H}}{\mathit{X}}_l^{\mathrm{R}}.$$

(5)

The SINR for vehicle k at the l-th RSU is given by

$${\gamma }_{k,l}^{\mathrm{ZF}}={\displaystyle \frac{p_k^{\mathrm{R}}}{{\sigma }^2{\left[{\left({\left({\mathit{G}}_l^{\mathrm{R}}\right)}^{\mathrm{H}}{\mathit{G}}_l^{\mathrm{R}}\right)}^{-1}\right]}_{k,k}}}.$$

(6)

2.2. Communication Model of the Multi-Antenna BS

In our vehicular system, the BS is equipped with an $N^{\mathrm{B}}$-antenna to serve the vehicles and RSUs. Therefore, all the RSUs and vehicles are users of the BS. Let M denote the BS’s users number, which can be regarded as a combination of L RSUs and K vehicles, i.e., $M=L+K$. Thus, the received signal at the BS can be defined as

$${\mathit{Y}}^{\mathrm{B}}={\left({\mathit{P}}^{\mathrm{B}}\right)}^{1/2}{\mathit{G}}^{\mathrm{B}}{\mathit{X}}^{\mathrm{B}}+{\mathit{Z}}^{\mathrm{B}},$$

(7)

where the diagonal matrix ${\mathit{P}}^{\mathrm{B}}$ is the transmit power matrix of the users, which can be defined as ${\mathit{P}}^{\mathrm{B}}=\mathrm{diag}[p_1^{\mathrm{B}},p_2^{\mathrm{B}},\dots ,p_M^{\mathrm{B}}]$, with $p_m^{\mathrm{B}}$ denoting the transmit power of the m-th user. ${\mathit{G}}^{\mathrm{B}}$ and ${\mathit{X}}^{\mathrm{B}}$ denote the channel matrix and the transmitted signal matrix between the users and the BS, respectively. ${\mathit{Z}}^{\mathrm{B}}$ denotes the additive complex Gaussian white noise at the BS. The channel matrix between the users and BS is given by

$${\mathit{G}}^{\mathrm{B}}=[{\mathit{g}}_1^{\mathrm{B}},{\mathit{g}}_2^{\mathrm{B}},\dots ,{\mathit{g}}_M^{\mathrm{B}}]={\mathit{H}}^{\mathrm{B}}{\left({\mathit{D}}^{\mathrm{B}}\right)}^{1/2},$$

(8)

where ${\mathit{H}}^{\mathrm{B}}=[{\mathit{h}}_1^{\mathrm{B}},{\mathit{h}}_2^{\mathrm{B}},\dots ,{\mathit{h}}_M^{\mathrm{B}}]$ and diagonal matrix ${\mathit{D}}^{\mathrm{B}}=\mathrm{diag}[{\beta }_1^{\mathrm{B}},{\beta }_2^{\mathrm{B}},\dots ,{\beta }_M^{\mathrm{B}}]$ are the small-scale and large-scale fading matrix between the users and BS, respectively. ${\mathit{h}}_m^{\mathrm{B}}$ and ${\beta }_m^{\mathrm{B}}$ denote the small-scale and large-scale fading between the m-th user and the BS. Thus, the channel vector between the m-th user and the BS is given by ${\mathit{g}}_m^{\mathrm{B}}=\sqrt{{\beta }_m^{\mathrm{B}}}{\mathit{h}}_m^{\mathrm{B}}$. In addition, in the vehicular system, the BS is equipped with a ZF receiver to eliminate the interference and provide higher communication rates of the V2B and infrastructure-to-BS (I2B) communication links [33]. The ZF receiving matrix can be defined as ${\mathit{A}}^{\mathrm{ZF}}={\left({\left({\mathit{G}}^{\mathrm{B}}\right)}^{\mathrm{H}}\mathit{G}\right)}^{-1}{\left({\mathit{G}}^{\mathrm{B}}\right)}^{\mathrm{H}}$. Thus, the received signal at the BS can be defined as

$${\mathit{Y}}^{\mathrm{ZF}}={\mathit{A}}^{\mathrm{ZF}}{\mathit{Y}}^{\mathrm{B}}={\left({\left({\mathit{G}}^{\mathrm{B}}\right)}^{\mathrm{H}}\mathit{G}\right)}^{-1}{\left({\mathit{G}}^{\mathrm{B}}\right)}^{\mathrm{H}}{\mathit{X}}^{\mathrm{B}}.$$

(9)

The SINR for user m at the BS can be defined as

$${\gamma }_m^{\mathrm{ZF}}={\displaystyle \frac{p_m^{\mathrm{B}}}{{\sigma }^2{\left[{\left({\left({\mathit{G}}^{\mathrm{B}}\right)}^{\mathrm{H}}{\mathit{G}}^{\mathrm{B}}\right)}^{-1}\right]}_{m,m}}}.$$

(10)

3. Computing Model of MEC-Assisted V2X Communication System

The processing capabilities of the vehicles, RSUs and BS executing one bit of data are $f^{\mathrm{V}}$, $f^{\mathrm{R}}$ and $f^{\mathrm{B}}$ (in cycles/bit) CPU cycles, respectively. The data packet size generated by the vehicle is $D_k$. In our vehicular system, the data model adopts a partial offloading model. Thus, the data packet can be regarded as a combination of local computation data $D_k^{\mathrm{L}}$ and offloading computation data $D_k^{\mathrm{O}}$, i.e., $D_k=D_k^{\mathrm{L}}+D_k^{\mathrm{O}}$. Let $D_{k,l}^{\mathrm{O},\mathrm{R}}$ and $D_k^{\mathrm{O},\mathrm{B}}$ denote the data offloading to the l-th RSU and the BS from the k-th vehicle, respectively. Thus,

$$D_k^{\mathrm{O}}=\left\{\begin{array}{cc}{D}_{k,l}^{\mathrm{O},\mathrm{R}},\hfill & x_k=0,\hfill \\ D_k^{\mathrm{O},\mathrm{B}},\hfill & x_k=1,\hfill \end{array}\right.$$

(11)

where $x_k$ denotes the server association of vehicle k. $x_k=0$ and $x_k=1$ represent the vehicle offloading tasks to the RSUs or the BS, respectively. It should be noticed that, when the k-th vehicle offloads a task to the RSUs, $D_k^{\mathrm{O},\mathrm{B}}=0$. Similarly, when the k-th vehicle offloads a task to the BS, $D_{k,l}^{\mathrm{O},\mathrm{R}}=0$.

3.1. Local Computing Model

When the k-th vehicle offloads partial data to MEC servers, the vehicle needs to compute partial tasks locally. The computation delay and energy (in Joules, J) at vehicle k can be defined as [19]

$$T_k^{\mathrm{L}}={\displaystyle \frac{D_k^{\mathrm{L}}}{f_k^{\mathrm{V}}}},E_k^{\mathrm{L}}=c_0{D}_k^{\mathrm{L}}(f_k^{\mathrm{V}}{)}^2,$$

(12)

where $f_k^{\mathrm{V}}$ and $c_0$ denote the processing capability of the k-th vehicle and the energy coefficient depending on the chip architecture [34], respectively.

3.2. RSU Computing Model

The k-th vehicle offloading computation-intensive tasks to the l-th RSU for edge computing consists of three steps: (1) the vehicle offloads tasks to the associated RSU via a V2I communication link, and the RSU computes the offloaded data, (2) the RSU transmits the computed data to the BS via an I2B communication link and (3) the BS transmits downlink data. The feedback data packet size is small; thus, the delay and energy consumption of downlink transmission are ignored.

(1)

V2I communication link

The MF and ZF receivers are adopted to provide higher communication rates of the V2I communication links. When the RSUs are equipped with the MF receivers, according to Equation (4), the corresponding communication rate between the k-th vehicle and the specific l-th RSU can be written as

$$R_{k,l}^{\mathrm{V}2\mathrm{I}}=B^{\mathrm{V}2\mathrm{I}}{log}_2(1+{\displaystyle \frac{p_k^{\mathrm{V}}\left|\right({\mathit{g}}_{k,l}^{\mathrm{R}}{)}^{\mathrm{H}}{\mathit{g}}_{k,l}^{\mathrm{R}}{|}^2}{{\displaystyle \sum _{i=1,i\ne k}}{p}_i^{\mathrm{V}}\left|\right({\mathit{g}}_{k,l}^{\mathrm{R}}{)}^{\mathrm{H}}{\mathit{g}}_{i,l}^{\mathrm{R}}{|}^2+{\sigma }^2\left|\right|{\mathbf{g}}_{k,l}^{\mathrm{R}}{\left|\right|}^2}}),$$

(13)

where $B^{\mathrm{V}2\mathrm{I}}$ is the bandwidth of the V2I communication link and $p_k^{\mathrm{V}}$ is the transmit power of the k-th vehicle. When the RSUs are equipped with the ZF receivers, according to Equation (6), the corresponding communication rate between the vehicle k and the specific l-th RSU can be defined as

$$R_{k,l}^{\mathrm{V}2\mathrm{I}}=B^{\mathrm{V}2\mathrm{I}}{log}_2(1+{\displaystyle \frac{p_k^{\mathrm{V}}}{{\sigma }^2{[{({({\mathbf{G}}_l^{\mathrm{R}})}^{\mathrm{H}}{\mathbf{G}}_l^{\mathrm{R}})}^{-1}]}_{k,k}}}).$$

(14)

Thus, the transmission delay and energy between vehicle k and the l-th RSU can be defined as [21,35]

$$T_{k,l}^{\mathrm{V}2\mathrm{I}}={\displaystyle \frac{D_{k,l}^{\mathrm{O},\mathrm{R}}}{R_{k,l}^{\mathrm{V}2\mathrm{I}}}},E_{k,l}^{\mathrm{V}2\mathrm{I}}=p_k^V{T}_{k,l}^{\mathrm{V}2\mathrm{I}}.$$

(15)

When the k-th vehicle offloads partial data $D_{k,l}^{\mathrm{O},\mathrm{R}}$ to the l-th RSU for edge computing, the computation delay and energy at the l-th RSU is given by [21]

$$T_{k,l}^{\mathrm{O},\mathrm{R}}={\displaystyle \frac{D_{k,l}^{\mathrm{O},\mathrm{R}}}{f_l^{\mathrm{R}}}},E_{k,l}^{\mathrm{O},\mathrm{R}}=c_1{D}_{k,l}^{\mathrm{O},\mathrm{R}}{\left(f_l^{\mathrm{R}}\right)}^2,$$

(16)

where $f_l^{\mathrm{R}}$ and $c_1$ denote the processing capability of the l-th RSU and the energy coefficient.

(2)

I2B communication link

The ZF receiver is adopted to eliminate the interference so that the communication rate of the I2B communication link can be improved. According to Equation (10), the communication rate between the l-th RSU and the BS is given by

$$R_l^{\mathrm{I}2\mathrm{B}}=B^{\mathrm{I}2\mathrm{B}}{log}_2(1+{\displaystyle \frac{p_l^{\mathrm{R}}}{{\sigma }^2{[{({({\mathbf{G}}^{\mathrm{B}})}^{\mathrm{H}}{\mathbf{G}}^{\mathrm{B}})}^{-1}]}_{K+l,K+l}}}),$$

(17)

where $B^{\mathrm{I}2\mathrm{B}}$ denotes the bandwidth of the I2B communication link, and $p_l^{\mathrm{R}}$ denotes the transmit power of the l-th RSU. Let $D^{\mathrm{prc}}$ denote the tasks after computing. The transmission delay and energy between RSU l and the BS are given by

$$T_l^{\mathrm{I}2\mathrm{B}}={\displaystyle \frac{D^{\mathrm{prc}}}{R_l^{\mathrm{I}2\mathrm{B}}}},E_l^{\mathrm{I}2\mathrm{B}}=p_l^{\mathrm{R}}{T}_l^{\mathrm{I}2\mathrm{B}}.$$

(18)

The delay for the k-th vehicle offloading tasks to the l-th RSU for edge computing includes $T_{k,l}^{\mathrm{V}2\mathrm{I}}$, $T_{k,l}^{\mathrm{O},\mathrm{R}}$ and $T_l^{\mathrm{I}2\mathrm{B}}$, and energy consumption includes $E_{k,l}^{\mathrm{V}2\mathrm{I}}$, $E_{k,l}^{\mathrm{O},\mathrm{R}}$ and $E_l^{\mathrm{I}2\mathrm{B}}$. Therefore, the delay and energy consumption for the k-th vehicle offloads tasks to the l-th RSU for edge computing can be written as

$$T_{k,l}^{\mathrm{R}}=T_{k,l}^{\mathrm{V}2\mathrm{I}}+T_{k,l}^{\mathrm{O},\mathrm{R}}+T_l^{\mathrm{I}2\mathrm{B}},E_{k,l}^{\mathrm{R}}=E_{k,l}^{\mathrm{V}2\mathrm{I}}+E_{k,l}^{\mathrm{O},\mathrm{R}}+E_l^{\mathrm{I}2\mathrm{B}}.$$

(19)

3.3. BS Processing Model

Vehicle k offloading tasks to the BS for edge computing consists of two steps: (1) the vehicle offloads task to the BS via the V2B communication link and the BS computes the offloaded data, and (2) the BS transmits downlink data. The delay and energy consumption of downlink transmission are ignored. The ZF receiver is adopted to eliminate interference so that the communication rate of the V2B communication link can be improved. According to Equation (10), the corresponding communication rate between vehicle k and the BS is given by

$$R_k^{\mathrm{V}2\mathrm{B}}=B^{\mathrm{V}2\mathrm{B}}{log}_2(1+{\displaystyle \frac{p_k^{\mathrm{V}}}{{\sigma }^2{[{({({\mathbf{G}}^{\mathrm{B}})}^{\mathrm{H}}{\mathbf{G}}^{\mathrm{B}})}^{-1}]}_{k,k}}}).$$

(20)

Thus, the transmission delay and energy between the k-th vehicle and the BS can be defined as

$$T_k^{\mathrm{V}2\mathrm{B}}={\displaystyle \frac{D_k^{\mathrm{O},\mathrm{B}}}{R_k^{\mathrm{V}2\mathrm{B}}}},E_k^{\mathrm{V}2\mathrm{B}}=p_k^{\mathrm{V}}{T}_k^{\mathrm{V}2\mathrm{B}}.$$

(21)

When the k-th vehicle offloads partial data $D_k^{\mathrm{O},\mathrm{B}}$ to the BS for edge computing, the computation delay and energy at the BS is given by

$$T_k^{\mathrm{O},\mathrm{B}}={\displaystyle \frac{D_k^{\mathrm{O},\mathrm{B}}}{f^{\mathrm{B}}}},E_k^{\mathrm{O},\mathrm{B}}=c_1{D}_k^{\mathrm{O},\mathrm{B}}{\left(f^{\mathrm{B}}\right)}^2,$$

(22)

where $c_1$ is the energy coefficient.

The delay for vehicle k offloading tasks to the BS includes $T_k^{\mathrm{V}2\mathrm{B}}$ and $T_k^{\mathrm{O},\mathrm{B}}$, and energy includes $E_k^{\mathrm{V}2\mathrm{B}}$ and $E_k^{\mathrm{O},\mathrm{B}}$. Thus, the delay and energy consumption for vehicle k offloading tasks to the BS for edge computing can be defined as

$$T_k^{\mathrm{B}}=T_k^{\mathrm{V}2\mathrm{B}}+T_k^{\mathrm{O},\mathrm{B}},E_k^{\mathrm{B}}=E_k^{\mathrm{V}2\mathrm{B}}+E_k^{\mathrm{O},\mathrm{B}}.$$

(23)

Thus, the delay and energy consumption of the k-th vehicle is related to server association $\{x_k=0,1\}$, offloading ratio and transmit power, and can be defined as

$$T_k^{\mathrm{total}}=\left\{\begin{array}{cc}{T}_k^{\mathrm{L}}+T_{k,l}^{\mathrm{R}},\hfill & x_k=0,\hfill \\ T_k^{\mathrm{L}}+T_k^{\mathrm{B}},\hfill & x_k=1,\hfill \end{array}\right.E_k^{\mathrm{total}}=\left\{\begin{array}{cc}{E}_k^{\mathrm{L}}+E_{k,l}^{\mathrm{R}},\hfill & x_k=0,\hfill \\ E_k^{\mathrm{L}}+E_k^{\mathrm{B}},\hfill & x_k=1.\hfill \end{array}\right.$$

(24)

3.4. Optimization Problem

In order to reduce the energy consumption of the V2X system while satisfying the requirements of communication delay, the energy consumption minimization problem is formulated under the delay constraint, which can be written as follows:

$$\begin{array}{cc}\hfill \left(\mathrm{P}1\right):& \underset{x_k,D_k^{\mathrm{O}},p_k^{\mathrm{V}}}{min}\sum _{k=0}^K{E}_k^{\mathrm{total}}\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill s.t.& T_k^{\mathrm{total}}\le T_{\max },k\in \mathcal{K},\hfill \end{array}$$

(25a)

$$\begin{array}{cc}\hfill & x_k\in \{0,1\},k\in \mathcal{K},\hfill \end{array}$$

(25b)

$$\begin{array}{cc}\hfill & 0\le p_k^{\mathrm{V}}\le P_{\max }^{\mathrm{V}},k\in \mathcal{K},\hfill \end{array}$$

(25c)

$$\begin{array}{cc}\hfill & 0\le D_k^{\mathrm{O}}\le D_k,k\in \mathcal{K}.\hfill \end{array}$$

(25d)

To ensure basic communication, the first constraint limits that the processing delay of the tasks generated by vehicle k must be less than the maximum delay. The second constraint ensures that the tasks can be offloaded to the RSUs or BS for edge computing. Constraint (25c) is to limit $p_k^{\mathrm{V}}$ to not be greater than the maximum transmit power $P_{\max }^{\mathrm{V}}$. Constraint (25d) restricts that, for the vehicles that adopt the partial offloading model, the offloading quantity must be not greater than the maximum value $D_k$. It is noted that, because of the coupling of server association, transmit power and offloading ratio, (P1) is a non-convex problem and difficult to solve by using the traditional convex optimization schemes while satisfying the delay and power constraints. Thus, the DRL algorithm is adopted to solve (P1), which can find the globally optimal or near-optimal solution of the complex optimization problem [36].

4. Proposed Scheme for Solving the Problem (P1)

In order to solve the highly non-convex problem, the DRL algorithm is proposed by optimizing the server association, transmit power and offloading ratio jointly. In addition, the improved KNN algorithm is used to assign vehicles and reduce the action space dimensions and complexity of the DRL algorithm.

4.1. Vehicle Grouping Based on the Improved KNN Algorithm

The optimization problem (P1) is aimed at minimizing the energy consumption of the MEC-assisted multi-vehicle V2X system while satisfying the delay constraint, which is related to the transmission rate and SINR of the wireless communication links. A higher SINR can effectively improve the communicate rate in the V2X system [37]. Thus, in order to minimize energy consumption, the SINR must be maximized. According to Equations (4), (6) and (10), the SINR is related to the vehicle’s transmit power in the same group. Therefore, assigning vehicles that are located within the coverage area of several RSUs to an RSU with a smaller user number can provide a higher SINR and reduce energy consumption.

The model-free KNN [38] method is one of the commonly used classification algorithms. It calculates the distance between initial center points and all training samples, and then obtains the test sample’s nearest neighbors sample and groups them together. But, it is hard for KNNs to correctly classify overlapping samples [39]. Removing an overlapped sample from the training set can increase the accuracy of the KNN classifier [40]. In this paper, the locations of RSUs and vehicles are regarded as initial center points and samples, respectively, and the improved KNN algorithm [41] is adopted to assign the vehicles within the coverage area of multiple RSUs to a specific RSU. The distance between the vehicle and RSUs is denoted by $d_{k,l}$, which can be defined as

$$d_{k,l}=\sqrt{{(x_k^{\mathrm{V}}-x_l^{\mathrm{R}})}^2+(y_k^{\mathrm{V}}-y_l^{\mathrm{R}})},$$

(26)

where $(x_k^{\mathrm{V}},y_k^{\mathrm{V}})$ and $(x_l^{\mathrm{R}},y_l^{\mathrm{R}})$ are the position of vehicle k and the l-th RSU, respectively. The k-th vehicle can communicate with the l-th RSU when $d_{k,l}\le R_{\mathrm{RSU}}$, with $R_{\mathrm{RSU}}$ denoting the radius of the l-th RSU. After calculating the distance between vehicles and RSUs, the improved KNN algorithm assigns a vehicle to the RSU with a smaller number of vehicles to decrease system energy consumption and computational complexity. The improved KNN algorithm is shown in Algorithm 1.

Algorithm 1 The Improved K-nearest Neighbors Algorithm

1: Initial: The location of vehicles and RSUs. The radius of RSUs and the number of vehicles in each RSU. 2: for vehicle $k=1:K$ do Initial the set ${\mathcal{I}}_k$ and number $i_k$ of RSUs that can be selected by the k-th vehicle. 3:       for RSU $l=1:L$ do Calculate the distance $d_{k,l}$ in (26). If $d_{k,l}\le R_{\mathrm{RSU}}$, ${\mathcal{I}}_k$ appends RSU l and $i_k+1$. 4:       end for 5:       if $i_k>1$ then Assigning vehicle k to the l-th RSU which is in ${\mathcal{I}}_k$ and has a lower number of vehicles. 6:       else Assigning vehicle k to corresponding RSU l which makes $d_{k,l}\le R_{\mathrm{RSU}}$. 7:       end if 8: end for

4.2. Joint-Optimized Server Association, Offloading Ratio and Transmission Power Scheme Based on DRL

DRL learns from the environment-to-action mappings and develops an optimal strategy via trial and error [24]; according to the reward function, it trains agents to make quick decisions and optimal policies. Due to the non-convexity of objective function (P1), we propose an MADRL scheme. Agents interact with the environment and update the learning strategies. Therefore, the agent can choose the optimal action to minimize the total energy consumption of the V2X system while satisfying the delay constraint.

(1)

Agent: In the system, vehicles work as agents. They know their own information—since the BS and RSUs have no direct link to exchange the information, it is difficult to make the optimal decisions for offloading and power. Thus, the training is performed in the optimization problem to make the adjustment to the dynamic environment. We choose the vehicle as the agent to make the decision in the uplink link, and the BS and RSUs will send the corresponding information to the associated vehicles—i.e., the packet size of the task $D_k$ generated by vehicle k, set ${\mathcal{I}}_k$ and the channel state information.

(2)

Action space: Due to the limited communication resources in the system, the server association, offloading ratio and transmit power affect the communication rate and determine the communication energy consumption. Thus, the action space of vehicle k comprises the server association, offloading ratio and transmit power. The server association action space of the k-th vehicle is given as $A_1=\{x_k=0,1\}$. The offloading and power control action space are given as $A_2=\{\frac{D_k}{d_p},\frac{2D_k}{d_p},\frac{3D_k}{d_p},\cdots ,D_k\}$ and $A_3=\{\frac{P{\max }^{\mathrm{V}}}{d_l},\frac{2P_{\max }^{\mathrm{V}}}{d_l},\frac{3P_{\max }^{\mathrm{V}}}{d_l},\cdots ,P_{\max }^{\mathrm{V}}\}$. $d_p$ and $d_l$ represent the length of offloading and the power control action space. Therefore, the k-th agent can choose action from the action space ${\mathit{a}}_k\in A=\{A_1,A_2,A_3\},k\in \mathcal{K}$.

(3)

State space: According to the communication model established in the system, the system energy consumption is related to transmission power, rate and delay; therefore, the state space comprises the transmit power $p_k^{\mathrm{V}}$, transmission rate $R_k^{\mathrm{des}}$ and communication delay $T_k^{\mathrm{total}}$ in Equation (24). When $x_k=0$, the transmission rate $R_k^{\mathrm{des}}$ is calculated by (14), and when $x_k=1$, the transmission rate $R_k^{\mathrm{des}}$ is calculated by (20). Thus, the state of vehicle k can be expressed as ${\mathbf{S}}_k=[P_k^{\mathrm{V}},R_k^{\mathrm{des}},T_k^{\mathrm{total}}],k\in \mathcal{K}$.

(4)

Reward: In the DRL algorithm, the agents are trained to adjust the action choice strategy and obtain the expected value by accumulating the maximum reward return. Aiming to minimize the total energy consumption of the V2X system and meet the delay constraint, the reward function of vehicle k is modeled as

$$r_k=\left\{\begin{array}{cc}{\displaystyle \frac{b}{{\sum }_{k=0}^K{E}_k^{\mathrm{total}}}},\hfill & \mathrm{if}\mathrm{satisfied}\left(25\mathrm{a}\right),\hfill \\ -T_k^{\mathrm{total}},\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(27)

where b is a positive number. When the corresponding delay constraint can be satisfied, the agent will obtain a greater reward related with the energy consumption of the V2X system. When the corresponding delay constraint cannot be satisfied, the agent will obtain a negative reward related with the delay. In order to obtain the long-term rewards, agents are trained to choose the action that can minimize the energy consumption and meet the delay constraint.

The detailed design framework based on the deep Q-network (DQN) joint server association, offloading ratio and transmit power optimization scheme in the MEC-assisted V2X system is shown in Figure 2. The framework of the DQN contains two neural networks and an experience replay buffer, and vehicles act as agents interacting with the environment. On the current environment state ${\mathit{s}}_k$, the k-th agent chooses and executes action ${\mathit{a}}_k$ and observes the next state ${\mathit{s}}_k^{\prime }$ and the greatest discount reward $r_k$. The experience replay buffer is used to store $<{\mathit{s}}_k,{\mathit{a}}_k,r_k,{\mathit{s}}_k^{\prime }>$. DQN trains the agent by randomly selecting a batch of samples from the experience replay buffer during the exploration cycle. Specifically, the main network takes current state ${\mathit{s}}_k$ and action ${\mathit{a}}_k$ from each data sample to obtain the predicted Q-value $Q({\mathit{s}}_k,{\mathit{a}}_k)$ of the particular action ${\mathit{a}}_k$. The target network takes next state ${\mathit{s}}_k^{\prime }$ and predicts the action ${\mathit{a}}_k^{\prime }$ with the largest Q-value from all actions that can be taken in that state, obtaining the target Q-value $Q^{\prime }({\mathit{s}}_k^{\prime },{\mathit{a}}_k^{\prime })$. The predicted and target Q-value is used to calculate the loss function and adjust the parameters of the neural network.

The loss function of the k-th agent is used to obtain the best Q-function by adjusting the neural network parameters, which is given by [42]

$$\mathbb{L}\left({\theta }_i\right)={(r_k+\gamma (\max Q^{\prime }({\mathit{s}}_k^{\prime },{\mathit{a}}_k^{\prime }|{\theta }_i^{\prime })-Q({\mathit{s}}_k,{\mathit{a}}_k|{\theta }_i)))}^2,$$

(28)

where ${\theta }_i$ and ${\theta }_i^{\prime }$ denote the weights parameters of the main network and target network during the i-th training, respectively. $\gamma \in \left[0,1\right)$ is a discount factor that can weight the future rewards. In order to obtain the maximum long-term reward, the Q-function is updated based on the Bellman equation, which is written as

$$\begin{array}{c}\hfill Q({\mathit{s}}_k,{\mathit{a}}_k)=(1-\alpha )Q({\mathit{s}}_k,{\mathit{a}}_k)+\alpha (r_k+\gamma \max \left(Q^{\prime }({\mathit{s}}_k^{\prime },{\mathit{a}}_k^{\prime })\right)),\end{array}$$

(29)

where $\alpha$ denotes the learning rate, which directly determines how quickly the agent can adapt to the environment. DRL combines reinforcement learning with the DQN neural network [43] and makes agents observe from the environment and intelligently select the action that interacts with the environment. With repeated iterations, the agent will choose a better server association, offloading ratio and transmit power to minimize the energy consumption of the MEC-assisted multi-vehicle V2X communication system. The DRL-based scheme is show in Algorithm 2.

Algorithm 2 DRL-based Multi-agent Transmit Power and Offloading Optimization Scheme

1: Inputs: The sate $\mathit{s}$ of the vehicles. 2: Initial: DQN-networks for all agents, including of policy strategy $\pi (\mathit{s},\mathit{a})$, $Q(\mathit{s},\mathit{a})$, and $\{\alpha ,\gamma ,ϵ,\theta \}$. 3: for each iteration episode do 4:     Agents observe state ${\mathit{s}}_t$, randomly selecting an action with probability $ϵ$, or $arg\underset{{\mathit{a}}_t\in \mathcal{A}}{\max }\left\{Q({\mathit{s}}_t,{\mathit{a}}_t)\right\}$ from the action space with probability $1-ϵ$. Interacting with the environment, obtaining the next state ${\mathit{s}}_t^{\prime }$ and reward (27). Storing $<{\mathit{s}}_t,{\mathit{a}}_t,r_t,{\mathit{s}}_t^{\prime }>$ in the experience replay buffer. 5:     for each training step do 6:         Agents select a random mini-batch data from the experience replay buffer, and transmit it to the neural network. The estimate network calculate the predicted Q-value according to ${\mathit{s}}_t,{\mathit{a}}_t$, the target network calculate the target Q-value according to $r_t,{\mathit{s}}_t^{\prime }$. According to the predicted and target Q-value, computing the loss function in (28), update the weights ${\theta }_i$ to the target network and Q-value $Q({\mathit{s}}_t,{\mathit{a}}_t)$ in (29). 7:     end for 8: end for 9: Outputs: The action $\mathit{a}$ of the vehicles used to optimize the offloading decision, offloading ratio and power control.

4.3. DRL Algorithm Complexity Analysis

Computational complexity is critical for evaluating the algorithm performance. For the Q-learning algorithm, the Q-table of agents consists of ${\left(\right|\mathit{S}\left|\right)}^{\mathrm{K}}$ rows with $\left|\mathit{S}\right|=3$ denoting the length of the environment states and ${\left(\right|\mathit{a}|}_p{)}^{\mathrm{K}}$ columns with ${\left|\mathit{a}\right|}_p=50$ denoting distinct possible agent actions in our simulation. Thus, the whole Q-learning algorithm is ${O\left(\right(\left|\mathit{S}\right|)}^{\mathrm{K}}·{\left(\right|\mathit{a}|}_p{)}^{\mathrm{K}})$, and the complexity of the Q-learning algorithm is primarily determined by the dimensions of state space, the distinct possible agent actions and the number of agents. From Algorithm 2, it can be known that the whole MADRL algorithm procedure mainly contains two parts:

(1)

Calculate the reward function: Agents, according to the state, select the beneficial action, interact with the environment and obtain the reward. Thus, the computational complexity of an agent calculating the reward is $O\left(\right|{\mathit{s}}_t\left|\right)$, with $|{\mathit{s}}_t|$ denoting the length of the state space for the t-th training step.

(2)

Select the beneficial action: For each agent, the numbers of layers in the DQN network and neurons in each layer are considered. For the DQN network, the number of layers in the DQN network is M, and neurons in the m-th layer is $U_m$. Thus, the computational complexity of the m-th layer is $O(U_{m-1}{U}_m+U_m{U}_{m+1})$, and, for the t-th step, the computational complexity of an agent selecting the beneficial action is $O_c=O\left(\right|s_t|·U_2+{\sum }_{m=3}^M(U_{m-1}{U}_m+U_m{U}_{m+1})+U_{M-1}·|a_t\left|\right)$, with $|{\mathit{a}}_t|$ denoting the length of the action space for the t-th training step. Therefore, the computational complexity of an agent is $O(O_r+O_c)$. The computational complexity for a complete episode is $O\left(N_{\mathrm{step}}(O_r+O_c)\right)$, where $N_{\mathrm{step}}$ is the number of steps in one episode. The computational complexity for the whole algorithm is $O(N_{\mathrm{episode}}·N_{\mathrm{step}}(O_r+O_c))$, where $N_{\mathrm{episode}}$ is the number of iteration episodes. In this paper, all agents have the same DQN network, and the computational complexity of all agents is $O\left(K(N_{\mathrm{episode}}·N_{\mathrm{step}}(O_r+O_c))\right)$. The complexity of the proposed algorithm is primarily determined by the network architectures of the DQN network structure and the dimensions of the state and action space. In addition, the complexity of the Q-learning algorithm is primarily determined by the number of agents, and the implementation of Q-learning is more complex compared to the MADRL algorithm in the multi-vehicle V2X communication system. Thus, the proposed MADRL algorithm is more suitable for solving the optimization problem (P1) in this work.

5. Simulation Results

Numerical results are provide to prove the effectiveness of the proposed DRL scheme in this section. Unless otherwise stated, the BS is located at the origin and its radius is 400 m. The number of RSUs $L=4$ and the radius of the RSUs is 200 m. The number of vehicles $K=6$. Note that the vehicles are in the coverage of the RSUs, which are in the coverage of the BS. Other parameters are shown in

Table 2. Simulation parameters.

Symbol	Description	Value
$T_{\max }$	Maximum vehicle delay	8 s
$f^{\mathrm{V}}$	Processing capability of the vehicles	0.1 Mbit/s
$f^{\mathrm{R}}$	Processing capability of the RSUs	5 Mbit/s
$f^{\mathrm{BS}}$	Processing capability of the BS	10 Mbit/s
$D_{\max }$	Data packet	2 M
$D^{\mathrm{prc}}$	Processed data packet size	1 M
$B^{\mathrm{V}2\mathrm{I}}$	Bandwidth of the V2I communication link	1 MHz
$B^{\mathrm{I}2\mathrm{B}}$	Bandwidth of the I2B communication link	5 MHz
$B^{\mathrm{V}2\mathrm{B}}$	Bandwidth of the V2B communication link	5 MHz
$P_{\max }^{\mathrm{V}}$	Maximum transmit power of the vehicles	23 dBm
$P^{\mathrm{R}}$	Transmit power of the RSUs	40 dBm
$d_o$	Length of offloading action space	5
$d_p$	Length of power control action space	5
${\sigma }^2$	Noise power	−90 dBm
$\alpha$	Learning rate	0.01
$\gamma$	Discount factor	0.9
$ϵ$	Exploration possibility	0.96
$N_{\mathrm{episode}}$	Iteration episode	350
$N_{\mathrm{step}}$	Training step in one episode	200

. In addition, in order to reflect the excellent performance of the joint optimization scheme, the maximum transmit power and full offloading schemes are adopted as benchmark schemes. Specifically, the maximum transmit power scheme represents that vehicles use their maximum transmit power offloading task, and the full offloading scheme represents that vehicles offload all tasks to the RSUs or BS for edge computing. The DQN has one input layer, one output layer and two hidden layers, and the number of neurons in each layer is 10. The simulation environment is Python 3.7.4 and TensorFlow 2.1.0.

The energy consumption of the MEC-assisted multi-vehicle V2X system versus the number of training episodes under different learning rates, i.e., $\alpha =0.005,0.01$ and 0.05, is shown in Figure 3. Figure 3a,b show that the energy consumption decreases when the RSUs are equipped with MF and ZF receivers, respectively. It can be seen that, whether the RSUs are equipped with MF or ZF receivers, as the training episode increases, the energy consumption decreases and is finally stable under the proposed scheme with different learning rates, especially when $\alpha =0.01$. Thus, $\alpha =0.01$ is adopted to train vehicles to select the beneficial server association, offloading ratio and transmit power to minimize the energy consumption. In addition, energy consumption when the RSUs are equipped with the ZF receivers is lower than when equipped with the MF receivers. This is because the ZF receivers eliminating the inter-user interference of the V2I, I2B and V2B communication links results in a higher transmission rate. The delay of each vehicles versus the number of training episodes under the proposed scheme when $\alpha =0.01$ is shown in Figure 4. Figure 4a,b show the delay of each vehicle when the RSUs are equipped with MF and ZF receivers, respectively. It is noted that the delay of each vehicles is less than the maximum vehicle delay; thus, the proposed scheme can decrease the energy consumption while satisfying the delay constraint.

The energy consumption versus the number of training episodes under different schemes is shown in Figure 5. Figure 5a,b show the energy consumption under different schemes when the RSUs are equipped with MF and ZF receivers, respectively. It is noted that the energy consumption decreases and is finally stable under different schemes. When the RSUs are equipped with the MF receivers, the proposed scheme can decrease the energy consumption by 55.89% and 64.67% more than the maximum transmit power and full offloading schemes, respectively. This is because the proposed scheme optimizes the offloading ratio and the transmission power jointly, which both have an impact on the energy consumption. When the RSUs are equipped with the ZF receivers, the proposed scheme can decrease the energy consumption by 20.00% more than the full offloading scheme, but has a similar performance compared to the maximum transmit power scheme. This is because ZF receivers can eliminate inter-user interference, making vehicles choose the maximum transmit power to offload tasks. In addition, RSUs equipped with ZF receivers under the proposed scheme can decrease the energy consumption by 35.04% more than MF receivers.

Figure 6 shows the impact of the number of vehicles on the energy consumption. It can be seen that as the number of vehicles increases, the energy consumption increases under the same scheme, which is independent of the RSUs equipped with the receiver. This is because the number of vehicles increases, the interference caused by other vehicles increases and transmission rate decreases. In order to satisfy the delay constraint, vehicles need to offload more tasks to the MEC server for edge consumption, which makes the energy consumption increase. When the number of vehicles K is lower than the number of RSUs L, the energy consumption of the maximum transmit power scheme is the same as the proposed scheme. This is because the improved KNN algorithm assigns vehicles to different RSUs, and the vehicles will use the maximum transmit power in the offloading task but not cause interference to each other. When the number of vehicles is larger than the number of RSUs, the proposed scheme has a lower energy consumption than the benchmark schemes. Specifically, when $K=5$ and RSUs are equipped with MF receivers, the proposed scheme can decrease the energy consumption by 42.86% and 54.29% compared with the maximum transmission power and full offloading schemes, respectively. When $K=9$, the energy consumption of the proposed scheme reduces by 42.26% and 47.27% compared with the maximum transmit power and the offloading schemes, respectively. When $K=9$ and RSUs are equipped with ZF receivers, the proposed scheme can decrease the energy consumption by 17.78% compared with the full offloading schemes, respectively. Thus, the proposed scheme has a better performance in reducing the energy consumption of the system when the number of vehicles increases.

The energy consumption versus the number of RSUs L under different schemes when the number of vehicles $K=6$ is shown in Figure 7. It can be seen that as the number of RSUs increases, the energy consumption decreases under the same scheme. This is because the number of RSUs increases and more vehicles can offload tasks to the RSUs and have a lower energy consumption. When the number of RSUs $L=8$, the proposed scheme can decrease energy consumption by 19.93% more than the full offloading scheme and has a similar performance compared to the maximum transmit power scheme. This is because the improved KNN algorithm assigns vehicles to different RSUs when $L>K$, making vehicles choose the maximum transmit power to offload tasks.

The energy consumption versus packet size increase under different schemes is shown in Figure 8. It is noted that as the packet size increases, the energy consumption increases under the same scheme, no matter what receivers the RSUs are equipped with. This is because the packet size increases and the process delay increases, and so the vehicles need to offload more tasks to the MEC server for edge consumption to satisfy the delay constraint. This makes the energy consumption increase. In addition, the energy consumption of the proposed scheme is lower than the benchmark schemes, and when the RSU is equipped with a ZF receiver, it can reduce more energy consumption. Specifically, when the packet size is 1.2 M and RSUs are equipped with MF receivers, the energy consumption of the proposed scheme reduces by 52.38% and 80.95% compared with the maximum transmit power and offloading scheme, respectively. When the packet size is 3M, the energy consumption of the proposed scheme reduces by 52.30% and 61.84% compared with the benchmark schemes. Thus, the proposed scheme has a better performance in reducing the energy consumption of the system when the packet size increases.

The DRL scheme is proposed to solve the non-convex energy consumption minimization problem. From the above simulation results, it can be concluded that the proposed DRL scheme can obtain superior performance in reducing the energy consumption of the system under a different number of vehicles, number of RSUs and packet size.

6. Conclusions

This paper studied the uplink MEC-assisted multi-vehicle V2X communication system in which vehicles can offload tasks to the RSUs and BS for edge computing. To minimize the energy consumption and satisfy the delay requirements, an energy consumption minimization problem was formulated under a delay constraint. Due to the coupling of the server association, offloading ratio and transmit power, as well as delay constraints, the minimization problem is highly non-convex and difficult to solve. Thus, the DRL algorithm was proposed to solve the optimization problem. Vehicles as agents were trained according to the reward function, which is related to the energy consumption of the system and processing delay, to obtain a better action containing server association, offloading ratio and transmit power. In addition, an improved KNN algorithm was used to assign vehicles within the coverage area of multiple RSUs to reduce the action space and complexity of the DRL scheme. Numerical results have shown that the proposed scheme outperforms the maximum transmit power and full offloading schemes regarding minimizing the energy consumption with a different number of vehicles and packet size. In addition, the RSUs equipped with ZF receivers can decrease more energy consumption than MF receivers. However, the collaborative RSUs for computation processing are not considered in this work due to the complicated action space and the computation complexity. It is interesting to investigate the collaborative RSUs for computation processing to reduce the energy consumption and reduce the latency, especially for the resource-limited V2X network. Joint offloading and RSU allocation optimization is difficult with communication delay and power constraints due to the action space being related to a discrete RSU.