DRL-Assisted Resource Allocation for NOMA-MEC Offloading with Hybrid SIC

Abstract

Multi-access edge computing (MEC) and non-orthogonal multiple access (NOMA) are regarded as promising technologies to improve the computation capability and offloading efficiency of mobile devices in the sixth-generation (6G) mobile system. This paper mainly focused on the hybrid NOMA-MEC system, where multiple users were first grouped into pairs, and users in each pair offloaded their tasks simultaneously by NOMA, then a dedicated time duration was scheduled to the more delay-tolerant user for uploading the remaining data by orthogonal multiple access (OMA). For the conventional NOMA uplink transmission, successive interference cancellation (SIC) was applied to decode the superposed signals successively according to the channel state information (CSI) or the quality of service (QoS) requirement. In this work, we integrated the hybrid SIC scheme, which dynamically adapts the SIC decoding order among all NOMA groups. To solve the user grouping problem, a deep reinforcement learning (DRL)-based algorithm was proposed to obtain a close-to-optimal user grouping policy. Moreover, we optimally minimized the offloading energy consumption by obtaining the closed-form solution to the resource allocation problem. Simulation results showed that the proposed algorithm converged fast, and the NOMA-MEC scheme outperformed the existing orthogonal multiple access (OMA) scheme.

1. Introduction

With fifth-generation (5G) networks being available now, the sixth-generation (6G) wireless network is currently under research, which is expected to provide superior performance to satisfy the growing demands of mobile equipment, such as latency-sensitive, energy-hungry, and computationally intensive services and applications [1,2]. For example, the Internet of Things (IoT) networks are being developed rapidly, where massive numbers of nodes are supposed to be connected together, and IoT nodes can not only communicate with each other, but also process acquired data [3,4,5]. However, such IoT and many other terminal devices are constrained by the battery life and computational capability, and thereby, these devices cannot fully support computationally intensive tasks. A conventional approach to improve the computation capability of mobile devices is mobile cloud computing (MCC), where computationally intensive tasks are offloaded to the central cloud servers for data processing [6,7]. However, MCC will cause significant delays due to the long propagation distance between the central server and the user equipment. To address the long transmission delay issue, especially for delay-sensitive applications in the future 6G networks, multi-access edge computing (MEC) has emerged as a decentralized structure to provide the computation capability close to the terminal devices, which is generally implemented at the base stations to provide a cloud-like task processing service [7,8,9,10].

From the communication perspective, non-orthogonal multiple access (NOMA) is recognized as a promising technology to improve the spectral efficiency and massive connectivity, which enable multiple users to utilize the same resource block such as time and frequency for transmission [11,12]. Taking power-domain NOMA as an example, the signals of multiple users are multiplexed in the power domain by superposition coding, and at the receiver side, successive interference cancellation (SIC) is adopted to successively remove the multiple access interference [13]. Hence, integrating NOMA with MEC can potentially improve the service quality of MEC including low transmission latency and massive connections compared to the conventional orthogonal multiple access (OMA).

1.1. Related Works

The integration of NOMA and MEC has been well studied so far, and researchers have proposed various approaches to optimal resource allocation to minimize the offloading delay and energy consumption. In [14], the author minimized the offloading latency for a multi-user scenario, in which the power allocation and task partition ratio were jointly optimized. The partial offloading policy can determine the amount of data to be offloaded to the server, and the remainder is processed locally. The author of [15] proposed an iterative two-user NOMA scheme to minimize the offloading latency, in which two users offload their tasks simultaneously by NOMA. Since one of the users suffers performance degradation introduced by NOMA, instead of forcing two users to complete offloading at the same time, the remaining data are offloaded together with the next user during the following time slot. Moreover, many existing works investigated the energy minimization of NOMA-MEC networks. For example, the joint optimization of central processing unit (CPU) frequency, task partition ratio, and power allocation for a NOMA-MEC heterogeneous network were considered in [16,17]. In [18], the author considered a multi-antenna NOMA-MEC network and presented an approach to minimize the weighted sum energy consumption by jointly optimizing the computation and communication resource.

In addition to the existing works on pure NOMA schemes as previously mentioned, a few works also combined NOMA and OMA together, which is denominated as hybrid NOMA [19]. In this paper, the author proposed a two-user hybrid NOMA scenario, in which one user is less delay tolerant than the other. The two users offload during the first time slot by NOMA, and the user with a longer deadline offloads the remaining data during an additional time duration by OMA. This configuration presents significant benefits and outperforms both OMA and pure NOMA in terms of energy consumption since the energy can be saved for the delay-tolerant user instead of finishing offloading at the same time in pure NOMA networks. In [20,21], the hybrid NOMA scheme was extended to multi-user scenarios, in which a two-to-one matching algorithm was utilized to pair every two users into a group. The whole bandwidth resource was divided into multiple sub-channels, and the users in each group offloaded simultaneously through a dedicated sub-channel.

For the resource allocation in NOMA-MEC networks, user grouping is a non-convex problem, which is solved by exhaustive search or applying matching theory. Moreover, deep reinforcement learning (DRL) is recognized as a novel approach to this problem, which is a powerful tool to solve the real-time decision-making tasks, and only a handful papers have utilized it for user grouping and sub-channel assignment, such as [22,23], which output the user grouping policy for uplink and downlink NOMA networks, respectively.

Moreover, in most of the NOMA works, the SIC decoding order is predetermined either by the channel state information (CSI) or the quality of service (QoS) requirements of the users [24,25,26]. A recent work [27] proposed a hybrid SIC scheme to switch the SIC decoding order dynamically, which showed significant performance improvement in uplink NOMA networks. The author of [28] integrated the hybrid SIC scheme with an MEC network to serve two uplink users, and the results revealed that the hybrid SIC outperformed the QoS-based decoding order.

1.2. Motivation and Contributions

Motivated by the existing research on MEC-NOMA, in this paper, we investigated the energy minimization for uplink transmission in multi-user hybrid NOMA-MEC networks with hybrid SIC. More specifically, a DRL-based framework was proposed to generate a user grouping policy, and the power allocation, time allocation, and task partition assignment were jointly optimized for each group. The DRL framework collects experience data including CSI, deadlines, and energy consumption as labeled data to train the neural networks (NNs). In association with the resource allocation, the proposed scheme can dynamically adapt the decoding order to achieve better energy efficiency. The main contributions of this paper are summarized as follows:

• A hybrid NOMA-MEC network was proposed, in which an MEC server is deployed at the base station to serve multiple users. All users are divided into pairs, and each pair is assigned into one sub-channel. The users in each group adopt NOMA transmission with the hybrid SIC scheme in the first time duration, and the user with a longer deadline transmits the remaining data by OMA in the following time duration. We proposed a DRL-assisted user grouping framework with joint power allocation, time scheduling, and task partition assignment to minimize the offloading energy consumption under transmission latency and offloading data amount constraints.
• By assuming that the user grouping policy is given, the energy minimization problem for each group is non-convex due to the multiplications of variables and a 0–1 indicator function, which indicates two cases of decoding orders. The solution to the original problem can be obtained by solving each case separately. A multilevel programming method was proposed, where the energy minimization problem was decomposed into three sub-problems including power allocation, time scheduling, and task partition assignment. By carefully analyzing the convexity and monotonicity of each sub-problem, the solutions to all three sub-problems were obtained optimally in closed-form. The solution to the energy minimization problem for each case can be determined optimally by adaptingthe decisions successively from the lower level to the higher level (i.e., from the optimal task partition assignment to the optimal power allocation). Therefore, the solution to the original problem can be obtained by comparing the numerical results of those two cases and selecting the optimal solution with lower energy consumption.
• A DRL framework for user grouping was designed based on a deep Q-learning algorithm. We provided a training algorithm for the NN to learn the experiences based on the channel condition and delay tolerance of each user during a period of slotted time, and the user grouping policy can be learned gradually at the base station by maximizing the negative of the total offloading energy consumption.
• Simulation results are provided to illustrate the convergence speed and the performance of this user grouping policy by comparing with random user grouping policy. Moreover, compared with the OMA-MEC scheme, our proposed NOME-MEC scheme can achieve superior performance with much lower energy consumption.

1.3. Organizations

The rest of the paper is structured as follows. The system model and the formulated energy minimization problem for our proposed NOMA-MEC scheme are described in Section 2. Section 3 presents the optimal solution to the energy minimization problem. Following that, the DRL-based user-grouping algorithm is introduced in Section 4. Finally, the simulation results of the convergence and average performance for the proposed scheme are shown in Section 5, and Section 6 concludes this paper.

2. System Model and Problem Formulation

2.1. System Model

In this paper, we considered a NOMA-MEC network, where a base station is equipped with an MEC server to serve K resource-constrained users. During one offloading cycle, each user offloads its task to the MEC server and then obtains the results, which are processed at the MEC server. Generally, the data size of the computation outcome is relatively smaller than the offloaded data in practice; thus, the time for downloading the results can be omitted [18]. Moreover, since the MEC server has a much higher computation capability than the mobile devices, the data processing time at the MEC server can be ignored compared to the offloading time [14]. Therefore, in this work, the total offloading delay was approximated to the time consumption of data uploading to the base station.

We assumed that all K users were divided into $\Phi$ groups to transmit signals at different sub-channels, and each group $\varphi$ contained two users such that $K=2\Phi$. In each group, we denote the user with a shorter deadline by $U_{m,\varphi }$ and the user with relevantly longer deadline by $U_{n,\varphi }$, which indicates ${\tau }_{m,\varphi }\le {\tau }_{n,\varphi }$, where ${\tau }_{i,\varphi }$ is the latency requirement of $U_{i,\varphi },\forall i\in \{m,n\}$ in group $\varphi$. Because $U_{m,\varphi }$ has a tighter deadline, it was assumed that the whole duration ${\tau }_{m,\varphi }$ would be used up, which means that the offloading time $t_{m,\varphi }={\tau }_{m,\varphi }$.

In this paper, we adopted the block channel model, which indicates that the channel condition remains static during each time slot. With the small-scale fading, the channel gain of a user in group $\varphi$ can be expressed as:

$$H_{i,\varphi }={\tilde{h}}_{i,\varphi }{d_{i,\varphi }}^{-\frac{\alpha }{2}},\forall i\in \{m,n\},\forall \varphi ,$$

(1)

where ${\tilde{h}}_{i,\varphi }\sim \mathcal{CN}(0,1)$ is the Rayleigh fading coefficient, $d_{i,\varphi }$ is the distance between $U_{i,\varphi }$ and the base station, and $\alpha$ is the pass loss exponent. The channel gain is normalized by the addictive white Gaussian noise (AWGN) power with zero mean and ${\sigma }^2$ variance, which can be written as:

$$h_{i,\varphi }=\frac{|H_{i,\varphi }{|}^2}{{\sigma }^2},\forall i\in \{m,n\},\forall \varphi .$$

(2)

As shown in Figure 1, since those two users have different delay tolerances, it is natural to consider that $U_{n,\varphi }$ is unnecessary to finish offloading within ${\tau }_{m,\varphi }$ via NOMA transmission, and potentially to save energy if $U_{n,\varphi }$ can utilize the spare time ${\tau }_{n,\varphi }-{\tau }_{m,\varphi }$. Hence, the adopted hybrid NOMA scheme enables $U_{n,\varphi }$ to offload part of its data at the same time when $U_{m,\varphi }$ offloads its task during ${\tau }_{m,\varphi }$. An additional time duration $t_{r,\varphi }$ is scheduled within each time slot to transmit $U_{n,\varphi }$’s remaining data. The task transmission for $U_{n,\varphi }$ should be completed within ${\tau }_{n,\varphi }$, which means that $t_{r,\varphi }$ should satisfy:

$$t_{r,\varphi }\le {\tau }_{n,\varphi }-{\tau }_{m,\varphi },\forall \varphi .$$

(3)

As previously mentioned, the users in each group will occupy the same sub-channel to upload their data to the base station simultaneously via NOMA. In NOMA uplink transmission, SIC is adopted at the base station to decode the superposed signal. Conventionally, the SIC decoding order is based on either the user’s CSI or the QoS requirement [27]. For the QoS-based case, to guarantee $U_{m,\varphi }$ can offload its data by ${\tau }_{m,\varphi }$, $U_{n,\varphi }$ is set to be decoded first, and the achievable rate should satisfy:

$$R_{n,\varphi }\le Bln\left(1+\frac{P_{n,\varphi }{\left|h_{n,\varphi }\right|}^2}{P_{m,\varphi }{\left|h_{m,\varphi }\right|}^2+1}\right),$$

(4)

where B is the bandwidth of each sub-channel. $P_{n,\varphi }$ and $P_{m,\varphi }$ are the transmission power of $U_{n,\varphi }$ and $U_{m,\varphi }$ during NOMA transmission, respectively. Based on the NOMA principle, the signal of $U_{m,\varphi }$ can then be decoded if (4) is satisfied, and the data rate for $U_{m,\varphi }$ can be written as:

$$R_{m,\varphi }=Bln\left(1+P_{m,\varphi }{\left|h_{m,\varphi }\right|}^2\right).$$

(5)

In contrast, $U_{m,\varphi }$ can also be decoded first by treating $U_{n,\varphi }$’s signal as interference if the following condition holds:

$$R_{m,\varphi }\le Bln\left(1+\frac{P_{m,\varphi }{\left|h_{m,\varphi }\right|}^2}{P_{n,\varphi }{\left|h_{n,\varphi }\right|}^2+1}\right).$$

(6)

Then, the data rate of $U_{n,\varphi }$ can be obtained by removing the information of $U_{m,\varphi }$, which is:

$$R_{n,\varphi }=Bln\left(1+P_{n,\varphi }{\left|h_{n,\varphi }\right|}^2\right).$$

(7)

If the same power is allocated to $U_{n,\varphi }$ for both decoding sequences, it is evident that the achievable rate for $U_{n,\varphi }$ in (7) is higher than that in (4) since the interference is removed by the SIC, and hence the decoding order in (7) is preferred in this case. However, since the constraint (6) cannot always be satisfied, the system has to dynamically change the decoding order accordingly to achieve better performance, which motivated us to utilize the hybrid SIC scheme.

In addition, during $t_{r,\varphi }$, $U_{n,\varphi }$ adopts OMA transmission, and the data rate can be expressed as:

$$R_{r,\varphi }=Bln\left(1+P_{r,\varphi }{\left|h_{n,\varphi }\right|}^2\right),$$

(8)

where $P_{r,\varphi }$ represents the transmission power of $U_{n,\varphi }$ during the second time duration $t_{n,\varphi }$.

In this work, the data length of each task is denoted by L, which is assumed to be bitwise independent, and we proposed a partial offloading scheme in which each task can be processed locally and remotely in parallel. An offloading partition assignment coefficient ${\beta }_{\varphi }\in [0,1]$ is introduced, which indicates the amount of data offloaded to the MEC server, and the rest can be executed by the local device in parallel. Thus, for each task, the amount of data for offloading to the server is ${\beta }_{\varphi }L$, and $(1-{\beta }_{\varphi })L$ is the data to be processed locally.

$U_{n,\varphi }$ can take the advantage of local computing by executing $(1-{\beta }_{\varphi })L$ data locally during the scheduled NOMA and OMA time duration $t_{m,\varphi }+t_{r,\varphi }$. Therefore, the energy consumption for $U_{n,\varphi }$’s local execution, which is denoted by $E_{n,\varphi }^{loc}$, can be expressed as:

$$E_{n,\varphi }^{loc}=\frac{{\kappa }_0{\left[C(1-{\beta }_{\varphi })L\right]}^3}{{\left(t_{m,\varphi }+t_{r,\varphi }\right)}^2},$$

(9)

where ${\kappa }_0$ denotes the coefficient related to the mobile device’s processor and C is the number of CPU cycles required for computing each bit.

The total energy consumed by $U_{n,\varphi }$ per task involves three parts, including the energy consumed by local computing and transmission during NOMA and OMA offloading. The power for offloading is scheduled separately during these scheduled two time duration according to the hybrid SIC scheme, and thereby, the offloading energy consumption $E_{n,\varphi }^{off}$ can be expressed as:

$$E_{n,\varphi }^{off}=t_{m,\varphi }{P}_{n,\varphi }+t_{r,\varphi }{P}_{r,\varphi }.$$

(10)

Hence, the total energy consumption can be expressed as:

$$E_{\varphi }^{tot}=E_{n,\varphi }^{loc}+E_{n,\varphi }^{off}.$$

(11)

2.2. Problem Formulation

We assumed that the resource allocation of $U_{m,\varphi }$ is given as a constant in each group since $U_{m,\varphi }$ is treated as the primary user whose requirements need to be guaranteed in priority, and we only focused on the energy minimization for $U_{n,\varphi }$ during both NOMA and OMA duration. Given the user grouping policy, which will be solved in Section 4, the energy minimization problem for each pair can be formulated as:

$$\begin{array}{cccc}\hfill \left(\mathcal{P}1\right):\underset{\begin{array}{c}{P}_{n,\varphi },P_{r,\varphi }\\ {\mathrm{t}}_{\mathrm{r},Œ},{\mathrm{fi}}_{Œ}\end{array}}{min}& \hfill & \hfill & \frac{{\kappa }_0{\left[C(1-{\beta }_{\varphi })L\right]}^3}{{\left({\tau }_{m,\varphi }+t_{r,\varphi }\right)}^2}+{\tau }_{m,\varphi }{P}_{n,\varphi }+t_{r,\varphi }{P}_{r,\varphi }\hfill \end{array}$$

(12)

$$\begin{array}{cccc}\hfill \mathrm{s}.\mathrm{t}.& \hfill & \hfill & {\tau }_{m,\varphi }{R}_{n,\varphi }^H+t_{r,\varphi }Bln\left(1+P_{r,\varphi }{\left|h_{n,\varphi }\right|}^2\right)\ge {\beta }_{\varphi }L\hfill \end{array}$$

(13)

$$\begin{array}{cccc}\hfill & \hfill & & {\tau }_{m,\varphi }Bln\left(1+\frac{P_{m,\varphi }{\left|h_{m,\varphi }\right|}^2}{P_{n,\varphi }{\left|h_{n,\varphi }\right|}^2+1}\right)\ge {\mathbf{1}}_{n,\varphi }L\hfill \end{array}$$

(14)

$$\begin{array}{cccc}\hfill & \hfill & & P_{n,\varphi }\ge 0,P_{r,\varphi }\ge 0\hfill \end{array}$$

(15)

$$\begin{array}{cccc}\hfill & \hfill & & 0\le t_{r,\varphi }\le {\tau }_{n,\varphi }-{\tau }_{m,\varphi }\hfill \end{array}$$

(16)

$$\begin{array}{cccc}\hfill & \hfill & & 0\le {\beta }_{\varphi }\le 1,\hfill \end{array}$$

(17)

where $R_{n,\varphi }^H={\mathbf{1}}_{n,\varphi }Bln\left(1+P_{n,\varphi }{\left|h_{n,\varphi }\right|}^2\right)+(1-{\mathbf{1}}_{n,\varphi })Bln\left(1+\frac{P_{n,\varphi }{\left|h_{n,\varphi }\right|}^2}{P_{m,\varphi }{\left|h_{m,\varphi }\right|}^2+1}\right)$, and ${\mathbf{1}}_{n,\varphi }$ is the indicator function. When ${\mathbf{1}}_{n,\varphi }=1$, $U_{m,\varphi }$ is decoded first, and vice verse. The constraint (13) and (14) ensures all the users should complete offloading of the designated amount of data within the given deadline. The constraint (16) limits that the additionally scheduled time slot should not be beyond $U_{n,\varphi }$’s delay tolerance. Constraints (15) and (17) set the feasible range of the transmission power and the offloading coefficient.

The problem $\left(\mathcal{P}1\right)$ is non-convex due to the multiplication of several variables. Therefore, in the following section, we proposed a multilevel programming algorithm to address the energy minimization problem optimally by obtaining the closed-form solution.

3. Energy Minimization for NOMA-MEC with the Hybrid SIC Scheme

In this section, the problem $\left(\mathcal{P}1\right)$ is solved separately for the case ${\mathbf{1}}_{n,\varphi }=1$ and ${\mathbf{1}}_{n,\varphi }=0$. Due to the non-convexity of the original problem for both cases, a multilevel programming method is introduced to decompose the problem $\left(\mathcal{P}1\right)$ into three sub-problems, i.e., power allocation, time slot scheduling, and task assignment, which can be solved optimally by obtaining the closed-form solution. By solving those three sub-problems successively, the optimal solutions for both cases can thereby be obtained, which are provided in the subsections below. The solution to the original problem $\left(\mathcal{P}1\right)$ can be determined by comparing the numerical result of both cases and choosing the more energy efficient one.

3.1. Power Allocation

Let $t_{r,\varphi }$ and ${\beta }_{\varphi }$ be fixed. The problem $\left(\mathcal{P}1\right)$ is regarded as a power allocation problem, which can be rewritten as:

$$\begin{array}{cccc}\hfill \left(\mathcal{P}2\right):\underset{\begin{array}{c}{P}_{n,\varphi },P_{r,\varphi }\end{array}}{min}& \hfill & \hfill & \frac{{\kappa }_0{\left[C(1-{\beta }_{\varphi })L\right]}^3}{{\left({\tau }_{m,\varphi }+t_{r,\varphi }\right)}^2}+{\tau }_{m,\varphi }{P}_{n,\varphi }+t_{r,\varphi }{P}_{r,\varphi }\hfill \end{array}$$

(18)

$$\begin{array}{cccc}\hfill \mathrm{s}.\mathrm{t}.& \hfill & \hfill & {\tau }_{m,\varphi }{R}_{n,\varphi }^H+t_{r,\varphi }Bln\left(1+P_{r,\varphi }{\left|h_{n,\varphi }\right|}^2\right)\ge {\beta }_{\varphi }L\hfill \end{array}$$

(19)

$$\begin{array}{cccc}\hfill & \hfill & & {\tau }_{m,\varphi }Bln\left(1+\frac{P_{m,\varphi }{\left|h_{m,\varphi }\right|}^2}{P_{n,\varphi }{\left|h_{n,\varphi }\right|}^2+1}\right)\ge {\mathbf{1}}_{n,\varphi }L\hfill \end{array}$$

(20)

$$\begin{array}{cccc}\hfill & \hfill & & P_{n,\varphi }\ge 0,P_{r,\varphi }\ge 0\hfill \end{array}$$

(21)

Since there exists an indicator function, $\left(\mathcal{P}2\right)$ is solved in two different cases, i.e., when ${\mathbf{1}}_{n,\varphi }=1$ and when ${\mathbf{1}}_{n,\varphi }=0$. The following theorem provides the optimal solutions of both cases.

Theorem 1.

The optimal power allocation to $\left(\mathcal{P}2\right)$ is given by the following two cases according to the indicator function:

1. 

For ${\mathbf{1}}_{n,\varphi }=1$, $U_{m,\varphi }$ is decoded first, and the power allocation for this decoding order is presented in the following three offloading scenarios:

(a) 

When $P_{n,\varphi }\ne 0$ and $P_{r,\varphi }\ne 0$, $U_{n,\varphi }$ offloads in both time durations, which is termed hybrid NOMA. Given the following two feasible ranges, the optimal power allocation can be expressed as follows:

i. 

If $P_{m,\varphi }>{\left|h_{m,\varphi }\right|}^{-2}{e}^{\frac{{\beta }_{\varphi }L}{B\left({\tau }_{m,\varphi }+t_{r,\varphi }\right)}}\left(e^{\frac{L}{B{\tau }_{m,\varphi }}}-1\right)$,

$$P_{n,\varphi }^{*}=P_{r,\varphi }^{*}={\left|h_{n,\varphi }\right|}^{-2}\left(e^{\frac{{\beta }_{\varphi }L}{B\left({\tau }_{m,\varphi }+t_{r,\varphi }\right)}}-1\right).$$

(22)

ii. 

If $|h_{m,\varphi }{|}^{-2}\left(e^{\frac{L}{B{\tau }_{m,\varphi }}}-1\right)\le P_{m,\varphi }\le {\left|h_{m,\varphi }\right|}^{-2}{e}^{\frac{{\beta }_{\varphi }L}{B{\tau }_{m,\varphi }}}\left(e^{\frac{L}{B{\tau }_{m,\varphi }}}-1\right)$,

$$\begin{array}{cc}\hfill & P_{n,\varphi }^{*}={\left|h_{n,\varphi }\right|}^{-2}\left[P_{m,\varphi }{\left|h_{m,\varphi }\right|}^2{\left(e^{\frac{L}{B{\tau }_{m,\varphi }}}-1\right)}^{-1}-1\right],\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill & P_{r,\varphi }^{*}={\left|h_{n,\varphi }\right|}^{-2}\left\{e^{\frac{{\beta }_{\varphi }L}{B{t}_{r,\varphi }}-\frac{{\tau }_{m,\varphi }}{t_{r,\varphi }}ln\left[P_{m,\varphi }{\left|h_{m,\varphi }\right|}^2{\left(e^{\frac{L}{B{\tau }_{m,\varphi }}}-1\right)}^{-1}\right]}-1\right\}.\hfill \end{array}$$

(24)

(b) 

When $U_{n,\varphi }$ only offloads during the first time duration ${\tau }_{m,\varphi }$, this scheme is termed pure NOMA, and the power allocation is obtained as:

$if$$P_{m,\varphi }\ge {\left|h_{m,\varphi }\right|}^{-2}{e}^{\frac{{\beta }_{\varphi }L}{B{\tau }_{m,\varphi }}}\left(e^{\frac{L}{B{\tau }_{m,\varphi }}}-1\right)$,

$$P_{n,\varphi }^{*}={\left|h_{n,\varphi }\right|}^{-2}\left(e^{\frac{{\beta }_{\varphi }L}{B{\tau }_{m,\varphi }}}-1\right).$$

(25)

(c) 

When $P_{n,\varphi }^{*}=0$, $U_{n,\varphi }$ chooses to offload solely during the section time duration $t_{r,\varphi }$, and the optimal power allocation is:

$if$$P_{m,\varphi }\ge {\left|h_{m,\varphi }\right|}^{-2}\left(e^{\frac{L}{B{\tau }_{m,\varphi }}}-1\right)$,

$$P_{r,\varphi }^{*}={\left|h_{n,\varphi }\right|}^{-2}\left(e^{\frac{{\beta }_{\varphi }L}{B{t}_{r,\varphi }}}-1\right).$$

(26)

2. 

For ${\mathbf{1}}_{n,\varphi }=0$, $U_{n,\varphi }$ is decoded first, and similarly, the power allocation for this decoding order is given in three scenarios:

(a) 

When $P_{n,\varphi }\ne 0$ and $P_{r,\varphi }\ne 0$, $U_{n,\varphi }$, the hybrid NOMA power allocation is given by:

if $P_{m,\varphi }\le {\left|h_{m,\varphi }\right|}^{-2}\left(e^{\frac{{\beta }_{\varphi }L}{B{t}_{r,\varphi }}}-1\right)$,

$$\begin{array}{cc}\hfill P_{n,\varphi }^{*}=& |h_{n,\varphi }{|}^{-2}\left(P_{m,\varphi }{\left|h_{m,\varphi }\right|}^2+1\right)\left[e^{\frac{{\beta }_{\varphi }L-t_{r,\varphi }Bln\left(P_{m,\varphi }{\left|h_{m,\varphi }\right|}^2+1\right)}{B\left({\tau }_{m,\varphi }+t_{r,\varphi }\right)}}-1\right]\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill P_{r,\varphi }^{*}=& |h_{n,\varphi }{|}^{-2}\left[\left(P_{m,\varphi }{\left|h_{m,\varphi }\right|}^2+1\right)e^{\frac{{\beta }_{\varphi }L-t_{r,\varphi }Bln\left(P_{m,\varphi }{\left|h_{m,\varphi }\right|}^2+1\right)}{B\left({\tau }_{m,\varphi }+t_{r,\varphi }\right)}}-1\right].\hfill \end{array}$$

(28)

(b) 

When $P_{r,\varphi }=0$, the pure NOMA case can be obtained as:

$$P_{n,\varphi }^{*}={\left|h_{n,\varphi }\right|}^{-2}\left(P_{m,\varphi }{\left|h_{m,\varphi }\right|}^2+1\right)\left(e^{\frac{{\beta }_{\varphi }L}{B{\tau }_{m,\varphi }}}-1\right).$$

(29)

(c) 

When $P_{n,\varphi }^{*}=0$, the OMA case is the same as (26).

Proof. 

Refer to Appendix A.    □

Remark 1.

Theorem 1 provides the optimal power allocation for both decoding sequences, i.e., $U_{m,\varphi }$ is decoded first when ${\mathbf{1}}_{n,\varphi }=1$, and $U_{n,\varphi }$ is decoded first when ${\mathbf{1}}_{n,\varphi }=0$. The optimal solution to $\left(\mathcal{P}1\right)$ is obtained by a numerical comparison between these two cases in terms of energy consumption. Both cases can be further divided into three offloading scenarios including hybrid NOMA, pure NOMA, and OMA based on different power allocations. For the hybrid NOMA case, $U_{n,\varphi }$ transmits during both ${\tau }_{m,\varphi }$ and $t_{r,\varphi }$, which indicates $P_{n,\varphi }>0$, $P_{r,\varphi }>0$, and $t_{r,\varphi }>0$. The pure NOMA scheme indicates that $U_{n,\varphi }$ only transmits simultaneously with $U_{m,\varphi }$ during ${\tau }_{m,\varphi }$, and therefore, $P_{r,\varphi }=0$ and $t_{r,\varphi }=0$. In addition, the OMA case represents that $U_{m,\varphi }$ occupies ${\tau }_{m,\varphi }$ solely, and $U_{n,\varphi }$ only transmits during $t_{r,\varphi }$.

Remark 2.

Appendix A provides the proof for the case ${\mathbf{1}}_{n,\varphi }=1$. The proof for the case ${\mathbf{1}}_{n,\varphi }=0$ similarly, and it can be referred to the previous work in [21]. Thus, the proof for the case ${\mathbf{1}}_{n,\varphi }=0$ is omitted for this and the following two sub-problems.

In this subsection, the optimal power allocation for the hybrid NOMA scheme is obtained when $t_{r,\varphi }$ is fixed, and then, the optimization of $t_{r,\varphi }$ is further studied to minimize $E_{n,\varphi }^{tot}$ in the following subsection.

3.2. Time Scheduling

The aim of this subsection is to find the optimal time allocation for the second time duration $t_{r,\varphi }$, which is solely utilized by $U_{n,\varphi }$ for OMA transmission. As previously mentioned in Theorem 1, the optimal power allocation for the hybrid NOMA scheme is given as a function of $t_{r,\varphi }$ and ${\beta }_{\varphi }$. Hence, by fixing ${\beta }_{\varphi }$, $\left(\mathcal{P}1\right)$ is rewritten as:

$$\begin{array}{cccc}\hfill \left(\mathcal{P}3\right):\underset{{\mathrm{t}}_{\mathrm{r},\varphi }}{min}& \hfill & \hfill & \frac{{\kappa }_0{\left[C(1-{\beta }_{\varphi })L\right]}^3}{{\left({\tau }_{m,\varphi }+t_{r,\varphi }\right)}^2}+{\tau }_{m,\varphi }{P}_{n,\varphi }^{*}+t_{r,\varphi }{P}_{r,\varphi }^{*}\hfill \end{array}$$

(30)

$$\begin{array}{cccc}\hfill \mathrm{s}.\mathrm{t}.& \hfill & & 0\le t_{r,\varphi }\le {\tau }_{n,\varphi }-{\tau }_{m,\varphi }\hfill \end{array}$$

(31)

Proposition 1.

The offloading energy consumption (30) is monotonically decreasing with respected to $t_{r,\varphi }$ for both the ${\mathbf{1}}_{n,\varphi }=1$ and ${\mathbf{1}}_{n,\varphi }=0$ cases. To minimize the energy consumption, the optimal time allocation is to schedule the entire available time before the deadline ${\tau }_{n,\varphi }$, i.e.,

$$t_{r,\varphi }^{*}={\tau }_{n,\varphi }-{\tau }_{m,\varphi }$$

(32)

Proof. 

Refer to Appendix B.    □

By assuming all the data are offloaded to the MEC server, the following lemma studies the uplink transmission energy efficiency of the two hybrid NOMA-MEC schemes for ${\mathbf{1}}_{n,\varphi }=0$ and ${\mathbf{1}}_{n,\varphi }=1$.

Lemma 1.

Assume all data are offloaded to the MEC server, i.e., ${\beta }_{\varphi }=1$. The solution in (27) and (28) for the case ${\mathbf{1}}_{n,\varphi }=0$ has higher energy consumption than the solution in (22) for the case ${\mathbf{1}}_{n,\varphi }=1$, if $|h_{m,\varphi }{|}^{-2}\left(e^{\frac{L}{B{\tau }_{m,\varphi }}}-1\right)\le P_{m,\varphi }\le {\left|h_{m,\varphi }\right|}^{-2}\left(e^{\frac{L}{B\left({\tau }_{n,\varphi }-{\tau }_{m,\varphi }\right)}}-1\right)$.

Proof. 

Without considering local computing, the energy consumption for (22) can be written as:

$$E_1={\tau }_{n,\varphi }{\left|h_{n,\varphi }\right|}^{-2}\left(e^{\frac{L}{B{\tau }_{n,\varphi }}}-1\right),$$

(33)

and the energy consumption for the power allocation scheme in (27) and (28) is given as:

$$\begin{array}{cc}\hfill E_2=& {\tau }_{m,\varphi }{\left|h_{n,\varphi }\right|}^{-2}\left(P_{m,\varphi }{\left|h_{m,\varphi }\right|}^2+1\right)\left[e^{\frac{L-\left({\tau }_{n,\varphi }-{\tau }_{m,\varphi }\right)Bln\left(P_{m,\varphi }{\left|h_{m,\varphi }\right|}^2+1\right)}{B{\tau }_{n,\varphi }}}-1\right]\hfill \\ & +\left({\tau }_{n,\varphi }-{\tau }_{m,\varphi }\right){\left|h_{n,\varphi }\right|}^{-2}\left[\left(P_{m,\varphi }{\left|h_{m,\varphi }\right|}^2+1\right)e^{\frac{L-\left({\tau }_{n,\varphi }-{\tau }_{m,\varphi }\right)Bln\left(P_{m,\varphi }{\left|h_{m,\varphi }\right|}^2+1\right)}{B{\tau }_{n,\varphi }}}-1\right].\hfill \end{array}$$

(34)

To prove that $E_2\ge E_1$, the inequality can be rearranged as:

$$-{\tau }_{m,\varphi }{P}_{m,\varphi }{\left|h_{m,\varphi }\right|}^2+{\tau }_{n,\varphi }{e}^{\frac{L}{B{\tau }_{n,\varphi }}}{\left(P_{m,\varphi }{\left|h_{m,\varphi }\right|}^2+1\right)}^{\frac{{\tau }_{m,\varphi }}{{\tau }_{n,\varphi }}}\ge {\tau }_{n,\varphi }{e}^{\frac{L}{B{\tau }_{n,\varphi }}}.$$

(35)

Define $\zeta \left(x\right)=-{\tau }_{m,\varphi }x+{\tau }_{n,\varphi }{e}^{\frac{L}{B{\tau }_{n,\varphi }}}{(x+1)}^{\frac{{\tau }_{m,\varphi }}{{\tau }_{n,\varphi }}}$. The first-order derivative of $\zeta \left(x\right)$ is given as:

$${\zeta }^{{}^{\prime }}\left(x\right)=-{\tau }_{m,\varphi }+{\tau }_{m,\varphi }{e}^{\frac{L}{B{\tau }_{n,\varphi }}}{(x+1)}^{\frac{{\tau }_{m,\varphi }}{{\tau }_{n,\varphi }}-1}.$$

(36)

Therefore, ${\zeta }^{{}^{\prime }}\left(x\right)$ is monotonically decreasing since ${\tau }_{m,\varphi }<{\tau }_{n,\varphi }$, and the following inequality holds:

$${\zeta }^{{}^{\prime }}\left(x\right)\ge {\zeta }^{{}^{\prime }}\left(e^{\frac{L}{B\left({\tau }_{n,\varphi }-{\tau }_{m,\varphi }\right)}}-1\right)=0.$$

(37)

Hence, for $0\le x\le e^{\frac{L}{B\left({\tau }_{n,\varphi }-{\tau }_{m,\varphi }\right)}}-1$, $\zeta \left(x\right)$ is monotonically increasing, and $\zeta \left(x\right)\ge \zeta \left(0\right)={\tau }_{n,\varphi }{e}^{\frac{L}{B{\tau }_{n,\varphi }}}$, which illustrates that $E_2\ge E_1$.    □

3.3. Offloading Task Partition Assignment

In this subsection, we focused on the optimization of the task partition assignment coefficient for $U_{n.\varphi }$ in group $\varphi$. Given the optimal power allocation and time arrangement, $\left(\mathcal{P}1\right)$ is reformulated as:

$$\begin{array}{cccc}\hfill \left(\mathcal{P}4\right):\underset{{\beta }_{\varphi }}{min}& \hfill & & \frac{{\kappa }_0{\left[C(1-{\beta }_{\varphi })L\right]}^3}{{\left({\tau }_{m,\varphi }+t_{r,\varphi }^{*}\right)}^2}+{\tau }_{m,\varphi }{P}_{n,\varphi }^{*}+t_{r,\varphi }^{*}{P}_{r,\varphi }^{*}\hfill \end{array}$$

(38)

$$\begin{array}{cccc}\hfill \mathrm{s}.\mathrm{t}.& \hfill & \hfill & 0\le {\beta }_{\varphi }\le 1,\hfill \end{array}$$

(39)

Proposition 2.

The above problem is convex, and the optimal task assignment coefficient can be characterized by those three optimal power allocation schemes for the hybrid NOMA model in (22), (23), and (27), which is given by:

$${\beta }_{\varphi }^{*}=1-\frac{2}{z_{2,\varphi }}\mathcal{W}\left(\frac{1}{2}{z}_{1,\varphi }^{-\frac{1}{2}}{z}_{2,\varphi }{e}^{\frac{z_{2,\varphi }}{2}}\right),$$

(40)

where $\mathcal{W}$ denotes the single-valued Lambert W function and $z_{1,\varphi }$ and $z_{2,\varphi }$ are determined by the different power allocation schemes, which are presented as follows:

(a) 

${\mathbf{1}}_{n,\varphi }=1$:

If (22) is adopted:

$$\left\{\begin{array}{cc}\hfill & z_1=\frac{3{\kappa }_{0}B{C}^3{L}^2{\left|h_{n,\varphi }\right|}^2}{{\tau }_{n,\varphi }^2},\hfill \\ \hfill & z_2=\frac{L}{B{\tau }_{n,\varphi }}\hfill \end{array}\right.$$

(41)

If (23) is adopted:

$$\left\{\begin{array}{cc}\hfill & z_1=\frac{3{\kappa }_{0}B{\left|h_{n,\varphi }\right|}^2{C}^3{L}^2{e}^{2u_{\varphi }}}{{\tau }_{n,\varphi }^2}\hfill \\ \hfill & z_2=\frac{L}{B\left({\tau }_{n,\varphi }-{\tau }_{m,\varphi }\right)}\hfill \end{array}\right.$$

(42)

where $u_{\varphi }=\frac{{\tau }_{m,\varphi }}{\left({\tau }_{n,\varphi }-{\tau }_{m,\varphi }\right)}ln\left[P_{m,\varphi }{\left|h_{m,\varphi }\right|}^2{\left(e^{\frac{L}{B{\tau }_{m,\varphi }}}-1\right)}^{-1}\right]$.

(b) 

${\mathbf{1}}_{n,\varphi }=0$:

$$\left\{\begin{array}{cc}\hfill & z_{1,\varphi }=\frac{3{\kappa }_{0}B{C}^3{L}^2{\left|h_{n,\varphi }\right|}^2{e}^{\frac{({\tau }_{n,\varphi }-{\tau }_{m,\varphi })ln\left(P_{m,\varphi }{\left|h_{m,\varphi }\right|}^2+1\right)}{{\tau }_{n,\varphi }}}}{{{\tau }_{n,\varphi }}^2\left(P_{m,\varphi }{\left|h_{m,\varphi }\right|}^2+1\right)}\hfill \\ \hfill & z_{2,\varphi }=\frac{L}{B{\tau }_{n,\varphi }}\hfill \end{array}\right.$$

(43)

Proof. 

Refer to Appendix C.    □

Remark 3.

Problem $\left(\mathcal{P}4\right)$ is the lowest level of the proposed multilevel programming method, which provides three task assignment solutions corresponding to the three power allocation schemes (22), (23), and (27), respectively. The final solution to the energy minimization problem $\left(\mathcal{P}1\right)$ can be obtained by substituting the optimal task assignment into the corresponding power allocation scheme. Then, the most energy-efficient scheme is selected among (22), (23), and (27) by comparing the numerical energy consumption for each scheme.

4. Deep Reinforcement Learning Framework for User Grouping

In the previous section, it was assumed that the user grouping is given, and the optimal resource allocation was obtained in closed-form. The user grouping can be obtained optimally by exploring all possible user grouping combinations and finding the one with the lowest energy consumption. Although this method provides the optimal user pairing scheme, the complexity of the exhaustive search method is high, and it is not possible to output real-time decisions. Therefore, we proposed a fast converging user pairing training algorithm based on DQN to obtain the user grouping policy, which is introduced in the following subsection, in which the state space, action space, and reward function are defined. Subsequently, the training algorithm for the user grouping policy is provided.

4.1. The DRL Framework

The optimization of user grouping is modeled as a DRL task, where the base station is treated as the agent to interact with the environment, which is defined as the MEC network. In each time slot t, the agent takes an action $a_t$ from the action space $\mathcal{A}$ to assign users into pairs according to an optimal policy, which is learned by the DNN. The action taken under current state $s_t$ results in an immediate reward $r_t$, which is obtained at the beginning of the next time slot, and then moved to the next state $s_{t+1}$. In this problem, the aforementioned terms are defined as follows.

(1)

State space: The state $s_t\in \mathcal{S}$ is characterized by the current channel gains and offloading deadlines of all users since the user grouping is mainly determined by those two factors. Therefore, the state $s_t$ can be expressed as:

$$s_t=\{h_1\left[t\right],h_2\left[t\right],...,h_k\left[t\right],...,h_K\left[t\right];{\tau }_1\left[t\right],{\tau }_2\left[t\right],...,{\tau }_k\left[t\right],...,{\tau }_K\left[t\right]\}.$$

(44)

(2)

Action space: At each time slot t, the agent takes an action $a_t\in \mathcal{A}$, which contains all the possible user grouping decisions $j_{k,\varphi }$. The action is defined as:

$$a_t=\{j_{1,1}\left[t\right],...j_{k,\varphi }\left[t\right],...j_{K,\Phi }\left[t\right]\},$$

(45)

where $j_{k,\varphi }=1$ indicates that $U_k$ is assigned to group $\varphi$. In our proposed scheme, two different users can only be assigned to each group.

(3)

Rewards: The immediate reward $r_t$ is described by the sum of the energy consumption of each group after choosing the action $a_t$ under state $s_t$. The numerical result of the energy consumption in each group can be obtained by solving the problem $\left(\mathcal{P}1\right)$. Therefore, the reward is defined as:

$$r_t=-\sum _{\varphi =1}^{\Phi }{E}_{\varphi }^{tot}\left[t\right]$$

(46)

The aim of the agent is to find an optimal policy that maximizes the long-term discounted reward, which can be written as:

$$\begin{array}{cc}\hfill R_t& =r_t+\gamma r_{t+1}+{\gamma }^2{r}_{t+2}+...\hfill \\ \hfill & =\sum _{i=0}^{\infty }{\gamma }^i{r}_{t+i},\hfill \end{array}$$

(47)

where $\gamma \in [0,1]$ is the discount factor, which balances the immediate reward and the long-term reward.

4.2. DQN-Based NOMA User Grouping Algorithm

To accommodate the reward maximization problem, a DQN-based user-grouping algorithm was proposed in this paper, which is illustrated in Figure 2. In conventional Q-learning, the Q-table is obtained to describe the quality of an action for a given state, and the agent chooses actions according to the Q-values to maximize the reward. However, it will be slow for the system to obtain Q-values for all the state–action pairs if the state space and action space are large. Therefore, to speed up the learning process, instead of generating and processing all possible Q-values, DNNs are introduced to estimate the Q-values based on the weight of DNNs. We utilized a DNN to estimate the Q-value denoted by Q-network, for which the Q-estimation is represented as $Q(s_t,a_t;\theta )$, and an additional DNN with the same setting to generate the target network with $Q(s_t,a_t;{\theta }^{-})$ for training, where $\theta$ and ${\theta }^{-}$ are the weights of the DNNs.

We adopted the $ϵ$-greedy policy with $0<ϵ<1$ to balance the exploration of new actions and the exploitation of known actions by either randomly choosing an action $a_t\in \mathcal{A}$ with probability $ϵ$ to avoid the agent sticking to non-optimal actions or picking the best action with the probability $1-ϵ$ such that [29]:

$$a_t=arg\underset{a_t\in \mathcal{A}}{max}Q(s_t,a_t;\theta ).$$

(48)

Generally, the threshold $ϵ$ is fixed, which indicates that the probability of choosing a random action remains the same throughout the whole training period. However, this brings a fluctuation when the algorithm converges and may lead to divergence again in extreme cases. In this paper, we adopted an $ϵ$-greedy decay scheme, for which a large ${ϵ}^{+}$ (greedier) is given at the beginning, and then, it decays with each training step until a certain small probability ${ϵ}^{-}$. The above policy encourages the agent to explore the never-selected actions at the beginning, and then, the agent intends to take more reward-guaranteed actions when the network has already converged.

The target network only updates every certain iteration, which provides a relatively stable label for the estimation network. The agent stores the tuples $(s_t,a_t,r_t,s_{t+1})$ as experiences to a memory buffer $\mathcal{R}$, and a mini-batch of samples from the memory is fed into the target network to generate the Q-values labels, which is given by:

$$y_i=r_i+\underset{a_{i+1}\in \mathcal{A}}{max}Q(s_{i+1},a_{i+1};{\theta }^{-}),\forall i\in \mathcal{R}$$

(49)

Hence, the loss function for the Q-network can be expressed as:

$$Loss\left(\theta \right)=\left(y_i-Q(s_i,a_i;\theta )\right),\forall i\in \mathcal{R}$$

(50)

The Q-network can be trained by minimizing the loss function to obtain the new $\theta$, and the weights of the target network are updated after ${\delta }_{up}$ steps by replacing ${\theta }^{-}$ with $\theta$. The whole DQN-based user grouping framework is summarized in Algorithm 1.

Algorithm 1 DQN-based user-grouping algorithm.

1: Parameter initialization: 2: Initialize Q-network $Q(s_i,a_i;\theta )$ and target network $Q(s_i,a_i;{\theta }^{-})$. 3: Initialize reply memory $\mathcal{R}$ with size $\left|\mathcal{R}\right|$, and memory counter. 4: Initialize $\gamma$, ${ϵ}^{+}$, ${ϵ}^{-}$, decay step, batch size, target network update interval ${\delta }_{up}$. 5: Training Phase: 6: for$episode=1,2,...,N_{ep}$do 7:  for$timestep=1,2,...,N_{ts}$do 8:   Input state $s_t$ into Q-network, and obtain Q-values for all actions. 9:   Generate a standard uniform distributed random number $\chi \sim \mathcal{U}(0,1)$ 10:   if$\chi >\gamma$then 11:    $a_t\leftarrow arg{max}_{a_t\in \mathcal{A}}Q(s_t,a_t;\theta )$ 12:   else 13:    Randomly select and action $a_t$. 14:   end if 15:   $r_t\leftarrow -{\sum }_{\varphi =1}^{\Phi }{E}_{\varphi }^{tot}\left[t\right]$ the observation to next state $s_{t+1}$. 16:   Store the experience tuple $(s_t,a_t,r_t,s_{t+1})$ into the memory $\mathcal{R}$. 17:   if memory counter $>\left|\mathcal{R}\right|$ then 18:    Remove the old experiences from the beginning. 19:   end if 20:   Randomly sample a mini-batch of the experience tuples $(s_t,a_t,r_t,s_{t+1})$ with batch size, and feed into the DNNs. 21:   Update the Q-network weights $\theta$ by calculating the loss function (50) 22:   Update target network weight ${\theta }^{-}\leftarrow \theta$ every ${\delta }_{up}$ steps. 23:  end for 24: end for

5. Simulation Results

In this section, several simulation results are presented to evaluate the convergence and effectiveness of the proposed joint resource allocation and user grouping scheme. Specifically, the impact of the learning rate, user number, offloading data length, and delay tolerance is investigated. Moreover, the proposed hybrid SIC scheme is compared to some benchmarks including the QoS-based SIC scheme and other NOMA and OMA schemes.

The system parameters were set up as follows. All users were distributed uniformly and randomly in a disc-shaped cell where the base station was located in the cell center. The total number of users was six, and each of them had a task containing 2 Mbit of data for offloading. As previously mentioned, the delay sensitive primary user $U_{m,\varphi }$ was allocated to a predefined power, which was $P_{m,\varphi }=0.5$ W for all groups in the simulation. The maximum delay tolerance for each user was $0.25$ s. In addition, the rest of the system parameters are listed in

Table 1. System parameters.

Effective capacitance coefficient	${10}^{-28}$
Number of CPU cycles required per bit	${10}^3$
Transmission bandwidth B	2 MHz
Path loss exponent $\alpha$	$3.76$
Noise spectral density $N_0$	$-174$ dBm/Hz
Maximum cell radius	1000 m
Minimum distance to the base station	50 m

.

To implement the DQN algorithm, the two DNNs were configured with the same settings, where each of them consisted of four fully connected layers, two of which were hidden layers with 200 and 100 neurons, respectively. The activation function we adopted for all hidden layers was the rectified linear unit (ReLU), i.e., $f\left(x\right)=max(0,x)$, and the final output layer was activated by tanh, for which the range was $(-1,1)$ [30]. The adaptive moment estimation optimizer (Adam) method was used to learn the DNN weight $\theta$ with the given learning rate [31]. The rest of the hyperparameters are listed in

Table 2. Hyperparameters.

$ϵ$-greedy coefficient	0.5–0.01
$ϵ$-greedy decay steps	2000
Discount factor $\gamma$	$0.7$
Reply memory size $\mathcal{R}$	20,000
Batch size	64
Target network update interval ${\delta }_{up}$	10
Number of episode $N_{ep}$	100
Number of time steps $N_{ts}$	500

. All simulation results were obtained with PyTorch 1.70 and CUDA 11.1 on the Python 3.8 platform.

5.1. Convergence of the Framework

In this part, we evaluated the convergence of the proposed DQN-based user-pairing algorithm. Figure 3 compares the convergence rate of the average reward for each episode under different learning rates, which was described by the average energy consumption. The learning rate controls how much the weights of a DNN based on the network loss should be adjusted, and we set the learning rate $=[0.1,0.01,0.001]$ to observe its influence on the convergence. The network with a $0.1$ learning rate converged slightly faster than the one with a $0.01$ learning rate, and both of them converged much faster than the network with a $0.001$ learning rate. However, when the learning rate was $0.1$, even though the higher learning rate had a better convergence, it overshot the minimum and therefore had higher energy consumption after convergence than the other two plots. Moreover, if the learning rate were too low, the network converged slower because it took more episodes to improve the loss function. Therefore, the most suitable learning rate for our proposed DQN network was $0.01$, which was adopted to obtain the rest of the simulation results in this paper.

Figure 4 illustrates the effectiveness of the DQN user-grouping algorithm proposed in this paper. By setting the numbers of users to $[6,8,10]$, the algorithm showed a similar performance that the average energy consumption decreased over training. Although the performance may be worse than the random scheme at the beginning of the training, which was due to the random actions and unstable NN weights, it converged within the first 20 episodes for all three cases. Moreover, more users in the network can result in higher energy consumption, and the algorithm showed superior performance over the random policy, which reduced the energy consumption significantly.

The application of the $ϵ$-greedy decay policy to the convergence performance is further investigated in Figure 5. The $ϵ$-greedy coefficient for the blue curve was set to $0.1$, which indicated that the probability of the NN to choose a random action was $0.1$, and the probability of choosing the action based on (48) was $0.9$. The red curve adopted the $ϵ$-greedy decay policy with the parameters in Table 2. Since the decay policy started with large $ϵ$, the network was more likely to choose the random action at the beginning, and hence, the energy consumption was higher at the beginning. With $ϵ$ decaying over the episodes, the network chose the actions that were selected before that guaranteed large rewards, and therefore, it was more stable afterwards. Meanwhile, the network without the decay policy had significant fluctuations during training. It had more of a chance to choose the random actions throughout the training, even when the NN had already converged, which may lead to the NN becoming divergent again. However, if a very small $ϵ$ were adopted, the network would be less likely to explore some actions, which may result in being stuck in non-optimal actions.

5.2. Average Performance of the Proposed Scheme

In this part, we present the average performance of the proposed NOMA-MEC scheme to show the impact of $P_{m,\varphi }$, the offloading data length, and the maximum delay tolerance. Meanwhile, our proposed scheme is compared with the one without task assignment and OMA offloading to show the superior performance. As shown in Figure 6, the energy consumption of both hybrid-SIC schemes rose and then decreased as $P_{m,\varphi }$ increased. Since $P_{m,\varphi }$ was relatively small at the beginning, $U_{m,\varphi }$ was not likely to be decoded first to satisfy the constraint (14) in the case ${\mathbf{1}}_{n,\varphi }=1$. Therefore, $U_{n,\varphi }$ was more likely to be decoded with priority, and increasing $P_{m,\varphi }$ caused more interference to $U_{n,\varphi }$ according to (4). With $P_{m,\varphi }$ continuing to increase, the power allocation schemes in (22) and (23) became feasible, and more groups in the system could adopt different decoding sequences where $U_{m,\varphi }$ was decoded first. Then, the energy consumption decreased with the increase of $P_{m,\varphi }$, which verified Lemma 1. Moreover, the hybrid-SIC scheme with task assignment outperformed the one without task assignment, shown with the blue line. The one with task assignment had a wider lower bound of the feasible range of power allocation for case ${\mathbf{1}}_{n,\varphi }=1$ in (22), which means that it could adopt the ${\mathbf{1}}_{n,\varphi }=1$ case with smaller $P_{m,\varphi }$. In addition, both hybrid SIC schemes had lower energy consumption than the OMA scheme.

In Figure 7, the energy consumption is presented as a function of the offloading data length. As the data length increased, the average energy consumption also grew. Our proposed hybrid-SIC scheme reduced the energy consumption significantly especially when the data length was large. Moreover, Figure 8 reveals the energy consumption comparisons versus the maximum delay tolerance for $U_{n,\varphi }$. With tighter deadlines, the energy consumption of the hybrid-SIC scheme was much lower than the OMA scheme, and a greater portion of the data was processed locally to save energy compared to the fully offloaded curve.

6. Conclusions

This paper studied the resource allocation problem for a NOMA-assisted MEC network to minimize the energy consumption of users’ offloading activities. The hybrid NOMA scheme had two durations during each time slot, in which NOMA was adopted to serve both users simultaneously during the first time duration, and a dedicated time slot was scheduled to solely offload the remaining part of the more delay-tolerant user by OMA. We assumed the user grouping policy was given at the beginning, the non-convex problem was decomposed into three sub-problems including power allocation, time allocation, and task assignment, which were all solved optimally by studying the convexity and monotonicity. The hybrid SIC scheme selected the SIC decoding order dynamically by the numerical comparison of the energy consumption among different decoding sequences. Finally, after solving those sub-problems, we proposed a DQN-based user-grouping algorithm to obtain the user grouping policy and minimize the long-term average offloading energy consumption. The convergence simulation results showed that the proposed DQN algorithm had similar convergence performance when different numbers of users were chosen, and the $ϵ$-decay policy was effective at stabilizing the network after convergence. In addition, by comparing with various benchmarks, the partial offloading scheme could reduce the energy consumption compared to full offloading, and the hybrid NOMA transmission outperformed the conventional OMA transmission. Hence, it proved the superiority of the proposed NOMA-MEC scheme in terms of energy consumption.