Exploring Reinforcement Learning for Scheduling in Cellular Networks ^†

Abstract

Cellular network scheduling is crucial for wireless deployments like 4G, 5G, and 6G and is a challenging resource allocation task performed by the scheduler located at the base stations. The scheduler must balance two critical metrics, throughput and fairness, which often conflict, as maximizing throughput favors users with better channel conditions, while ensuring fairness requires allocating resources to those with poorer channel conditions. The proportional fairness metric is a prominent scheduling approach that aims to balance these competing metrics with minimal compromise. The common strategy to attain proportional fairness relies on a greedy approach in which each resource block is allocated to the user who maximizes the proportional fairness criterion. With such a strategy, the scheduler can ensure that the resources allocated to the users at each time instance maximize the proportional fairness metric. However, users can usually tolerate some delay and are willing to accept temporary fairness imbalances if they ultimately improve their performance, provided that the fairness criterion is maintained over time. In this paper, we propose a new scheduler that uses reinforcement learning to enhance proportional fairness. The suggested scheduler considers both current and predicted future channel conditions for each user, aiming to maximize the proportional fairness criterion over a set of predefined periodic time epochs. Specifically, by learning patterns in channel fluctuations, our reinforcement learning-based scheduler allocates each resource block not to the user who maximizes the instantaneous proportional fairness metric, but to the user who maximizes the expected proportional fairness metric at the end of the current time epoch. This approach achieves an improved balance between throughput and fairness across multiple slots. Simulations demonstrate that our approach outperforms standard proportional fairness scheduling. We further implemented the proposed scheme on a live 4G eNodeB station and observed similar gains.

1. Introduction

The rapid and continuous growth in the user population, coupled with the dramatic increase in demand per user, has significantly heightened the pressure on air time, turning the wireless channel into a bottleneck and making resource allocation one of the most critical challenges in wireless communication. Resource allocation in wireless communication typically refers to intra-technology scenarios, where resources must be fairly distributed among users utilizing the same technology. Resource allocation can also involve inter-technology scenarios, where the wireless channel is shared among different technologies and protocols, necessitating the distribution of resources across them. For example, the coexistence of LTE and Wi-Fi in unlicensed bands requires fair air time sharing between different protocols (e.g., [1,2]). Unlike inter-technology scenarios, where the challenge lies in distributing resources among users operating under different technologies and protocols with varying syntax and semantics, intra-technology scenarios involve users operating under the same protocol and following the same rules.

The need to manage the rapid expansion of services and applications like the Internet of Things (IoT) and machine-to-machine (M2M) communications, combined with the increasing complexity of global mobile data traffic, has pushed the broadband cellular network industry (5G and the upcoming 6G) to embrace machine learning (ML) and Artificial Intelligence (AI). This shift is particularly evident in the vision for 6G, which aims to support AI by design through a cost-efficient, scalable, and flexible architecture that can abstract and distribute complexities. Resource allocation, especially scheduling, is a prime candidate for leveraging these AI-based frameworks. This potential is amplified by the emergence of new use cases, such as private and enterprise 5G networks, which are expected to become even more prevalent with 6G and bring more demanding Quality of Service (QoS) requirements. In light of these developments, there is a growing need for practical, low-complexity schemes that can be distributed and implemented at the network edge. These solutions would be crucial in managing the increasing complexities of resource allocation in future cellular networks.

In cellular networks, scheduling is one of the base station’s primary tasks, determining which users receive resources like channels and power for each uplink and downlink data transmission opportunity. This function has been essential across all generations of cellular technology (e.g., [3]). Typically, schedulers operate based on predefined performance criteria and goals, which may include maximizing overall data rates, minimizing average latency, or meeting specific deadline constraints (e.g., [4]). Several traditional scheduling algorithms, such as round-robin (RR) and Weighted Fair Queuing (WFQ), have been adapted for wireless systems [5]. However, in wireless networks, users are not homogeneous; different users not only have different abilities but also experience different channel conditions due to factors like fading, multipath propagation, and interference. Thus, schedulers that are oblivious to the users’ channel conditions and avoid the burden (and overhead) of acquiring any information often compromise performance. Accordingly, modern cellular networks (3G and beyond) have given rise to opportunistic schedulers that rely on channel state requisition from users and take into account physical layer information [4]. However, an approach that favors users based on their wireless channel quality and abilities can result in severe unfairness and even starvation of some users. To avoid such significant differences between the throughput attained by various users, a scheduler should adopt a notion of fairness. However, trying to maintain fairness among the users faces two obstacles. First, the notion of fairness is ambiguous and can assume different definitions, e.g., Max-Min fairness criterion, Jain’s fairness index, and Weighted Fair Queuing (WFQ) [6]. Second, attaining fairness can come at the expense of overall throughput. For example, balancing the throughput attained by all users can result in severe sum throughput degradation, as a user with poor channel conditions can consume most of the resources. Accordingly, a scheduler should try to balance attained throughput and fairness, recognizing the potential tradeoff between throughput and fairness (e.g., [7]).

One of the most accepted fairness metrics that tries to cope with this tradeoff is the proportional fairness (PF) criterion, which balances the tradeoff between overall system throughput and fairness among users. The notion behind PF is that the resources allocated to each user are proportional to their respective channel conditions and past throughput. Accordingly, users with better channel conditions receive more resources, yet not to the extent that it leads to significant imbalances. Formally, the proportional fair distribution is the one that maximizes ${\sum }_{k=1}^{K}log{T}_k$, where $T_k$ denotes the throughput that user k has received so far and K denotes the total number of users. Note that the logarithmic function ensures that the utility gained from allocating additional resources to a user decreases as the user’s throughput increases, which helps in allocating more resources to users with better channel conditions without starving users with poor channel conditions (more details on PF are given in Section 3.2).

Proportional fairness-based scheduling has been extensively researched, with numerous variations and implementations. Conventional schedulers typically maximize proportional fairness metrics for each scheduling instance, relying on current user channel states and cumulative throughput per user. Specifically, the cellular base station scheduler employs user feedback on current channel conditions and a rolling average of individual user throughput to assign the next resource block (time slot and subchannel). This allocation goes to the user that optimizes the proportional fairness metric for that particular scheduling instance (e.g., [8,9]). However, standard QoS requirements do not necessitate optimizing performance metrics at every single time instance. Instead, key performance indicators (KPIs) are usually assessed periodically rather than at each time slot. Consequently, there is potential to enhance proportional fairness performance at evaluation points by considering projected future channel conditions for users. This approach might involve postponing allocation for a user with suboptimal current channel conditions (despite maximizing the current proportional fairness metric) and waiting for improved channel conditions within the KPI cycle.

In this paper, we explore the potential gains of incorporating the reinforcement learning (RL) paradigm into the proportional fair (PF) downlink scheduler used in operational cellular networks. An RL-based scheduler can exploit the fact that performance in typical cellular networks is evaluated at the end of predefined cycles rather than after every individual time slot. It can anticipate future users’ channel conditions and delay the allocation of resources to a user, waiting for more favorable conditions. Accordingly, an RL-based PF scheduler allocates resources not only based on the average rate achieved so far and the current channel conditions of each user, as conventional schedulers do, but also relies on future channel state predictions.

We emphasize that the PF objective scheme proposed here is an illustrative example that can be extended to other scheduling objectives. For instance, the same scheme can be easily adapted to handle traffic with due-date constraints.

In this paper, our focus is on exploring the potential of leveraging RL methodology for scheduling purposes and understanding the mechanisms behind such an approach. Therefore, we rely on Q-learning, which is suited for small-scale setups but still provides valuable insights into the scheme.

We evaluated the proposed RL-PF algorithm through simulations, demonstrating its advantages over both the greedy PF scheduler and a round-robin-based scheduler (which allocates equal time shares to each user) across various network scenarios. Additionally, we implemented the RL-PF algorithm in an operational LTE base station, assessing its performance in scheduling users with different path losses. This served as a proof of concept for employing such a scheduler in a cellular base station.

Specifically, the main contributions of this study are as follows: (i) We highlight the significance of fairness criteria in scheduling by analyzing various channel quality indicators collected from a wide range of operational 4G deployments. (ii) We provide a formal definition of PF scheduling that evaluates performance at predetermined periodic intervals, occurring after a fixed number of time slots. (iii) We develop an RL-based downlink scheduler that forecasts and incorporates near-future user channel conditions. This scheduler allocates each resource block to the user expected to maximize the PF metric at the interval’s conclusion, utilizing Q-learning to understand individual user channel fluctuations and make appropriate user selections for each time slot. (iv) Using a Python-based simulation platform, we demonstrate that our algorithm outperforms the standard PF at the predefined time instances. (v) We implement the RL-PF algorithm in an operational LTE base station, proving its practicality and viability for real-world deployment.

This paper is organized as follows. Section 2 reviews related work. Section 3 provides a brief background on relevant topics discussed throughout this paper. In Section 4 we present various channel quality indicators, which are collected in multi-operational 4G deployments. The model is formulized in Section 5. The RL-based scheduler is presented in Section 6. The evaluation parts comprising simulation and implementation results are given in Section 7 and Section 8, respectively.

2. Related Work

In this study, we examine the utilization of the RL paradigm for a prescient scheduler that can learn and predict the channel states of the backlogged users associated with it and allocate resources based on these predictions. As a representative example, we consider the ubiquitous proportional fair scheduler (PFS). Accordingly, we cover related works in both aspects. We first discuss related works in the scope of ML. The second part of this section reviews related works in the scope of schedulers. Since the epitomizing example is related to fairness-based schedulers, we mostly concentrate on fairness-metric-oriented schedulers and in particular PF-based schedulers.

Reinforcement Learning Applications—The growing diversity and complexity of mobile network architectures have made monitoring and managing the multitude of network elements increasingly challenging. Consequently, embedding versatile machine learning-based solutions into future mobile networks is drawing unparalleled research interest. Reinforcement Learning (RL), a subset of ML, focuses on determining the actions needed to maximize a well-defined objective within an environment. Next-generation wireless technology will heavily rely on RL as it enables continuous, real-time optimization and decision-making in complex and dynamic network environments. Specifically, RL learns and adapts from interactions with the network, making autonomous decisions that enhance performance and resource allocation. This capability allows for adaptive management of network resources, efficient spectrum utilization, and proactive adjustments by predicting and addressing issues before they affect users. Accordingly, RL can significantly boost the performance of next-generation wireless networks in many aspects.

Over the last few years, numerous studies have incorporated RL into wireless communication protocols, demonstrating its effectiveness in various applications such as dynamic spectrum access, power control, load balancing, and more. This showcases RL’s potential to revolutionize wireless communication. For instance, recent examples include a survey on RL-based medium access control (MAC) protocols and methodologies for applying RL within the context of the MAC reference model [10]. Additionally, a deep reinforcement learning-based decentralized resource allocation scheme is suggested in [11] for the vehicle-to-everything (V2X) communication network, where vehicle-to-infrastructure (V2I) and vehicle-to-vehicle (V2V) networks share resources. This scheme utilizes a deep Q-network (DQN) to determine the resource blocks (RBs) and transmit power of vehicles in the V2V network.

Current and next-generation wireless communication systems are characterized by high diversity, heterogeneity, and density. This complexity leads to resource scarcity, necessitating sophisticated resource allocation mechanisms. RL has emerged as a fertile field for innovations in addressing these challenges. Ref. [12] focuses on optimizing system throughput and fairness in an underlay device-to-device (D2D) communication system using device-centric joint spectrum and power allocation schemes. The study utilizes a distributed RL-based iterative resource allocation scheme that overcomes the unknown and dynamic nature of channel parameters by allowing the system to learn about the environment through trial and error, without requiring any prior knowledge. Ref. [13] studies the problem of energy-saving optimization in 5G base stations (BSs) by implementing multi-sleep mode (SM) levels. The paper investigates the tradeoff between energy saving and QoS and suggests a distributed Q-learning algorithm that can learn the best policy to control the activity of the small base stations (SBSs) in order to minimize energy consumption and data dropping. Ref. [14] investigates energy-efficient resource allocation and power control techniques in 5G networks. The paper presents a 5G base station (BS) sleep mode management based on reinforcement learning that balances a set of metrics to find the best tradeoff between energy reduction and QoS constraints.

A literature review on deep reinforcement learning (DRL) techniques and applications used for energy optimization and resource allocation in future wireless and mobile networks is provided in [15]. Ref. [16] proposes an adaptive DRL-based resource allocation scheme to jointly allocate downlink power and bandwidth, thus optimizing total throughput while considering the impact of power-consumption outage problems. Ref. [17] investigates user association in multi-unmanned aerial vehicles (UAVs)-assisted cellular networks, allowing users to connect with different UAVs or macro base stations for uplink (UL) and downlink (DL) transmissions. The paper formulates the joint optimization problem for UL-DL association and beamforming as a partially observable Markov decision process (POMDP) and employs multi-agent deep reinforcement learning (MADRL) for distributed optimization. Ref. [18] formulates the problem of maximizing the sum rate and fairness for non-orthogonal multiple access (NOMA)-based cellular users as a mixed-integer nonlinear programming (MINLP) problem, and a learning framework was employed to address it. Ref. [19] utilizes the multi-agent reinforcement learning (MARL) paradigm to develop distributed strategies to reduce processing complexity and energy use in large-scale multiple-input multiple-output (XL-MIMO) systems. In their multi-agent cell-free XL-MIMO system, base stations (BSs), user equipment (UE), and antennas are viewed as agents that learn to allocate resources and transmit signals efficiently. Two use cases of the MARL-aided multi-agent cell-free XL-MIMO system were studied: antenna selection and power control.

Additional state-of-the-art reinforcement learning (RL) algorithms related to resource allocation have been explored in various studies. In [20], researchers investigate RL for downlink inter-cell interference mitigation through power control and rate adaptation. They propose an RL framework to develop an efficient power control policy that optimizes the downlink transmission power of LTE cells, aiming to maximize network-wide utility for users. Their RL state representation includes LTE cell measurements such as cell downlink transmission power, average Reference Signal Received Power (RSRP), and average user interference within each cell. Actions are based on this representation adjust cell transmission power, with a focus on maximizing utility functions like average sum log-throughputs across cells. In [21], the authors design and implement an RL testbed for Multi-User MIMO (MU-MIMO) user selection in IEEE 802.11ax. Their approach applies Q-learning with the $ϵ$-Greedy policy [22] to select user groups for scheduling. In [23], a Q-learning algorithm for allocating RBs to users is devised. The algorithm relies on the scheduler receiving CQI feedback from all users before each RB allocation, assigning the RB to the user that maximizes Jain’s Fairness Index (JFI) [24].

In [25], the authors propose a multi-agent Q-learning algorithm for joint power and resource allocation in 5G networks. In this algorithm, each general eNodeB (gNodeB) performs joint power and resource block allocation at every scheduling interval. The states and rewards are formulated to consider two critical factors: inter-cell interference and queuing delay. The authors in [26] use a deep reinforcement learning (DRL) approach to optimize resource allocation in OFDMA systems. Specifically, they propose a centralized DRL-based scheduling scheme that is traffic- and channel-aware. This scheme relies on real-time network traffic and channel conditions to manage downlink OFDMA time-frequency resources at a wireless access point. In [27], a DRL-based resource allocation approach is developed for ultra-dense networks (UDNs) in the presence of very limited channel state information (CSI). The multi-objective problem that considers the tradeoff between spectrum efficiency (SE), energy efficiency (EE), and fairness is decomposed into two components. Specifically, the deep neural network is designed to maximize SE, while the EE and fairness considerations are incorporated as rewards to guide the training process. Ref. [28] explores the use of DRL framework to maximize user fairness in terms of delay. Ref. [29] proposes a 5G downlink scheduling mechanism that employs Q-learning techniques to improve wireless transmission quality and efficiently manage the radio resources of a BS. The mechanism aims to enhance various Quality of Service (QoS) parameters, including throughput, delay, and fairness, as measured by Jain’s fairness index. Ref. [30] presents a novel method to integrate fairness into actor-critic reinforcement learning for network optimization tasks. Their algorithm adapts the reward system by using a weighting factor that accounts for both the structure of the fairness utility function and historical reward statistics. This approach ensures that the rewards are adjusted dynamically based on these two elements.

Scheduling in Wireless Communications— The scheduler in a cellular-based network plays a fundamental role; it is the “brain” of the whole system [3]. Traditionally, its basic function is to allocate resources for the connected users, taking into account QoS and channel condition. Different schedulers consider different utility metrics, e.g., max throughput and various types of fairness (proportional, equal share, Max-Min, etc.), and therefore the network performance is influenced by the objective. Scheduling algorithms that were commonly adapted in implemented wireless systems disregarded users’ channels, for the sake of simplicity, e.g., round-robin (RR), which serves users in a circular manner without any other consideration, [31], and Weighted Fair Queuing (WFQ), which allocates resources with respect to weights that are associated with every user [32]. In contrast, 4G and 5G cellular networks have given rise to schedulers that take advantage of physical layer information, such as user channel states. Such schedulers are named opportunistic schedulers. Proportional fair scheduler [33] is one of the popular channel-aware schedulers implemented in today’s cellular systems and being as such, much work has been performed on the subject.

Opportunistic schedulers take into account information such as the users’ channel conditions and QoS, for the purpose of allocating resources for each connected user. Opportunistic scheduling was first coined in [34], showing that by utilizing multi-user diversity, a significant improvement in capacity can be made. Their approach was to schedule the user (or users) with the best channel state in any given time slot. The work of [35] was continued and improved in [34] by considering delay aspects, when they treat the situation of starvation. Best Quality Indicator (BCQI) [36] is a simple LTE scheduling algorithm that aims to maximize the cumulative throughput by assigning the network resources to users that report the best channel quality in the current time slot. Max-Weight scheduler [37], selects the user with the highest product of queue length and transmission rate. The algorithm was considered throughput-optimal until [38] proved otherwise under flow level dynamics. However, Max-Weight is still throughput-optimal with fully backlogged queues. We refer the interested reader to [3,4] for more details on the different scheduling algorithms. As mentioned, the aforementioned scheduling solutions are throughput optimized; however, optimizing the cell capacity comes with the cost of unfair resource allocations to those users that experience bad channel quality.

Since opportunistic schedulers can favor users over substantial time intervals, fairness is a major concern in communication systems, e.g., scheduling users according to their channels’ conditions might lead to starvation of users with poor channel quality, which is not only unfair but can lead users to unsubscribe from the service provider. The notion of fairness is ambiguous (i.e., what is a fair allocation and how we can measure it), and different metrics were suggested over the years to define fairness, e.g., Jain’s index, Max-Min, proportional fairness, and more.

Jain’s index is a widely known metric which we utilize in our simulations (Section 7). This is a quantitative measure, first proposed by [24] that aims at quantifying the fairness of resource allocation among traffic flows. Jain’s index values range between one divided by the number of flows and one, where a unity value indicates maximal fairness. Such a metric, however, disregards the heterogeneity of the users and their channel conditions, and tries to balance the rates attainable by all users, irrespective of their capabilities. Max-Min fairness is a qualitative measurement of fairness that has been studied widely and implemented in various applications [39], and its main motivation is that all users should attain the same performance (e.g., throughput). Specifically, a system reaches Max-Min fairness if it cannot increase any individual’s share without decreasing another’s that has a lower one. Accordingly, Max-Min fair allocation can allocate most of the resources to users with very poor capabilities (e.g., poor channel conditions), sacrificing overall performance in order to improve the performance for degraded users.

One of the most diffused opportunistic approaches with fairness constraints is the proportional fair scheduler (PFS), first introduced by Kelly in [8]. A set of attainable throughput (rate) allocations $x_k$ is proportionally fair if it maximizes ${\sum }_{k\in N}log{x}_k$, where N is the set of users. Later, [9] adapted the proportional fairness criteria and suggested a scheduling algorithm that assigns priorities to users based on the ratio of two components: the first component accounts for the rate potentially achievable in the current time slot, while the second component accounts for the historical average of the user’s throughput. Such an algorithm was proven to maximize the logarithmic sum of capacities. The work of [40] suggested that [9] does not fully address the problem of starvation due to a constant average window size of the UE rate, which relates to the time a UE can be starved, in scenarios, for example, where a user experiences sudden drops in channel quality, forcing the PFS to starve the user for a certain time period. Instead of using a constant average window size, they offered to update the average UE rate by monitoring the UEs’ starvation time period, giving rise to a much more starvation-aware scheduling.

In [41], four PF scheduling algorithms were introduced that are based on the PF algorithm for handling cases where users are not fully backlogged and a user can be in an idle or active state. The first two algorithms only update the average throughput of active users, instead of updating average throughput for active and idle users. The third algorithm suggests a different approach for a user’s throughput update, according to the status of the user in the previous slot, i.e., active or idle, and the last time the user was scheduled. The fourth algorithm dynamically changes the averaging time window with respect to the backlog of users. Ref.  [42] extended PF scheduling to multi-carrier transmissions systems, proposing an assignment of users to each carrier while maximizing the sum of logarithmic average user rates.

PF channel-aware schedulers, such as those proposed by [43], rely on channel stationarity and achieve fairness by employing a moving average scheme. At each time slot, the channel is allocated to the UE that maximizes the average PF metric at the end of the current slot. These schedulers achieve fairness over long time intervals and can even converge to the optimal fair allocation as time approaches infinity (see [33]). However, typical operational networks measure performance and report key performance indicators (KPIs) periodically, after finite and predefined time intervals. Consequently, “greedy” schedulers that maximize fairness at every time slot may compromise overall performance. This is because it may sometimes be beneficial to postpone scheduling a user who maximizes the current fairness metric within the KPI time interval, waiting for that user to experience better channel conditions.

A strategy that allows the postponement of an allocation to a deserving UE, waiting for better channel conditions, does not ensure fairness after each allocation. However, such a strategy can boost overall performance at the end of each KPI time interval. To deploy this strategy effectively, a mechanism for predicting users’ channel conditions is required. Analysis of the gains from knowing future channel conditions is proposed in [44]. Specifically, the authors assume that future channel states for all users are known a priori (i.e., accurate estimates of the users’ attainable data rates for the following $L-1$ time slots are available to the scheduler at each time slot). They compute the optimal allocations for the finite L time slots and compare these to the PFS greedy allocation.

The authors of [45] extend the single-channel prediction-based PFS scheme suggested in [44] to evaluate the effects of considering predicted data rates on the performance of multi-carrier networks. Specifically, ref. [45] investigated a multiple subchannel model in which the rates each user can attain on each channel are known a priori, and the scheduler determines which user is allocated to each subchannel (a user may be assigned more than one subchannel). The authors show that the resulting optimization problem is a complex combinatorial problem in both the time and frequency dimensions and propose two suboptimal algorithms.

3. Preliminaries

This section provides a brief background.

3.1. Q-Learning Background

We provide a concise description of the Q-learning algorithm we utilize in this paper (a comprehensive background can be found in [46] and many other references).

Q-learning [22] is an off-policy reinforcement learning algorithm that seeks to find the optimal policy that maximizes the expected accumulated rewards starting from the current state. It does not require the transition probability distribution or the reward function hence it is termed a “model-free” algorithm. The basic idea behind Q-learning is to approximate the values $Q(s,a)$ of state–action pairs through samples obtained during the interactions with the environment, enabling the agent to select the optimal action ($ma{x}_{a}Q(s,a)$) for a given state (s). Accordingly, Q-learning maintains a two-dimensional matrix termed a Q-table, which stores the estimated Q-value of each state–action pair ($\mathbb{Q}(s,a)$), independent of the policy being followed. This Q-table serves as a reference table for the agent to select the best action needed to be taken in a given state based on the Q-values associated with these states. The table’s dimensions are $\left|\mathbb{S}\right|\times \left|\mathbb{A}\right|$, such that each state has an entry for each possible action (the table index represents a state and an action). The table entries are initialized to arbitrary values (in the results presented herein all Q-values were initialized to zero), and are updated after each action based on the result. These updates balance the estimated current value with the instantaneous reward gained by taking a specific action and the predicted Q-value of the new state. Specifically, these updates are defined as follows:

$${\mathbb{Q}}_{t^{+}}(s,a)={\mathbb{Q}}_t(s,a)+\alpha \left[{\mathbb{R}}_t(s,a)+\gamma max{\mathbb{Q}}_{t+1}(s,a)-{\mathbb{Q}}_t(s,a)\right]$$

(1)

where ${\mathbb{R}}_t(s,a)$ is the immediate reward obtained as a result of being in state s and taking action a, $\gamma$ is the discount factor that determines the importance of future rewards, and $\alpha$ is the learning rate that is used to determine the impact of the new information upon the existing Q-value. The learning rate ($\alpha$) can be chosen to be constant or adjusted dynamically during the learning process. The algorithm can terminate when all Q-values converge or after a certain number of iterations. The process can also restart (or continue indefinitely) in order to encounter changes in the environment (e.g., channel state distribution changes, different backlogged users).

During the learning stage, the agent explores different actions for each state and performs them multiple times to obtain an accurate Q-value estimation. To balance exploration and exploitation, we used the prevalent epsilon-greedy strategy. At the start, when the agent has little knowledge about Q-values, it mostly explores by selecting actions at random (high epsilon). As the agent gains more confidence in its Q-value estimates, epsilon decreases, and the agent exploits more by selecting the action with the max Q-value for a given state. Eventually, the agent terminates the learning stage and exploits its Q-table to choose the best action for each state.

Q-learning, being a tabular method for storing the Q-values, is limited by system complexity. As a result, the Q-learning algorithm might not be practical when trying to design a scheduler for a large cellular network. Other RL concepts such as neural networks can be utilized to handle such limitations.

3.2. Proportional Fairness (PF) Background

The proportional fairness criterion was introduced in Kelly’s seminal paper [47] as an alternative to the Max-Min fair metric, which can often allocate a significant amount of system resources to marginally improve the performance of the lowest-performing user. In the context of throughput distribution, the proportional fairness criterion can be expressed as ${\sum }_{k=1}^{K}log{r}_k$, where K represents the total number of users sharing the link and $r_k$ denotes the achievable rate for user k. Communication networks, particularly cellular networks, operate with indivisible resource units (see Appendix A.1 for details). This necessitates a modification to the fluid model, where service can be partitioned to allow all users to receive allocations simultaneously. Instead, the proportional fairness metric in these networks considers the average throughput a user receives over a specified time interval. Let $T_k$ represent the average rate received by user k. The proportionally fair distribution is then defined as the one that maximizes:

$$\begin{array}{c}\hfill \sum _{k=1}^{K}log{T}_k\end{array}$$

(2)

To achieve the proportional fairness criteria outlined in (2), ref. [33] proposes a user selection algorithm. This algorithm considers two key factors: the users’ channel quality in time slot t, which is converted to an instantaneous data rate $R_k\left(t\right)$, and a moving average of each user’s achieved throughput $T_k\left(t\right)$. The scheduling algorithm then selects the user $k^{*}$ that:

$$\begin{array}{c}\hfill k^{*}=\underset{k\in K}{argmax}{\displaystyle \frac{R_k\left(t\right)}{T_k\left(t\right)}}\end{array}$$

(3)

The average throughputs $T_k\left(t\right)$ are updated using an exponentially weighted low-pass filter

$$T_k(t+1)=\left\{\begin{array}{cc}(1-{\displaystyle \frac{1}{t_c}})T_k\left(t\right)+{\displaystyle \frac{1}{t_c}}{R}_k\left(t\right),\hfill & k=k^{*}\hfill \\ (1-{\displaystyle \frac{1}{t_c}})T_k\left(t\right),\hfill & k\ne k^{*}\hfill \end{array}\right.$$

(4)

Viswanath et al. [33] prove that the proposed proportional fair algorithm maximizes (2) when the time window approaches infinity ($t_c\to \infty$). However, practical networks like LTE and 5G evaluate performance metrics (e.g., KPIs) over finite time scales. Consequently, ensuring fairness over infinite or extended periods is often impractical and insufficient for real-world applications. Moreover, key performance indicators (KPIs) are typically assessed periodically at predetermined intervals rather than on a per-time-slot basis. This periodic assessment eliminates the need to maintain fairness at every individual time slot; instead, fairness needs to be ensured only at the conclusion of these specified evaluation periods.

Proportional fairness is the conventional fairness metric utilized by operational BSs, and numerous studies have suggested different proportional fairness-based algorithms; several examples are provided in Section 2.

3.3. From Theory to Practice: Implementing Proportional Fairness in 4G and 5G Networks

Opportunistic scheduling methods, such as proportional fairness, rely on users’ channel quality reports to optimize scheduling decisions. Specifically, the algorithm described earlier requires knowledge of all users’ instantaneous channel conditions, which are translated to the appropriate transmission rates, allowing the selection of the user that maximizes the equation $\underset{k}{argmax}{\displaystyle \frac{R_k\left(t\right)}{T_k\left(t\right)}}$. In wireless networks, this channel state information is obtained through reports that are sent occasionally by the users to the BS. Practical systems support only a finite set of rates. Accordingly, the channel’s condition should be adapted to the set of transmission rates supported by the users. Specifically, each user’s transmission rate should be chosen from its supported transmission rates that are compatible with its channel conditions, i.e., the transmitted packet is most likely to be received successfully by the BS under the current channel condition. Link Adaptation is the process by which transmission code rates and modulation schemes are chosen based on prevailing channel conditions and recent transmission success rates. This mechanism ensures that the most appropriate data rate is selected for each user given their current channel state.

The downlink link adaptation (DLLA) procedure is a crucial part of wireless communication systems. It can dramatically affect the performance of the network by controlling the data rate that can be reliably transmitted [48]. In Appendix A.1, we briefly explain the concepts and processes of a typical LTE DLLA that we utilized in the evaluation part, both in the simulation and experimental results. Since the description follows the common terminology and accepted acronyms, and is somewhat cumbersome, we have included it in the Appendix A.

4. Operational 4G Deployments’ Statistics

In this study, we propose an opportunistic RL-based scheduler that considers both current and anticipated future user channel quality. The advantages of the proposed scheme lie in two key aspects: its opportunistic approach to resource allocation and its ability to predict users’ channel states. While our focus is on fairness, we also recognize the applicability of these aspects to a broad range of objectives, such as due date-constrained traffic and QoS guarantees.

To understand the significance of these aspects, particularly the potential benefits of opportunistic schedulers, we analyze channel statistics collected from various operational 4G deployments from two perspectives: intra-cell and inter-cell UEs’ channel quality variability. Intra-cell channel quality variability highlights the differences in channel quality experienced by UEs within the same cell, thereby justifying the need for opportunistic resource allocation schemes that adapt to the varying channel qualities of UEs. Additionally, considering that the channel quality of each UE may fluctuate over time due to factors like user movement or environmental changes (e.g., movement of other UEs in the vicinity), this underscores the predictive nature of the proposed scheme. Inter-cell channel quality variability, on the other hand, underscores the diversity of the cell environments themselves and the variability in UE distribution within cells, emphasizing the importance of a distributed learning-based scheduling scheme conducted independently by each cell.

We utilized two datasets to illustrate the high diversity in users’ channel quality characteristics, which are correlated with the users’ attainable rates. The first dataset comprises data gathered by a 4G vendor from 90 operational 4G deployments across diverse locations in the United States, including both urban and rural environments. The second dataset consists of data collected from various 4G operators in the states of New York and Florida.

We examine three common KPIs: (i) Channel quality indicator (CQI): This KPI maps the users’ channel condition into a quantized discrete value between 0 and 15 (further details on CQI can be found in Appendix A.1). The same CQI values were used in both the simulation and implementation. (ii) RSRP is the average of the base station’s reference signals measured by the users, indicating the strength of the base station signal to a user. The farther the distance between the user and the base station, the lower the RSRP value received, and vice versa. We also compare RSRP with SNR in rural and urban deployments to provide further insights into how the environment type affects users’ channel conditions. (iii) Average resource utilization: Measured in physical resource blocks (PRBs), this KPI is assessed for both urban and rural deployments (further information on LTE resource allocation can be found in Appendix A.1).

We note that the schedulers deployed in the base stations that collected the data are PF-based, similar to those used in our simulation and implementation results for comparison with the proposed scheme. Additionally, the users’ channel characteristics obtained in this subsection are utilized in our simulations and implementation (Section 7 and Section 8, respectively).

Figure 1 depicts the average number of CQI reports collected for each CQI value during one day, averaged over the inspected cells, i.e., all measurements collected were divided by the total number of cells.

Figure 1 confirms the diversity in channel quality attained by users (recall that the measurements are taken throughout one day, hence the diversity shown encompasses both a variety of users as well as users whose channel state has been changed). As seen in the figure, even though most measurements reported good channel conditions and only a few experienced inferior ones, a substantial number of measurements encountered moderate channel conditions. Assuming that the distribution to CQI roughly reflects the temporal CQI distribution of backlogged users during a given time in the day, Figure 1 highly motivates the utilization of a fairness metric that balances between maximizing the attained users’ sum rate while providing each user a reasonable share of the resources. The observation that the diversity also relates to users with varying channel conditions motivates the use of an RL-based scheduler. The proportional fair scheduler, which we utilize in this study, is a channel-aware scheduler that utilizes users’ channel diversity for a better resource allocation policy and aims to balance the overall attainable rate and fairness, giving more RBs to users with high CQI values, yet allocating sufficient RBs to the low-CQI users such that the share each user is given conserves the ratio between the users’ average achievable rate. Note that the high standard deviation shown in the figure, representing the high variability, further emphasizes the need for an opportunistic channel-aware scheduler leveraging a learning-based solution.

Reference Signal Received Power (RSRP) is a key performance indicator (KPI) that assesses the average power of the reference signals received by the UE across a specific bandwidth. It is measured in dBm and typically ranges from about −140 dBm (very weak signal) to −44 dBm (very strong signal). The RSRP reports collected at urban cells and rural cells are depicted in Figure 2a and Figure 2b, respectively.

As can be seen in the figures, in both types of deployments, the received signal strength is widely dispersed, ranging from very poor RSRP ($-140$ dBm to $-110$ dBm) to excellent RSRP ($-80$ dBm to $-44$ dBm). The average RSRP values for the two kinds of deployments differ significantly: for urban cells it is −97 dBm, while for rural cells it is −104 dBm. This difference can be explained by the characteristics of the different deployments. In a more urban environment, more base stations are deployed, and therefore, users are mostly closer to their associated base stations. In contrast, in a rural area, fewer users are connected, and fewer base stations are deployed, such that each base station serves users who are farther away. Moreover, the RSRP distribution (especially in the rural deployment) seems to follow a right-skewed distribution. This result can be explained by the fact that most users in such deployments are in low-to-medium coverage areasfrom the base stations, while few users are in excellent coverage areas, extending the right tail into high RSRP values.

Overall, the RSRP measurements depicted in Figure 2 further support the CQI measurements in demonstrating the high diversity in signal path loss experienced by UEs, which indicates considerable differences in the received signal strength in both types of deployments.

Next, rather than averaging over a large number of cells, showing significant variance between the different cells, we focus on specific zones showing variability within a zone. We distinguish between urban and rural deployments. Figure 3 depicts a heat map indicating RSRP and SINR in the urban region of New York and the more rural region of Florida. The heat maps are divided into different zones according to zip codes. Specifically, Figure 3a and Figure 3c depict the RSRP of the urban region of New York and the rural region of Florida, respectively. We defined an RSRP threshold of $-116$ [dBm], which was taken from lab results and indicated a user with poor coverage, i.e., a user with a low CQI (a CQI that corresponds to QPSK modulation) and colored each zone based on the percentage of RSRP reports which were below this threshold (denoted by $\varphi$). For example, the dark green color indicates that less than 5% of the RSRP reports in the zone were below $-116$ [dBm], and zones marked in red are zones in which more than 20% of the collected RSRP reports were below the $-116$ [dBm] threshold. Similarly, Figure 3b and Figure 3d show the average SINR reporting in the urban region of New York and the rural region of Florida, respectively, where the SINR threshold was set to 2 [dB], i.e., $\varphi$ denotes the percentage of SINR reports in a zone which are below 2 [dB].

The RSRP heat maps depicted in Figure 3a,c confirm the assertion that the received signal strength in an urban environment is expected to be stronger than in a rural one. In contrast, Figure 3b,d indicate that in the urban environment, there is a large number of zones where users suffer from poor SINR conditions, even though the RSRP indicated is mostly good, whereas, in the rural area of Florida, an inverse effect is observed, more zones are associated with poor RSRP. However, most users experience good SINR. The seemingly contradicting observation can be explained by the fact that despite the high (low) received signal strength (RSRP) in an urban (rural) environment users in the urban (rural) environment are more (less) prone to inter-cell interference, and therefore have low SINR, such that counterintuitively, users’ SINR in the urban environment might experience worse channel conditions than in the rural environment.

Last, we examine the daily average system resource utilization per scheduling time unit in both rural and urban deployments. User resource allocations are measured in PRBs, which are the basic time-frequency resource units that can be allocated to a UE. The number of PRBs available in the system per scheduling time unit is determined by the LTE base station’s bandwidth. At each scheduling epoch, the available PRBs are distributed among the selected UEs. The rate, and therefore the number of bytes attained by each scheduled UE per PRB, depends on its channel conditions. The number of PRBs allocated to each scheduled UE is determined by its target number of bytes and its channel condition. A user in poor channel conditions will require more PRBs to support the required number of bits, and vice versa. Therefore, a higher average PRB utilization can be attributed to a higher number of UEs with poor channel conditions and to a higher number of UEs scheduled at each scheduling epoch.

The measurements show that the average number of PRBs allocated to each UE in rural terrain is $5.9$, while in urban terrain it is $12.4$. This indicates that the average PRB allocation in an urban deployment is approximately twice as much as in the rural counterpart. These results can be attributed to the observation that more UEs in an urban environment experience poor channel conditions than in the rural counterpart, as can be seen in Figure 3b,d. It can also correlate with the characteristic that the population density is typically higher in urban environments compared to rural ones.

In this section we illustrated the high variability in users’ channel quality, which highly motivates for an efficient fairness metric that balances between maximizing the throughput yet giving each user a substantial share of the available resources, one that shares the resources proportionally to its capabilities, yet allows each user to operate (run applications) satisfactorily.

5. Model

While cellular base stations support bidirectional data traffic, in this study we focus on the downlink traffic to K users. We generally follow the conventional cellular resource units in which the time is slotted into constant intervals, and each slot is divided into RBs. Each RB comprises a bandwidth and time duration, e.g., in LTE each RB comprises 12 sub-carriers in the frequency domain and 1 ms in the time domain. The RBs cannot be shared, and a single user can use each RB. The BS (scheduler) distributes these RBs between the users such that a subset is allocated to a single user. For simplicity, throughout most of this study, we will assume that the bandwidth cannot be partitioned, i.e., a single user has the entire bandwidth at each time slot (in this study, time slot and RB are interchangeable). However, we shall also present results in which the BS selects a single user for a subset of RBs in a time slot. Note that similar results will apply to a setup where the BW can be shared among different users and the BS selects a user for each subset of RBs in a specific time slot. We assume that users’ downlink transmission rates, which are based on each user’s channel quality and link adaptation module, are already determined by the BS prior to each time slot (see Appendix A.1 for details). We further assume a stochastic wireless channel, such that there is a probability that the channel between each UE and the BS can change. However, we assume that the channel remains static within the time slot, as in [49,50,51]. Prior to each time slot, the scheduler selects a user and transmits to it at a certain rate that with some probability might fail, despite the DLLA mechanism. Note that this loss probability can capture a model in which the channel can change during a time slot.

We assume that time is divided into L time slot cycles. The rate values received by the users are reset at the beginning of each cycle and are evaluated at the end of the cycle. The key performance indicators (KPIs) are measured at the end of each cycle based on the proportional fairness metric. Specifically, in this study we concentrate on proportional fairness, hence denoting the throughput user k received at the j-th slot of cycle i by $T_k^i\left(j\right)$, the accumulated throughput of user k at the end of cycle i, denoted by $T_k^i$, is $T_k^i={\sum }_{j=1}^L{T}_k^i\left(j\right)$. The system’s proportional fairness utility metric at the end of the i-th cycle is:

$${\mathrm{U}}^i=\sum _{k=1}^{K}log{T}_k^i$$

(5)

Throughout this study, we primarily focus on scheduling within a single cycle. To simplify our notation, we will omit the cycle index when discussing a general cycle. Consequently, $T_k\left(j\right)$ represents the throughput achieved by user k in the j-th time slot, and  $T_k$ denotes user k’s total accumulated throughput at the cycle’s conclusion.

6. Reinforcement Learning Proportional Fairness-Based Scheduler

In this section, we present the RL-PF-based scheduler. The motivation for an RL-based scheduler relies on the understanding that since the performance, and specifically the proportional fairness criterion (2), is measured only at the end of each cycle, it is sometimes beneficial to delay an allocation for a user, waiting for better channel conditions within the same cycle. We illustrate this with a simple example depicted in Figure 4.

Assume a BS serving two users ($K=2$), and that each cycle comprises ten time slots ($L=10$). The blue and orange curves in Figure 4a,b depict the attained rates by users UE-1 and UE-2, respectively, over an exemplary cycle. The asterisks in Figure 4a depict the scheduled user at each time slot according to the greedy PF algorithm described in Section 3.2. For example, the first slot is allocated to the user that maximizes the current PF criteria, which is UE-1 (since both users have received an equal share so far, zero bytes, the user with better channel quality is given the allocation); similarly, the second slot is allocated to UE-2 that recieved nothing so far in the cycle, hence contributes the most to the PF criteria, etc. Figure 4b depicts the allocations that are based on a scheduler that aims at maximizing the proportional fairness criteria at the end of the cycle, hence postpones the allocation for UE-2, waiting for it to attain better channel conditions. The blue and orange curves in Figure 4c depict the sum log throughput of the two users (Equation (2)) prior to every time slot according to the greedy PF algorithm and the delay-tolerant PF algorithm, respectively. Time slot j on the x-axis corresponds to the sum log throughput at the end of slot $j˘1$ and the beginning of slot j, just before the scheduler selects the UE to be allocated slot j according to the two schedulers. As can be seen in the example, allocating slots 2 and 4–8 to UE-2 by the greedy PF scheduler provides a better instantaneous gain in (2) at the end of the first 8 slots out of the 10-slot cycle; however, allocating it slots 9–10 when it has better channel conditions attains higher PF utilization at the end of the cycle. Additional motivation can be found in [44], which shows that the availability of future channel estimations could help improve the proportional fairness criterion.

Maximization of $\mathbb{U}$, as defined in (5) encounters two main challenges: (i) the scheduler (BS) needs to know all users’ channel conditions for the entire cycle at the beginning of the cycle, which necessitates users’ channel prediction, and (ii) attaining optimality requires an exhaustive search over all users’ scheduling possibilities. To address these challenges, we employ a Reinforcement Learning (RL) scheme characterized by four elements: agent, environment (states), actions, and rewards (see Section 3.1). The agent, in our case the BS or more specifically, the scheduling algorithm deployed at the BS, determines to which user to allocate each resource block. The environment encompasses the current channel states of all the users. The action space, which determines which user will be allocated the next resource block, comprises all users, and the reward is the sum log throughput that is attained only at the end of the cycle. The policy, which is the aim of the suggested scheme, defines the agent’s strategy for associating actions with states. In the following, we formalize the main components of the PF scheduling setup:

• The state space is represented by a vector with K elements, where K is the number of users. Each entry in the vector consists of a 2-tuple $(T_i\left(n\right),CQ{I}_i\left(n\right))$, where $T_i\left(n\right)$ is the accumulated throughput that user i has achieved up to the start of time slot n in the current cycle, and $CQ{I}_i\left(n\right)$ indicates the channel quality for user i at the beginning of time slot n. This structure results in a distinct set of states for each time slot within the cycle. Specifically, denoting by ${\mathbb{S}}_n$ the set of states associated with the beginning of time slot n, we have:

  $$\begin{array}{c}{\mathbb{S}}_n=\{\left(T_1\left(n\right),CQ{I}_1\left(n\right)\right),\dots \left(T_i\left(n\right),CQ{I}_i\left(n\right)\right),\dots \hfill \\ \hfill \dots ,\left(T_K\left(n\right),CQ{I}_K\left(n\right)\right)\}n=1,2,...,L\end{array}$$

  (6)
• The action space, denoted by ${\mathbb{A}}_n$, is defined by a specific user scheduled at the beginning of time slot n, who will be allocated the entire bandwidth. The key challenge lies in the fact that the action (choosing which user to schedule next) taken at each time slot must be evaluated based on future rewards (delayed rewards). It is important to note that in this study, we deal with a deterministic action space, meaning the scheduler must choose a specific user to assign the resource block to. This contrasts with a stochastic action space, where the scheduler can allocate time slot n to different users according to some probability distribution, i.e., the time slot is allocated to each user i with a probability $p_i$ that is not necessarily 0 or 1 ($0\le p_i\le 1$).
• The reward, denoted by $\mathbb{R}$, is given only at the end of each scheduling cycle and is computed based on the proportional fairness criterion as formulated in Equation (5), i.e., $\mathbb{R}={\sum }_{k=1}^{K}log{T}_k$. An important characteristic of this reward structure is that intermediate actions do not yield instantaneous rewards. Instead, the reward is determined only once the entire cycle has been completed.

In this paper, we focus on understanding the potential gains of deploying RL for the PF-based scheduler, and we leave the challenge of handling an enormous state space (e.g., large user populations, high number of supported rates, etc.) for future work. Accordingly, we utilize Q-learning, a model-free RL algorithm that learns the value of each action for each state.

We employ the epsilon-greedy approach for our scheduling strategy. In this method, the scheduler follows the learned policy for the current state with a probability of $1-ϵ$ or, with a probability of $ϵ$, deviates from this policy to explore various actions. Specifically, when exploring, it selects a user randomly and allocates the resource block (RB) to that user (e.g., [52]). After each scheduling decision at time slot n, we calculate and store a Q-value in the Q-table. This value, denoted by ${\mathbb{Q}}_n({\mathbb{S}}_n,a_n)$, corresponds to the current state–action pair $({\mathbb{S}}_n,a_n)$. The Q-value is updated based on the previous Q-value, the maximal Q-value of the next state multiplied by the discount factor, and scaled by the learning rate. The specific formula for this update is detailed in Section 3.1.

Algorithm 21 depicts the pseudo-code of the proposed Q-learning-based scheduling algorithm.

Algorithm 1 Q-Learning PF-Scheduler

1: Initialize Q-table   2: while Training do   3:     Update ${\mathbb{S}}_n$ with UEs’ CQI reports   4:     if ${\mathbb{S}}_n$ not in Q-table then   5:         Add ${\mathbb{S}}_n$ to Q-table   6:     end if   7:     With probability $ϵ$ select a random UE to schedule   8:     Otherwise select the UE that corresponds to:       $a_k={argmax}_{a\in \mathbb{A}}\mathbb{Q}({\mathbb{S}}_n,a)$   9:     Allocate DL transmissions to UE-k 10:     Update UE-k throughput with the corresponding rate:       $T_k=T_k+r_k$ 11:     if $n<L$ then 12:         proceed to next time slot $n=n+1$ 13:         Receive CQI reports 14:         $R=0$ 15:         Transit to next state ${\widehat{\mathbb{S}}}_n$ 16:     else if $n=L$ then 17:         Calculate reward $R={\sum }_{k=1}^{K}log{T}_k$ 18:         Initialize $T_k,\forall k\in K$ 19:     end if 20:     Update $\mathbb{Q}({\mathbb{S}}_n,a_k)=\mathbb{Q}({\mathbb{S}}_n,a_k)+\alpha [R+\gamma max\mathbb{Q}(\widehat{{\mathbb{S}}_n},\widehat{a})-\mathbb{Q}({\mathbb{S}}_n,a_k)]$ 21: end while

The Q-learning scheduler algorithm we implemented consists of the following key components:

• User Equipment (UE) Selection (lines 7–9): At each time slot, the algorithm decides which UE to schedule using the exploration/exploitation policy. It either chooses a random UE (exploration) or selects the UE that maximizes the learned reward (exploitation).
• State Update (lines 10–15): After downlink (DL) transmission to the chosen UE, the algorithm updates the next state with the observed throughput and new channel quality indicator (CQI) reports.
• Reward Calculation (not explicitly shown): The reward is calculated at the end of the scheduling cycle. There are no mid-cycle reward calculations, so the reward is zero during the cycle.
• Next Cycle Initialization (lines 16–18): These lines depict the state initialization for the subsequent cycle.
• Q-value Update (line 20): The current Q-value for time slot n is updated using the current reward and the difference between the maximum possible discounted future Q-value and the current Q-value. This adjustment steers the Q-value towards the maximum reward at the cycle’s end.

It is important to note that the reward calculated at the end of the cycle backward-propagates to all states involved in that cycle, influencing their Q-values. This structure allows the algorithm to learn and improve its scheduling decisions over time based on the outcomes of its choices.

The computational complexity of the BS for each update includes action selection, action execution, and Q-value update. In an exploration iteration, the BS randomly chooses which user to schedule next, with a time complexity of $\mathcal{O}\left(1\right)$ (line 7 in Algorithm 1). The exploitation iteration involves a lookup in the Q-table at the index of the current state and choosing the action with the highest value. Since the BS can choose between N possible actions (N users to choose from), it needs to perform $N-1$ comparisons to find the one with the greatest Q-value among the N users, resulting in $\mathcal{O}\left(N\right)$ mathematical operation (line 8 in Algorithm 1). The Q-table update for both exploration and exploitation involves updating a single Q-value according to the Q-value equation (line 20 in Algorithm 1). To update the Q-value, the BS first updates the accumulated attained rate of the scheduled user by adding the rate attained in the last slot to the aggregate rate received by the user so far in the current cycle (line 10 in Algorithm 1), which requires a single arithmetic operation. To update the Q-value, the BS retrieves the previous Q-value $Q(s,a)$ and the current Q-values $Q(s^{\prime },a^{\prime })$ for the new state $s^{\prime }$, with time complexities of $\mathcal{O}\left(1\right)$ and $\mathcal{O}\left(\right|N\left|\right)$, respectively. Note that the new state is determined based on the channel state collected from the users (line 13 in Algorithm 1). The max operation to find ${max}_{a^{\prime }}Q(s^{\prime },a^{\prime })$ takes $\mathcal{O}\left(\right|N\left|\right)$ time. The arithmetic operations, which include the temporal difference calculation $\delta =r+\gamma {max}_{a^{\prime }}Q(s^{\prime },a^{\prime })-Q(s,a)$ and the update $Q(s,a)\leftarrow Q(s,a)+\alpha \delta$, are $\mathcal{O}\left(1\right)$ each. Overall, the time complexity per Q-learning update is $\mathcal{O}\left(\right|N\left|\right)$ and includes $N+1$ lookups, 1 comparison, and 5 arithmetic operations. Note that a reward is given only at the end of the cycle ($n=L$). Accordingly, at the end of each cycle, the BS needs to calculate the reward (line 17 in Algorithm 1), which includes taking the logarithm of N values and summing them, resulting in $2N-1$ arithmetic operations.

The space complexity (memory complexity) for storing the Q-table requires storing the Q-values for all the state–action tuples for the entire cycle. The state comprises both the accumulated rate each user has attained so far in the cycle as well as its current channel state. Since the throughput a user attains when scheduled is based on the transmission rate, which is directly associated with its channel state, the number of different throughput gains a user can accumulate when scheduled is equal to the number of Index of Transport Block Size (ITBS-see Appendix A.1). Accordingly, denoting the number of different ITBSs by ${\eta }_{ITBS}$, each cycle stage can branch into $N·{\eta }_{ITBS}$ different states. Since each cycle is composed of L stages (allocations), the table contains ${\left(N·{\eta }_{ITBS}\right)}^L$ elements. Note that if the transmission can fail, the number of elements in each stage (inside the parenthesis) should be multiplied by 2, corresponding to successful and unsuccessful transmissions. Even though the latter leaves the accumulated rates unchanged, it corresponds to a different stage and hence relates to a different state.

Overall, we observe that the computational complexity remains manageable as the system size increases and is comparable to other schedulers, as any selective scheduling algorithm must account for all users. However, the space complexity poses the most significant scalability barrier. Specifically, the exponential growth of the state space with respect to cycle length indicates that this approach does not scale efficiently for large systems or longer cycles. For example, a small-scale deployment serving 50 users with 26 different ITBS values (as defined by the LTE standard) and a cycle of 10 slots would require a Q-table containing ${\left(50·26\right)}^{10}\sim \mathcal{O}\left({10}^{31}\right)$ elements, which is clearly infeasible. Moreover, beyond the memory limitations, an extremely long learning phase would be required to explore all these states sufficiently.

7. Simulation

In this section, we evaluate the performance of the proposed RL-based algorithm through simulations under different parameters and configurations. We begin by presenting the simulation setup.

7.1. Simulation Setup

We developed a Python-based simulation platform to explore the proposed Q-learning-based algorithm for downlink scheduling. The simulation model consisted of a single base station (BS) serving K User Equipment (UE) devices. All UEs were assumed to have fully backlogged downlink traffic, ensuring the BS always had data to transmit to each UE. The system operated on a time-slotted basis, with one UE scheduled per time slot. We employed a time-varying block-fading channel model, where each UE’s channel was represented by a two-state Markov chain, following the Gilbert–Elliott model [53,54]. Each UE supported two transmission rates, $r_1$ and $r_2$, with $r_1>r_2$ corresponding to the traditional notation of ‘good’ (G) and ‘bad’ (B) channel states, respectively. Specifically, $r_1$ and $r_2$ allowed for transmission of 4587 bytes and 225 bytes per resource block (RB), corresponding to CQI values of 15 and 1 in the LTE standard, respectively. The channel state transition probabilities, denoted by $Prob(G\to G)=p_k$ and $Prob(B\to B)=q_k$ (i.e., $Prob(G\to B)=1-p_k$ and $Prob(B\to G)=1-q_k)$, were independently chosen for each UE. The channel state remained constant throughout each time slot. The BS maintained channel state estimations for each UE (detailed in Section 3.3) and transmitted at rates corresponding to these estimations. The simulation was structured in cycles, each comprising L time slots. We ran each configuration for a minimum of 1000 cycles. Performance was evaluated according to the average PF achieved across all simulations. We explored various combinations of $p_k$ and $q_k$ values.

Table 1. Python-based simulation parameters and notations.

Parameter	Value
Good (G) channel state rate $r_1$	4587 Bytes per RB (CQI = 15)
Bad (B) channel state rate $r_2$	225 Bytes per RB (CQI = 1)
Channel transition probabilities: $G\to B$	$Pro{b}_{G\to B}=1-p_k$
Channel transition probabilities: $B\to G$	$Pro{b}_{B\to G}=1-q_k$
Time slots per cycle	L time slots

summarizes the key simulation parameters and notations used in this section.

The Q-learning parameters used in the simulation were as follows: the discount rate $\gamma =0.9$, and epsilon was decayed exponentially according to $ϵ\left(t\right)={ϵ}_{min}+({ϵ}_{max}-{ϵ}_{min})\times e^{-kt}$, where t is the current iteration number, ${ϵ}_{max}=1$, ${ϵ}_{min}=0.1$, and $k=0.9$ is the decay parameter. The learning rate was $\alpha =0.1$. The number of iterations used in the simulation was approximately 100,000.

We evaluated our algorithm against three benchmark approaches: (i) The traditional round-robin scheme, which serves as a baseline. This method allocates time slots to UEs in a fixed, cyclical order, disregarding channel conditions. It achieves fairness simply by ensuring each UE receives approximately the same number of RBs. (ii) The widely used opportunistic PF scheme, as described in Section 3.2. (iii) The optimal scheduling, which retrospectively searches at the end of each cycle for the scheduling that would have maximized the PF criterion (5) based on the UEs’ attained channel states. It is important to note that the optimal scheduling is not only computationally intensive and thus not scalable, but more importantly, it is not feasible in practice. This is because it determines which UEs should have been scheduled after the cycle has already finished, using information about channel states that would not have been available at the time of actual scheduling. We utilize this optimal scheduling approach to provide an upper bound on PF performance.

Since the scheduler’s objective is to maximize the PF metric (${\mathrm{U}}^i={\sum }_{k=1}^{K}log{T}_k^i$), the results presented depict the average PF attained per user per cycle ($Av{g}_i{\mathrm{U}}^i$). Additionally, we present the average accumulated throughput per user per cycle, $Av{g}_i\left({\sum }_{k=1}^K{T}_k^i\right)$, which provides complementary insights into the system’s performance, particularly in understanding the tradeoff between efficiency and fairness. In some of our results, we also present an additional fairness-related performance metric, Jain’s fairness index, which is a widely recognized measure in network resource allocation, defined as:

$$\begin{array}{c}\hfill Jai{n}^{\prime }sfairnessindex={\displaystyle \frac{{\left({\sum }_{k=1}^K{T}_k\right)}^2}{K\times {\sum }_{k=1}^K{T_k}^2}}\end{array}$$

(7)

where $T_k$ represents the throughput of user k, and K is the total number of users.

7.2. Simulation Results

Due to the enormous state space involved, we focus on a small number of users, starting with only two and later extending to three and four users.

7.2.1. 2-UE Scenario

To gain some insight, we start with a simple 2-UE setup where one user (UE-1) is constantly in a good channel state (i.e., $p_1=1$, and $q_1=0$), while the second user (UE-2) alternates between good and bad channel states ($p_2=0.3$, and $q_2=0.7$). Both UEs have backlogged traffic at the BS at all times (fully backlogged model).

Figure 5 compares the average sum log throughput, the average Jain’s fairness index, and the average cell throughputs for a scheduling cycle of $L=5$. The average of all three metrics was calculated over all scheduling cycles.

Figure 5 presents a comparative analysis of the suggested RL-based algorithm against the other three scheduling approaches with respect to three metrics: PF, Jain’s fairness index, and cell throughput. PF metric performance: Figure 5a shows that the RL-based scheduler nearly matches the performance of the optimal scheduler, achieving the optimal schedule in over $95\%$ of cycles. It attained a $3\%$ improvement in the PF metric compared to the standard PF scheduler. Jain’s fairness index: Figure 5b shows that in terms of fairness, the RL-based scheduler not only outperforms the PF scheduler in the proportional fairness measure but also as measured by the Jain’s fairness index. Overall cell throughput: Figure 5c shows that the RL-based scheduler yields a $12\%$ increase in aggregated cell throughput compared to the standard PF scheduler. When compared to the round-robin (RR) scheduler, which does not consider UEs’ channel conditions or attainable rates, the RL approach shows a $36\%$ throughput improvement. These results highlight the effectiveness of our RL-based approach in balancing fairness and throughput optimization, outperforming traditional scheduling methods across multiple performance metrics, even in small-scale environments.

In order to gain some insight into the last result, we compare the decisions made by the two schedulers, PF and RL-PF, during a single cycle. Specifically, in

Table 2. Parameters and values attained by the PF and the RL-PF schedulers for an exemplifying cycle.

	UE-1	UE-2	PF-Scheduler				RL-PF-Scheduler
Slot	CS-TB *	CS-TB *	PF(a → UE-1)	PF(a → UE-2)	$\mathsf{\Sigma }{\mathbf{T}}_{\mathbf{UE}-\mathbf{1}}$	$\mathsf{\Sigma }{\mathbf{T}}_{\mathbf{UE}-\mathbf{2}}$	Q(a → UE-1)	Q(a → UE-2)	$\mathsf{\Sigma }{\mathbf{T}}_{\mathbf{UE}-\mathbf{1}}$	$\mathsf{\Sigma }{\mathbf{T}}_{\mathbf{UE}-\mathbf{2}}$
1	G-4587	B-225	12.16334	7.813781	0	0	16.72	16.51	0	0
2	G-4587	B-225	13.16334	19.97712	4587	0	18.24	17.99	4587	0
3	G-4587	B-225	20.97712	20.97712	4587	225	19.71	19.63	9174	0
4	G-4587	B-225	21.56208	21.97712	9174	225	20.91	21.06	13761	0
5	G-4587	G-4587	22.56208	25.46168	9174	450	21.98	25.98	13.761	225
* CS-Channel State, TB-Transport Block					9174	5037			13761	4812

we present a representative 5-slot cycle in which UE-1 experiences a good channel condition in all time slots, while UE-2 experiences a good channel state only in the fifth slot. The table depicts the channel state (CS) realization of both UEs in each time slot (G\B channel), the respective attainable transport block (TB), and the accumulative number of bytes sent by each UE so far in the cycle ($T_{current}^i={\sum }_{k=1}^{Current}{T}_k^i$). The table also presents the relevant parameters for the two schedulers, the PF values attained by assigning the slot to each of the UEs, computed by the PF scheduler prior to each slot and the Q-values for the two actions, assigning the slot to UE-1 or to UE-2. The highlighted cells indicate the chosen UE for allocation based on the two schedulers, specifically, the UE that provides the highest increase in PF for the PF scheduler and the one (the action) with the highest Q-value for the RL-PF scheduler.

Each cycle begins with initializing the aggregate throughput of all UEs to zero. Since the PF metric is a concave function (the logarithm is a concave function), the lower the aggregate bytes sent by a UE, the higher the marginal PF gain for additional $\delta$ bytes. Consequently, the PF scheduler, which maximizes the PF metric at each transmission opportunity, is likely to distribute the first few slots among the UEs that have not yet received any allocations. The allocation order will be determined based on their instantaneous attainable rates. Indeed, it can be seen that the PF scheduler assigns the first slot to UE-1, which is given the highest rate, and the second time slot to UE-2 even though it experiences a poor channel and a much lower attainable throughput, as it provides a better instantaneous PF value (6.01 vs. 3.96).

Note that even if UE-2 had experienced good channel conditions in the first slot, since its attainable rate under good channel conditions is the same as that of UE-1, the slot would have been assigned arbitrarily to one of the UEs, regardless of the likelihood of the UEs experiencing good channel conditions. Additionally, if the rate attainable by UE-2 is only slightly lower than that of UE-1, the slot would be allocated to UE-1, even if the probability of UE-2 achieving such high rates is rare and much lower than the probability of UE-1 achieving this high rate.

In contrast to the RL scheduler, the RL-PF scheduler has learned that it can postpone UE-2’s allocations while waiting for it to attain better channel conditions. Consequently, the first two slots are allocated to UE-1, which has good channel conditions.

Examining the retrospective optimal strategy (oracle scheduler) reveals that if UE-2 had only experienced bad channel conditions throughout the cycle, it should have been allocated two time slots. If UE-2 had obtained a single good channel condition time slot, it should have been allocated that slot and only that slot. If UE-2 had obtained two good channel condition time slots, it should have been allocated those two slots and only those two. If UE-2 had obtained three or more good channel condition time slots, one of the UEs should have received three time slots while the other received only two. Since the good channel condition rates are the same for both UEs, which UE received the three time slot allocation would be arbitrary.

Even though the RL-PF scheduler cannot know a priori the future channel conditions of the UEs, it has learned that UE-2 should be allocated at least one and at most three time slots. Accordingly, since in the presented cycle UE-2 was not allocated any of the first three slots (attained only bad channel conditions), and in the fourth time slot it also attained bad channel conditions, the BS needs to determine whether to allocate this slot to UE-1 or UE-2. Note that, in order to converge to the retrospective optimal strategy, the allocation should be based on UE-2’s next slot channel condition. Specifically, if the channel state in the last slot eventually turns out to be bad, both the fourth and last slots should be allocated to UE-2. However, if the channel state in the last slot eventually turns out to be good, the fourth slot should be allocated to UE-1, and only the last slot should be allocated to UE-2. Since the RL-PF scheduler cannot know a priori what channel conditions UE-2 will experience in the last time slot, it relies on the learned probability that UE-2 is more likely to experience bad channel conditions than good ones, and therefore allocates the slot to UE-2. Consequently, the PF scheduler assigned two time slots to UE-1 and three time slots (one good and two bad) to UE-2, while the RL-PF scheduler assigned three time slots to UE-1 and only two time slots (one good and one bad) to UE-2, resulting in a better PF value at the end of the cycle. Specifically, the PF values attained at the end of the cycles were 25.46 for the PF scheduler and 25.98 for the RL-PF scheduler (see Remark 1 below).

Note that the schedule attained by the RL-PF scheduler for this channel realization is not the optimal allocation. As previously mentioned, the optimal allocation would assign to UE-2 only the last time slot, in which it experienced good channel conditions.

Clearly, the policy attained by the RL-PF scheduler is related to the distribution of rates that each UE can attain in each channel condition and the ratio between these rates. Different rates will result in different learned policy. Typically the policy learned cannot guarantee optimal allocation for all channel realizations, as usually there is no such policy when the future is not known a priori. Note that despite the nonoptimal allocation example discussed above that stresses the fact that no policy can guarantee the optimal allocation, the optimal schedule was reached only in $95\%$ of the scheduling cycles (see Figure 5).

Remark 1.

Note that the Q-values at the end of each cycle, i.e., $\mathbb{Q}(\widehat{s_L},\widehat{a})\forall s_L\in {\mathbb{S}}_L,a_k\in \mathbb{A}$ converge to the actual PF values for the associated states.

Specifically, the Q-value is updated every time the system visits a state and chooses an action. Let us count the number of times each state and action tuple is visited, and denote the Q-value of state $s_n$ and choose action a, after visiting the state and choosing the same action for the m-th time, by ${\mathbb{Q}}_m(s_n,a)$. Let us concentrate on the state and action tuples that are related to the last (L-th) slot of the cycle (${\mathbb{Q}}_m(s_L,a)$).

Since the accumulated throughput of each user is reset at the end of each cycle, the Q-value of the next step is zero ($\mathbb{Q}({\widehat{s}}_{L+1},\widehat{a})=0\forall {\widehat{s}}_{L+1}\in \mathbb{S},a_k\in \mathbb{A})$. Consequently, the L-th and last update at the end of the cycle (Algorithm 1, line 20) is ${\mathbb{Q}}_m(s_L,a)=(1-\alpha ){\mathbb{Q}}_{m-1}(s_L,a)+\alpha R(s_L,a)=(1-\alpha )\left({\mathbb{Q}}_{m-2}(s_L,a)+\alpha R(s_L,a)\right)+\alpha R(s_L,a)=\cdots ={(1-\alpha )}^m{\mathbb{Q}}_0(s_L,a)+\alpha {\sum }_{i=0}^{m-1}{(1-\alpha )}^{i}R(s_L,a)={(1-\alpha )}^m{\mathbb{Q}}_0(s_L,a)+\left(1-{(1-\alpha )}^m\right)R(s_L,a)$. When the number of times that each state is visited goes to infinity, ${(1-\alpha )}^m\to 0$. Accordingly, ${(1-\alpha )}^m{\mathbb{Q}}_0(s_L,a)\to 0$ regardless of the initial Q-value of ${\mathbb{Q}}_0(s_L,a)$, and ${\mathbb{Q}}_m(s_L,a)\to R(s_L,a)$, which is the sumlog throughput associated with $(s_L,a)$.

Accordingly, the last two Q-values depicted in Table 2 for the RL-PF scheduler (21.98 and 25.98) are the actual values attained by the PF metrics for allocating four time slots to UE-1 and one bad channel slot to UE-2 (21.98) and allocating three time slots to UE-1 and two slots, one bad and one good channel slot to UE-2 (25.98).

7.2.2. 3-UE Scenario

We repeat the comparison described above for three UEs for two different setups. Specifically, we keep the cycle length on 5 slots ($L=5$) and examine two different scenarios, as follows: (i) UE-1 is kept at a constant good state, i.e., $p_1=1,q_1=0$, UE-2 is kept at a constant bad state, i.e., $p_2=0,q_2=1$, and the last UE alternates between good and bad with probabilities, $p_3=0.3$, and $q_3=0.7$. (ii), UE-1 is kept at a constant good state, $p_1=1,q_1=0$, while the other two UEs alternate between good and bad states, i.e., $p_2,p_3=0.3$ and $q_2,q_3=0.7$.

Figure 6a,b depict the average sum log throughput, the average cell throughput, respectively, for both setups. Since the differences in the Jain’s fairness index are negligible (a negligible gain was observed favoring the RL-based scheduler), they were omitted from the figures. Figure 6a, which depicts the sum log throughput of the four algorithms, shows the benefit of the RL-based scheduler over the PF standard scheduler with a $4\%$ increase in setup (i), and $5\%$ increase in setup (ii). Figure 6b shows the average cell throughput of all three scheduling algorithms for each case; an $8\%$ increase in performance was observed with the RL scheduler compared with the PF for setup (i), while a much higher increase of $21\%$ was observed for (ii).

As previously mentioned, providing insight into the learned scheduler policy can be quite challenging. However, the first three-user setup, in which two UEs have fixed channel states, was chosen to allow for a better understanding of the attained policy. Specifically, it is clear that the scheduler will distribute the slots among the UEs, aiming to allocate a slot to a UE when its channel is good. Since, in a cycle of length $L=5$, the slots cannot be equally distributed among the UEs, and since each UE must be allocated at least one time slot, the scheduler (policy) must determine which slots to allocate to which users (how much to postpone UEs with bad channel conditions) and to whom to allocate the “extra” two time slots (preferably to the one experiencing good channel conditions).

The attainable rates are such that the PF metric value for allocating two time slots to a UE with good channel conditions, and two and one slots to UEs with bad channel conditions, is higher than allocating three time slots to a UE with good channel conditions and only one slot to each of the other UEs (with bad channel conditions). Accordingly, the RL-PF scheduler has learned the following:

(i)

Allocate two and only two time slots to UE-1, which always experiences good channel conditions, regardless of the channel states of the other UEs. These two time slots can be allocated at any point in the cycle. Thus, whenever one of the other UEs experiences good channel conditions, the RL-PF scheduler delays the allocation to UE-1, even though it also experiences good channel conditions.

(ii)

Allocate only one time slot to UE-2, which always experiences bad channel conditions. This slot can be allocated at any point in the cycle.

(iii)

Allocate the remaining two time slots to UE-3, preferably when it is in a good channel state.

Thus, similar to the two-user setup, the allocations to UE-3 will be delayed until it is in a good channel state; whenever it experiences good channel conditions, it receives the slot. If it does not receive its two slot allocations earlier, these allocations will be delayed toward the end of the cycle (one or two time slots, depending on how many it has received so far), regardless of whether the channel state is good or bad.

Setup (II) presents a bit more of a challenge. Similar to the previous setup, UE-1, which always has a good channel state, will receive two slots, while the other two UEs will share the remaining three slots. Since both UE-2 and UE-3 have the same channel characteristics (the same probability of being in each state and the same rate attained in each state), there is no advantage in favoring one over the other. Consequently, if both UEs experience the same channel conditions (good or bad), the RL-PF scheduler’s decision of which UE to allocate a slot to is arbitrary.

The scheduler will allocate one of the two slots to UE-2 and UE-3 whenever they are in a good channel state. If no slots were assigned to these two UEs in the first two time slots (both having experienced bad channel conditions), the first two slots would be allocated to UE-1. The remaining three slots will be assigned as follows:

(i)

Better Channel Conditions: If one of the UEs has better channel conditions, it will be assigned the slot.

(ii)

Equal Channel Conditions: If they experience the same channel state, the allocation will be arbitrary, ensuring that each one obtains at least one slot.

(iii)

Final Slot Allocation: If, prior to the last slot, one of the UEs has received no allocations, it will be allocated the last slot regardless of its channel state.

Obviously, for this setup, there is no policy that can ensure an optimal allocation, as when the two UEs (UE-2 and UE-3) are in a tiebreaker and experience the same channel conditions, the scheduler cannot know which one to favor since it does not know a priori what channel conditions they will experience in the following slots. To illustrate this assertion, we examine the Q-values obtained for a cycle in which UE-2 experiences bad channel conditions in the first three time slots and good channel conditions in the last two time slots, while UE-3 experiences bad channel conditions in all five slots.

Table 3. Q-table for the last three time slots following the initial allocations to UE-1. The marked cells indicate the UE selected based on the highest Q-value.

	UE-1	UE-2	UE-3	RL-PF-Scheduler
Slot	CS-TB *	CS-TB *	CS-TB *	Q UE-1	Q UE-2	Q UE-3	$\mathsf{\Sigma }{\mathbf{T}}_{\mathbf{UE}-\mathbf{1}}$	$\mathsf{\Sigma }{\mathbf{T}}_{\mathbf{UE}-\mathbf{2}}$	$\mathsf{\Sigma }{\mathbf{T}}_{\mathbf{UE}-\mathbf{3}}$
3	G-4587	B-225	B-225	26.51	26.85	26.72	9147	0	0
4	G-4587	G-4587	B-225	27.66	31.07	28.48	9147	225	0
5	G-4587	G-4587	B-225	25.98	26.36	33.20	9147	4812	0
* CS-Channel State, TB-Transport Block							9174	4812	225

shows the Q-table for the last three time slots after the first two slots were assigned to UE-1, with both UE-2 and UE-3 having experienced bad channel conditions.

As can be seen, the RL-PF scheduler makes a “wrong” decision compared to an Oracle scheduler by assigning the third slot to UE-2 instead of UE-3. However, as previously explained, the expected channel states of the UEs cannot be known a priori, and the scheduler must allocate slots based on its accumulated experience. In this specific scenario, since UE-2 experiences good channel conditions in the last two slots, it would have been more beneficial to allocate the third slot to UE-3 and reserve the last two slots for UE-2. This strategy would effectively exchange one bad channel condition slot for one good channel condition slot for UE-2. However, note that in cycles where the channel conditions experienced by all three UEs are the same as in the cycle depicted in Table 3, but with UE-2 and UE-3 switching positions (i.e., UE-2 experiences bad channel conditions in the last two time slots while UE-3 experiences good channel conditions), the scheduler’s decision to allocate the third slot to UE-2 would be the “right” choice.

Interestingly, it can be observed that in the depicted cycle, even though neither UE received allocations in the first two time slots and both experienced bad channel conditions in the third time slot, the Q-values for assigning the third time slot to UE-2 or UE-3 are not the same, with a preference for allocating it to UE-2. This discrepancy in Q-values can be attributed to the finite learning process of the RL-based scheduler. During learning, the scheduler may have gathered slightly different statistics for the channel conditions of the two UEs, leading it to favor UE-2 for the third slot. This suggests that, based on the experience accumulated so far, the scheduler perceives a higher likelihood that UE-3 will encounter better channel conditions in the subsequent slots, which influences the Q-values and, consequently, the allocation decision.

7.2.3. 4-UE Scenario

We further scale the number of users, and in this section we examine a similar setup as in the previous sections in which users experience a good–bad channel model, $L=5$, for four UEs. We examine three different setups: (i) UE-1 and UE-2 always experience good channel state, i.e., $p_1=p_2=1$, UE-3 always experiences bad channel state, $q_3=1$, and the last UE alternates between good and bad with probabilities, $p_4=0.3$, and $q_4=0.7$. (ii) UE-1 always experiences good state, $p_1=1,q_1=0$, while UE-2 and UE-3 alternate between good and bad channel states, i.e., $p_2,p_3=0.3$ and $q_2,q_3=0.7$, while UE-4 always experiences a bad channel state, i.e., $q_4=1$. (iii) UE-1 always experiences good channel state, i.e., $p_1=1$ while UE-2, UE-3, and UE-4 alternate between good and bad channel state with $p_2=p_3=p_4=0.3$ and $q_2=q_3=q_4=0.7$. Similar to previous scenarios, we first present the average sum log throughputs and the average cell throughput in Figure 7.

Similar to previous scenarios, in both performance metrics, RL-PF outperforms the common PF scheduler. Specifically, Figure 7a, which depicts the average sum log throughput, shows gains of $3\%$, $5\%$, and $6\%$ by using RL scheduling for the three setups, respectively. Figure 7b, which shows the results of the average cell throughput, indicates a $3\%$, $8\%$, and $16\%$ gain over the standard PF for the three setups, respectively. As in the 2-UE scenario, the RL-PF scheduler attains almost the same PF metric results as the optimal one.

7.2.4. Unreliable Channel

In this section, we extend our analysis to a more realistic scenario where downlink (DL) transmissions may fail. We introduce a probability of successful transmission, denoted as $P_k^{succ}<1$, for each UE-k. To further reflect actual operational cellular networks, we account for delayed feedback (ACK/NACK) due to processing and uplink allocation delays. Specifically, since in operational cellular networks the receiver’s feedback is not received after each and every transmission, we have assumed in our simulations that the feedback is delayed and received only after the cycle has ended. Note that under this setup, the scheduler cannot recover a failed transmission and compensate the UE by allocating to it a subsequent slot, as it is not aware of such failed transmissions until the cycle has already ended. Furthermore, under this framework, a UE can achieve zero throughput at the end of a cycle if all its received allocations result in failed transmissions. To keep the sum log throughput results in scale, for cases where a UE attains zero throughput, we assign a value of 0 (rather than negative infinity) to the $logT$ term when a UE’s throughput (TT) is zero. This more realistic model allows us to assess the robustness of our scheduling algorithms under imperfect transmission conditions, providing insights into their performance in practical network environments.

We revisit the simple 2-UE setup from Section 7.2.1, where UE-1 is near the BS, consistently experiencing good channel conditions and no losses. UE-2 is closer to the cell edge, with its channel state alternating between good and bad (probabilities: $p_2=0.3$, $q_2=0.7$). We examine two transmission failure scenarios for UE-2: (i) $P_2^{succ}=0.3$ (most DL allocations for UE-2 fail) and (ii) $P_2^{succ}=0.9$ (most UE-2 transmissions succeed). For the optimal scheduler to compute attainable rates, it needs to know in advance the result of all possible scenarios (not only the channel state of each UE at each time slot but also whether a transmission to this UE at this time slot will succeed). Accordingly, we simulate a transmission for both UEs in every time slot, determining success based on each UE’s probability, and allow the optimal scheduler to examine all possible allocations and choose the best among them. It is important to note that since the optimal scheduler allocates slots a posteriori with knowledge of transmission success, it will not assign a slot to a UE whose transmission will fail if the other UE’s transmission succeeds.

Figure 8 illustrates the average sum log throughput and the average cell throughput for both setups. Figure 8a shows that the RL-based algorithm outperforms the PF algorithm in terms of average sum log throughput (PF metric), with a $25\%$ gain in setup (i) (16.5 for RL-PF vs. 12.3 for PF and 20.1 for optimal scheduling) and a $6\%$ gain in setup (ii) (24.7 for RL-PF vs. 23.4 for PF and 25.6 for optimal scheduling). Figure 8b reveals that this PF metric improvement comes at the cost of average cell throughput. PF scheduling achieves higher cell throughput than the RL-based scheduler in both setups: 14.9, 8.7, and 20 Mb/s for PF, RL-PF, and optimal, respectively, in setup (i), and 17.4, 17.3, and 20.8 Mb/s for PF, RL-PF, and optimal, respectively, in setup (ii). This performance difference can be attributed to the RL-based scheduler’s learning process. It recognizes that UE-2’s transmissions are not always successful and consequently allocates more resources (time slots) to it. Some of these extra transmissions occur at a low rate, and some fail (especially in setup (i)). In contrast, the PF-based scheduler, which does not take failed transmissions into account, favors UE-1, whose transmissions are always successful and at a high rate. As a result, even though the proportional fairness criterion aims to balance fairness and channel utilization, the RL-based scheduler, which is aware of the loss probability of disadvantaged UEs, allocates a substantial number of resources to these less-advantaged UEs to compensate for potentially unsuccessful transmissions. Notably, both scheduling algorithms fall short of the optimal scheduler’s performance. The optimal scheduler, which can evaluate all $2^5$ possible schedules and their outcomes a priori, takes into account both the successful transmissions of each UE and the rates of these transmissions.

8. Implementation

To demonstrate the feasibility of the proposed scheme, we implemented the Q-learning RL-PF algorithm within an operational LTE base station serving three connected users.

The implementation testbed features a single-cell 4G network operating on a 10 MHz bandwidth. It employs a split architecture where the RF component, known as the remote radio head (RRH), is separated from the upper layers (such as PHY and MAC). These upper layers are implemented on a commercial off-the-shelf (COTS) server, referred to as the baseband unit (BBU). In this setup, the RRH is connected to two RF splitters, enabling the 2 × 2 RRH to interface with three RF cages housing the users. To precisely control each user’s path loss to the LTE cell, variable attenuators (V/A) are integrated into each RF connection. The MAC layer hosts the scheduler entity, which is responsible for making scheduling decisions. It is configured to allocate resources to one user per time slot. Figure 9 provides a visual representation of this testbed architecture.

Our system evaluation used a typical three-user configuration, representing diverse user channel state characteristics. Specifically, UE-1 had near-ideal channel conditions ($98\mathrm{dB}$ path loss), UE-2 had moderate channel conditions ($118\mathrm{dB}$ path loss), and UE-3 had challenging channel conditions ($138\mathrm{dB}$ path loss). To ensure continuous data demand, we generated UDP downlink traffic for each UE, maintaining a fully backlogged state across all users.

In addition to the proposed RL-PF scheduler, we also implemented the PF scheduler as formulated in Section 3.2. Furthermore, similar to our simulation approach, we tracked the channel states attained by each UE throughout the cycle. At the cycle’s end, we computed the optimal schedule that could have been achieved if all channel states (ITBSs) were known in advance. These implementations enable us to compare the performance of the RL-based scheduler against both the implemented PF scheduler and the theoretical optimal scheduling scheme under realistic network conditions.

The implementation of the RL-PF-based scheduler on 4G base station required several code adaptations in the MAC layer where the scheduler and the downlink link adaptation are implemented. Specifically, besides the modules that are associated with the RL, which were coded from scratch, some modules such as the queuing, DLLA, UE scheduling, and the DL TB assignment needed to be modified in order to support the RL-PF-based scheduler. A further detailed description of the modifications is given in Appendix A.2.

The RL-PF-based scheduler described in Section 6 required a minor adaptation for practical implementation. The original algorithm assumes that the channel quality between the BS and each UE is available prior to every time slot. However, in operational systems, Channel State Information (CSI) reports are typically received from UEs every 80 ms, not every time slot. To maintain accurate channel quality estimation, which is crucial for rate adaptation, operational BSs utilize additional feedback mechanisms. Specifically, they use ACKs/NACKs from each DL packet transmission to update channel quality estimates and adjust transmission rates accordingly. This process, known as downlink link adaptation (DLLA), is detailed in Appendix A.1. To address the limited granularity of CQI reporting, we modified the RL-PF state representation. Instead of relying directly on CQI, we use the ITBS, which is the output of the link adaptation mechanism. This ITBS incorporates both CQI reports and ACK/NACK feedback, providing a more frequently updated measure of channel conditions. Consequently, the implemented state space is redefined as follows:

$$\begin{array}{c}\hfill {\mathbb{S}}_n=\{\left(T_1\left(n\right),ITB{S}_1\left(n\right)\right),\dots ,(T_i\left(n\right),ITB{S}_i\left(n\right)),\dots ,(T_K\left(n\right),ITB{S}_K\left(n\right))\}n=1,2,\dots L\end{array}$$

(8)

where ${\mathbb{S}}_n$ denotes the state at the beginning of time slot n, and is composed of the accumulated number of bytes attained so far in the cycle and the ITBS of each UE-k connected to the base station.

The learning process is conducted offline following a measurement session where the UEs’ ITBS and throughputs for each time slot were recorded. ACK/NACK feedback, which is received by the scheduler 4 ms after the downlink transmission as specified by the 4G standard, is taken into account. After collecting all feedback from each UE, the actual throughputs at the end of a scheduling cycle are compiled. These throughputs are then used to calculate the PF criteria, which are utilized in the learning process. The Q-learning parameters used were the same as those used in the simulation results.

Implementation Results

The testbed results for the three UEs, using a cycle length of 5 slots (L = 5), comparing the RL-PF scheduler with the PF scheduler and the optimal schedule, are presented in Figure 10. Specifically, Figure 10a shows the average sum log throughputs, while Figure 10b displays the average cell throughput.

Figure 10a shows that the average sum log throughputs for the PF scheduler, RL-PF scheduler, and optimal schedule are 25.7, 30.4, and 36.3, respectively, demonstrating an $18\%$ improvement of the RL-PF scheduler over the PF scheduler. Figure 10b shows average cell throughputs of 19.4, 12.6, and 31.1 Mb/s for the PF scheduler, RL-PF scheduler, and optimal schedule, respectively, indicating a $35\%$ cell throughput loss for the RL-PF scheduler compared to the PF scheduler. This is consistent with the simulation observations in Section 7.2.4.

This tradeoff between fairness and performance is well recognized. In our setup, as explained in Section 7.2.4, it is attributed to the RL-PF scheduler’s strategy of allocating extra RBs to UEs with higher packet drop ratio (PDR) in order to mitigate potential transmission failures. This strategy has a dual effect: not only does the RL-PF-based scheduler assign more slots to UEs with lower success probabilities, increasing the average number of unutilized slots, but these UEs are typically also the ones with poor channel quality (lower data rates). Consequently, the PF scheduler achieves a higher number of utilized slots for UEs with high data rates (greater overall cell throughput) at the cost of reduced fairness (lower sum log throughputs). Note that even though the downlink link adaptation (DLLA) mechanism, explained in Appendix A.1, is responsible for converging the users to a target block error rate (which is, as per design, the same for all users), some users still suffer from higher loss probability.

Specifically, in our setup, UE-1 has consistently superior channel conditions. Hence, the RL-PF scheduler allocates to it a single time slot. UE-2 and UE-3, with different channel conditions but similar probabilities of successful transmission due to the DLLA mechanism, share the remaining time slots (each is allocated two time slots in our setup). Furthermore, since the RL-based scheduler cannot know a priori which transmissions will fail and must rely on anticipated loss probabilities, in some cycles it can underperform in both metrics. For example, when all five slots are successfully transmitted, the PF scheduling policy outperforms the RL-based policy in both the sum log throughputs and the aggregate cell throughput metrics. On the other hand, since the PF scheduler allocates only a single transmission (time slot) to one of the UEs with inferior channel conditions, which occasionally fails, in many cycles one of these UEs will attain zero throughput. Overall, as can be seen in Figure 10a, on average the RL-PF-based scheduler attains much higher proportional fairness scores.

9. Conclusions

This research aimed to explore the potential of reinforcement learning in scheduling procedures, investigating possible gains from learning system parameters such as channel quality, packet loss, and mobility. We focused on a small-scale deployment using Q-learning, which is suitable for such scenarios. Specifically, in this paper we introduced an innovative proportional fair (PF) scheduling algorithm that leverages the estimation and prediction of users’ future channel rates. The proposed scheme relies on Q-learning, a reinforcement learning technique, to optimize user selection allocation for each time slot.

To evaluate the performance of the suggested scheme, we developed a simulation framework and tested various scenarios, including different numbers of users under varying channel conditions. The simulation results demonstrate that the suggested scheme outperforms other scheduling algorithms, such as round-robin and traditional PF, in terms of the proportional fairness metric. We also examined the tradeoff between throughput and fairness under the suggested scheme.

To provide proof of concept (PoC), we implemented the proposed scheduler in a functional 4G base station, demonstrating the achievable gains with a real base station and users. The testbed results coincide with the simulation results.

Results indicate that if short-term fairness is the objective, an RL-based scheduling framework can offer significant benefits. However, if long-term fairness is considered, the traditional opportunistic PF scheduler can converge to the performance of the RL-based scheduler and is preferable due to its lower complexity.

It is crucial to note that our study focuses on a small-scale deployment, which aligns well with the capabilities of Q-learning. The 4G technology used as our infrastructure already necessitated simplifications in parameters such as CQI granularity, state space representation, number of users, and cycle length. These challenges would be significantly amplified in more complex environments like 5G or 6G networks.

Despite these constraints, we believe our work provides valuable insights and lays a foundation for future research. Subsequent studies could explore more advanced machine learning techniques, such as deep learning or actor–critic methods, to handle the complexities of larger-scale scenarios with more users and longer cycles. Another approach could involve narrowing the state space through state aggregation, which groups several states into meta-states. While this improves scalability, it may come at the expense of optimality (e.g., [55]).

Overall, this research demonstrates the potential of reinforcement learning-based scheduling for downstream traffic in cellular networks, while acknowledging the need for further advancements to meet the demands of next-generation technologies. We believe this approach can be extended to various other scheduling scenarios, including different setups and objectives, such as scheduling upstream traffic, wireless sensor networks with power constraints, traffic with due date constraints, and downstream scheduling with noise rise constraints (e.g., [56]).