A Systematic Study on Reinforcement Learning Based Applications

Abstract

We have analyzed 127 publications for this review paper, which discuss applications of Reinforcement Learning (RL) in marketing, robotics, gaming, automated cars, natural language processing (NLP), internet of things security, recommendation systems, finance, and energy management. The optimization of energy use is critical in today’s environment. We mainly focus on the RL application for energy management. Traditional rule-based systems have a set of predefined rules. As a result, they may become rigid and unable to adjust to changing situations or unforeseen events. RL can overcome these drawbacks. RL learns by exploring the environment randomly and based on experience, it continues to expand its knowledge. Many researchers are working on RL-based energy management systems (EMS). RL is utilized in energy applications such as optimizing energy use in smart buildings, hybrid automobiles, smart grids, and managing renewable energy resources. RL-based energy management in renewable energy contributes to achieving net zero carbon emissions and a sustainable environment. In the context of energy management technology, RL can be utilized to optimize the regulation of energy systems, such as building heating, ventilation, and air conditioning (HVAC) systems, to reduce energy consumption while maintaining a comfortable atmosphere. EMS can be accomplished by teaching an RL agent to make judgments based on sensor data, such as temperature and occupancy, to modify the HVAC system settings. RL has proven beneficial in lowering energy usage in buildings and is an active research area in smart buildings. RL can be used to optimize energy management in hybrid electric vehicles (HEVs) by learning an optimal control policy to maximize battery life and fuel efficiency. RL has acquired a remarkable position in robotics, automated cars, and gaming applications. The majority of security-related applications operate in a simulated environment. The RL-based recommender systems provide good suggestions accuracy and diversity. This article assists the novice in comprehending the foundations of reinforcement learning and its applications.

1. Introduction

Machine learning supports various applications, such as data prediction, classification, regression analysis, and clustering. In supervised learning (SL) [1], input and output data (labeled data) are fed into the system that generates a model. Unsupervised learning (UL) [2] finds associations, similarities, and variations in unlabeled data and extracts hidden information. SL, UL, and RL [3] are three types of machine learning. The generated model can predict the output for new unknown data. Spam detection [4], image and object detection [5], predictive analytics [6], text classification, and other applications make use of SL. SL algorithms include logistic regression, linear regression, decision tree, random forest, and support vector machine (SVM) [7]. Clustering with the k-means clustering algorithm and dimension reduction with principal component analysis (PCA) are two UL applications. Deep learning has been able to overcome the shortcomings of machine learning. Machine learning extracts features separately, whereas deep learning extracts features automatically and without human intervention. Machine learning cannot handle large amounts of data, but deep learning can take millions of data points. Deep learning can process unstructured data such as images and audio. Because of their dynamic nature, specific real-time scenarios, such as recommendation systems and self-driving cars, do not have sufficient datasets for training. Reinforcement learning is introduced to address this issue. The third type of machine learning is RL [8]. RL is appropriate for sequential decision-making processes and learns through direct interaction with the environment to achieve long-term goals without external motivation or complete knowledge of the environment. Figure 1 depicts the RL framework.

The RL agent examines the condition of the environment and chooses an appropriate action. If the RL agent performs the correct action, it receives a positive reward. If the agent makes the wrong move, a negative is generated. RL must balance exploration and exploitation. Exploitation occurs when an agent attempts to maximize the reward based on a previously established route. If the agent always tries to explore a new way to reach the destination, it is called exploration. There are several applications for RL. RL The model does not need a large dataset and learns by trial and error.

A recommendation system is designed to suggest items valuable to the user. Recommendation systems (RS) are used by most e-commerce [9] websites to increase revenue. Different types of recommendation systems are content-based, collaborative-based, and hybrid-based. Another type of RS is location-based recommendation [10,11]. Traditional RS systems suffer from a cold start, a warm start, a long tail, data sparsity, and scalability. To address these shortcomings, RL is introduced in RS to provide recommendations that are accurate, relevant, and diverse. Reinforcement learning-based RS [12] is used in various applications, including news article recommendations, course recommendations, product recommendations, movie recommendations, and so on. In conjunction with deep learning, RL handles high-dimensional sensory input in applications such as Atari games.

The deep Q network (DQN) is the most widely used algorithm in deep reinforcement learning. Multi-armed bandit comprises multiple arms, each with a different chance of winning. Contextual bandit solves the exploration-exploitation problem. For personalized recommendations, the contextual bandit is used. The most widely used algorithms in contextual bandit are upper confidence bound (UCB), Thomson sampling, and LinUCB. In IoT applications, RL is used to protect users from security threats. Most RL works in security applications are simulation-based, making the transition into the real world expensive. The most efficient robots are those built with RL. Social robots are primarily used in healthcare to take care of the elderly. Cognitive empathy is also used in the social robot to understand human emotions better and make them comfortable with the response. Natural language processing works much better with RL. Inverse reinforcement learning (IRL) [13] and the multi-agent RL approach are considered different forms of RL. In some instances, the rewards and transition probability of the systems are unknown. In such cases, IRL has been applied to solve this problem. IRL, on its own, learns the reward function. IRL observes the domain expert’s behaviours to implement the observed behaviour.

One of the critical applications of RL are EMSs. Energy demand continues to increase, and optimizing energy consumption while maintaining comfort is challenging. Most EMSs are implemented with the help of RL. In this review, we briefly discuss how the RL algorithm is built for EMSs. The RL algorithm learns through trial and error, adjusting the control policy based on the HEV’s performance feedback. The algorithm can consider various factors, including the battery’s present condition, the vehicle’s speed and power demands, and the expected driving conditions. This method can result in more effective use of the HEV’s power sources, leading to longer battery life and better fuel economy. RL can be used in HVAC control to teach an agent the optimal control approach for the HVAC system by interacting with a simulated environment that models the building and its thermal dynamics. The agent can perform actions such as adjusting the temperature setpoint or changing the airflow and is rewarded or punished based on energy consumption and thermal comfort level. The agent understands how to make decisions that conserve energy while sustaining thermal comfort over time. Eco-driving, also referred to as fuel-efficient driving, is the driving practice aimed at lowering fuel consumption and emissions. RL can be used to optimize the control of a vehicle’s powertrain to achieve eco-driving. The vehicle’s powertrain and its surroundings, such as the traffic and road conditions, are monitored closely in an RL-based eco-driving system. The RL algorithm then uses this information to determine the best actions for the driver to take, such as adjusting the speed, gear selection, and engine power to reduce fuel consumption. The algorithm also considers the driver’s behaviour and preferences, such as the driver’s comfort level and the urgency of the trip, to provide personalized eco-driving recommendations.

Section 2 of this review paper defines reinforcement learning algorithms. We discuss the application of reinforcement learning in various fields in Section 3.

2. Reinforcement Learning Algorithms

An RL process that satisfies the Markov property (MP) is called a Markov decision process (MDP) [14]. Different RL algorithms are available in the literature, and the hierarchy of algorithms is given in Figure 2.

RL algorithms can be value-based [15] or policy-based [16]. The value-based algorithm begins with a random starting value. Initially, the RL agent chooses a spontaneous action for a specific state and computes the value function. The RL agent can find the best policy using the value function. It does not save the policy but preserves the value function. Value-based RL requires a long time to compute because the q table has more states and actions. In policy-based RL, the agent begins with a random policy. It selects an action for a specific state based on policy. Then it computes the value function for the randomly chosen policy. The policy-based algorithm has a higher rate of convergence. Policy-based RL is appropriate for large state-space applications. It is possible to learn stochastic policies using policy-based RL. However, policy-based RL suffers from high variance. It requires few iterations to converge, but the algorithm is complex. An actor-critic algorithm is a combination of both value-based and policy-based algorithms.

2.1. Three Approaches to RL

The other three approaches of RL are Monte Carlo [17,18], temporal difference (TD) [19], and dynamic programming (DP) [20]. Dynamic programming requires the model to learn and follow a recursive approach. The model has to define states, actions, rewards, and transition probability. Dynamic programming is used for solving complex problems. It resolves a problem by breaking it into sub-tasks/sub-problems after determining the solution and combining the sub-tasks to form the solution. It can solve interlaced sub-problems, optimal substructure, and MDP. Monte Carlo (MC) and TD require experience and can learn by exploration (trial and error) without the model. MC follows the learning approach and is used for uncertain environments. TD determines the value estimates based on the estimates of other values, a process called bootstrapping. TD works for continuous tasks (non-terminating) and does not require a model. After every step, it updates the values estimates of the value. It does not have to wait till the entire trial and only updates the values for the visited path. TD is used in online and incremental learning and has low variance and some bias.

2.2. Different Types of RL Algorithms

Monte Carlo (MC) follows a learning approach based on interacting with the environment. MC is suitable for episodic tasks and has a starting and ending state (destination state). No matter what action it takes, the episode will terminate. MC does not require a model. It learns from the complete episode, and no bootstrapping is performed. When an agent visits a state for the first time in an episode, it is called the first visit. The episode’s return can be calculated based on the average of the samples concerning the first visit. In every visit, the return can be calculated based on the average of state values concerning every visit. A state is visited multiple times in every episode. The average of every visit is the sum of all the samples divided by the number of visits. Two types of approaches are employed: MC Control and MC prediction. MC control is in charge of calculating the state value function. MC prediction is responsible for finding the optimal policy based on the value estimates. MC has high variance and zero bias. MC and TD use sampling, but DP does not perform sampling.

Q-learning is one of the most widely used algorithms and follows a model-free approach and off-policy. The RL agent tries to find the maximum Q value. Q-learning is a value-based algorithm. Initially, all the q values are zero or random. The states and actions are represented as the rows and columns of the Q table. The Bellman equation is used to derive the Q value. The best action is selected based on the max Q value for a particular state using the ϵ-greedy policy. It starts exploring during ϵ times and (1-ϵ) times it finds the best possible q value (optimal policy). Q-learning has been used in gaming [21] and robotics [22]. State action reward state action (SARSA) is identical to Q-learning except that it does not find the max q value and uses on-policy. Utilizing the current set of actions carried out in the current state enhances the agent’s learning process. Previous states and rewards are not considered for a new set of states. Electric vehicle route optimization has been implemented using the SARSA algorithm [23]. It is used in non-stationary applications. Actor-critic (A2C/A3C) is a combination of both policy-based and value-based approaches. An actor performs specific actions, and the critic analyzes the value function of the corresponding action. By using TD error, the critic evaluates the action taken by the RL agent. Q-learning is not sufficient for vast state spaces and action spaces. For many states and actions, more computations and time are required. Deep neural networks are used in deep Q networks (DQN). The Q value is determined using a neural network. DQN is suitable for significant state space problems.

Proximal policy optimization (PPO) is an on-policy algorithm that generates data and updates the policy using the existing policy. PPO comprises two parts: a policy network that maps states to actions and a value network that computes the value of a state or a state-action combination. The policy network is trained to maximize the predicted cumulative reward, whereas the value network is taught to evaluate the worth of states and actions. It is reasonably straightforward to deploy and robust to hyperparameter selection. It has been employed in various applications, such as robots, gaming, and continuous control. Trust region policy optimization (TRPO) is another type of RL algorithm. It is similar to PPO. It is an on-policy algorithm that generates data and updates a policy using the existing policy. The trust region is used to update the policy settings, and the algorithm uses a natural gradient rather than a standard gradient. It outperforms the other RL algorithms in terms of performance. TPRO is utilized in various applications, including gaming, robotics, and continuous control. Deep deterministic policy gradient (DDPG) is another type of RL algorithm. The DDPG approach combines Q-learning with policy gradient methods. It employs the actor-critic network, which comprises two neural networks. The actor network is in charge of choosing an action, and the critic network examines the action selected by the actor network. The actor network seeks to maximize the total payoff. It is suitable for use in environments with continuous action spaces and is utilized in various applications, including robotics, gaming, and self-driving cars.

Soft actor critic (SAC) is another type of RL algorithm. SAC is a model-based reinforcement learning (RL) technique that integrates actor-critic architecture with entropy regularization. The actor-network chooses actions depending on the current state, whereas the critic network is employed to assess the worth of those actions. The entropy regularization element is introduced to the objective function to stimulate exploration and prevent premature convergence. SAC has been demonstrated to be highly successful on a vast scope of continuous control tasks and has been utilized in various applications, including robotics, gaming, and simulated physical systems. Twin delayed deep deterministic policy gradient (TD3) is the next step in the evolution of DDPG. It carries out the postponed policy update. It employs two criteria. The critic network computes the current state-activity pair’s value, whereas the actor-network selects the best course of action given the current scenario. TD3 reduces the overestimation of the Q value and increases its accuracy. The delayed policy update can lessen the overfitting of the network.

C51 (Categorical DQN) is a well-known Q-learning algorithm for discrete action spaces. In classic Q-learning, the Q value is expressed as a single value for each state-action pair, which tackles the problem of classical Q-learning discretization. Instead of employing a single Q value for each action, C51 extends the DQN by including a categorical distribution across the Q values. QR-DQN (quantile regression DQN) is an enhancement to the DQN algorithm. QR-DQN updates the network’s parameters using a quantile regression loss function, which differs from the standard mean squared error loss function used in DQN. It is defined as the sum of the Huber losses for each quantile of the action-value distribution. This loss function enables the agent to learn various possible action values rather than just one optimum action. Hindsight experience replay (HER) is another type of RL algorithm. It enables the agent to learn from previous failures and overcome them by focusing on the present objective. Additionally, the same experience is replayed with the new aim in mind. It is beneficial for tasks with sparse rewards and is appropriate for various suboptimal goal states. HER uses experience replay, which stores all previous experiences and is compatible with algorithms such as DDPG, DQN, and A3C.

World models are unsupervised algorithms consisting of three parts: a variational auto encoder (VAE), a recurrent neural network (RNN), and a decoder. The variational auto encoder encodes environmental observations. The RNN simulates the dynamics of the environment, and the decoder provides observations based on the VAE and RNN output. The RNN may be used to predict future environmental conditions. It is employed in various applications, namely robotics and computer vision. Model-based meta-learning with flexible computation (MBMF) was proposed in 2021 by a researcher from OpenAI. The agent learns from a collection of tasks. It quickly learns the new task by sharing computation across the tasks. It takes advantage of a computing model which is made up of a collection of previously learned activities. The agent is instructed on a series of tasks using an environment model called the task model, and the neural network serves as the task model. This network predicts the following state environment based on the present state and action. It contributes to the RL algorithm’s sampling efficiency. Model-based value expansion (MBVE) is a technique for improving the sample efficiency of model-based algorithms. It forecasts the future using the model environment. This algorithm calculates the value estimates. Based on the value estimates, it predicts the next state of the environment and can be combined with model predictive control (MPC) or guided policy search (GPS) to improve the sample efficiency. Imagination-augmented agent (I2A) consists of two components: RNN and DQN. RNN acts as the imagination module, and DQN acts as the control module. The imagination module generates the imaged scenarios by simulating the environment. The control module forecasts the future based on the generated scenarios. The control module estimates the return of state and action using the Q-learning algorithm. AlphaZero is a computer program developed by DeepMind that can play chess, shogi, and the game of go. It combines deep neural networks and Monte Carlo tree search algorithms.

3. Reinforcement Learning Applications

RL has been used in many applications in recent years. In this paper, we list several domains and their applications, which are shown in Figure 3.

3.1. Reinforcement Learning Applications in Recommendations

In course recommendations, user interest belongs to different categories of subjects. A novel method for course recommendations using the dynamic attention model using RL has been proposed [24]. This model represents the interaction between the user’s preferences and the profile reviser. To enhance recommendation accuracy, their model adaptively modifies the attention weight of the related course during various sessions to monitor changes in user preferences. They use two real-time datasets from massive open online courses. It has a high level of recommendation accuracy. In [25], a novel course recommendation model is proposed, which uses a profile constructor with self-learning capabilities to recommend courses specifically for each student. They offer a unique policy gradient method to overcome the exploration-exploitation trade-off problem in generating user profiles. The model uses a context-aware recurrent approach to capitalize on the available knowledge to investigate the user’s potential future preferences. Two real-world datasets were used for extensive tests to validate the effectiveness of the proposed HELAR model. The results prove that HELAR performs better than cutting-edge course recommender systems. Another method providing a personalized learning experience to students using reinforcement learning has been proposed [26]. The authors employ the Q-learning algorithm to recommend educational materials to students based on their present situation rather than their records, building the policies from collected data. At the initial state, the model generates a random policy, and later, the optimal policy is obtained based on the Q Value. To overcome the cold start problems, grey sheep, and sparsity of RS, ref. [27] proposed an RL-based dynamic recommendation. In [28], an RL model is designed that recommends learning objectives (LO) to students based on sensors that capture students’ heartbeats, quiz scores, blinks, and facial expressions.

3.1.1. Deep Recommender Systems in Recommendations

Deep reinforcement learning (DRL)-based RS was developed to provide more accuracy than traditional RS. DRL can manage more states and actions in a dynamic environment. A recent study proposed a DRL with user-commodity state representation called UCSRDRL [29]. They interpret RS as a sequential process and use the actor-critic algorithm, which includes long and short-term incentives. The cold start problem and data sparsity were overcome by their proposed DRL based on a prioritized experience replay network for capturing user interest changes [30]. Promotion of user engagement using deep reinforcement learning-based RS has been proposed [31]. To handle massive action sequences, a deep hierarchical category-based RS was implemented, and a two-layered DQN was developed. The first level selects the item category, and the second recommends the item to the user. Four real-time datasets are used, including Netflix and, Movielens (ML) (20 M, 10 M, and 1 M). The recommendation accuracy is higher based on the hit ratio and normalized discounted cumulative gain. To solve the cold and warm start problem, ref. [32] proposed a long-term recommendation model using a recurrent neural network with reinforcement learning. This model was evaluated using the hit ratio and normalized discounted cumulative gain. The recommendations can be fine-tuned based on the feedback of the user. A DQN for analyzing both negative and positive feedback of users was proposed by [33]. Positive feedback is given whenever a user clicks or orders an item; negative feedback is given when the user skips the item. A DRL framework for the interactive suggestion was developed, comprising convolution neural network and GAN [34]. The proposed combinatorial product recommendation system includes a consumer behaviour simulator and utilizes DRL to find appropriate product combinations that can improve the platform’s sales [35]. A user-specific and biased user-specific DQN for interactive recommendation problems for explicit feedback has been developed [36]. Matrix factorization is used to create user-specific states to learn the best recommendation policies. A biased user-specific DQN was developed to model the user-specific information by including the bias to analyze the Q values of every user.

3.1.2. Recommendation Using Contextual Bandits

The contextual bandit is an extension of the multi-armed bandit. Contextual bandit is used for solving the exploration-exploitation dilemma. The RS considers items or users as one of the arms in the contextual bandit framework. Contextual bandit aims to minimize regret. Based on previous experience, the bandit chooses the arm which maximizes the reward. The newly trained RS tries to exploit the products that the users are already engaged with. To overcome this problem, ref. [37] proposed an advertisement recommendation system using contextual bandits that uses two exploration algorithms: upper confidence bound (UCB) and Thomson sampling. They built a hybrid method combination of bootstrap and dropout. Based on the CTR, the model recommends an item to the users, and instead of training the RS, the model learns by dynamic interaction with users. Ke-LinUCB is a model proposed for personalized recommendations in a changing domain [38]. The dynamic preferences of the users are captured by this proposed system. The intention-selection method is used to analyze the behaviour of users. LinUCB uses an exploitation and exploration strategy to recommend the items to the users. Amazon-book and Yelp2018 were benchmark datasets used in this work. The recommendation accuracy of this model depends on the information of items and users. In [39], a news recommendation system is proposed that uses three different algorithms: CoLin, LinUCB, and Hybrid-LinUCB. Each algorithm is implemented on different datasets, such as MovieLens20M, Yahoo FrontPage Today Module, Synthetic, and LastFM. Each news article is considered an arm, and the reward is different for different users. The confidence bound is calculated based on the three algorithms. This system makes more accurate and diverse recommendations. An implicit feedback-based recommendation system that uses multi-armed bandit (MAB) was suggested by [40]. In this system, clicks and favourites are considered implicit feedback, which is divided into three categories: strong interaction, non-interaction, and weak interaction. The item category is considered an arm, and the number of arms is fixed. Thomson sampling is applied to balance the exploration-exploitation dilemma. To handle the cold start problem, the best-selling or popular items are recommended to new users. For old customers, items are recommended based on their behaviour history. The authors successfully verified the model on three datasets. In most of the recommendations using contextual bandit, either the item or user is used as an arm. The binary upper confidence bound can be employed to regard users and things as arms of each other [41]. This method also solves the cold start problems for items and users. The method was compared using five baselines. Considering precision, BiUCB provides a high precision value when compared with the five baselines. In another work, ref. [42] applied a contextual bandit algorithm to recommend personalized online learning objectives. The algorithm makes recommendations based on the student’s history and current state. Using a dataset from e-learning systems, which contained 365 students, 2519 events, and 78 Learning Outcomes(LO), they found the correlation of two actions by conditional probability. The RL agent recommends the LO, and the reward is generated based on the click through rate (CRT). Contextual bandit algorithms resulted in a higher CRT than the ϵ-greedy, greedy optimal, and upper confidence bound algorithms.

Table 1. Reinforcement learning applications for recommendation systems.

S. No	References	Product	Datasets	Model
1	[24]	Courses	MOOC	Dynamic attention and hierarchical reinforcement Learning (DARL)
2	[25]	Courses	MOOCCourse and MOOCCube	Hierarchical reinforcement learning with a dynamic recurrent mechanism
3	[29]	E-commerce items	Item-info, trainset, and track2_testset	Actor-critic with state representation
4	[30]	Movies	MovieLens	Deep Q network
5	[31]	Movies	MovieLens and Netflix	Deep Q network
6	[32]	Movies	MovieLens 100K, MovieLens 1M and Steam	Recurrent neural network
7	[33]	E-commerce items	JD.com(E-commerce website)	Deep Q network
8	[34]	E-commerce items	E-commerce website	Deep Q network with CNN and GAN
9	[35]	E-commerce items	E-commerce website	LSTM and DDPG
10	[36]	Movies and Music	ML100K, ML1M, and YMusic	User-specific deep Q-learning
11	[37]	Advertisements	ADS-16	Deep bayesian bandits
12	[38]	E-commerce items	Amazon book and Yelp2018	Knowledge-enhanced Ke-LinUCB
13	[39]	News and Movies	Yahoo Front Page Today Module, Lastfm, and MovieLens20M	LinUCB, Hybrid-LinUCB, and CoLin
14	[40]	Item	Yoochoose, IJCAI-15 Retailrocket	Thompson sampling
15	[41]	Movies	MovieLens	BiUCB (Binary upper confidence bound)

provides an overview of RL recommendation applications using different datasets and models.

3.2. Reinforcement Learning in Gaming

RL is widely used in many gaming applications. RL learns the game through trial and error based on short- and long-term rewards. Sometimes the RL agent has to sacrifice the immediate reward for the long-term reward, which yields good results. An RL model to learn the Othello game without the intervention of human knowledge has been proposed [21]. In [43], the authors implemented a DRL to play Atari games. They used Q-learning with convolutional neural network (CNN). Image pixels were used as input, and value functions were the outputs. They developed the algorithm for seven Atari games with this CNN and Q-learning and found that it performs better than the previous methods. In other research, the DDPG algorithm was proposed to track the pursuer and seize the evader quickly [44]. RL can also provide game-based learning in which students can efficiently learn through the interactive RL environment. In their article, ref. [45] presented game-based learning in a reinforcement learning environment. Many gaming scenarios have successfully deployed self-play RL, where the RL agents learn by interacting with themselves [46]. The game of go is a classical game involving two players, and one player has to surround more places than the other. The AlphaGo program, which comprises neural networks and Monte Carlo, was developed by [47]. AlphaGo performed better than the other Go programs. A general reinforcement learning framework called AlphaZero has been developed for three games: go, chess, and shogi (Japanese chess) [48]. The game rules were provided to the RL model without domain knowledge, and the RL model learned by self-play. AlphaZero defeated the world champions in these games. To prevent time delays during online execution, automated decision-making with DQN has been developed in the Boulder Dash game [49]. Self-play is one of the applications of RL games. The model plays against itself and learns the environment by interacting, and experience replay has been added to improve the effectiveness of the actor-critic algorithm [50]. This model was implemented with benchmark Atari and Mujoco. The value function is calculated without using Q-learning, and the model can also be used for both continuous and discrete jobs.

Table 2. Reinforcement learning applications in gaming.

S. No	References	Application	Algorithm
1	[21]	Othello game	Q-learning
2	[43]	Atari 2600 games	Convolutional neural network with Q-learning
3	[44]	Pursuit-Evasion differential game	Deep deterministic policy gradient
4	[47]	AlphaGo	Deep neural network and Monte Carlo
5	[48]	AlphaZero	Monte Carlo tree search
6	[49]	Boulder Dash game	Deep Q network

lists the gaming applications implemented using RL.

3.3. Reinforcement Learning in Automated Vehicles

Autonomous cars use technology to replace human drivers to reduce road accidents and ensure road safety. These cars work based on rules and imitation. In a model based on rules, the decisions of the cars are based on the rules framed. In an imitation model system, supervised learning methods are used to train the model. Deep learning-based automated vehicles require excessive data to train the model and cannot correct accumulated faults. RL has been introduced into automatic vehicles to learn by trial and error and overcome these issues. Self-driving cars are one of the critical application areas of reinforcement learning. Two tasks of these cars are perception and decision making systems (DMS). The car’s state is observed using the sensors, cameras, global positioning system (GPS), etc. [51]. Decision-making systems using perception systems are in charge of moving the car from the starting position to the desired goal. A model for self-driving cars using confidence-aware reinforcement learning has been proposed [52]. These researchers created an RL policy as well as a benchmark rule-based policy. The RL policy deals with the situation when the classic rule-based approach fails. Sometimes RL agents produce the wrong decision that is not encountered during the learning, leading to failure of the systems. To handle such uncertainty, ref. [53] proposed a model for uncertain environments using DeepSet-Q with a Gaussian mixture (DwGM-Q). Their experiments showed that uncertainty was well detected in the simulation environment, and the computation time was shorter than the existing ensemble method.

3.4. Reinforcement Learning in Natural Language Processing (NLP)

NLP is a notable AI application in which the computer recognizes human language and responds to its queries. Some examples of NLP are intelligent assistants, language translation, text analytics, etc. In their review article, ref. [54] discuss NLP applications such as syntactic parsing, language understanding, text generation systems, and machine translation using RL. In syntactic parsing, states are considered as parse trees of all possible combinations. Grammar rules are considered as actions, and the reward depends on the number of arcs identified correctly in the final parse tree. In text generation, states are formed with the feature vector and by adding/deleting the word as an action. Rewards are generated based on feedback from the user. Machine translation takes input in one language and converts it into another language with a similar meaning. States are the set of all possible input strings, actions are added/deleted, and rewards are generated based on a match between the input and output meanings. The complexity of natural language and its constant change make it challenging to build an NLP model. DRL can store many grammatical structures as a neural model. RL models are suitable for problems which continue changing. RL models jointly applied with DRL can provide better results. There are two types of text summarization: abstract and extract. Extract summarization highlights the critical sentence. Paraphrasing is performed in abstract summarization, which is complex. In their review article, ref. [55] discuss papers on automatic text summarization using RL and transfer learning in terms of algorithms, datasets, challenges, solutions, and performance metrics. Chatbots can perform conversations with customers to answer queries. DRL has been used with an ensemble-based model to build a chatbot [56].

3.5. Reinforcement Learning in the Internet of Things Security

Security is a significant problem in IoT systems due to the many smart devices linked to the internet, and protecting users’ data from security threats is a challenge. Supervised machine learning algorithms can detect only trained threats. RL can learn by trial and error without training and does not need a massive volume of data. In their review article, ref. [57] discuss RL methods to overcome security threats such as jamming, spoofing, and denial of service attacks. Other authors [58] have reviewed the DRL methods for identifying the intrusions in the system and their challenges. Most of the studies implemented the IDS in a simulated environment. Training the DRL for IDS in a real-world environment is costlier and more complex.

3.6. Reinforcement Learning in Finance

The dynamic nature of the financial systems can cause difficulty in framing equations, a problem called the curse of modeling. RL learns by trial and error to overcome the curse of modeling by interacting with the environment. The authors of [59] proposed portfolio management in finance using DRL. They studied how the total rewards were influenced by earlier states and actions using RNN (recurrent neural network). The optimal portfolio management policy was obtained by combining RL with DRL.

3.6.1. Trading

Framing adaptive stock trading policies is difficult due to the dynamic properties of stocks. An adaptive stock trading technique using DRL has been suggested [60]. To extract the characteristics of the financial data, a gated recurrent unit (GRU) is used. The model designers developed strategies for performing quantitative stock trading using GRU along with a deterministic policy gradient and gated DQN. They solved the disadvantages of traditional trading strategies which are constrained to single market patterns. In the ever-changing stock market, a critic-only network with GDQN is less stable than GDPG with an actor-critic structure. A multi-model RL for trading, which considers price fluctuations and sentiment analysis on news articles, has been proposed [61]. Other authors have suggested algorithmic trading based on historical data, correlation of features, and technical analysis indicators [62]. A deep neural network (DNN) model was proposed to forecast the number of shares to trade [63]. These authors used DQN to find suitable action techniques for improving the profit in the market. A long short term memory (LSTM) model along a with proximal policy optimization algorithm for constructing a Bitcoin trading strategy has been proposed [64]. ResNet and LSTM provide outperforming results for automatic trading compared with RL algorithms [65]. A random neural network based on DRL was proposed to forecast market data’s upward, downward, and similar trends [66]. Short-term memory is required in stock market analysis rather than long-term history. A multi-agent DQN was proposed for automatic trading [67], and the DQN was fine-tuned by adjusting the hyper-parameters, such as activation function, number of q networks, learning rate, discount factor, etc., using a Forex (EUR/USD) dataset. The model uses performance metrics such as sharp ratio, average cumulative return, maximum cumulative return, minimum cumulative return, etc. A DQN model for automated trading, trained on the same dataset multiple times to predict future market conditions was suggested [68]. This model was applied to intraday trading. The majority of research in trading applications employs value-based RL. An asynchronous advantage actor-critic (A3C) algorithm is proposed for stock selection [69], which makes use of both policy-based and value-based models.

Table 3. Reinforcement learning applications in finance.

S. No	References	Datasets Source	Model	Performance Measures
1	[62]	Yahoo Finance	Time-driven feature-aware jointly deep reinforcement learning model	Total profit, transaction times, the annualized rate of return, and sharp ratio
2	[63]	Thomson Reuters and Yahoo Finance	Deep neural network regressor and DQN (Deep Q network)	Total profit
3	[64]	Cryptodatadownload	LSTM and PPO	Profit rate
4	[66]	UK house prices, Gold, Bitcoin, FTSE, and Brent oil Market Validation	DRL with random neural network	Accuracy, RSME, MAE, MAPE
5	[67]	Forex	Multi-agent DQN	Sharp ratio, average cumulative return, maximum cumulative return, minimum cumulative return, etc.
6	[68]	Standard & Poor’s 500 (S&P500) and the German stock index (DAX)	Q-learning	Equity curve, accuracy, coverage, maximum drawdown, and Sortino ratio

shows the references and their corresponding datasets, Modes, and performance metrics for trading applications.

3.6.2. Comparison between ML and RL in Credit Risk

Plenty of research work has been published about using machine learning to analyze credit risk. In their research, ref. [70] categorized credit card defaulters by combining DALEX and XGBoost. The performance metrics employed were sensitivity, specificity, and accuracy. In other research [71], a deep learning and machine learning strategy for categorizing credit risk was proposed. They used machine-learning approaches such as random forest, logistic regression, and gradient boosting. In addition, the four DL model vary in the neurons, hidden layer, and regularization methods. In their review of credit risk analysis and its limitations, ref. [72] concluded that DL models provide better prediction accuracy than ML and statistical methods. Credit, market, and operational risks can be assessed using ML, as reviewed by [73]. In their work, ref. [74] provided a suggestion for banking risk management utilizing ML and AI. These suggestions apply to small and mid-sized banks in developed and developing markets. They evaluated the client risk based on the credit score before granting a loan to a customer. A credit risk analysis using an RL model was recommended by [75]. The credit score can be used to analyze the customer risk category. Another paper calculated credit scores using DQN and the changing reward function [76]. They tested the model on five different datasets including Australia (AU), Chongqing (CH), credit card fraud (CR), German (GE), and Leadingclub (LE). The evaluation measures used by them were true negative rate (TNR), area under the ROC curve, percentage correctly classified (PCC), true positive rate (TPR), precision, and F1 score. The specific threshold was defined to differentiate between good and bad credit.

3.7. Reinforcement Learning in Robotics

The most popular application field of RL is robotics. The authors of [77] examined the use of social robots, which are designed with a different form of reward mechanism. They addressed three types of reward mechanisms: interactive reinforcement learning, intrinsic motivation, and task performance. In classical RL, an agent obtains the reward from the environment by utilizing a predetermined reward function. Interactive RL interacts with humans to obtain explicit and implicit feedback. Explicit feedback was direct based on ratings and labels. Implicit feedback was indirect and based on non-verbal cues such as emotions, speech, and gestures. The third form of reward was based on the robot’s performance while interacting with a human. Nowadays, Robots are used in various applications such as manufacturing, packing, disaster management, healthcare, logistics warehouses, space, etc. Robots can also be used to understand human emotions and make them feel comfortable. The authors of [78] suggested a framework to introduce cognitive empathy in social robots. This model identifies the user’s affective state based on their facial expression and sends them an empathic behaviour. If the user’s affective state changes from a negative to a positive or neutral state, the robot receives a positive reward; otherwise, it receives a negative reward. In this experiment, the researchers investigated four basic emotions and three types of people. RL robots have been deployed in airways, waterways, and on land. The authors of [3] reviewed the applications of RL in different domains. A DRL-based algorithm was proposed to explore underwater [79]. With the help of RL, mobile robots that are moved from one location to another to perform specific tasks have become popular. Such robots face a slew of navigational challenges. The authors of [80] review listed mobile robots’ challenges and solutions to these challenges. The authors of [81] discuss the challenges of motion planning for mobile robots. Finding the optimal route for the mobile robot without colliding with an obstacle is difficult. In their work, ref. [82] proposed a method to optimize the indoor path for mobile robots. Other researchers proposed development of a DQN for planning the most efficient route for the mobile robot [83]. They made a comparison of conventional DQN and enhanced DQN. Excellent outcomes were achieved by increasing the reward value and decreasing the loss function. Mobile robots suffer from the problem of deadlock and redundant paths. To overcome this problem, ref. [84] proposed a fusion model which consists of fuzzy logic, long short term memory (LSTM), and RL algorithms. In their work, ref. [85] proposed continual learning with RL to optimize a mobile robot’s trajectory and reach the current destination in the real world and simulation environment. Another review discusses the advantages and disadvantages of robotics in RL [86].

3.8. Reinforcement Learning in Healthcare

A large number of people have lung cancer, but early diagnosis and discovery can lower the death rate. In their work, ref. [87] suggested a computer-aided diagnosis system for detecting lung cancer using deep reinforcement learning. They used value-based algorithms such as DQN, hierarchical DQN, and deep successor Q-network. In other research, supervised learning and RL gave a dynamic treatment recommendation [88]. These researchers used two signals: an indicator signal and an evaluation signal. The indicator signal matches the signal with the doctor’s prescription, and the evaluation signal is the overall reward obtained from the survival rate. They also used the actor-critic algorithm and recurrent neural network for treatment recommendations. The authors of [89] reviewed methods to analyze disease detection and recommended the medications to the patients using RL. In their article, ref. [90] suggested context-aware RL for analyzing human health using sort retention double DQN. They compared DQN and sort retention double DQN and concluded that this proposed method achieves better results. Precision medicine is one of the essential applications of RL. It provides personalized treatment recommendations based on the disease’s symptoms. In their work, ref. [91] proposed a precision medicine model using RL. They clustered the patients based on their same states and recommended treatments to them. Diabetes affects a large number of people and necessitates lifelong medication. A treatment recommendation for diabetic patients is given using an RL-based approach [92].

3.9. Inverse Reinforcement Learning (IRL)

MDP and reward functions are not known in some applications. In such cases, inverse reinforcement learning is used. IRL learns the reward function on its own. There is no need to specify the reward function explicitly. IRL is a demonstration learning in which the learner tries to learn the reward function of the instructor giving the lessons. In earlier forms of IRL, the expert provided demonstrations, whereas in later IRL, demonstrations are given as trajectories (state-action pairs). Different categories of IRL algorithms are max margin, Bayesian method, and maximum entropy [13]. In specific environments, the transition probability is unknown, a situation called model-free. IRL can be applied to the model-free approach. IRL is applied in dynamic route recommendations [93]. In the model-based approach, the reward function is estimated in linear and non-linear systems [94]. When IRL is used in a system where the developer cannot frame the rewards explicitly, expert behaviour is transferred to the RL agent so that it can perform well in the desired task. Inverse RL has been applied in a multiplayer, non-cooperative environment [95]. The authors of [96] proposed data-driven IRL for multiplayer environments. IRL with Dijkstra’s algorithm was modelled to optimize the route for food delivery applications [97]. Based on the delivery staff’s preferences, it recommends the optimized route. The authors of [98] suggested a model using IRL to predict commenting behaviour among the users and inattentive user groups on YouTube. They considered each user as an individual contextual bandit problem. Based on the commenting behaviour of the user, they grouped the users into different clusters. Their primary research conclusion was that viewers were eager to leave comments on popular videos.

3.10. Multi-Agent Reinforcement Learning (MARL)

In multi-agent RL, multiple agents are involved in decision-making. Each agent is solely accountable for their actions. There are three types of environments for MARL: competition between agents, cooperation between agents, and a combination of both. Agents compete against each other to win the game in a competition situation. In a cooperative situation, agents work together to attain a common goal. Intelligent traffic light control has been implemented with multi-agent DRL [99]. This model solves traffic congestion in less time. The experiments were conducted on three different datasets and achieved good results. Job scheduling has been implemented using multi-agent deep reinforcement learning [100]. Dynamic route optimization for human drivers using multi-agent DQN was implemented by [101]. Predictive maintenance has been performed with multi-agent RL [102]. The agent observed the machine’s state and performed the maintenance task based on the prediction, which improved performance by 75%. Multiple agents try to learn simultaneously to improve the cumulative reward. In this case, the agent’s policies were difficult to converge. An entropy regularizer is used along with an actor-critic algorithm to overcome this scenario [103].

3.11. Energy Management

EMSs are used for achieving various goals, such as lowering energy consumption, controlling energy supply and demand, enhancing the use of renewable energy and lowering energy expenses. Energy management in a smart grid is a difficult challenge. The authors of [104] proposed a smart grid modelled as a Markov decision process and used Q-learning to reduce energy consumption and cost. The authors of [105] investigated power grid operation and maintenance implemented using a Q-learning-based artificial neural network (ANN). Reinforcement learning has been used in energy management in various contexts, such as building energy management systems, reducing the consumption of electric vehicles, and renewable energy integration. RL has been used in building energy management systems to control heating, ventilation, and air conditioning (HVAC), thereby minimizing energy use while keeping the consumer in their comfort zone. RL has been applied to reduce the fuel consumption of electric vehicles. RL can also be used to optimize the control and management of renewable energy. The authors of [106] reviewed the advantages and disadvantages of using RL in the energy consumption of intelligent buildings. Real-time deployment of RL in intelligent buildings to reduce energy usage may pose difficulties in real-world scenarios as training takes a long time.

3.11.1. HVAC Control in Buildings

Most people like to spend time inside their homes, and indoor air quality affects human health. Higher levels of CO₂ cause ill health in human beings. It is essential to maintain the level of CO₂. RL-based approaches have been proposed to monitor and manage the level of CO₂ [107,108]. These authors proposed smart home-based energy management using RL. The smart home consisted of thermal storage systems, rooftop photovoltaics, battery storage systems, etc. They designed the energy management with a constrained MDP. A primal-dual deterministic policy gradient algorithm was used to reduce consumption and cost function. Similarly, thermal comfort is necessary for the employees in an office environment to work comfortably and also to avoid health issues. The system has to provide both comfort and consume less energy. The authors of [109] proposed a cooling water system using multi-agent reinforcement learning based on a model-free algorithm. The energy consumption of their proposed method was better than the rule-based model and close to the model-based design. A multi-actor attention critic approach has been proposed for controlling HVAC [110]. These researchers used a model-free algorithm and framed their own Markov decision process, which consists of states, actions, and rewards. They used comfort-related performance metrics such as average temperature deviation and average CO₂ concentration deviation. In another work [111], it was suggested that control systems do not need models since they learn through interaction with their environment. These researchers used a double deep Q network and performed the hyperparameter tuning in the double deep Q network to obtain the desired results. In addition, they compared the results with a model predictive control. Their proposed method outperformed the model predictive control regarding the deviation of temperature and period.

An actor-critic-based model was proposed for regulating ventilation, air conditioning, and heating [112]. These authors investigated the relationship between energy usage and thermal stability and found that the SAC algorithm outperformed the existing algorithms. The authors of [113] framed the occupant behaviour as a Markov decision process. The thermostat was adjusted based on the behaviour of the occupants. In the winter season, the model had to raise the temperature. In this case, the model receives positive rewards. When the occupant feels a neutral temperature, the model increasing or decreasing the temperature results in negative rewards. The researchers performed the simulation for four different seasons with reinforcement learning. In addition, they used transfer learning to transfer the trained RL model to a separate building with only a few pieces of information.

The authors of [114], applied a DRL-based system to minimize energy use while keeping the occupant in their comfort zone using a DDPG algorithm. Another work reviewed the challenges of managing the energy consumption system in buildings [115]. These researchers divided the work into single-building RL, clusters of buildings, and MARL. MDP modifications, off-policy algorithms, and model-based RL can improve the sample efficiency of an RL model. Expert knowledge can reduce the RL’s training time. Therefore, they used transfer learning instead of starting from scratch in different buildings. The conclusions of their review were that model-based RL provides good results in terms of sample efficiency. They also mentioned that transfer learning shows promising results. Most of the research work focused on single-variable control.

The authors of [116] proposed a multi-variate control using a branching dueling Q-network. This agent was pre-trained in the simulated environment and then deployed in a real-world environment. They reduced the cooling energy by 14% and improved thermal acceptability by 11%. The authors of [117] proposed an attention-based multi-agent DRL which does not require prior knowledge of an uncertain environment. They promoted the coordination between the personal comfort system and HVAC. In another work [118], researchers framed a non-stationary MDP. Whenever there is a degradation in the performance, this DRL model relearns to improve the performance. To improve the sample efficiency, they used elastic weight consolidation. They compared it with the classic rule-based model, model predictive control, PPO, and DDPG models.

Table 4. Reinforcement learning applications for HVAC control.

S. No	References	Applications	Algorithms
1	[109]	Water cooling system	Multi-agent DRL
2	[110]	HVAC control in commercial buildings	Multi-agent DRL with actor attention critic
3	[111]	Control of HVAC considering dynamic occupant patterns	Double deep Q networks
4	[112]	HVAC control by maintaining the thermal stability	Actor-critic
5	[113]	HVAC control based on occupant behaviour for different buildings	Q-learning
6	[114]	Multi-zone HVAC control	Deep deterministic policy gradient
7	[116]	Multi-variant occupant-centric HVAC	Branching dueling Q-network
8	[117]	HVAC control in office buildings	Attention-based multi-agent DRL
9	[118]	HVAC control for non-stationary buildings	A deep reinforcement learning model

provides an overview of HVAC control in buildings using RL algorithms.

3.11.2. Energy Management in Vehicles

Vehicle fuel management has received attention due to the scarcity of energy and oil and environmental issues such as the greenhouse effect. The authors of [119] proposed a reverse RL based on the energy management of hybrid vehicles. In [120], a DRL-based automated guided vehicle (AGV) is presented that controls its speed according to the environment. These researchers designed a deep deterministic policy gradient according to the actor-critic algorithm. This method reduced energy consumption by 4.6%. In hybrid electric vehicles, hydrogen is used to reduce pollution. Fuel cell vehicles typically need a combination of various power sources to satisfy driving demands. Different power sources can be optimized with an EMS. A deep Q-learning system with priority experience replay and DDPG with priority experience replay to reduce consumption has been proposed [121]. Fuel cells have some disadvantages such as delays in response and the inability to recover braking energy. Researchers have proposed a TD3 algorithm for logistic trucks to reduce hydrogen consumption and extend the fuel and battery life span [122]. They compared a deep deterministic policy gradient and a non-linear programming algorithm. Their proposed algorithm minimizes the ageing of core components and hydrogen consumption. Eco-driving is an effective technology for reducing energy consumption in vehicles. An EMS using the Q-learning algorithm has been implemented [123]. These authors compared their algorithm with dynamic programming. The RL-based model adjusts the vehicle’s speed based on road conditions and maintains a suitable distance from the leading vehicle. The RL model simulation results provided near-optimal performance compared with dynamic programming. Other researchers have proposed a SAC algorithm [124]. They employed cooperative optimization and implemented the MARL for eco-driving by optimizing energy management. They framed the reward function concerning collision avoidance, driving comfort, and energy efficiency. This model outperformed the hierarchical model used as a predictive control.

Table 5. Reinforcement learning applications of energy management in vehicles.

S. No	References	Applications	Algorithms
1	[120]	Automated guided vehicle	Actor-critic and DDPG
2	[121]	Reducing the fuel consumption in hybrid vehicles	Deep Q-learning and DDPG
3	[122]	Energy management strategy for logistic trucks	TD3
4	[123]	Energy-efficient eco-driving by controlling the speed in electric vehicles	Q-learning
5	[124]	Eco-driving in hybrid electric vehicles	MARL using SAC

provides the RL algorithms for reducing energy consumption in hybrid vehicles.

3.11.3. Renewable Energy Management

Renewable energy contributes to an eco-friendly environment and sustainable use. Recently, renewable energy has been used in buildings. Solar energy is one of the fastest-growing sources of renewable energy. Traditional rule-based and model predictive controllers manage use for optimizing energy resources. Renewable energy optimization has been examined by [125]. In this study, the Gaussian distribution was used to frame the reward function. They compared the two DRL algorithms TD3 and DDPG. They tested the algorithms under both random and extreme conditions. In another work [126], the authors proposed a DRL-based renewable energy optimization. Their goal was to produce hydrogen and sell it to increase revenue, and they created this model to be used in conjunction with OpenAI Gym and Ray/RLib for deep reinforcement learning applications. They used three algorithms including TD3, PPO, and SAC. PPO provided stable performance over the other two algorithms.

Researchers have proposed a storage system for renewable energy based on hydrogen using DRL [127]. They contrasted this strategy to rule-based and dynamic programming. The DRL model outperformed the rule-based method in simulation results. They employed the PPO method with the stochastic gradient descent (SGD) optimizer and fine-tuned the parameters with hyperparameters. Wind energy is one of the most environmentally friendly sources of electricity. The authors of [128] applied actor-critic RL for wind turbine control. Wind turbines are devices that transform kinetic energy into electrical energy. The simulation was carried out using the OpenFAST simulator.

Table 6. Reinforcement learning application of renewable energy.

S. No	References	Applications	Algorithms
1	[125]	Off-grid optimization of renewable energy	TD3 and DDPG
2	[126]	Nuclear renewable integrated energy system optimization	TD3, PPO, SAC
3	[127]	Storage systems of renewable energy	PPO
4	[128]	Control of wind turbines	Actor-Critic

shows how renewable energy can be managed using RL algorithms.

4. Conclusions

In this research, we reviewed the literature on reinforcement learning. Because RL can learn independently, it is well suited to dynamic contexts. In both critical and non-critical applications, RL is used. Critical applications include self-driving cars, security, healthcare, energy management systems, and finance. The non-critical applications are those such as Gaming. RL has the potential to revolutionize energy management by allowing systems to adapt and optimize energy use in real time. RL algorithms may use previous data to forecast future energy demand, allowing energy systems to modify energy output and consumption accordingly. RL-based EMS can result in more efficient energy consumption and cost savings.

Furthermore, by forecasting and compensating for swings in energy output, RL may improve the functioning of renewable energy sources such as wind and solar power. Overall, RL has the potential to increase the efficiency and efficacy of energy management systems significantly. Most EMS applications have been tested in simulation environments. In the future, more EMS should be implemented in real-world world settings. RL has a broad scope in the future with the potential to enhance the energy efficiency of HVAC systems dramatically. Still, additional research is needed to build RL algorithms that can manage the complexity and unpredictability of real-world HVAC systems.

Recommendation systems built with RL work effectively and have high prediction accuracy. In security-related work, RL is used in a simulated environment, but it may be used in the real world in the future. With the help of RL, gaming applications are expanding. RL is also being used in financial applications. More works have been published in trading applications using RL, but fewer papers have been published about credit risk analysis using RL. In the future, RL has a broad scope for applications in all fields