BEERL: Both Ends Explanations for Reinforcement Learning

Abstract

Deep Reinforcement Learning (RL) is a black-box method and is hard to understand because the agent employs a neural network (NN). To explain the behavior and decisions made by the agent, different eXplainable RL (XRL) methods are developed; for example, feature importance methods are applied to analyze the contribution of the input side of the model, and reward decomposition methods are applied to explain the components of the output end of the RL model. In this study, we present a novel method to connect explanations from both input and output ends of a black-box model, which results in fine-grained explanations. Our method exposes the reward prioritization to the user, which in turn generates two different levels of explanation and allows RL agent reconfigurations when unwanted behaviors are observed. The method further summarizes the detailed explanations into a focus value that takes into account all reward components and quantifies the fulfillment of the explanation of desired properties. We evaluated our method by applying it to a remote electrical telecom-antenna-tilt use case and two openAI gym environments: lunar lander and cartpole. The results demonstrated fine-grained explanations by detailing input features’ contributions to certain rewards and revealed biases of the reward components, which are then addressed by adjusting the reward’s weights.

1. Introduction

Reinforcement learning (RL) is a method by which an autonomous agent learns from its interaction with the environment. This agent gathers information about its environment via state variables as input features and, based on it, calculates the quality of the current state to select an action to be generated as the output. In each step, a reward is given in return to measure how good the state is, especially in achieving the main goal of the task. One of the RL methods is deep reinforcement learning (DRL), and it has been used to solve various tasks with outstanding performance [1,2]. The main driving factor for high-performance DRL is the use of a deep neural network (DNN). In general, DNN is used in many complex tasks, such as identifying images, recognizing voices, generating fake videos, etc. Each node in DNN is a mathematical operation constructed in a certain way, and following all operations to generate the output is not intuitive. Since the process of how DNN produces a specific output is hard to understand and explain, explainable reinforcement learning (XRL) methods produce information about the AI model’s behavior. Attributive explanation is the most common type of explanation [3], where the importance of an input features indicates how it affects the output of the model without informing the inner part of the model. Alternatively, a contrastive explanation shows why one action is preferred over the other, which may require model decompositions for RL the system. When the output of an RL model is decomposed into several components, a model may prefer one component over the other; we refer to this as component bias. Please note that, the bias of the reward component is derived from the base value of [4] where it previously showed the bias of each class/action. However, the correlation of the input features importance with reward components has not been found in the literature. Additionally, the effect of reward prioritization is also not explored with respect to generating explanations. We present a novel “Both Ends Explanations for RL (BEERL)” method that produces correlations between input features and output reward components. The goals of our work are to generate more detailed explanations, to show the correlations among them, to summarize extensive information for the user, and to identify and mitigate the bias of the RL model.

The main contribution of BEERL method is to generate fine-grained explanations (a fine-grained explanation consists of input features and reward components pairs instead of either only input features or only reward components) by performing the following: (1) Exposing the effects of reward prioritization resulting in two different levels of the RL agent (Q-function and RL level); (2) Identifying the bias of the reward components and demonstrating a method to mitigate it; (3) Extending the contrastive explanation method to highlight a certain input feature(s) or reward component(s); (4) Formulating a focus value that provides a quantification of the explanation by measuring how much the desired behavior is satisfied. We apply our method to a remote electrical antenna-tilt use case and two openAI gym environments to evaluate the results. The result showed that the generated explanations can be used to evaluate the performance of the components of the RL agent. With these explanations, we also demonstrate several possible options to reconfigure the RL agent, including a bias reduction in the reward components.

Paper outline:Section 2 presents the background and related works. Section 3 describes the details of our BEERL method. Section 4 presents the application of BEERL to the use case and the experimental result. Section 5 summarizes the paper and points to possible extensions of future work. Appendix A presents the improved MSX formulation that is used to compress contrastive explanation. Appendix B and Appendix C present the parameters of the RL agent and antenna tilt configuration. Appendix D shows the implementation of BEERL in two openAI gym environments.

2. Background and Related Works

This section elaborates the motivation of our work, the problem we are addressing in the current XRL method, and the relationship with other explainable artificial intelligence (XAI) methods.

2.1. Background

In XAI, generating an explanation can be performed intrinsically if the model is transparent (such as decision tree and a rule-based system) or after the AI model has produced its output (post hoc methods). Several post hoc methods such as SHAP [4] and LIME [5] are model-agnostic, which means that they are applicable to any black-box model such as neural networks (NN), ensemble models, etc. One type of XAI methods focuses on feature importance where knowing the contribution of different input variables is important to understand the characteristics of certain input variables. It may also indicate which factors/variables are not important and can be changed without affecting the output or which ones are important and must be retained to keep the same output. In the RL problem, this information is closely related to the input data used, specifically the states of the environment during the training phase. Deeplift [6], SHAP [4], and LIME [5] are methods that can be used to explain the feature importance of the DRL agent with respect to the actions. However, a method to measure the contribution of input features to the components of the RL reward function is lacking in the literature.

In RL, the reward function is designed to quantify the desired situation as feedback to update the agent’s parameters. Designing and obtaining feedback on the formulation of the reward function is challenging. Its calculation generally considers several factors, but without a reward decomposition method, such as Hybrid Reward Architecture (HRA) [7], only a single value is given to the agent. HRA uses several reward values (respective of the components) to update the RL agent’s parameters. The HRA authors mentioned that the reward component typically depends on a subset of features without explaining this dependency further. Juozapaitis et al. [8] used HRA to explain the output layer of a DRL agent, specifically the RL reward components. However, it only presents the values of reward components without showing the bias and effect of different weights for each component.

A correlation of input and output explanations is lacking in the literature. With the available methods, the contributions of the input features are not reflected in the explanation of the reward component. Conversely, the feature importance does not inform us about the reward component to which the state variable contributes. In this work, we present a method for expanding these techniques not only to correlate input and output explanations but also to expose the effect of the reward weight and to identify and mitigate the bias on different reward components.

2.2. Related Works

This section presents related work on methods that focus on measuring the importance of the input feature to the output of the model. Then, it discusses the reward decomposition to complement the generated explanation in the output or the final layer.

2.2.1. Explainability for Input Features

Several XAI methods [6,9,10,11,12] have been developed for a specific type of AI model. Methods that explain the DNN model’s [6,11,12] operation by tapping or backpropagating the inference process, resulting in faster computations than methods with repetitive model inferences. In order to understand a Deep Q-network (DQN), Zahavy et al. [13] introduced a method that requires manual clustering to generate an explanation in the form of a saliency map. On the other hand, Atrey et al. [14] manipulated the RL environment and used the saliency map to verify the counterfactual explanation and showed that the saliency explanation is subjective and insufficient. In that work, several attributive XAI methods were analyzed, but SHAP [4] was not included. Wang et al. [15] highlighted the importance of the baseline data that are required for the SHAP method in the RL problem but did not provide a detailed data selection method nor its verification. For a machine learning problem, SHAP [4] is a model-agnostic method in which it is observed to produce the best results when applied to the telecommunications use case [16]. RL-SHAP [17] is an example where SHAP [4] can be applied to the RL problem to explain the contribution of the actor’s input features. A recent survey by Hickling et al. [18] also identified that SHAP is highly adopted in RL, especially in the field of vehicle guidance, robotic manipulation, and system control. In the aforementioned methods, they used the entire model to generate the explanation, while our method splits the model into each reward component, and the importance of the input feature is now generated per reward component, thus providing fine-grained explanations.

2.2.2. Explainability for Output/Reward Components

When a reward function considers multiple factors, the reward value can be decomposed by exposing its calculation [7]. Lin et al. decomposed the reward while disentangling the representation without domain knowledge [19]. These methods did not consider the contribution of every input feature, while our work connects it to the reward components. Juozapaitis et al. implemented reward decomposition and used the difference between Q-values of the contrasted actions to explain the agent’s decision [8]. They introduced the concept of minimal sufficient explanation (MSX) to compress the contrastive explanation. Lin et al. in [20] used this concept to generate contrastive explanations by implementing an embedded self-prediction model via general value functions (GVFs) [21] that are located between two NNs used by the agent. In the above methods, all components are treated as the same (having equal weights, i.e., 1) and only final reward components values are obtained. Our method decouples the reward weights and Q-functions approximators, allowing each component to be treated equally and shows the weights explicitly. It uses the contributions of the input features to each reward component to explain why an action is preferred over the other. Since our method generates a more granular explanation than earlier methods, we also propose a method to compress it. Another work by Bica et al. proposed a method to learn the reward weight using the counterfactual inverse RL algorithm [22] while our method maintains a static weight that encourages transparency during training.

2.2.3. Quantification of the Explanations

In XAI, the quality of the explanation is measured on a case-by-case basis because the explanations can be presented in various forms. Mohseni et al. in [23] summarized the evaluation measures for XAI, comprising different aspects such as the mental model, usefulness and satisfaction, user trust and reliance, human-AI task performance, and computational measures. Schmidt and Biessmann [24] presented the quality and trust metrics for AI explanation that are based on response time, mutual information, and agreement between XAI and humans. Zhang et al. [25] used KL-divergence to measure suboptimal attributes in an explanation to detect bias and failure. Anderson et al. [26] performed a more similar evaluation to our work, where they evaluated the improvement of the human mental model after being presented with a certain type of explanation. In that work, both input and output explanations were presented without any correlation. Our proposed work presents not only the correlation between input and output explanations but also a focus value to quantify the desired properties set by the user.

2.2.4. Other XRL Methods

In addition to the above methods, there are several other XRL methods which use the causal model [27], and distill the trained agent into a transparent model [28,29]. Madumal et al. [27] proposed a method that uses a causal model to explain the cause and counterfactual explanation of the behavior of the RL agent. This method requires that the causal model be prepared manually, which can inhibit implementation, especially when an expert is not available to build or verify it. Verma et al. [28] proposed an RL method in which they initially trained a DRL agent and derived transparent programmatic policies. Liu et al. [29] approximated a DRL Q-function using linear model U-trees where the leaf node of the tree is a linear model. Our method is different from the above because, instead of distilling the RL policy into a transparent model, it exposes the reward components and their proportion to make the second-last layer of the NN transparent, producing explanations at a finer level of granularity. With the existing methods [4,5,6,8,11,12], the quantity of the generated explanation will be the sum of the number of input features ($N_I$) and the number of reward components ($N_C$), while our method can produce $N_I\times N_C$ explanations.

2.2.5. Explainability of Remote Electrical Antenna Tilt

We apply our method to the remote electrical antenna tilt use case, which has previously been explored from different angles using state-of-the-art methods. Vanella et al. [30] applied a safe RL method to avoid the risk of performance degradation, especially in the exploration phase. In [31], Vanella et al. also introduced the offline RL method to learn the agent’s optimal policy from the real-world data. Furthermore, Vanella et al. [32] used a contextual linear bandit method to find an optimal policy with fewer data than naive or rule-based algorithms. Bouton et al. [33] introduced an RL method that uses a predefined coordination graph to allow coordination among agents in controlling the antennas. Lastly, Jin et al. [34] proposed graph-attention Q-learning where the graph-attention network [35] is combined with DQN [1] to capture broader network information without having a large state for the agent. All these previous works do not focus on the explainability of the agent. Our work addresses this matter, including how to detect and mitigate bias. We implement our method and analyze our results on this use case and it can be implemented in any other environment, such as for OpenAI gyms (Cartpole and Lunar lander examples are shown in Appendix D).

3. Both End Explanations for Reinforcement Learning (BEERL) Method

The system designer determines the RL system to be used based on the task and the environment of the use case. In case the task provided by the system designer is of a high complexity, the produced machine learning model will be of a high complexity as well, and its understanding will be challenging for a human being. One of the main goals of explainability is to improve the human understanding of AI [36]. Therefore, we categorize the three main aspects of XRL: the environment in which the RL agent operates, the human user of the system, and the RL framework that performs its operation. The Both End Explanations for Reinforcement Learning (BEERL) method and the interplay between the XRL aspects are presented in Figure 1 and discussed in the following subsections.

3.1. RL Environment

An RL agent is deployed in a certain environment where it observes the state and performs its actions. In the proposed method, the environment should be constructed using several components to decompose the total reward calculation into different reward functions, as is performed in [7,8]. The state and action formulations do not need to be modified and are similar to the RL without reward decomposition.

3.2. Reward Functions

In a non-decomposed RL, the reward is provided as a single numeric value, while our method follows [7,8] to decompose the reward value into several components($r_c$). In our method, each reward component must be defined in a normalized manner, that is, within the same range (e.g., within the range $[-1,1]$). The normalized rewards are fed to the Q-function during the training process to update Q-function parameters. Additionally, the output range of the Q-function is set at a normalized value to avoid the vanishing or exploding gradient of DNN parameters. It can happen when the output range is too large or too small, as addressed in [37] and also exemplified in Lunar Lander in Appendix D.2. However, the weighted reward values are stored in a replay buffer in order to be summarized when a global explanation is generated. The summary of the reward component is calculated by taking the absolute mean of all the recorded reward components:

$$R_c=\frac{{\sum }_{t=1}^T\left|r_{ct}\right|}{N_T}$$

(1)

Throughout this paper, $i\in I$ indicates input feature and $c\in C$ indicates the reward component. Additionally, N indicates the count of the referred annotation, e.g., $N_T$ annotates the number of time steps, $N_I$ annotates the number of input features, $N_C$ annotates the number of reward components, etc.

3.3. User Configuration

Reward prioritization is required to encode the magnitude of each component in calculating the total reward because the reward functions generate values in a normalized manner. Furthermore, our method also requires a relevance table, which encodes the desired behavior of the RL agent. In return, after completing the entire process, the user will receive three different types of explanation, which are (1) attributive explanations that consists of Q-function explanation ($U_{ic}$), RL explanation ($V_{ic}$), and normalized explanation ($L_{ic}$); (2) contrastive explanation ($\Delta V_{ic}$); and (3) focus values ($F_c$), which are explained in Section 3.5 and Section 3.6.

3.3.1. Reward Prioritization

We decouple reward prioritization ($Z_c\left(r\right)$) from reward function ($R_c\left(o\right)$) to generate two levels of explanation (Q-function and RL level). Thus, the normalized value is used as the target value for the NN that approximates the Q-value that the respective reward prioritization is then applied to configure the contribution of each reward component. $Z_c\left(r\right)$ may implement a complex formula (e.g., quadratic or exponential), while a linear function is preferred because it can be interpreted as a ratio among different reward components (e.g., $Z_{c1}\left(r\right):Z_{c2}\left(r\right):Z_{c3}\left(r\right)=k_{c1}:k_{c2}:k_{c3}$). Reward priorities are maintained during training and can be adjusted after completion for transfer learning purposes, as discussed in Section 4.6.

3.3.2. Relevance Table

In designing an RL agent, the user commonly has expectations of how the agent will behave in performing its task. Normally, this expectation is encoded in a reward function where a fraction of the state variables contribute to a reward component, and another fraction of the state variables contributes to another reward component. Our method requires the user to define the relevance of each input feature (state variable) to each reward component in the form of a relevance matrix $M_{ic}$ where i represents the input feature and c represents the reward component. This matrix quantifies the desired behavior and the value of each element should be within the range of $[0,1]$ where 1 quantifies the complete relevance of an input feature to a reward component.

3.4. RL Agent

The RL is formulated as Markov decision processes (MDPs) with a tuple of $(S,A,H,R,\gamma )$ where S represents the state space, A represents the action space, H represents the transition function ($H(s,a)$), R represents the reward function, and $\gamma$ represents the discount factor. The goal of the RL agent (as shown within the blue box in Figure 1) is to maximize the expected return ($J_t={\sum }_{t=1}^{\infty }{r}_t{\gamma }_t$) when performing its task. A Q-function $Q^{\pi }(s,a)$ generates Q-values that measure the quality of the policy given the current state (s), and the action (a) executed on the policy $\pi$ is formulated as $Q^{\pi }(s,a)={\mathbb{E}}_{\pi }\left[J_t|s_t=s,a_t=a\right]$. The component aggregator applies the reward priority, and the output layer combines all weighted Q-values to select the action as described in the following subsections.

3.4.1. Q-Function

When the reward is decomposed, the Q-function is defined per component $Q_c(s,a)$. In DRL, a reward component can be implemented as an independent neural network structure that shares only input and output layers. This architecture allows every component to be modular (i.e., a component can be transferred to or taken from another trained model) at a cost of a higher number of parameters. Alternatively, a branched network can also be implemented, where the shared initial layers can be randomly initialized network or taken from a trained model as the feature extractor. In addition to allowing transfer learning, the branching architecture also reduces the number of parameters for the NN model.

In the proposed solution, Q-functions are trainable and can be considered as hidden layers of the NN when DRL is used. The input layer takes the state of the observed environment when performing its task or the perturbation data from the explainer when generating the explanation. The output of this element is the approximated Q-values for each action per reward component. During training, the last hidden layer generates the prediction of Q-values to be calculated with a normalized reward to update the NN parameters. In this manner, the NN training operates at a normalized value and has the benefit of avoiding vanishing/exploding gradients.

3.4.2. Component Aggregator

This element connects the output/last layer with the second-last layer. It is worth noting that every node on every component’s last-layer is connected only to one node on the output layer (instead of fully connected to all output nodes). The main function of this component is translating normalized Q-values($q_c$) to weighted Q-values($y_c$) and vice versa (for training). To calculate the weighted Q-values, it receives and stores the reward prioritization ($y_c=Z_c\left(q_c\right)$). This prioritization is another major difference of this component from the summation of the reward component in [8], where they do not explicitly weight/prioritize the components.

3.4.3. Output Layer

This element aggregates the weighted Q-values by taking a sum of all components.

$$Y=\sum _{c=1}^C{Y}_c$$

(2)

When the action space is continuous, these values are then performed as the RL action. Otherwise, if the action space is discrete, it selects the action that maximizes the reward in the exploitation phase. The selected action from this part will be performed in the environment to obtain the effect of it and also stored in the replay buffer for training purposes.

3.5. Generating Explanation

Our work generates two types of explanations, which are attributive and contrastive explanations. The former calculates the contribution towards the predicted output while the latter calculates the difference of two compared actions. Since the contrastive explanation compares two different actions, it is only produced in a local scope, whereas the attributive explanation is produced in both local and global scopes.

3.5.1. Attributive Explanation

Our presented approach is independent of the XAI method used to generate feature importance, i.e., any method (e.g., SHAP [4], LIME [5], etc.) can be applied. When a deep network is employed for the RL agent, the neural network explainer can also be applied to fit the required characteristics, e.g., the complexity of the operation, consistency, the accuracy of the explanation, etc. Without reward decomposition, the XAI methods generate a feature importance for the entire Q-function (i.e., the entire DRL network), as depicted in Figure 4a. In the proposed solution, the explainer analyzes the Q-function by calculating the contribution of every input feature to every reward component:

$$U_{ic}=E(Q_c(s,a),s,a)$$

(3)

where $E(h,x,y)$ is the explainer that takes h as the model to be explained (Q-function of reward component c, $Q_c(s,a)$) together with the input data x and the predicted output y (state s and action a in DRL, respectively). The generated attribution values, $U_{ic}$, explain the contribution in a normalized domain, which are independent of reward prioritization, and we refer to them as the Q-function explanations. Furthermore, we apply the reward prioritization to these explanations to generate the RL-level explanations ($V_{ic}$).

$$V_{ic}=Z_c\left(U_{ic}\right)$$

(4)

In this manner, the effect of reward prioritization is shown and each reward component can be individually analyzed. Any scope of explanations can be produced using this method, where the global explanation is an aggregation of local explanations using the absolute mean formula, as shown in Equation (5) ($P_{ic}$ refers to either level of explanation, i.e., $U_{ic}$ or $V_{ic}$).

$$P_{ic}=\frac{{\sum }_{t=1}^T\left|P_{ict}\right|}{N_T}$$

(5)

The above generated explanations show the importance of input feature towards different reward components, but currently, there is no mechanism to compare the importance of input features with reward components. With these detailed explanations ($U_{ic}$ and $V_{ic}$), the normalized input feature importance ($L_i$) is calculated by summing its importance to all reward components and dividing it by the number of reward components ($N_C$), as shown in Equation (6). Similarly, Equation (7) shows the calculation of normalized reward component importance ($L_c$) where all input feature importance are summed and then divided by the number of input features ($N_I$). In this manner, both input feature ($L_i$) and the reward component’s ($L_c$) importances are measured on the same basis and can be properly compared.

$$L_i=\frac{{\sum }_{c=1}^C{P}_{ic}}{N_C}$$

(6)

$$L_c=\frac{{\sum }_{i=1}^I{P}_{ic}}{N_I}$$

(7)

3.5.2. Contrastive Explanations

The proposed method generates $N_C\times N_I$ of reasons because each explanation consists of a tuple of input feature and reward component. When comparing two actions, the contrastive explanation can be compressed by following the MSX concept [8]. We propose two possible compressions of the explanations. First, compression is performed by aggregating the reward component ($MS{X}^C$) resulting in explanations similar to the original MSX where only a subset of reward components’ Q-values are highlighted, and second, we compress the MSX by aggregating the input feature ($MS{X}^I$). Details of the modified MSX calculations are presented in Appendix A.

Automatic compression is also proposed to generate the simplest contrastive explanations by implementing the Algorithm 1. We denote a set of compressed explanations by implementing the MSX concept as $MS{X}^{BEERL}$. The main task of this algorithm is to select one of $MS{X}^{BEERL}$, $MS{X}^I$, or $MS{X}^C$ to be highlighted. We first collect these MSX sets in the list of explanations(P). Second, the length of reasons (N) and the advantage values (A) of each MSX set are also stored. The advantage value is the difference between the total MSX contribution and the sum of all negative contributions, as formulated in Equation (A10). The selection is started by comparing the number of reasons in each MSX set and choosing the least if a single solution is obtained. When all MSX sets have the same number of reasons, it selects the set with the highest advantage. When two sets of MSX have the same number of reasons, it removes a set of MSX with the most reasons and chooses the set with the highest advantage value between the two sets.

Algorithm 1 Automatic MSX Compression

Require:  $MS{X}^{BEERL},MS{X}^I,MS{X}^C$ $P\leftarrow [MS{X}^{BEERL},MS{X}^I,MS{X}^C]$ $N\leftarrow [N_{MS{X}^{BEERL}},N_{MS{X}^I},N_{MS{X}^C}]$ $A\leftarrow [Adv\left(MS{X}^{BEERL}\right),Adv\left(MS{X}^I\right),Adv\left(MS{X}^C\right)]$        ▹$Adv$ is advantage function (A10) $N_{min}\leftarrow min\left(N\right)$                   ▹ Check the fewest reason from the MSXs if$|N_{min}\in N|==1$then               ▹ only 1 MSX has the simplest explanation     $d=arg\; minN$                ▹ Select the fewest reason for the final explanations else if$|N_{min}\in N|==3$then     $d=arg\; maxA$              ▹ Select the explanations with the highest advantage else     $d=arg\; maxN$           ▹ Select the explanations with most reasons to be removed     pop($P_d$); pop($P_d$); pop($P_d$);                 ▹ Remove the explanation from the lists     $d=arg\; maxA$              ▹ Select the explanations with the highest advantage end if return$M_d$

3.6. Evaluator

Humans evaluate the presented explanation mostly qualitatively. When an explanation consists of many elements, humans may be overwhelmed by the amount of information. On the other hand, no quantification of how the explanation satisfies the desired outcome is available in the existing work. We propose a focus value that evaluates the generated explanation by quantifying how much the explanation satisfies the desired behavior. The focus value ($F_c$) is the result of element-wise multiplication between the relevance table ($M_{ic}$) and the explanation of the Q-function ($U_{ic}$) then it is averaged per component.

$$F_c=\frac{{\sum }_{i=1}^I{M}_{ic}\odot U_{ic}}{{\sum }_{i=1}^I{U}_{ic}}$$

(8)

In addition, the unweighted and weighted mean values are also calculated. The unweighted mean value ($F_{unweighted}$) is merely the mean of all components:

$$F_{unweighted}=\frac{{\sum }_{c=1}^C{F}_c}{N_c}$$

(9)

while the weighted mean ($F_{weighted}$) considers the reward prioritization:

$$F_{weighted}=\frac{{\sum }_{c=1}^C{Z}_c\left(F_c\right)}{{\sum }_{c=1}^C{Z}_c\left(1\right)}$$

(10)

where $Z_c$ is the reward prioritization.

3.7. Data Flow

In generating local explanations, the explainer takes a state that is then modified to calculate the feature importance. On the other hand, all recorded states are taken by the explainer and are then fed to the Q-function to generate global explanations. The first generated explanation is the feature importance at Q-function level ($U_{ic}$ or Q-FI in Figure 2). The feature importance at the RL level ($V_{ic}$ or RL-FI in Figure 2) is generated by applying reward prioritization to the Q-level explanation. The focus values are then generated after applying focus value formulas (Equations (8)–(10)) to the explanations.

4. Experimental Evaluations and Results

We experiment with a realistic use case, where DRL agents are used in a remote electrical antenna tilt environment, as shown in Figure 3. It is a multi-agent environment where each antenna is controlled individually. In this experiment, we trained a shared policy using DQN with reward decomposition (drDQN) where each agent obtains its observation, but uses the same policy as the others.

4.1. Remote Electrical Antenna Tilt Environment

Remote Electrical Antenna Tilt is a joint-optimization problem in which the agent controls the tilt of the antenna to optimize several key performance indicators (KPI). The antenna performance is evaluated mainly in three different aspects, namely the quality, capacity, and coverage of the traffic delivered by each antenna. The quality KPI is measured by the Signal-to-Interference and Noise Ratio (SINR), where the user’s equipment (UE) signal strength is compared to noise and interference, and capacity KPI measures the throughput (Tput) delivered from the antenna to the UE, while the coverage is indicated by the Reference Signals Received Power (RSRP) showing the received power level of a reference signal. The challenge of this setup lies in the opposing KPIs that are used. When an antenna is fully tilted down, the signal beam covers a narrow spot, delivering high capacities due to fewer UEs being served, and high-quality signal due to less interference from the opposing antenna. However, when the antenna is fully tilted up, it will cover a wider area, but it will deliver low capacities to the UEs and low quality due to interference from other antennas. Therefore, it is important to understand the factors of the performing agent when controlling the tilt of the antenna.

In this work, the inputs (state variables) of the RL agent consist of the tilt position, RSRP, SINR, and throughput of the antenna. DeepliftSHAP [4] is applied to analyze the contribution of these input features to the prediction generated by the RL agent. The reward for the agent is constructed with factors similar to the input feature, which are SINR ($R_{SINR}$), RSRP ($R_{RSRP}$), and throughput ($R_{Throughput}$) metrics of the network. The reward calculation is shown in Equation (11) where the RSRP has twice the weight of other KPIs. This reward component has higher priority because the other two components have opposing properties, as mentioned above. We implement our drDQN model with three branches (one for each reward component) without shared layer, and each of them has two hidden layers where each layer has 32 nodes and rectified linear unit (ReLU) activation function.

$$R_{total}=R_{SINR}+R_{Throughput}+2\times R_{RSRP}$$

(11)

Once the RL training converged, we ran an evaluation where five episodes with 100 time steps in total, resulting in 2100 data points to collect the observations as the baseline for the Deeplift-SHAP explainer. Each datum is then fed to the trained agent to calculate the Q-values, select the optimal action, and generate a local explanation. The global explanation is calculated by taking the absolute mean value of all local explanations.

4.2. Explanations Using Existing XAI Methods

By implementing existing methods, we obtain the results shown in Figure 4. The importance of every input feature generated from DeepLiftSHAP [4] is shown in Figure 4a without showing its contribution to different components. We can see that the RSRP feature contributes significantly to the prediction of the agent, but there is no information on how it is distributed to the reward components. Similarly, Figure 4b shows the explanation of how each component constructs the total reward, as in [8] without showing the contribution of every input feature. For example, when the RSRP reward component ($R_{RSRP}$) contributes the most to the final reward calculation, the contributions of the input features are not shown.

Figure 4. Explanations generated using the available XAI methods. (a) Feature importance generated using [6], which produces similar explanation format as [4,5]. (b) Reward components summary generated using [8].

Figure 4. Explanations generated using the available XAI methods. (a) Feature importance generated using [6], which produces similar explanation format as [4,5]. (b) Reward components summary generated using [8].

4.3. Attributive Explanations with BEERL

Our method leverages the transparency of the last layers by implementing the reward decomposition method and separating the reward weight so that the reward components can be compared in a fair manner. This approach benefits in two aspects, which are as follows: (1) each reward component is trained in the same range, which minimizes the explosion or vanishing gradient; (2) the effect of reward prioritization is exposed so it helps humans understand the identification of unwanted behavior.

Figure 5 shows the explanation on the Q-function level, i.e., how the Q-function (which is implemented in a neural network model) performs regardless of the priority of the reward component. From Figure 5a, we can see that the RSRP, SINR, and throughput input features have dominant importance to their respective reward components in comparison to other input features. Similarly, the respective input feature becomes the most important for each reward component (other input features do not outweigh their importance in every reward component), as shown in Figure 5b.

When NN is employed for the RL agent, it is desirable that the Q-functions work in a normalized range to avoid the exploding or vanishing gradient phenomenon. Explanations at this level show the behavior of NN in approximating Q-values and the fairness of it (i.e., when all components are treated/trained in equal numeric values). Ideally, each feature has a significant contribution to at least one reward component. It is a waste of resources when an input feature is used, but it does not have any impact on the output. Similarly, each reward component should have at least an input feature that significantly affects it.

The explanation at the RL agent level shows the correlation between the input and output explanations, where the priority/weights of the reward components are considered. In this manner, we can clearly see which aspect affects the final action of the agent (e.g., whether the neural network as the Q-function, the reward prioritization, or the combination of them). When reward weights are applied and the RSRP reward component is prioritized more than others ($2.0:1.0:1.0$ for $k_{RSRP}:k_{SINR}:k_{Throughput}$), we can see that each input feature still dominates its importance for the respective reward component and vice versa, as shown in Figure 6. Unlike Q-function explanations, an expert may accept RL explanations where a reward component does not contribute significantly to the total reward. The contribution of the reward component can be adjusted using reward prioritization, and the proportion of reward prioritization can be justified by the domain knowledge expert.

The detailed feature importance is shown in Figure 5 and Figure 6 without comparing the importance between the input features and the reward components. Figure 7 shows the importance of them in the same plot so that they can be compared which is also known as a normalized explanation ($L_{ic}$). We can see from Figure 7a that the RSRP input feature and the reward component become the top-two most important element in this agent. When reward weights are applied, their importances are pushed further, as shown in Figure 7b.

In a local scope (i.e., explanation of a single action), the detailed information produced by our method exposes the contribution that has not been seen in existing methods. For example, information about negative contribution of RSRP and SINR input features to the throughput reward component cannot be inferred from feature importance or reward decomposition methods alone. This information is important, especially when several input features have opposing contributions. With an existing method, a feature may not be described as important because the attribution value is small or zero. However, it may actually contribute to different components in an opposing way (e.g., high positive contribution to reward component 1 and high negative contribution to reward component 2), as shown by SINR input feature importances on Figure 8 (top plots, second row). By using our method, hidden information, as mentioned, can be revealed and the user may have a better understanding of the performing agent.

4.4. Analysis of Shapley-Based Explainer

If we sum the attribution values of input features towards the RSRP reward component on Figure 8 (bottom-right), we will obtain a negative value of the RSRP reward component. However, the predicted Q-value of this component is positive, as shown as a black bar on Figure 9b. When the explainer is built on the Shapley-concept (e.g., SHAP and its variants), the sum of attribution (i.e., feature importance) values equals to the output (or prediction) of the performing model. This value includes the baseline value, which is the average model output of the given dataset, where we present it as the purple bar in Figure 9b. In this manner, we have a complete explanation in which the predicted Q-value (black bar) is equal to the sum of all input features contributions including the base value of the reward component.

In an implementation without decomposition, the baseline values are different for possible actions (or classes in a supervised learning problem). Thus, we can consider this baseline value as the bias towards a different action. In a model with reward decompositions, the difference in the base value appears not only for different actions but also for reward components, as shown by

Table 1. Base value for each action and reward component at the RL level.

Action	RSRP	SINR	Throughput
Tilt-down	12.522070	8.453173	17.904276
No-tilt	12.652522	9.103990	18.448515
Tilt-up	12.470411	8.746243	18.138926
Mean	12.548335	8.767802	18.163904
Range	0.182111	0.650817	0.544239

. Although these values are constant for any instance/state/observation, they play an important role in selecting the action. The differences in the base value among different actions are not as large as the difference between different reward components, as shown by the range on Table 1 (the range is the subtraction of the maximum by the minimum value of each reward component). However, the mean values show the average base value of each reward component, where the SINR has the lowest average base value and the throughput component has the highest value.

When analyzing a local explanation, it is important that the explanation be accompanied by the base value as a reference to show all contributions leading to the predicted Q-value. In this manner, the explanation shows how each input feature impacts the model’s output from the average situation. Although understanding why a specific base value is obtained remains limited to the fact that it is the average output value of the model from the given dataset, exposing it informs the complete analysis of the model. The dataset becomes the scope or corridor for generating the explanation. In connection with the normalized explanations, in Figure 7b, the information shows the importance of different model elements in terms of the variation of input values regardless of the bias of the reward component. In connection with the RL level explanation Figure 6b, the bias of the reward component may be overcome by the contribution made by the input feature(s). Specifically, for the experiment above, the mean bias (base value) for the throughput reward component (18.164) is higher than the RSRP (12.548). However, the mean absolute contribution of the RSRP input feature to the RSRP reward component is 15.501 where the sum of it and its bias (28.049) can overcome the throughput bias. This explanation shows that the model has bias towards different components, and the variation of the input values may overcome it.

4.5. Contrastive Explanations with BEERL

In the exploitation phase, an RL agent chooses an action that maximizes the cumulative reward, and understanding why it is preferred over the other is important. When contrasting the possible actions (e.g., why no tilt is preferred than tilting up), the generated explanations from our method present detailed information as shown in Figure 10 where they consist of input feature and reward component tuples. Figure 10a shows the explanation of why no tilt is preferred than tilting down where the tilt input feature contributes negatively to all reward components. On the other hand, the sum of positive contributions is greater than the disadvantage of not tilting up. When the $MS{X}^{+}$ concept is applied, simpler explanations are produced, such as the following: the contribution of ‘RSRP[I] to SINR[R]’ and ‘base value[I] to SINR[R]’ ([I] and [R] indicate input feature and reward component, respectively) are enough to overcome the disadvantage of not selecting action tilting down (shown as textured bar in Figure 10). Since both reasons contribute to the SINR reward component, the Algorithm 1 further compresses it by aggregating the contribution of each reward component($MS{X}^C$). Thus, the compressed explanation can be interpreted as ‘The contribution of SINR reward component is enough to overcome the negative impact of not choosing tilting down action’.

Figure 10b shows another constrastive explanation of why no tilt is preferred than tilting up. The $MS{X}^{+}$ of these explanations informs that the contribution of ‘Tilt[I] to AvgSINR[R]’, ‘Tilt[I] to AvgRSRP[R]’, ‘base val[I] to AvgSINR[R]’, and ‘base val[I] to AvgThroughput[R]’ are enought to overcome the negative contribution of not tilting up. Furthermore, they are compressed by aggregating the contribution of the input feature resulting in an explanation that can be interpreted as ‘The contribution of Tilt input feature and base values are enough to overcome the negative impact of not choosing tilting up action’. It should be noted that the base values in Figure 10 are the difference among the actions that the reward weight adjustment does not address. Adjusting the reward weight cannot change the proportion of it within each reward component (i.e., making it less dominant than the input features).

4.6. Usages with BEERL

By presenting the explanations as above, we can gain more insight about the trained agent than the available methods by noting some findings, such as the following:

• The RSRP input feature is the most contributing feature and significantly affects the RSRP reward component. This explanation shows in detail the contribution of each input feature to every output component and how they are correlated, as shown in Figure 5a and Figure 6a.
• SINR and throughput reward components are correctly focused on SINR and throughput input features, respectively. It shows that the Q-function performs well in focusing on the respective feature.
• The SINR and throughput reward components have less contribution than RSRP reward component even on the Q-function level. Applying the reward weights pushes the contribution further, where RSRP has a significantly larger contribution than other components.
• The base values show the bias of different actions and reward components. Even though the RSRP reward component is weighted twice that of others, the throughput reward component remains as the highest. This means that a higher contribution from the input feature(s) is required if other component(s) need to be overruled in selecting the action.
• If there is a situation where the priority of the reward components needs to be adjusted, one can reuse the trained model and then only adjust the reward weight and continue the training without retraining it from scratch with a random policy. With the given explanations, we are confident that the model is performing well because each reward component focuses on the correct features. This information presents the correctness of the proposed method, which not only increases human trust in the AI used but also allows for the transfer of knowledge with stronger reason.

From the above points, the explanations improve human knowledge to understand the behavior of the model in different elements of the RL model. The effect of the reward weight, the bias of actions and the reward component, and the contribution of the input feature and the reward components are produced to inform the detailed properties of the model’s behavior. An indication of a better understanding is that the agent can be adjusted or reconfigured to fit the necessary goal. In the following, we show the adjustment of the reward weights to reduce the bias (difference of the base value) among different reward components.

As the mean base values at the Q-function level are 6.274168, 8.767802, and 18.163904 for the RSRP, SINR, and throughput reward component, respectively, we adjusted the reward weight to achieve similar values. We choose an arbitrary value (in this case, it is 10) that we divide by the mean values of the base values of the reward components. Therefore, to keep two decimal values, we set the new reward weight at 1.60, 1.14, and 0.55 for RSRP, SINR, and throughput, respectively. Subsequently, we conducted an evaluation process and obtained the explanations, including bias, as shown in Figure 11 and

Table 2. Base value for each action and reward component at the RL level after the reward weights are adjusted to reduce the bias of the reward components.

Action	RSRP	SINR	Throughput
Tilt-down	9.273874	9.247864	9.758694
No-tilt	9.375855	9.965568	10.043088
Tilt-up	9.231198	9.528765	9.867805
Mean	9.293643	9.580732	9.889862
Range	0.144657	0.717704	0.284394

.

As the reward adjustment is made relative to the trained agent, the explanations on the Q-function level are similar to Figure 5. From Figure 11, we can see that the throughput input feature has a relatively similar contribution to the SINR and throughput reward components. The importance of the characteristics in each reward component is also changed following the updated reward weights, as shown in Figure 11b. Table 2 shows the base values after the reward weights were adjusted, resulting in a reduction in the bias of the reward components. This weight adjustment exemplifies the possibility of reconfiguring the RL agent after training to achieve the desired properties.

Furthermore, we evaluated antenna telecommunication metrics (RSRP, SINR, and throughput) before and after adjusting the agent reward weights as shown in Figure 12. We can see that the agent with the adjusted reward weights has a better RSRP KPI than the original agent, while SINR and throughput KPIs are slightly compromised. Since the range of the base values of the adjusted agent is low, the action selection can depend more on the contribution of input features to the reward components and less on the base values of the reward components.

4.7. Focus Value: Quantifying the Desired Propreties

To measure the fulfilment of the generated explanation with the desired properties as calculated by the focus value formula Equation (8), the user must set a relevance table ($M_{ic}$). In this experiment, the configured relevance table is presented in

Table 3. Relevance table configured by the user quantifying the desired properties.

Input	Reward Component
Feature	RSRP	SINR	Throughput
Tilt	1.0	1.0	1.0
RSRP	1.0	0.5	0.5
SINR	0.5	1.0	0.5
Throughput	0.5	0.5	1.0

with the rationale explained in the following sentences. The RSRP, SINR, and throughput reward component are highly correlated with the RSRP, SINR, and throughput input features, respectively. The tilt input feature also has a high correlation because it directly affects the antenna performance in all aspects. However, RSRP, SINR, and throughput are also correlated with each other in a moderate manner. Therefore, we set the correlation value at 0.5 since we want each reward component to focus on its most correlated input feature. The relevance table can be set by the user as she/he formulates the desired properties, and the rationale of the given example (Table 3) is explained above.

The generated explanations (Figure 5 and Figure 11) are evaluated with the relevance table (Table 3) to generate the focus values shown in

Table 4. Focus values of the agent with original and adjusted reward weight, which quantifies how much the desired properties are fulfilled.

Focus	Reward Component			Weighted	Unweighted
Value	RSRP	SINR	Throughput	Mean	Mean
Original weight	0.942084	0.824311	0.884382	0.898215	0.883592
Adjusted weight	0.929772	0.823575	0.878465	0.884397	0.877271

. The focus values for both experiments are relatively similar because adjusting the weight does not affect the proportion of the input features contribution to each reward component. The RSRP reward component has the highest focus value because the RSRP input feature has a significant contribution compared to other features. On the other hand, the SINR reward component has the lowest focus value because it has significant contributions from the tilt and throughput input features.

The weighted mean has a higher value than the unweighted mean focus value, which means that reward prioritization mitigates unwanted behavior because components with high focus values have a higher priority. On the other hand, the unweighted focus value is lower, indicating that it has less desired behavior if we treat all components equally (i.e., do not apply prioritization).

5. Conclusions

This paper presents a novel “Both Ends Explanations for RL (BEERL)” method that generates fine-grained explanations, which connects and compares the input and output explanations of a black-box model. In this manner, the contribution of every feature to every reward component is revealed explicitly, allowing each element to be evaluated individually. The explanations are generated at two different levels as a result of exposing reward priority. The first is at the Q-level, where all components generate values in the same range to minimize the risk of vanishing or exploding gradients, as exemplified in Appendix D.2. The second is at the RL-level where the reward priority is applied to explain the rationale behind choosing an action. Our method also produces a detailed contrastive explanation that explains why one action is chosen instead of another. Additionally, the contrastive explanation can be compressed to highlight the important element(s) that make it more comprehensible to humans. To identify the most important factor of the agent, the input feature and the reward component are compared by aggregating their contributions. The implemented agent can be modified by adjusting the reward priority to fulfill the desired properties, such as reducing the bias of the reward components. A further adjustment of the RL agent can also be performed, such as retraining the Q-function partially per component, removing non-contributing input feature(s), removing misbehaving or suboptimal reward component(s), and tuning reward prioritization without restarting the training process, etc.

We implemented our method with respect to the remote electrical antenna tilt use case (as described in Section 4) and to two openAI gym environments (see Appendix D) to demonstrate the benefits of resultant explanations. A fine-grained explanation implies that the amount of information received by the user increases. Using the relevance table set by the user, the focus value summarizes the fulfillment of the desired properties. In the future, we intend to investigate a method to improve the focus value of the RL agent. This can be performed by incorporating the explanation during the training process. More experiments can be conducted on top of this work to mitigate bias between different actions.

6. Patents

A patent with the title “Explaining Operation of a Neural Network” has been filed as a US provisional application on 11 January 2022.