Modeling Theory of Mind in Dyadic Games Using Adaptive Feedback Control

Abstract

A major challenge in cognitive science and AI has been to understand how intelligent autonomous agents might acquire and predict the behavioral and mental states of other agents in the course of complex social interactions. How does such an agent model the goals, beliefs, and actions of other agents it interacts with? What are the computational principles to model a Theory of Mind (ToM)? Deep learning approaches to address these questions fall short of a better understanding of the problem. In part, this is due to the black-box nature of deep networks, wherein computational mechanisms of ToM are not readily revealed. Here, we consider alternative hypotheses seeking to model how the brain might realize a ToM. In particular, we propose embodied and situated agent models based on distributed adaptive control theory to predict the actions of other agents in five different game-theoretic tasks (Harmony Game, Hawk-Dove, Stag Hunt, Prisoner’s Dilemma, and Battle of the Exes). Our multi-layer control models implement top-down predictions from adaptive to reactive layers of control and bottom-up error feedback from reactive to adaptive layers. We test cooperative and competitive strategies among seven different agent models (cooperative, greedy, tit-for-tat, reinforcement-based, rational, predictive, and internal agents). We show that, compared to pure reinforcement-based strategies, probabilistic learning agents modeled on rational, predictive, and internal phenotypes perform better in game-theoretic metrics across tasks. The outlined autonomous multi-agent models might capture systems-level processes underlying a ToM and suggest architectural principles of ToM from a control-theoretic perspective.

1. Introduction

How do autonomous social agents model the goals, beliefs, and actions of other agents they interact with in complex social environments? This has long been a central question in cognitive science and philosophy of mind. It forms the basis of what is known as Theory of Mind (ToM), a long-standing problem in cognitive science concerned with how cognitive agents form predictions of mental and behavioral states of other agents in the course of social interactions [1,2]. The precise mechanisms by which the brain achieves this capability are not entirely understood. Furthermore, how a ToM can be embodied in artificial agents is very relevant for the future course of artificial intelligence (AI), especially with respect to human-machine interactions. What can be said concerning architectural and computational principles necessary for a Theory of Mind? As a small step in that direction, we propose and validate control-based cognitive architectures to predict actions and models of other agents in five different game theoretic tasks.

It is known that humans also use their ToM to attribute mental states, beliefs, and intentions to inanimate objects [3], suggesting that the same mechanisms governing ToM may be integral to other aspects of human cognition. We argue that in order to understand both how ToM is implemented in biological brains and how it may be modeled in artificial agents, one must first identify its architectural principles and constraints from biology.

Several notable approaches addressing various aspects of this problem already exist, particularly those based on artificial neural networks [4,5,6]. Recent work in [7] on Machine Theory of Mind is one such example. This work proposes that some degree of theory of mind can be autonomously obtained from Deep Reinforcement Learning (DRL). This deep learning approach can be made symmetric by allowing the agents to observe each other [8] and training them to develop a mutual Theory of Mind. However, the use of Black-Box Optimization (BBO) algorithms in DRL hinders the understanding one can obtain at the mechanistic level. Because BBO algorithms are able to approximate any complex function (as suggested for Long-Short Term Memory (LSTM)-based neural networks [9]), one cannot use that to decipher specific mechanisms that may underlie a ToM in these systems.

Another interesting approach to modeling ToM follows the work of [10,11,12,13,14] on Bayesian Theory of Mind. These methods are cognitively-inspired and suggest the existence of a “psychology engine” in cognitive agents to process ToM computations. Nevertheless, the challenge for this approach remains in explaining how the computational cost of running these models might be implemented in biological substrates of embodied and situated agents.

ToM can also be modeled as inverse reinforcement learning (IRL), a popular machine learning method designed to operate within the Markov Decision Process (MDP) framework [15]. This approach, known as ‘ToM as Inverse Reinforcement Learning’, aims at predicting other agents’ actions by simulating an RL model with the hypothesized agent’s beliefs and desires, while mental-state inference is achieved by inverting this model [16]. This approach to ToM has seen a resurgence of interest in the machine learning community [17,18]. However, one of the key challenges for IRL-based ToM models is to improve their ability to handle vast and complex state spaces, a requirement for real-world success [18]. Moreover, while most IRL approaches assume a simplified form of human rationality, it is important to note that humans often deviate from these assumptions [19,20].

A third approach to this problem follows from work on autonomous multi-agent models (see [21] for a recent survey). This approach has its roots in statistical machine learning theory and robotics. It includes agent models capable of policy reconstruction, type-classification, planning, recursive reasoning, and group modeling. It focuses on multi-agent communication and cooperation tasks where the ToM agent not only observes but also acts in the environment [22,23]. In a sense, this is closest to the approach we take in this work, even though our models are grounded in cognitive control systems. Open challenges in the field of autonomous multi-agent systems include modeling fully embodied agents that operate with only partial observability of their environment and are flexibly able to learn across tasks (including meta-learning) when interacting with multiple types of other agents.

The cognitive agent models we propose here advance earlier work on Control-based Reinforcement Learning (CRL) [24], where we studied the formation of social conventions in the Battle of the Exes game. The CRL model implements a feedback control loop handling the agent’s reactive behaviors (pre-wired reflexes), along with an adaptive layer that uses reinforcement learning to maximize long-term reward. We showed that this model was able to simulate human behavioral data in the above-mentioned coordination game.

The main contribution of the current paper is to advance real-time learning and control strategies to model agents with specific phenotypes that can learn to either model or predict the opposing agents’ actions. Using top-down predictions from the adaptive to reactive control layers and bottom-up error feedback from reactive to adaptive layers, we test cooperative and competitive strategies for seven different multi-agent behavioral models. The purpose of this study is to understand how the architectural assumptions behind each of these models impact agent performance across standard game-theoretic tasks and what this implies for the development of autonomous intelligent systems.

2. Methods

2.1. Game Theoretic Tasks

Benchmarks inspired by game theory have become standard in the multi-agent reinforcement learning literature [4,5,7,25]. However, most of the work developed in this direction presents models that are tested in one single task or environment [24,26,27,28,29,30,31], at best two [32,33]. This raises a fundamental question about how these models generalize to deal with a more general or diverse set of problems. So far, this approach does not readily enable one to extract principles and mechanisms or unravel the dynamics underlying human cooperation and social decision-making.

In this work, we want to go a step further and propose a five-task benchmark for predictive models based on classic normal-form games extracted from game theory: the Prisoner’s Dilemma, the Harmony Game, the Hawk-Dove, the Stag Hunt, and the Battle of the Exes.

In their normal form, games are depicted in a matrix that contains all the possible combinations of actions that players can choose, along with their respective rewards (Table 1). The most common among these games are dyadic games, known as social dilemmas, in which players have to choose between two actions: cooperate or defect. One action (cooperate) is more generous and renders a good amount of reward to each player if both choose it, but it gives very poor results if the other player decides to defect. On the other hand, the second action (defect) provides a significant individual reward if it is taken alone, but a very small one if both choose it.

Although simple in nature, these dyadic games are able to model key elements of social interaction, such as the tension between the benefit of cooperative action and the risk (or temptation) of free-riding. This feature can be described in a general form, such as:

Table 1. A social dilemma in matrix-form.

	Cooperate	Defect
Cooperate	R, R	S, T
Defect	T, S	P, P

Table 1. A social dilemma in matrix-form.

	Cooperate	Defect
Cooperate	R, R	S, T
Defect	T, S	P, P

This matrix represents the outcomes of all possible combinations of actions between the row player and the column player. R stands for “reward for mutual cooperation”; T for “temptation of defecting”; S for the “sucker’s payoff for non-reciprocated cooperation”; and P for “punishment for mutual defection”. By manipulating the relationship between the values of this matrix, many different situations can be obtained that vary in terms of what would be an optimal solution (or Nash equilibria [34]).

The Prisoner’s Dilemma (

Table 2. Prisoner’s Dilemma.

	Cooperate	Defect
Cooperate	2, 2	0, 3
Defect	3, 0	1, 1

) represents the grim situation in which the temptation of defecting (T) is more rewarding than mutual cooperation (R), and the punishment for mutual defection (P) is still more beneficial than a failed attempt of cooperation (S). This relationship among the possible outcomes can be stated as $\mathrm{T}>\mathrm{R}>\mathrm{P}>\mathrm{S}$. Mutual defection is the only pure Nash equilibrium in this game since there is no possibility for any player to be better off by individually changing its own strategy.

On the Stag Hunt (

Table 3. Stag Hunt.

	Cooperate	Defect
Cooperate	3, 3	0, 2
Defect	2, 0	1, 1

), we face a context in which mutual cooperation (R) gives better results than individual defection (T), but at the same time, a failed cooperation (S) is worse than the punishment for mutual defection (P). In this case, $\mathrm{R}>\mathrm{T}>\mathrm{P}>\mathrm{S}$, there are two Nash equilibria: mutual cooperation and mutual defection.

The Hawk-Dove game (

Table 4. Hawk-Dove.

	Cooperate	Defect
Cooperate	2, 2	1, 3
Defect	3, 1	0, 0

) presents a scenario in which temptation (T) is more rewarding than cooperation (R), but a mutual defection (P) is less desirable than non-reciprocated cooperation (S). So, for a relationship of $\mathrm{T}>\mathrm{R}>\mathrm{S}>\mathrm{P}$ like this one, there are three Nash equilibria: two pure anti-coordination equilibria, in which each player always chooses the opposite action of their opponent, and one mixed equilibrium, in which each player probabilistically chooses between the two pure strategies.

The fourth type of social dilemma is the Harmony Game (

Table 5. Harmony Game.

	Cooperate	Defect
Cooperate	3, 3	1, 2
Defect	2, 1	0, 0

). In this case ($\mathrm{R}>\mathrm{T}>\mathrm{S}>\mathrm{P}$), the game has only one pure equilibrium, pure cooperation, since mutual cooperation (R) renders better outcomes than the temptation to defect (T), and also the penalty for failing to cooperate (S) is less than the punishment for mutual defection (P).

Finally, as the last game, we will introduce a coordination game, the Battle of the Exes [35,36]. In this type of game, the main goal is to achieve coordination between two players (either congruent coordination on the same action or incongruent coordination upon choosing different actions) since the failure to do so is heavily penalized (see

Table 6. Battle of the Exes.

	A	B
A	0, 0	1, 4
B	4, 1	0, 0

). Following the previous nomenclature applied in the social dilemmas, we could define this game as $\mathrm{T}>\mathrm{S}$; $\mathrm{R},\mathrm{P}=0$. This game has two pure-dominance equilibria, in which one player chooses the more rewarding action and the other the low rewarding action, and a turn-taking equilibrium, in which players alternate over time in choosing the more rewarding action.

This selection of games provides enough variability to test how learning agents can perform across different contexts, so we avoid problems derived from over-fitting on a specific payoff distribution or related to the possibility of a model to exploit/capitalize on certain features of a game that could not have been predicted beforehand. Moreover, a similar selection of games has been tested in human experiments, proving to be sufficient to classify a reduced set of behavioral phenotypes across games in human players [37,38]. Using the above games as benchmarks, below we describe the control-theoretic framework and seven agent models that capture phenotypes relevant for testing ToM capabilities across games.

2.2. Control-Based Reinforcement Learning

Our starting point is the Control-based Reinforcement Learning (CRL) model presented in previous work [24]. The CRL is a biologically grounded cognitive model composed of two control layers (Reactive and Adaptive, see Figure 1), and based on the principles of the Distributed Adaptive Control (DAC) theory [39]. The Reactive layer represents the agent’s pre-wired reflexes and serves for real-time control of sensorimotor contingencies. The Adaptive layer endows the agent with learning abilities that maximize long-term reward by choosing which action to perform in each round of the game. This layered structure allows for top-down and bottom-up interactions between the two layers [40], resulting in an optimal control at different time-scales: within each round of play and across rounds [24].

The Reactive layer is inspired by Valentino Braitenberg’s Vehicles [41] and presents the intrinsic mechanic behaviors that are caused by a direct mapping between the sensors and the motors of the agent. The two reactive behaviors are based on a cross excitatory connection and a direct inhibitory connection between the reward sensors and the two motors. This results in an approaching behavior in which the agents will turn towards a reward location increasingly faster the closer they detect it. Since the agents are equipped with two sets of sensors specifically tuned for detecting each reward location, they also have two instances of this reactive behavior, one for the ’cooperative’ and one for the ‘individual’ reward (see Figure 1, green box).

The Adaptive layer is in charge of prediction, learning, and decision-making. Each of the agent models described in the next section instantiates a different Adaptive layer. Its main role is to learn from previous experience and to decide at the beginning of each round which action the agent will take. When playing normal-form games, the Adaptive layer receives as inputs the state of the environment, which corresponds with the outcome of the last round of the game, and the reward obtained because of that final state. This information is used to produce an action as the output. For all experiments conducted in this paper, the possible states s can be either ‘R’, ‘S’, ‘T’ or ‘P’ (see Table 1), and determines what reward the agent will obtain based on the specific payoff matrix of the game being played, as specified in Table 1, Table 2, Table 3, Table 4, Table 5 and Table 6. The possible actions a are two: “cooperate” or “defect”.

Depending on the action that the Adaptive layer selects, there is a top-down selective inhibition that affects the reactive behaviors of the Reactive layer. If the action selected is“cooperate”, the “approach defect location” reactive behavior is inhibited, thus focusing the agent’s attention only on the cooperative reward and on the other agent. Conversely, if the action selected is “defect”, the reactive behavior inhibited will be “approach cooperate location”. This mechanism aims to mimic how biological systems execute top-down control over a hierarchy of different control structures [42,43,44].

The error monitoring function works in real-time as a bottom-up mechanism that signals an error function from the Reactive layer sensors to the Adaptive layer decision-making module. The error signal is only triggered when the agent detects an inconsistency between its initial prediction of the opponent’s action and the real-time data obtained by its sensors. If a prediction error occurs, the error monitoring function will update the current state of the agent, and this will make the decision-making module output a new action. Along with the top-down inhibitory control, this module is inspired by evidence from cognitive science about the role of bottom-up sensory stimuli in generating prediction errors [42,45,46].

2.3. Agent Models

In this section, we describe the four learning models that we will compare in this study. All of them use the classic TD-learning algorithm as the fundamental component of their cognitive architectures. The different combinations of this element with other functions allow these models to learn to predict their opponent’s actions and adapt its behavior accordingly.

2.3.1. TD-Learning Model

The TD-learning model (see Figure 2) uses an Actor-Critic version of the Temporal-Difference (TD) learning algorithm [47] for maximizing long-term rewards (see [24] for a detailed description of the implementation).

In brief, for each observed state $s\in S$, the TD-learning algorithm selects an action $a\in A$ according to a given policy $P(a=a_t|s=s_{t-1})$. Once a round of play is finished, the reward $r\left(s_t\right)$ obtained by the agent will update the TD-error e signal, following: $e\left(s_t\right):=r\left(s_t\right)+\gamma V_{\mathsf{\Pi }}\left(s_t\right)-V_{\mathsf{\Pi }}\left(s_{t-1}\right)$ where $\gamma$ is a discount factor set to $0.99$. and $V_{\pi }\left(s_t\right)=\gamma r\left(s_t\right)$ is the Critic. Finally, the policy (or Actor) will be updated following $\mathsf{\Pi }(a_t,s_{t-1}):=\mathsf{\Pi }(a_t,s_{t-1})+\delta e\left(s_{t-1}\right)$, where $\delta$ is a learning rate that is set to $0.15$. The experiments conducted in this paper use the same parameter values as [24], where this parameter configuration showed the best fit to human behavioral data when playing the Battle of the Exes under similar experimental conditions.

2.3.2. Rational Model

The Rational model is a predictive model that represents an ideal perfectly-rational and self-interested player provided that its predictions are correct. Its function is to serve as a benchmark for the other predictive models since once it learns to predict its opponent’s actions accurately; it will always respond automatically with the optimal response to that predicted action. It is composed of two main functions: a prediction module and a deterministic utility maximization function (see Figure 3). The first module learns to predict the next action of the opponent using the opponent’s previous state as input. Once the prediction is made, the second module calculates the action that will render the highest reward assuming that the opponent has chosen the predicted action.

The predictive module also uses a TD-learning algorithm but substitutes the explicit reward for an implicit reward that comes from the error in predicting the other agent’s action. We call this reward ‘implicit’ because it is calculated internally and is based on whether the action of the opponent was accurately predicted or not. That way, an accurate prediction will render a positive reward, while an incorrect one will receive a negative one.

The implicit reward signal is calculated by the function:

$$R_i=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{a}_o\left(t\right)={\widehat{a}}_o(t-1)\hfill \\ -1,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(1)

2.3.3. Predictive Model

The Predictive model has two distinct RL modules: one that learns to predict and one that chooses an action based on that prediction (see Figure 4). The predictive RL algorithm is identical to the one used by the Rational agent described above, so it also learns from the implicit reward described in Equation (1).

The prediction generated at the beginning of each round by the predictive-RL module is sent as an input to the second RL algorithm in charge of decision-making. The decision-making RL module uses the combined information of the opponent’s predicted action and the state of the previous round to learn the optimal action. For its update function, it uses the explicit reward obtained in that round of the game.

2.3.4. Internal Model

The Internal Model is also composed by RL algorithms, one predictive and one for learning the optimal policy (see Figure 5). In terms of function, the decision-making RL module is identical to the one used by the Predictive model: it integrates both the previous state of the game and the opponent’s predicted action in order to learn the optimal policy.

However, it differs from the Predictive model in the way its first module is designed. In this case, the predictive−RL algorithm is updated by the explicit reward that the opponent has obtained in an attempt to create an internal model of the other agent’s policy. At the functional level, it is very similar to the Predictive model, but with an important difference: while the Predictive model tries to predict the opponent’s action by focusing on its overt behavior, the Internal Model does it trying to learn the internal policy of its opponent.

2.3.5. Deterministic Agent Models

In order to test the correct functioning of the predictive capacities of the ToM agents described above, we have implemented a number of agents with a fixed behavior or policy. The first two represent two behavioral phenotypes observed in humans [37] while the third one is a classic benchmark in game theory [48].

Greedy

This model implements a simple behavioral strategy of pure self-utility maximization: it always chooses the action that renders the highest reward to itself without taking into account the opponent’s action. So for the games described in this work, it will always choose to ‘defect’ with the exception of the Stag Hunt and the Harmony Game, where the cooperative reward can give a higher reward than the temptation to defect ($\mathrm{R}>\mathrm{T}$).

$${\pi }_{greedy}=\left\{\begin{array}{cc}cooperate,\hfill & \mathrm{if}\mathrm{R}>\mathrm{T}\hfill \\ defect,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(2)

Cooperative/Nice

As a deterministic counterpart of the Greedy model, the Nice model executes an even simpler deterministic strategy: it will always choose cooperation.

$${\pi }_{nice}=P(a=\mathrm{cooperate}|s_t)=1$$

(3)

Tit-for-Tat

This simple yet powerful strategy became popular in the famous Axelrod’s tournament [49] where it won against all competing algorithms and strategies in a contest based on the Iterated Prisoner’s Dilemma. We have introduced this opponent because we expect that this agent will give problems to the predictive models. Tit-for-tat always starts by cooperating, and from then on, it always chooses the last action made by its opponent. Since its capacity to switch actions does not depend on a specific policy, it cannot be learned by just taking into account variables such as its previous state or its reward. It can be described by the following equation, where $a_{\mathrm{opponent}}$ is the action made by the opponent at $t-1$:

$${\pi }_{tft}=\left\{\begin{array}{cc}cooperate,\hfill & \mathrm{if}t=0\hfill \\ a_{\mathrm{opponent}}(t-1),\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(4)

2.4. Experimental Setup

First, we perform four different experiments in which we test the four learning models described above (TD-learning, Rational, Predictive, Internal) in each of the five games described in Section 2.1 (Prisoner’s Dilemma, Hawk-Dove, Stag Hunt, Harmony Game, and Battle of the Exes) and against opponents of different levels of complexity (Greedy, Nice, Tit-for-tat, and TD-learning), resulting in a 4 × 5 × 4 experimental setup. This way, in experiment one, all models are tested against the Greedy agent; in experiment two, against the Nice agent; in experiment three, against the Tit-for-tat agent; and in experiment four, against the TD-learning agent.

In all experiments, each model plays 50 times each of the five games during a total amount of 1000 rounds per iteration (i.e., 5 games, 50 dyads per game, 1000 rounds per dyad). On each iteration, every model starts learning from scratch, with no previous training period. After that, we perform a fifth experiment to study a real-time spatial version of the games’ special cases in which the models encountered problems in the previous setting. So, in the first four experiments, we use the discrete-time version of the games, and for the fifth experiment, a continuous-time version.

In the discrete-time version studied in Experiments 1 to 4, both agents simultaneously choose an action at the beginning of each round. Immediately after that, the outcome of the round is calculated, and each agent receives a reward equal to the value stated in the payoff matrix of the game.

In the continuous-time version studied in Experiment 5, we implement the games in a 2D simulated robotic environment (see Figure 6). The two agents can choose to go to one of the two equally distant locations that represent the two actions of a 2-action matrix-form game. In this version, agents also choose an action at the beginning of the round, but then they have to navigate towards the selected location until one of them reaches its destination. The real-time dimension of this implementation allows the possibility of changing the action during the course of the round as well as receiving feedback in real-time. Each round of the game begins with both agents in their initial positions, as shown in Figure 6B. When the agents reach a reward location, the round ends, the rewards are distributed accordingly to the payoff matrix, and the game restarts with the agents back in their starting positions. The agents of the continuous version are embodied in virtual ePuck robots (see Figure 6A). They are equipped with two motors (for the left and right wheels) and three pairs of proximity sensors; one pair is specialized in detecting the other agent, and the other two in detecting the two distinct reward locations (‘cooperate’ or ‘defect’).

3. Results

In order to study the performance of the four agents (TD-learning, Rational, Predictive, and Internal), we focus on three aspects of the interaction: Efficacy, Stability and Prediction Accuracy. Efficacy tells us how competent the models were in obtaining rewards on average and in relation to each other. It is computed by averaging the total score obtained per game by each agent (i.e., compute the accumulated score over 1000 rounds of game for each agent, averaged by the 50 agent models of the same kind). Stability measures how predictable the behavior was or, equivalently, whether the models converged to a common strategy or alternated between non-deterministic states. In other words, Stability quantifies how predictable are the outcomes of the following rounds based on previous results by using the information-theoretic measure of surprisal (also known as self-information), which can be defined as the negative logarithm of the probability of an event [50]. By computing the average surprisal value of a model over time, we can visually observe how much did it take for a model to converge to a stable strategy and for how long it was able to maintain it for more details on the computation of surprisal see [24,35]. Finally, Prediction Accuracy gives us a better understanding of how well the predictive models were able to actually predict their opponent’s behavior. It is measured by computing for each agent the accumulated prediction errors (i.e., the number of times the agent fails to predict the opponent’s action) across the 1000 rounds of a game and then calculating the running average over the total sample (n = 50) of each predictive model.

In the context of each conducted experiment, the findings are presented in their corresponding figures, wherein the above three key metrics are examined: Efficacy in the left panels, Stability in the center panels, and Prediction Accuracy in the right panels. The reported results show the mean values obtained from a sample size of n = 50 for each game and condition.

3.1. Experiment 1: Versus a Deterministic-Greedy Agent

The results of this experiment show that, on average, all models were equally effective since they obtained a similar amount of rewards (see Figure 7). The Rational model only achieves a slightly better performance than the rest in the Stag Hunt and the Harmony Game, so this is a sign that the rest of the models were performing almost optimally. Of course, if we pay attention to the overall score obtained among the five games, we can observe a significant difference between the performance obtained by all models in the Harmony Game and the Stag Hunt when compared to the other three games. This salient difference is not due to a malfunctioning of the predictive models, as we can see by the results in Stability and Prediction Accuracy. It is caused by the constraints imposed by the Greedy opponent strategy in those games were $\mathrm{R}>\mathrm{T}$, since they are the ones in which it always chooses to ‘cooperate’. We see that all models learn to predict or to adapt their strategy to this opponent. In terms of Stability, we can observe how Rational Agents outperform all the other models. This result is not surprising, given the simplicity of the opponent’s greedy strategy. Against these simple, predictable agents, we consider the Rational agent a reference point of optimal performance, against which the other models can be compared. In this line, it is relevant to note that overall, both Predictive and Internal models converge to a stable state faster than the vanilla TD-learning model, who lacks the ability to predict the opponent’s action. In terms of Prediction Accuracy, no significant difference is observed among predictive models.

3.2. Experiment 2: Versus a Deterministic-Nice Agent

In this experiment, the overall Efficacy of all models has increased significantly compared to that of the previous experiment (see Figure 8). The explanation was again that all models were able to successfully learn the optimal policy against a simple deterministic strategy such as the one exhibited by the Nice agent. However, in this case, the best response strategy against a Nice agent is the one that renders the highest possible benefit in all games, as opposed to the best response strategy against the Greedy model, that on many occasions, was sub-optimal. In terms of Stability, we see again how the Predictive and the Internal agents perform slightly better than the baseline TD-learning. This time the difference is smaller, but since predicting the opponent’s behavior, in this case, is almost trivial, we do not find this result unexpected. Again we can also observe how the TD-learning, the Predictive, and the Internal agents differ from optimal by comparing their convergence rate with that of the Rational agent.

3.3. Experiment 3: Versus a Tit-for-Tat Agent

The results of the models against the Tit-for-tat agent show a significant amount of variability in terms of Efficacy (see Figure 9). The most remarkable result in this aspect is the superior Efficacy of the non-predictive TD-learning model in the Battle of the Exes. This model is able to beat the more complex predictive models in this case precisely because the predictive models encounter more problems in accurately predicting the TFT agent. This accumulated error in prediction, in turn, drives their behavior toward a more unstable regime than the one achieved by the TD-learning model. Moreover, given that this model has a more reduced state-space than the Predictive models and it only ’cares’ about the previous state of the game, it capitalizes better than the rest on the anti-coordination structure of the Battle of the Exes.

In terms of Stability, predictive models perform better than standard TD-learning in the Prisoners Dilemma and in the Harmony Game, but this result is reversed in the Stag Hunt and the Hawk-Dove and the Battle of the Exes, where the TD-learning agent reaches lower levels of surprisal, therefore indicating higher Stability. We can see how in the cases that the game has one pure Nash equilibrium (the Prisoner’s Dilemma and the Harmony Game), the predictive models perform as well in terms of Stability as against the simpler deterministic agents studied before. However, in games with a wider variety of equilibria, such as the Stag Hunt or the Hawk-Dove, predictive models have overall more difficulties in falling into a stable state with the TFT opponent.

Finally, the most salient feature regarding the results of Prediction Accuracy is the elevated accumulation of errors of the Rational model in the Stag Hunt and the Battle of the Exes, which almost reached 50%. These results may seem counter-intuitive, but actually, they highlight one of the weak points of the predictive models and the strength of the tit-for-tat strategy. As stated in the model’s description, the TFT agent selects its action based on the last action performed by its opponent, so attempting to predict a policy of such an agent can become impossible for a predictive agent unless the dyad falls into an equilibrium state very soon or if the predictive agent was able to integrate its own action into its prediction algorithm.

3.4. Experiment 4: Versus the TD-Learning Agent

In this experiment, we also observe a similar performance of the models in terms of their Efficacy in obtaining rewards, with the exception of the Battle of the Exes, where the Predictive and Internal models achieve the best results (see Figure 10). Remarkably, the Rational model achieves a lower score than the rest. This is due to the fact that this model has fallen in a pure-dominance equilibrium with the TD-learning model, where the best response for him is to perform the low-rewarding action. Again, in this case, the TD-learning model capitalizes on the fact that it is solely driven by its own self-utility maximization function, so it had the initiative to go to the higher-rewarding action faster than the Rational model. This, on the other hand, is predicting accurately and responding in the best optimal way to that accurate prediction. However, in this case, this behavioral strategy makes it fall victim to the pure-dominance equilibrium enforced by its opponent.

Regarding Stability, the results show that overall the Predictive and the Internal models converge faster than the vanilla TD-learning toward an equilibrium. In all cases, we can observe that these two predictive models fall in between the ’optimal’ level of convergence of the Rational model and the more unstable level of the non-predictive TD-learning model. The results of the Prediction Accuracy show the worse overall performance of the four experiments studied so far. This, however, was an expected outcome since the predictive agents were facing a non-deterministic learning agent, which is objectively more difficult to predict.

3.5. Experiment 5: Continuous-Time Effects on Prediction Accuracy

As an additional control experiment, we investigate whether the instances where the Rational agent showed poor Prediction Accuracy against the TFT agent in Experiment 3 are solely attributed to the discrete-time nature of ballistic games or if similar errors persist under continuous-time interactions with real-time feedback.

The findings presented in Figure 11 demonstrate that Prediction Accuracy is indeed restored under real-time conditions. This improvement is attributed to the Rational agent’s ability to update its predictive model more rapidly and adjust its chosen course of action in real-time before the round concludes. Consequently, this reduced prediction error against the TFT opponent allows the Rational agent to adopt a more effective strategy, leading to the establishment of a stable equilibrium within the dyad (see Figure 11, middle panel).

3.6. Comparison against Human Data

Finally, in order to inspect which model is closer to human behavior, we compare the results of our models with human data obtained in a study using a similar experimental setup. We analyzed a human behavioral dataset of participants playing the Battle of the Exes game in dyads, published in [35]. In this behavioral experiment, participants played against each other during 50 rounds, so unfortunately, the amount of available data limits the conclusions we can draw from it. In Figure 12, we reproduce the behavioral results of that study against the results of our models playing the Battle of the Exes under a similar payoff structure. When compared with human data, we can see that from the models presented in this study, the Predictive and the Internal models are the closest to human behavior in terms of Stability. Interestingly, this similarity with human behavior occurs only when the models were tested against a greedy opponent. On the other hand, it is important to note that the models showing more diverging results from human data are the pure TD-learning and the Rational predictive model.

4. Conclusions

This work presents a novel control-based cognitive modeling approach to autonomous multi-agent learning models, which combines adaptive feedback control with reinforcement learning. Based on a layered-control architecture, we have designed and implemented agent models characterized by seven different behavioral phenotypes. This includes both deterministic as well as probabilistic learning models. The former category includes three phenotypes: Cooperative, Greedy, and Tit-for-tat, whereas the latter includes four: TD-learning, Rational, Predictive, and Internal. We tested these agent models in dyadic games against each other across five different game theory settings (Harmony Game, Stag Hunt, Hawk-Dove, Prisoner’s Dilemma, and Battle of the Exes). From human behavioral experiments, it is known that the scope of these games sufficiently encompasses behavioral phenotypes in human social decision-making [35,37]. Our proposed agent models were designed and implemented with the aim of deepening our understanding of how cognitive agents learn and make decisions when interacting with other agents in social scenarios (in the context of the game-theoretic tasks stated above).

Additionally, we also implemented our agent models as virtual ePuck robots. The simulated robots are Braitenberg vehicles [41] equipped with sensors and fully embodied controllers (which are the proposed control-based learning modules). These agents are situated in the sense that they only have partial observability of the environment and other agents depending on their location and sensors during rounds. We tested these simulated robots within a spatial continuous-time version of the games involving real-time feedback. We showed that under fully embodied and situated conditions, limitations encountered by agents in ballistic scenarios, such as lack of convergence to an optimal solution or an increase in the number of ties, can be overcome.

Our results show that agents using pure reinforcement-learning strategies are not optimal in most of the tested games (as shown by the Efficacy and Stability metrics). On the other hand, ToM models that predict actions or policies of opposing agents showed faster convergence to equilibria and increased performance. Interestingly, the Rational agent (which knows the best action to take in each game and situation if the opponent’s action is foreseen) outperforms others in many games. However, upon examining human data obtained in the Battle of the Exes game by [35], we found that the Stability profiles of human players matched closest to our Predictive and Internal models rather than the Rational or vanilla TD-learning agents. Although further validation with behavioral data is required to draw any significant conclusion, this result indicates that the models that more closely match human behavior are those that deviate slightly from rationality but do not rely solely on reinforcement learning, thus supporting the notion of bounded rationality in decision-making processes [19].

An especially noteworthy outcome is that both the Predictive and Internal ToM agents successfully acquire the ability to accurately predict their opponent’s actions and policies, respectively, across all games. This shows how autonomous, embodied, and situated multi-agent systems can be equipped with simple Theory of Mind capabilities (in the context of the specified tasks). For future work, it would be interesting to extend these agents with meta-learning models so that they may be able to choose which behavioral phenotype to enact in a given social scenario and flexibly switch in changing environments.

Even though the different behavioral strategies modeled in this study consist of some of the simplest building blocks of human behavior, it is by no means a complete list. Additionally, typical human behavior is not rigidly tied to a given model but rather can flexibly shift between behavioral strategies and learning mechanisms [51,52,53]. Model flexibility is definitely an important consideration for further developing artificial systems capable of ToM. Another issue that is relevant for many real-world tasks is the cost or penalty of being engaged too often in less desirable outcomes such as ties or low rewards [54]. In a sense, this could serve as an incentive to improve performance in complex tasks and should be taken into account when modeling ToM. Yet another typical human feature is trust when dealing with repeated social interactions [55]. Humans tend to form social conventions by agreeing on simple rules or deals and forming a trust relation. At the moment, our models currently lack the ability to form teams that agree on collective goals and work cooperatively to achieve success.

Finally, our control-based reinforcement learning architecture provides an interesting possibility to implement complex social behaviors on multi-agent robotic platforms, such as humanoid robots [56] or other socially competent artificial systems [57]. Another domain that may benefit from the type of agent models described in this paper is the study of evolutionary dynamics; in particular, the evolution of cognitive and intelligent agents. Social interactions involving cooperation and competition are known to play a key role in many evolutionary accounts of biological life and consciousness [58,59]. Lastly, autonomous multi-agent models, of the type described above, with ToM capabilities, might provide useful tools for modeling the dynamics of global socio-political and cultural phenomena such as selective social learning and cumulative cultural evolution [60,61].