Matrices Based on Descriptors for Analyzing the Interactions between Agents and Humans

Abstract

The design of agents interacting with human beings is becoming a crucial problem in many real-life applications. Different methods have been proposed in the research areas of human–computer interaction (HCI) and multi-agent systems (MAS) to model teams of participants (agents and humans). It is then necessary to build models analyzing their decisions when interacting, while taking into account the specificities of these interactions. This paper, therefore, aimed to propose an explicit model of such interactions based on game theory, taking into account, not only environmental characteristics (e.g., criticality), but also human characteristics (e.g., workload and experience level) for the intervention (or not) of agents, to help the latter. Game theory is a well-known approach to studying such social interactions between different participants. Existing works on the construction of game matrices required different ad hoc descriptors, depending on the application studied. Moreover, they generally focused on the interactions between agents, without considering human beings in the analysis. We show that these descriptors can be classified into two categories, related to their effect on the interactions. The set of descriptors to use is thus based on an explicit combination of all interactions between agents and humans (a weighted sum of 2-player matrices). We propose a general model for the construction of game matrices based on any number of participants and descriptors. It is then possible to determine using Nash equilibria whether agents decide (or not) to intervene during the tasks concerned. The model is also evaluated through the determination of the gains obtained by the different participants. Finally, we illustrate and validate the proposed model using a typical scenario (involving two agents and two humans), while describing the corresponding equilibria.

1. Introduction

Interactions between humans and software agents in the context of complex tasks have been studied in different research domains, notably in human–computer interaction (HCI) and in multi-agent systems (MAS). For example, we find works in the fields of road traffic management [1,2], autonomous cooperative robotics [3], and workflow modeling [4], and more recently on the problems of explanation of reasoning [5]. From the point of view of MAS, (autonomous) agents are usually defined as entities capable of acting (and interact) without human intervention [6,7,8,9,10]. From the point of view of human–machine interaction, and particularly intelligent interaction, studies focus on the characterization of interactions between a human being and an intelligent system [11,12,13,14]. Beyond the visual aspects of interfaces, interaction models involve software mechanisms based on an adaptation principle, to ensure intelligent interactions. The cross-fertilization between these two fields of research has led to different perspectives, considering teams composed of different human beings and intelligent agents [15,16,17,18]. Research has led to models of new interactions (and tools to enable these interactions), making them more explicit. Collaboration and cooperation between software agents and humans seems to be a promising solution. Indeed, Badeig et al. [19] highlighted that these interactions require essential properties (autonomy, proactiveness, context awareness, and situatedness) to model real applications. For example, these properties supported tangible human–agent interactions with an interactive tabletop [20]. In [21], the authors proposed a model of interaction between these different participants, with the objective of improving the artificial intelligence component by relying on human expertise. The design of such (human–compatible) agents is still an open issue, as underlined by [22,23,24].

Overall, few research works have taken into account the explicit relationships between humans and software agents, with each participant having its own characteristics. The majority have taken ad hoc approaches, hence our interest in an explicit formulation of these interactions. In our previous work, we proposed interactions between an agent (driving a simulated vehicle) and a human being (driving a vehicle in a virtual environment) [25]. We also studied the context of road traffic congestion: agents and humans were trying to reduce the number of conflicting situations in a road traffic simulation [1,2]. These works focused on the modeling of particular interactions between agents and humans, which were described using matrix games.

The game theory approach is a well-known method for studying and understanding different types of interaction between individuals, and particularly their social relations [26]. The idea is based on finding a strategy that helps a group of players to maximize their own benefits (utilities). This mathematical approach has been explored in different applications based on agents (without direct intervention of humans beings); for example, in modeling (i) the land change and spatial and temporal dynamics of the urban environment [27,28]; (ii) specific behaviors of people (e.g., extremist behaviors [29], spatial segregation [30,31], and dissemination of culture [32]); (iii) social networks [33,34,35]; and (iv) resource allocation [36,37,38]. In their study, Kaviari et al. [27] claimed that “urban land development is the result of the game between different players representing different human behaviors, thus game theory can improve the efficiency of the simulation of such a problem”. These authors showed that a game theory model (in this context, to predict the growth of Zanjan city) with temporal resolution gave better results for urban planning. In this context, the decision of resident agents (to seek the best land relatively to the income level of people) was based on different criteria, such as accessibility, land price, etc. Another illustrative example was introduced by [29]: The authors proposed an agent-based model of the emergence and escalation of anxiety in situations in which individuals from two different groups encounter various hazards. This model is characterized by different criteria, called social identity and identity fusion. The model is not directly based on game theory; nevertheless, the agents make their decision by estimating a utility, which is based on the evaluation of the anxiety level and the perception of a hazard in the group (individual and collective criteria). Lemos et al. [33] studied network formation for agents from different groups. The game proposed in this approach was defined by the payoff of a social dilemma game a particular case of a 2-player matrix. This study showed that the influence on the formation of social networks, depends on the size of the minority group, the frequency with which agents react to adversarial agents, and a cooperation barrier. Noori et al. [38] dealt with water allocation policies and demand management. More specifically, the water demand and the interactions of agricultural agents (concerning products such as rice and citrus) were estimated using utilities corresponding to the level of satisfaction of stakeholders. In these studies, the authors proposed different criteria for estimating the utilities, but their approach, which was well-suited for these applications, proposed ad hoc criteria, without the direct intervention of humans in the loop. The objective of this paper was to build these decision matrices into a more general structure, with a consideration of direct interactions between agents and human participants. The problem is, thus, to allow agents to make decisions to assist or not the human participants.

The paper is organized as follows: Section 2 presents the state of the art related to different concepts useful for modeling human–agent interactions. Section 3 proposes a model based on game matrices for two players/participants. Section 4 generalizes the 2-player model to any number of participants. We show that matrices depend not only on the interactions between participants, but also on predefined criteria/descriptors. The model is then evaluated in Section 5, through a scenario involving two agents and two humans. Section 6 discusses our approach in a more general context. Finally, the last section concludes and gives some perspectives.

2. Background

We propose to list in a non-exhaustive manner the main descriptors that can be involved in intelligent assistance, and which are available in the literature (Section 2.1). We also show that these descriptors can be classified into two categories, which are associated with the relative importance of their numerical value. We provide a description of different steps of the principles considered for the modeling of interactions based on descriptors (Section 2.2). Finally, we explain the concept of a matrix game and the way to determine an equilibrium (Section 2.3).

2.1. Main Existing Criteria (or Descriptors)

Privacy is the first criterion (or descriptor) that can be taken into consideration. The greater the need to respect privacy, the less relevant it will be for a system to interact with a human to provide assistance, with the risk of transmission of confidential information [39]. For example, assistive systems in smart homes, for people in general or people with disabilities in particular, must protect their privacy. The disability of the human, whether physical (e.g., visual) or cognitive (difficulties in understanding, memorizing, etc.), is also an important criterion. For example, the greater the visual impairment (ranging from visually impaired to blind), the more crucial is the need for assistance when interacting with the system [40]. In addition, the weaker the performance of a user, the more useful the assistance of an agent [41]. For example, human performance with office software consisting of hundreds of functions and/or options may be poor for many users for complex or non-routine tasks. Another example is the saturation of road traffic (involving inter-blocking situations) controlled by human operators. In general, the lower the usability of an interactive system, the more useful an assistive system should be for the user [42]. For example, an intelligent aid can be associated with interactive software composed of multiple functionalities(for instance a CAD interactive system) and can guide the user according to the tasks to be performed. The user can be confronted with an environment having a stochastic character, leading to random events. This is the case, for instance, in power plants, production lines, or in multimodal transportation networks. In this case, the higher the level of stochasticity, the more help in anticipating events should be useful for the human [43]. In the same way, the higher the criticality of a situation, the more useful it is help the human with interventions. The goal can be, for example, to avoid possible incidents or accidents, or even disasters [44] (air traffic control is a typical example). Indeed, in a highly sensitive environment, any human error can have serious consequences, and it is useful to have assistance in detecting them or even to anticipate them [45,46]. Depending on the field of application, human error can lead to the loss of a document, an erroneous financial transaction, or an explosion at a chemical plant. In an uncertain environment, an agent can act to evaluate the reliability of the information transmitted to the human. Depending on the experience level of the user of a system, assistance may be more or less useful. This can be the case in certain social networks or in war situations. For example if the user is a novice, assistance may be crucial [47]. The higher the workload of a user, the more useful the assistance can be in reducing it [48,49]. This is the case for control tasks with complex dynamic systems composed of hundreds or sometimes thousands of variables (multi-modal transportation networks, nuclear power plants, etc.).

These criteria are called descriptors in the following. They can be classified into two categories of descriptors:

• Category 1: an increase in the value associated with the descriptor requires a cooperative situation;
• Category 2: a descriptor with a low value implies a situation of assistance (an intervention is recommended).

Table 1. Classification of descriptors into two categories (those in bold are used to illustrate the proposed approach).

Category 1	Category 2
Criticality	Experience level
Workload	Privacy
Disability	Usability
Stochastic environment	Performance
Human errors	Reliability of the system

classifies the main descriptors introduced previously, according to these two categories. This list of descriptors is not exhaustive. It is representative and could certainly be extended by analyzing in depth the specificities of certain fields of application. It aims, above all, to help readers appreciate the complexity of the problem domain and the diversity of possible descriptors.

Hypothetically, numerical values can indicate the relative importance of each category of descriptors. Indeed, for some descriptors (such as criticality), the higher the value, the more useful it will be to set up cooperation between the different participants (in order to reduce criticality). For example, let us imagine values ranging from 1 to 5, with 5 being the maximum value of the descriptor of Category 1. If the criticality has the maximum value (here 5), then an intervention with an assistance goal can be considered essential. For other descriptors (Category 2), a high value (e.g., experience level) would not necessarily require cooperation between participants. For example, if the experience level has the maximum value (here 5), then an intervention with an assistance goal can be considered unnecessary.

The characteristics of the environment, as well as the essential aspects considered by the different actors, are then defined using a set of descriptors. They thus describe certain information that is crucial in the decision-making processes and joined actions selected by the participants (with the aim of global effectiveness being obtained by the latter). These descriptors are proposed by the designer. In order to decide which descriptors to exploit, the designer can start with a global analysis of the application domain [50]. A literature review can also be conducted in parallel, to study the descriptors implemented with the purpose of assistance in the domain concerned (e.g., assistance in power plant supervision). It is also necessary to have discussions with experts of the domain, as well as with users having experience of situations in which assistance may be necessary.

Each descriptor should be computable (and/or estimable) in a reasonable time. Similarly, the descriptors depend on the weights defined a priori by the designer. For example, a designer may consider that criticality is more important than performance in a risk area (e.g., control of major-accident hazards involving dangerous substances [51]).

Initially, our previous studies dealt with three descriptors: workload, experience level, and criticality. These studies concerned a concrete application domain: traffic management by humans assisted by software agents [1,2]. In this paper, even if we have shown that many possible descriptors exist (classified into two categories), we propose to use these three descriptors. They will be sufficient to show the feasibility of the proposed model.

2.2. Methodology Principles

We assume that the environment has its own temporal dynamics, which is correlated to the actions of the different participants. It is therefore necessary to define a reasonable temporal window for the decision regarding assistance (or not) of the different agents. We will assume that this temporal window is defined by the designer. The principle of the model is shown in Figure 1. In the general case (for each cycle), the participants evaluate/assess the selected descriptors, build the game matrix, and then select a Nash equilibrium to make their decision. However, the proposed principle is necessarily partial, since the human actors do not have to build the game matrix. Indeed, we try to propose the best decision (according to Nash equilibrium determination) for the agents. In parallel, human beings make their own decision, independently of the other actors. This first figure gives an overview of the steps at the decision level, and the second figure complements this one at the temporal level.

This principle assumes a cyclical process for the reasoning of the agents (Figure 2). In this figure, we consider a temporal window $\left[t,t+\Delta t\right]$ for Agent $A_i$ and Human $H_j$; the principle remains similar for any number of agents and humans. As such, in each cycle, the participants are able to initialize and evaluate their own descriptors: for example, the workload (descriptor from Category 1 according to Table 1) and experience level (descriptor from Category 2). It is also assumed that each actor (agent or human) is able to evaluate the criticality of the environment. $Ma{j}_{A_i,t}$ (resp. $Ma{j}_{H_j,t}$) represents the update at time t of the descriptor $Maj$ for $A_i$ (resp. $H_j$). Let us therefore decompose the temporal window $\Delta t$ into different phases:

1.

The different actors perceive the evolution of the environment and determine the level of criticality: $cr{t}_{A_i,t}$ for $A_i$ (in the same way, $cr{t}_{H_j,t}$ for $H_j$);

2.

Agent $A_i$ estimates the criticality from the point of view of $H_j$ (denoted $cr{t}_{H_j,t}^{{}^{\prime }}$), which may differ from that determined by $H_j$ (There is no reason why we should have the equality $cr{t}_{H_j,t}^{{}^{\prime }}=cr{t}_{H_j,t}$). In the following, we assume that the criticality (based on the evaluation of the environment) of the human and that of the agent are identical;

3.

In the same way, $A_i$ estimates the workload and the experience level of each $H_j$, denoted by ${\mathit{WL}}_{H_j,t}^{{}^{\prime }},{\mathit{expl}}_{H_j,t}^{{}^{\prime }}$. In the following, we assume that these two descriptors of the agent are identical to those of $H_j$;

4.

Each agent $A_i$ builds the matrix $M_{AxHy,t}$. We will come back to this in the following;

5.

$A_i$ determines the Nash equilibrium for the matrix computed from the $\mathit{S}{\left({\mathit{s}}_{\mathit{i}},\overline{{\mathit{s}}_{\mathit{i}}}\right)}_t$ strategies, where $s_i$ is the strategy of Agent $A_i$ and $\overline{s_i}$ the strategy of any actor other than $A_i$;

6.

We assume that there are exchanges between the different actors (for example, informative acts for the chosen strategies);

7.

We also assume that there are exchanges between the different actors, for example requests about the action to be carried out (the strategy that an actor would like to be selected by another actor). Note that these last two phases are a cyclical process that should converge fairly quickly to a consensus;

8.

The actors perform their respective actions (doing nothing is also an action), which take a certain time;

9.

We assume that the workload and the experience level can be updated by $H_j$. The most difficult problem is to consider an update of the two descriptors for $A_i$. Depending on the strategy selected $\mathit{S}{\left({\mathit{s}}_{\mathit{i}},\overline{{\mathit{s}}_{\mathit{i}}}\right)}_t$, the experience level may increase (a failure could also bring additional knowledge) depending on the success or failure of the action chosen by the players. Similarly, in the previous steps of estimating the two descriptors of $H_j$, Agent $A_i$ could propose an estimation of the experience level according to the success/failure of $H_j$, as well as the strategy considered optimal. In the end, the updating of these two descriptors leads to their valuation at $t+1$.

The principle of the interaction model thus leads to the determination of numerical values for each descriptor during the building of the matrices. These matrices allow the agents to make a rational decision about the necessity to cooperate or not with the humans. We briefly detail the equilibrium search model, and thus the way to select an action.

2.3. Hypothesis and Concepts of Equilibria for a Matrix Game

We assume that assistant agents and humans share the same common environment. It is accepted that like humans, assistant agents also have a limited competence for the task to be completed. This task is in fact a priori a cooperative task. However, nothing prevents us from thinking that the humans may be in a competitive interaction. Let us note n assistants agents defined by $\mathcal{A}=\left\{A_1,A_2,\cdots ,A_i,\cdots ,A_n\right\}$, and m humans defined by $\mathcal{H}=\left\{H_1,H_2,\cdots ,H_j,\cdots ,H_m\right\}$. We also define the set of participants $p_k\in \mathcal{P}$ such as $\mathcal{P}=\mathcal{A}\cup \mathcal{H}$.

Let us take just two players, a software agent and a human being. We consider by convention that the actions of the first player (Assistant Agent A) are represented in the rows, and the ones of the second player (Human H) in the columns. Each actor can decide to cooperate or not, so we will use the usual convention of actions (by considering the usual notation used to deal with the problem of the prisoners’ dilemma): C (for cooperation) in the first row and column; D (for defection, the willingness not to cooperate) in the second row and column. For example, the first row and first column are associated with the strategy $CC$ (this notation first indicates the agent strategy and then the human one). Each player can choose between two actions, $\forall i,S_i\in \left\{C,D\right\}$. A matrix game for a 2-player 2-action game is defined as:

$$\begin{array}{ccc}A\setminus H\hfill & C& D\\ C\hfill & (v_{A_{cc}},v_{H_{cc}})& (v_{A_{cd}},v_{H_{cd}})\\ D\hfill & (v_{A_{dc}},v_{H_{dc}})& (v_{A_{dd}},v_{H_{dd}})\end{array}$$

(1)

Let us note any positive values for the different utilities or gains ($v_{A_{cc}}$, $v_{H_{cc}}$, …, $v_{A_{dd}}$, $v_{H_{dd}}$). When Agent A chooses the strategy D and Human H the strategy C, we can determine the utilities of these two players using the couple $(v_{A_{dc}},v_{H_{dc}})$, i.e., $u_A\left(DC\right)=v_{A_{dc}}$ and $u_H\left(DC\right)=v_{H_{dc}}$. Moreover, as we mentioned in the previous subsection, we reinitialize the matrix by calculating/estimating the different utilities, as in the work proposed in [17,25]. This is called an iterated game.

To ensure the rational behavior of the agents, one method for finding the right decision is to look for Nash equilibria [26,52,53,54,55,56]. To determine these equilibria, we choose the algorithm described in [57,58]. This algorithm essentially consists of two consecutive steps: (i) the gradual elimination of dominated strategies (if we obtain a single profile by successively eliminating (strictly) dominated strategies); (ii) the determination of the different equilibrium. Its application is possible to search for pure strategies, which seems satisfactory in this context. For a non-zero-sum game, we know that the number of Nash equilibria in pure strategies is many, and we assume that the agent then selects an equilibrium from those obtained. Note that human beings do not calculate their respective matrix (the game can be considered partially uncooperative), but they are necessary for software agents.

Recall that the Nash equilibrium is defined using a joint strategy $s=〈s_i^{*},\overline{s_i^{*}}〉\in S_1\times S_2\times \cdots \times S_n$ for n players, where, knowing the strategy chosen by the other players, each player seeks to maximize their gain, i.e., $\forall i,\forall s_i^{\prime }\in S_i,u_i(s_i^{*},\overline{s_i^{*}})\ge u_i(s_i^{\prime },\overline{s_i^{*}})$. In this context, a player i has no interest in changing strategy unilaterally. Therefore, for two players, we consider the following inequalities according to the pair of winning strategies in a 2-player game, H and A:

• $CC$: $v_{A_{cc}}\ge v_{A_{dc}}$ and $v_{H_{cc}}\ge v_{H_{cd}}$, knowing that H cooperates, A has an interest in cooperating (for example, the task is complex enough that H felt the need to call on A and A detects an interest in cooperating with the human);
• $CD$: $v_{A_{cd}}\ge v_{A_{dd}}$ and $v_{H_{cd}}\ge v_{H_{cc}}$, knowing that H is defecting (does not cooperate), A has an interest in cooperating (for example, the task is complex enough for A to feel an interest in cooperating with the human, even if the latter was acting individually);
• $DC$: $v_{A_{dc}}\ge v_{A_{cc}}$ and $v_{H_{dc}}\ge v_{H_{dd}}$, knowing that H cooperates, A has an interest in not cooperating (for example, H felt the need to call on A but A does not consider the task complex enough and, occupied by other tasks, A does not detect an interest in cooperating with the human);
• $DD$: $v_{A_{dd}}\ge v_{A_{cd}}$ and $v_{H_{dd}}\ge v_{H_{dc}}$, knowing that H is defecting (does not cooperate), it is in A’s interest not to cooperate (for example, H did not feel the need to call on A, and A does not consider the task complex enough to offer cooperation or assistance).

For reasons of simplification and without loss of generality, we assume that the descriptors from Category 1, as well as the parameters of the matrix ($v_{A_{cc}}$, $v_{H_{cc}}$, …, $v_{A_{dd}}$, $v_{H_{dd}}$) are defined according to the same ordered scale of discrete values $\left[v_{min}..v_{max}\right]$ (we assume in the following, that $v_{min}=$ 1 and $v_{max}=$ 5). The challenge is then to know if the intervention of the agent remains necessary for the intermediate values. We, therefore, also assume that there is a threshold set a priori (which we note as $v_{fixed}$) that could trigger the cooperative intervention of the agent. The definition of this threshold depends on the designer and the level of freedom desired by the human actors (for example, reducing the amount of intervention of an assistant agent allows humans to act more and progress in their learning of the complex system they manipulate), which is in line with the learning by doing [59] approach. We, thus, define $v_{fixed}=\delta ·v_{max}$. Considering that the human beings must not lose their expertise, this parameter $\delta$ (in the following, we assume that this parameter $\delta$ is identical for the different descriptors) thus sets the threshold for the descriptor where the agent can begin to intervene. A value $\delta =0$ means that it intervenes very quickly; its intervention is delayed when $\delta =1$.

Construction of the matrix for the second category is based on the notations described previously. Suppose the following notations: (i) the value of any descriptor varying from $v_{min}^{\prime }$ to $v_{max}^{\prime }$; (ii) existence of a threshold set a priori that can trigger a software agent intervention, defined by $v_{fixed}^{\prime }$; (iii) $v_A^{\prime }$ for the current value of Agent A, and $v_H^{\prime }$ for Human H.

We will now describe how to construct decision matrices for two participants/players.

3. Building Two-Player Matrix Game

We have shown that the construction of matrices depends on the interpretation of the descriptors, which have been defined in two categories (cf. Table 1). We present matrix models for two players (Section 3.1 and Section 3.2). Then, we study the behavior of the agent for various descriptors (Section 3.3), in the case where there are two players. Finally, we propose three illustrations combining two and three descriptors (Section 3.4).

3.1. Representation of the Two-Player Matrix for Category 1

We propose to study the first category of descriptors, and in particular its matrix game (Section 3.1.1). We also illustrate our point for the following two descriptors: criticality and workload (Section 3.1.2).

3.1.1. Building the Matrix Game for Category 1

To the notations already proposed, we add $v_A$ for the current value of assistant Agent A and $v_H$ for that Human H for the studied criterion (i.e., the descriptor). For convenience, these are also considered to be the utility values: the payoffs that can be obtained by A and H, respectively, in the matrix game. Let us consider the two corresponding extreme situations:

• When the value of the descriptor is equal to $v_{max}$ for A, the agent should decide to intervene to assist the human. The strategy in this case would be a cooperative situation, denoted $CC$ (or $CD$ if this value is low for the human). A high value of the descriptor should therefore lead to a cooperative strategy on the part of the agent. By using the notations of Equation (1)), we have: $v_{A_{cc}}=v_A$, $v_{A_{dc}}=v_{fixed}$, $v_{H_{cc}}=v_H$, and $v_{H_{cd}}=v_{fixed}$. Thus, having this inequality $v_A\ge v_{A_{fixed}}$ (a value greater than or equal to the fixed threshold), $s_A=\left\{C\right\}$ will be the chosen strategy for the agent A;
• Similarly, when the value of the descriptor is low for A, it is not in the interest of the agent to intervene. Strategies such as $DC$ and $DD$ are then necessary, to indicate its non-intervention. To obtain the $DC$ strategy, the first inequality $v_{H_{dc}}\ge v_{H_{cc}}$ is satisfied as soon as the current value of the descriptor, for A, is less than (or equal to) the fixed threshold. The second inequality $v_{H_{dc}}\ge v_{H_{dd}}$ must also be satisfied. If we consider the current value ($v_{H_{dc}}=v_H$), this will be verified for the operation where $v_{H_{dd}}=v_{min}$ (H considers the descriptor to be of relative importance). Strategy $DD$ supposes the satisfaction of $v_{A_{dd}}\ge v_{A_{cd}}$, with $v_{A_{cd}}=v_A$ and $v_{A_{dd}}=v_{min}$ (H partially considers the descriptor).

Therefore, the assignment of matrix values associated with constructing a descriptor from Category 1 is defined as follows, where only the values $v_A$ and $v_H$ are free ($v_{fixed}$ being defined by $\delta ·v_{max}$):

$$\left(\begin{array}{cc}(v_A,v_H)& (v_A,v_{fixed})\\ (v_{fixed},v_H)& (v_{min},v_{min})\end{array}\right)$$

Making assumptions about the possible values of descriptors, a reasonable setting of $\delta$ seems to appear when $\delta \in [0.6,0.7]$. For small values of $\delta$ (i.e., $v_{fixed}$), the Nash equilibrium is the cooperative strategy for A for any $v_H$, if A deems the descriptor higher than the minimum. For values of $\delta$ close to 1, there are many winning strategies, and a cooperative strategy is more difficult to obtain. It should also be noted that the choice of $v_{min}=0$ as the lowest value would increase (with intervention rate) the willingness of the assistant agent to cooperate. More generally, we want to evaluate the percentage of intervention of the agent by varying $v_A$ on the same scale of values. For each value of $v_A$, we set the value of $\delta$, while varying $v_H$ (see Figure 3). We can see that an increase in values of $v_A$ increases the rate of intervention of the agent; which ends approximately at a rate of $50\%$ for $\delta =1$. Therefore, the more A deems the descriptor to be important, the more it decides to cooperate (the more A finds an interest in it). Note that for a zero value of $v_A$, the intervention rate converges quickly towards the non-intervention of the agent. We also note that the variation of $v_A$ in relation to the value of $v_H$ tends to reduce the intervention of the latter.

3.1.2. Illustration of the Criticality and Workload Descriptors

The criticality level of the environment ranges from 1 (normal state) to 5 (hazardous state). This assessment essentially depends on the application, and therefore we will admit that the common criticality is perceived in the same way by assistant agents as by human beings: $v_A=v_H=crt$. In the same way, we set the notations: (i) $cr{t}_{fixed}$ for a fixed criticality, (ii) $crt$ for the current criticality, and (iii) $cr{t}_{min}$ for the minimal criticality (likewise $cr{t}_{max}$ for the maximal criticality).

The workload depends on the different actors and their analysis of their ability to carry out the task. A value 5 for the workload means that the actor feels overwhelmed by how the system works; A minimum value of 1 characterizes a task that the actor can perform without stress and/or difficulty. We note $w{l}_A$ and $w{l}_H$ for the values of the current workload of the assistant agent and the human being; $w{l}_{fixed}$ is the threshold of acceptability of the workload (above this threshold, agents will have to intervene); and $w{l}_{min}$ (respectively $w{l}_{max}$) for a minimum workload value (resp. maximum workload value). By applying the analysis of Section 3.1.1, the matrices associated with criticality and workload for two players are defined by:

$$\left(\begin{array}{cc}(crt,crt)& (crt,cr{t}_{fixed})\\ (cr{t}_{fixed},crt)& (cr{t}_{min},cr{t}_{min})\end{array}\right)\left(\begin{array}{cc}(w{l}_A,w{l}_H)& (w{l}_A,w{l}_{fixed})\\ (w{l}_{fixed},w{l}_H)& (w{l}_{min},w{l}_{min})\end{array}\right)$$

In the following, we consider that software agents have no personal workload; they are still able to perform a task at each iteration; at worst, we consider that the agent estimates its workload like the human, and so we set $w{l}_A=w{l}_H$. The strategic behavior of the assistant agent for the two descriptors described below are those described by $v_A=v_H$ of Figure 3.

3.2. Representation of the Two-Player Matrix for Category 2

This category of descriptors (in which the experience level falls, for example) assumes that the maximum value is not critical; while the minimum value may raise a particular concern for the proper completion of the task. Using an approach similar to the previous category, we plan to study this second category of descriptors, and in particular the matrix (Section 3.2.1), and we illustrate our proposal with a particular descriptor, the experience level of participants (Section 3.2.2).

3.2.1. Building the Matrix Game for Category 2

A reasoning similar to the previous case (Category 1) leads, for this one, to the study of the two extreme situations:

• When the value of the descriptor is low, the assistant agent may have an interest in intervening (strategies $CC$ or $CD$), before the global system deteriorates. To respect these constraints, one solution would be to swap the values proposed in Section 3.1.1. Let us then take $v_{A_{cc}}=v_{fixed}^{\prime }$, $v_{A_{dc}}=v_A^{\prime }$, $v_{H_{cc}}=v_{fixed}^{\prime }$ and $v_{H_{cd}}=v_H^{\prime }$;
• When the value of the descriptor tends towards $v_{max}^{\prime }$, the agent will select a non-intervention action. The strategies in this case would be $DC$ and $DD$. Similarly, the permutation of values proposed in Section 3.1.1 follows the same analysis. For $v_{A_{cd}}$ and $v_{H_{dc}}$, several values are then possible; for example, with two values: $v_{A_{cd}}=v_{H_{dc}}=v_{min}^{\prime }$ and $v_{A_{cd}}=v_{H_{dc}}=v_{fixed}^{\prime }$.

For the descriptors from Category 2, the two-player matrix game is therefore represented by:

$$\left(\begin{array}{cc}(v_{fixed}^{\prime },v_{fixed}^{\prime })& (v^{\prime },v_H^{\prime })\\ (v_A^{\prime },v^{\prime })& (v_A^{\prime },v_H^{\prime })\end{array}\right)\mathrm{where}{v}^{\prime }\in \left\{v_{min}^{\prime },v_{fixed}^{\prime }\right\}$$

As we use the same scale and the same parameter $\delta$, we can assume that $v_{fixed}^{\prime }=v_{fixed}$ and $v_{min}^{\prime }=v_{min}=1$. Figure 4 thus presents the rate of intervention of the agent for any value $v_A^{\prime }$. For each of these values, we vary $v_H^{\prime }$ by setting $\delta$. The different values $v_A^{\prime }$ tend to converge towards the expected values; i.e., we obtain an approximately $50\%$ chance that the agent will intervene and therefore decide to assist the humans.

3.2.2. Illustration of Experience Level

Experience level is a descriptor from Category 2. Remember that an experience level of 1 means that the actor does not really know the system and its evolution, while an experience level of 5 indicates that this actor has perfect mastery of the evolution of the system. We set $exp{l}_A$ and $exp{l}_H$ for the current experience level of the agent and the human being; $exp{l}_{fixed}$ is the threshold of acceptability of the experience level (below this threshold, agents should/could intervene); and $exp{l}_{min}$ (respectively $exp{l}_{max}$) for a low level (resp. a high value for experience). Continuing the analysis of Section 3.2.1, the matrix concerning the experience level for two players is represented by:

$$\left(\begin{array}{cc}(exp{l}_{fixed},exp{l}_{fixed})& (expl,exp{l}_H)\\ (exp{l}_A,expl)& (exp{l}_A,exp{l}_H)\end{array}\right)\mathrm{where}expl\in \left\{exp{l}_{min}^{\prime },exp{l}_{fixed}^{\prime }\right\}$$

In the following, we consider that software agents do not have a variable level of experience, as for human beings; they are always able to intervene wisely. Assuming then $exp{l}_A=exp{l}_H$, the strategic behavior of the agent would correspond to that proposed by Figure 4 for $v_A=v_H$.

By calculating the Nash equilibria for each value of $v_H$ and $v_A$ in $[0,5]$ for a given $\delta$, and by accumulating the occurrences of the cooperative strategy (C), we obtain Figure 5 (Source code that produced these figures can be found here: https://github.com/EmmanuelADAM/Julia/blob/main/majJeux.jl, accessed on 17 March 2023). Figure 5a (respectively, Figure 5b) accumulates for all $\delta$ in $[0,1]$ the cooperation distributions of A for Category 1 (respectively, Category 2); the darker the color, the greater the level of agent involvement.

• For Category 1, we notice that the agent cooperates when it evaluates the descriptor characterizing the situation more strictly than the human. In this case, the agent finds it more useful not to cooperate, even if the human asks for it; because it judges the descriptor weaker than the human values it. For example, if the agent judges the situation as critical, whatever the human says, it will offer assistance.
• For Category 2, we note that the agent cooperates when it evaluates the descriptor describing the situation as weaker than the human. In this case, the agent thinks it is more useful not to cooperate, even if the human asks for it; because it judges the descriptor more strictly than the human. For example, if the agent judges the user to be very inexperienced, whatever the human says, it will offer assistance.

The proposed matrix game is therefore in line with what was expected: the agent assists the human when it feels the need. On the other hand, it lets the human act alone, and therefore learn and gain experience when the agent does not judge the situation to be problematic.

3.3. Combination of Descriptors for the Two-Player Matrix

Combining all the descriptors to obtain a matrix, while giving a reasonable/acceptable interpretation of the Nash equilibria, is not obvious. Indeed, these descriptors are independent features, and depend on the correspondence of the different scales of values. Nevertheless, we consider an empirical approach to their construction. Thus, we can establish a matrix (denoted $M_{A_x{H}_y}$ for an agent $A_x$ and a human $H_y$) as a weighted sum of 2-player $M_i$ associated to $k>0$ descriptors (Cf. Section 3.1.1 and Section 3.2.1):

$$M_{A_x{H}_y}=\frac{1}{{\sum }_{i=1}^k{\lambda }_i}·\sum _{i=1}^k\left({\lambda }_i·M_i\right)\mathbf{with}{\lambda }_i\in \mathbf{N}$$

where ${\lambda }_i$ is the parameter of a descriptor $des{c}_i$ (associated to the 2-player matrix $M_i$) fixed by the designer. As the matrix has a different behavior according to the category, it is essential to differentiate the two-player matrices, $M_i$. Let us take for example that for $k=k_1+k_2$ descriptors, $k_1$ descriptors (respectively $k_2$) belong to the first category (resp. the second category).

For a given matrix $M_i$, we made different assumptions ($\forall i,v_{fixed}^i=v_{fixed}^{{}^{\prime }i}=\delta ·v_{max}$ and $\forall i,v_{min}^i=v_{min}^{{}^{\prime }i}=v_{min}=1$) that should lead us to the need to evaluate the two following quantities, in order to simplify our notations:

$$\left\{\begin{array}{cc}\sum _{i=1}^{k_1}{\lambda }_i·v_{fixed}^i-\sum _{j=1}^{k_2}{\lambda }_j^{\prime }·v_{fixed}^{{}^{\prime }j}\hfill & =\left(\sum _{i=1}^{k_1}{\lambda }_i-\sum _{j=1}^{k_2}{\lambda }_j^{\prime }\right)·\delta ·v_{max}\hfill \\ \sum _{i=1}^{k_1}{\lambda }_i·v_{min}^i-\sum _{j=1}^{k_2}{\lambda }_j^{\prime }·v_{min}^{{}^{\prime }j}\hfill & =\left(\sum _{i=1}^{k_1}{\lambda }_i-\sum _{j=1}^{k_2}{\lambda }_j^{\prime }\right)·v_{min}\hfill \end{array}\right.$$

(2)

We can then state, to simplify the writing: ${\Lambda }^k={\left[{\lambda }_1,\dots ,{\lambda }_k\right]}^T$, ${\Lambda }^{{}^{\prime }k}={\left[{\lambda }_1^{\prime },\dots ,{\lambda }_{{}^{\prime }k}\right]}^T$, $V_A^k=\left[v_A^1,\dots ,v_A^k\right]$, $V_A^{{}^{\prime }k}=\left[v_A^{{}^{\prime }1},\dots ,v_A^{{}^{\prime }k}\right]$, $V_H^k=\left[v_H^1,\dots ,v_H^k\right]$, $V_H^{{}^{\prime }k}=\left[v_H^{{}^{\prime }1},\dots ,v_H^{{}^{\prime }k}\right]$, $V_{value}^k=\left[v_{value},\dots ,v_{value}\right]$, with $card\left(V_{value}^k\right)=k$ and $value\in \{min,max,fixed\}$. Equation (2) then becomes:

$$\left\{\begin{array}{cc}{\Lambda }^{k_1}·V_{fixed}^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·V_{fixed}^{{}^{\prime }{k}_2}\hfill & =\left({\Lambda }^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}\right)·V_{max}^k·\delta \hfill \\ {\Lambda }^{k_1}·V_{min}^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·V_{min}^{{}^{\prime }{k}_2}\hfill & =\left({\Lambda }^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}\right)·V_{min}^k\hfill \end{array}\right.$$

The general framework describes the use of different descriptors with different interpretations. We thus give the results associated with the four possible strategies:

$$CC:\left\{\begin{array}{c}{\Lambda }^{k_1}·V_A^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·V_A^{{}^{\prime }{k}_2}\ge \left({\Lambda }^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}\right)·V_{max}^k·\delta \hfill \\ {\Lambda }^{k_1}·V_H^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·V_H^{{}^{\prime }{k}_2}\ge \left({\Lambda }^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}\right)·V_{max}^k·\delta \hfill \end{array}\right.$$

(3)

$$CD:\begin{array}{cc}\mathrm{If}{v}_x^{{}^{\prime }j}=v_{fixed}^{{}^{\prime }j}\hfill & \left\{\begin{array}{c}{\Lambda }^{k_1}·V_A^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·V_A^{{}^{\prime }{k}_2}\ge {\Lambda }^{k_1}·V_{min}^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·V_{max}^{k_2}·\delta \hfill \\ \left({\Lambda }^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}\right)·V_{max}^k·\delta \ge {\Lambda }^{k_1}·V_H^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·V_H^{{}^{\prime }{k}_2}\hfill \end{array}\right.\hfill \\ \mathrm{If}{v}_x^{{}^{\prime }j}=v_{min}^{{}^{\prime }j}\hfill & \left\{\begin{array}{c}{\Lambda }^{k_1}·V_A^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·V_A^{{}^{\prime }{k}_2}\ge \left({\Lambda }^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}\right)·V_{min}^k\hfill \\ \left({\Lambda }^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}\right)·V_{max}^k·\delta \ge {\Lambda }^{k_1}·v_H^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·v_H^{{}^{\prime }{k}_2}\hfill \end{array}\right.\hfill \end{array}$$

(4)

$$DC:\begin{array}{cc}\mathrm{If}{v}_x^{{}^{\prime }j}=v_{fixed}^{{}^{\prime }j}\hfill & \left\{\begin{array}{c}\left({\Lambda }^{k_1}·V_{max}^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·V_{max}^{k_2}\right)·\delta \ge {\Lambda }^{k_1}·V_A^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·V_A^{{}^{\prime }{k}_2}\hfill \\ {\Lambda }^{k_1}·V_H^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·V_H^{{}^{\prime }{k}_2}\ge {\Lambda }^{k_1}·V_{min}^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·V_{max}^k·\delta \hfill \end{array}\right.\hfill \\ \mathrm{If}{v}_x^{{}^{\prime }j}=v_{min}^{{}^{\prime }j}\hfill & \left\{\begin{array}{c}\left({\Lambda }^{k_1}·V_{max}^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·V_{max}^{k_2}\right)·\delta \ge {\Lambda }^{k_1}·V_A^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·V_A^{{}^{\prime }{k}_2}\hfill \\ {\Lambda }^{k_1}·V_H^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·V_H^{{}^{\prime }{k}_2}\ge {\Lambda }^{k_1}·V_{min}^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·V_{min}^{k_2}\hfill \end{array}\right.\hfill \end{array}$$

(5)

$$DD:\begin{array}{cc}\mathrm{If}{v}_x^{{}^{\prime }j}=v_{fixed}^{{}^{\prime }j}\hfill & \left\{\begin{array}{c}{\Lambda }^{k_1}·V_{min}^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·V_{max}^{k_2}·\delta \ge {\Lambda }^{k_1}·V_A^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·V_A^{{}^{\prime }{k}_2}\hfill \\ {\Lambda }^{k_1}·V_{min}^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·V_{max}^{k_2}·\delta \ge {\Lambda }^{k_1}·V_H^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·V_H^{{}^{\prime }{k}_2}\hfill \end{array}\right.\hfill \\ \mathrm{If}{v}_x^{{}^{\prime }j}=v_{min}^{{}^{\prime }j}\hfill & \left\{\begin{array}{c}{\Lambda }^{k_1}·V_{min}^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·V_{min}^{k_2}\ge {\Lambda }^{k_1}·V_A^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·V_A^{{}^{\prime }{k}_2}\hfill \\ {\Lambda }^{k_1}·V_{min}^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·V_{min}^{k_2}\ge {\Lambda }^{k_1}·V_H^{k_1}-{\Lambda }^{{}^{\prime }{k}_2}·V_H^{{}^{\prime }{k}_2}\hfill \end{array}\right.\hfill \end{array}$$

(6)

We discuss the two special cases, for which the descriptors are of the same nature: (i) $k=k_1$ (with $k_2=0$) and (ii) $k=k_2$ (with $k_1=0$). For these particular cases, we can see that the behavior of the matrix $M_{AxHy}$ is similar to the initial two-player matrices for a given descriptor. Furthermore, the different Nash equilibria lead to a weighted sum of the evaluations relative to the thresholds (more precisely to the minimum or fixed thresholds).

3.4. Illustration with Combinations of Two and Three Descriptors

We propose to illustrate different cases for two descriptors of the same or different category (Section 3.4.1 and Section 3.4.2), and for three descriptors (Section 3.4.3). Note that the approach is not limited to three descriptors. It should be remembered that the model proposed in the previous section is based on any number of descriptors that are weighted by two-player matrices.

3.4.1. Combination of Two Descriptors of the Same Category

Let us take the two previous descriptors from the same category (Category 1), namely criticality and workload levels ($k_1=2$ and $k_2=0$). The ratio $\alpha =\frac{{\lambda }_{wl}}{{\lambda }_{crt}}$ allows us to evaluate the sensitivity of the two descriptors, according to the parameter $\delta$. We show that for a ratio of 1000, we can see that it corresponds exactly, not only to the number of interventions of the agent, but also to the min/max/average number of equilibria according to the value of $\delta$ and for a descriptor of this category. Above this value, the intervention rate of the agent remains the same. Figure 6 shows the influence of the weights on the agent’s intervention rate. An increase of $\delta$ leads to a decrease of the number of interventions for the agent, corresponding to the importance of keeping the knowledge learning for the human being (the closer $\delta$ is to 1, the more the agent’s intervention percentage is around $50\%$). For example, for $lambd{a}_{wl}={\lambda }_{crt}=1$ with the same assumptions on the valuations of the two descriptors, we shift from $96\%$ (for $\delta =0$) to $50\%$ (for $\delta =1$). The difference between the two extreme configurations, according to the ratio of the two coefficients associated with the two descriptors: (i) $\frac{{\lambda }_{crt}}{{\lambda }_{wl}}=1$ and (ii) $\frac{{\lambda }_{crt}}{{\lambda }_{wl}}=1000$ is maximal (approximately $18\%$ for $\delta =0.2$). For values of $\delta >0.8$, the approach is not really significant for the different weightings of these coefficients.

3.4.2. Combination of Two Descriptors from Different Categories

We wish to illustrate with the combination of the following two descriptors ($k_1=k_2=1$): (i) criticality or workload levels (from Category 1) and experience level (from Category 2). Since the choice of criticality or workload does not change the following analysis, we propose to select criticality to illustrate the first category.

To address the sensitivity of the coefficients of the two descriptors (${\lambda }^{crt}$ and ${\lambda }^{expl}$), we have to provide different values for the weight coefficients. The results (assuming $v^{\prime }=v_{fixed}^{\prime }$) obtained for the different weights of the coefficients associated with the descriptors again show that a ratio of $\frac{{\lambda }_{crt}}{{\lambda }_{expl}}=1000$ allows us to find the same percentages as for the criticality descriptor (a binary search allows us to again find this value). We see that the number of min/max/average equilibria is also identical to the criticality level. Above this value, we observe that the percentages do not change (it is thus not relevant to take values higher than 1000). Figure 7 shows the agent intervention rates for the three configurations: (i) ${\lambda }_{crt}=1$ and ${\lambda }_{expl}=1$, (ii) ${\lambda }_{crt}=1000$ and ${\lambda }_{expl}=1$, (iii) ${\lambda }_{crt}=1$ and ${\lambda }_{expl}=1000$. The differences tend to decrease significantly for $\delta =0.6$; but they are almost $40\%$ for a value of $\delta =0$. It is then possible to describe a range of values up to a ratio of 1000 between these two coefficients.

For the hypothesis $v^{\prime }=v_{min}^{\prime }=1$ and coefficients equal to 1, the result obtained is identical for any value of the parameter $\delta$. The behaviors are thus stable, with a $50\%$ of chance of intervening for the agent. Indeed, we show that from Equations (3)–(6): (i) Strategy $CC$ is a Nash equilibrium when the criticality level is greater than or equal to the experience level, (ii) if the values of the descriptors are equal, Strategy $DC$ wins; and (iii) the experience level is greater than the criticality level for $DD$. In contrast, the results obtained by adjusting the different parameters (including $v^{\prime }=v_{fixed}^{\prime }$), and with the contextual conditions of these two descriptors show that the level of assistance for the agent increases according to the parameter $\delta$ from $44\%$ to $65\%$ (essentially due to the decreasing number of Strategies $DD$ as Nash equilibria).

3.4.3. Illustration with a Combination of Three Descriptors

We adapt the general matrix $M_{A_x{H}_y}$ for $k_1=2$, $k_2=1$, $v_A^{crt}$, $v_A^{wl}$, $v_A^{expl}$, $v_H^{crt}$, $v_H^{wl}$, and $v_H^{expl}$. The different simulations underline the consistency between the analytical result and its contextual interpretation. Figure 8 gives some results obtained for the agent’s intervention rate.

Figure 9a illustrates the degree of cooperation of the agent when $k_1=2$ and $k_2=0$, and ${\lambda }_1={\lambda }_2=1$; with the criticality and workload descriptors. Logically enough, the degree of cooperation of the agent follows the same curve as in the case of a unique descriptor. Figure 9b represents the degree of cooperation of the agent when $k_1=2$ and $k_2=1$ (workload, experience level, criticality descriptors), and ${\lambda }_1={\lambda }_2=1$ and ${\lambda }_2^{\prime }=2000$. We observe a lower degree of cooperation of the agent. Indeed the descriptor from Category 2 (for example, the experience level of the human) is strongly supported by ${\lambda }_2^{\prime }$.

We now propose to generalize to a team of n agents and m humans.

4. Generalization for $\mathit{N}$ Agents and $\mathit{M}$ Humans

We are only interested in agent–human interactions, and thus in the construction of a general matrix (denoted ${\overline{M}}_{AnHm}$ for n agents and m humans) determining the different interactions between the different participants. Two approaches can be defined, namely a centralized approach (Section 4.1) and a distributed approach (Section 4.2) for the building of the resulting matrices.

4.1. Centralized Approach for the Decision of Agents

We obtain a formulation that seems sufficient (see Equation (7)) for the centralized matrix, which we will designate by ${\overline{M}}_{AnHm}$:

$${\overline{M}}_{AnHm}=\frac{1}{{\sum }_{i=1,j=1}^{i=n,j=m}\left|{\mu }_{i,j}\right|}\left(\sum _{i=1,j=1}^{i=n,j=m\overline{+}}{\mu }_{i,j}·M_{A_i{H}_j}\right)$$

(7)

Let us set the different parameters ${\mu }_{i,j}$ corresponding to the degrees of trust related to the joint activities of the participating actors $A_i$ and $H_j$. This degree could evolve according to a positive or negative interaction between them (1 would indicate a positive relationship; 0 would indicate no interaction, and $-1$ a negative interaction). We could also imagine, in a future perspective, reinforcement learning to determine these coefficients according to the result of the joint activity. Let us denote the operator ${\sum }^{\overline{+}}$ as the sum of the valuations computed by generating the combination of strategies of a set of players. For example for three participants (Agents $A_1$ and $A_2$, Human $H_1$), $M_{A_1{H}_1}\overline{+}{M}_{A_2{H}_1}$ generates a $(2,4)$ matrix where lines correspond to the strategies C and D for Agent $A_1$, and columns corresponds to the different strategies ($CC,CD,DC,DD$) for the other participants (Agent $A_2$, and then Human $H_1$). Moreover, we denote $M_{AnHm}{\downarrow }_{〈s_i,\overline{s_i}〉}$, the value obtained by the projection of the strategy $s=〈s_i,\overline{s_i}〉$ for the matrix. In the following, we distinguish the strategy of the assistant agent $A_i$ by the notation $s_i$ and that of the human being $H_j$ by the notation $s_j^{\prime }$. The gains obtained for the assistant agents (Section 4.1.1) and human beings (Section 4.1.2) are given for a centralized approach.

4.1.1. Determination of Gains for Assistant Agents

Let us consider the creation of an intermediate matrix that aggregates for each agent $A_i$ the information on the different $H_k$; the matrix thus aggregates its matrices against each human, weighted by its trust on each relation. This matrix that we call $M_{A_i\overline{H}}$ allows us to determine the gains for the assistant agents (Cf. Equation (8)):

$$M_{A_i\overline{H}}=\frac{1}{{\sum }_{j=1}^{j=m}{\mu }_{i,j}}·\left({\mu }_{i,1}·M_{A_i{H}_1}\overline{+}{\mu }_{i,2}·M_{A_i{H}_2}\overline{+}\cdots \overline{+}{\mu }_{i,m}·M_{A_i{H}_m}\right)$$

(8)

Let us build an intermediate matrix that aggregates for each human $H_k$ the information about the different assistant agents. This matrix that we call $M_{\overline{A}{H}_k}$ determines the gain for the assistant agents (Equation (9)) for a human being $H_k$:

$$M_{\overline{A}{H}_k}=\frac{1}{{\sum }_{i=1}^{i=n}{\mu }_{i,k}}·\left({\mu }_{1,k}·M_{A_1{H}_k}\overline{+}{\mu }_{2,k}·M_{A_2{H}_k}\overline{+}\cdots \overline{+}{\mu }_{n,k}·M_{A_n{H}_k}\right)$$

(9)

We are mainly interested in the control of the cooperation of the agents. Thus, we propose a matrix representing the global behavior of the agents regarding the group of humans:

$$M_{\overline{A}\overline{H}}=\frac{1}{{\sum }_{i=1,j=1}^{i=n,j=m}{\mu }_{i,j}}·\left(\sum _{j=1}^{j=m}{\mu }_{1,j}·M_{A_1\overline{H}}\overline{+}\cdots \overline{+}\sum _{j=1}^{j=m}{\mu }_{n,j}·M_{A_n\overline{H}}\right)$$

(10)

Recall the viewpoint consistency assumption: $v_{A_i}^{desc}=v_{A_j}^{desc}=v_{H_k}^{desc}$ for any two assistant agents $A_i$ and $A_j$ and one human being $H_k$. We can easily show that:

$$M_{A_i{H}_k}=M_{A_j{H}_k},\mathrm{for}\mathrm{all}i,j\mathrm{and}k\mathrm{given}$$

(11)

Equation (9) then changes by using Equation (11), and in this case: $M_{\overline{A}{H}_k}=M_{A_1{H}_k}$. Considering this assumption, the utility of a strategy compared to the strategies taken by human players can then be simplified:

$$\begin{array}{cc}u(s_i,\overline{s_i})\hfill & =\frac{1}{{\sum }_{i=1,j=1}^{i=n,j=m}\left|{\mu }_{i,j}\right|}·({\mu }_{1,1}·M_{A_1{H}_1}{{\downarrow }_{〈s_1,\overline{s_1^{\prime }}〉}+\cdots +{\mu }_{1,m}·M_{A_1{H}_m}\downarrow }_{〈s_1,\overline{s_m^{\prime }}〉}\hfill \\ & +{\mu }_{2,1}·M_{A_1{H}_1}{{\downarrow }_{〈s_2,\overline{s_1^{\prime }}〉}+\cdots +{\mu }_{2,m}·M_{A_1{H}_m}\downarrow }_{〈s_2,\overline{s_m^{\prime }}〉}\hfill \\ & +\cdots \hfill \\ & +{\mu }_{n,1}·M_{A_1{H}_1}{{\downarrow }_{〈s_n,\overline{s_1^{\prime }}〉}+\cdots +{\mu }_{n,m}·M_{A_1{H}_m}\downarrow }_{〈s_n,\overline{s_m^{\prime }}〉})\hfill \end{array}$$

It is then possible to determine the number of cooperative agents and those who do not want to intervene, for a given strategy. Thus, we could group the values according to the strategies of the assisting agents, in order to obtain a reformulation:

$$\begin{array}{cc}u(s_i,\overline{s_i})\hfill & =\frac{1}{{\sum }_{i=1,j=1}^{i=n,j=m}\left|{\mu }_{i,j}\right|}·(\sum _{i=1/s_i{=}^{\prime }{C}^{\prime }}^{j=n}·\left(\sum _{j=1/s_j^{\prime }{=}^{\prime }{C}^{\prime }}^{j=m}{\mu }_{i,j}·M_{A_1\overline{H}}{{\downarrow }_{〈C,C〉}+\sum _{j=1/s_j^{\prime }{=}^{\prime }{D}^{\prime }}^{j=m}{\mu }_{i,j}·M_{A_1\overline{H}}\downarrow }_{〈C,D〉}\right)\hfill \\ & +\sum _{i=1/s_i{=}^{\prime }{D}^{\prime }}^{j=n}·\left(\sum _{j=1/s_j^{\prime }{=}^{\prime }{C}^{\prime }}^{j=m}{\mu }_{i,j}·M_{A_1\overline{H}}{{\downarrow }_{〈D,C〉}+\sum _{j=1/s_j^{\prime }{=}^{\prime }{D}^{\prime }}^{j=m}{\mu }_{i,j}·M_{A_1\overline{H}}\downarrow }_{〈D,D〉}\right))\hfill \end{array}$$

Let us set the following parameters such that $n=ca+da$ and $m=ch+dh$:

• $ca$: Number of agents wishing to intervene
• $da$: Number of agents not wishing to intervene
• $ch$: Number of humans wishing to cooperate with agents
• $dh$: Number of humans not wishing to be assisted by agents

Let us assume that the coefficients ${\mu }_{i,j}$ are identical (if these weights are unitary ${\mu }_{i,j}=1$ for all $i,j$, for example). In this case, we can again simplify the previous formulation:

$$\begin{array}{cc}u(s_i,\overline{s_i})\hfill & =\frac{1}{n·m}·(ca·\left(ch·M_{A_1\overline{H}}{{\downarrow }_{〈C,C〉}+dh·M_{A_1\overline{H}}\downarrow }_{〈C,D〉}\right)\hfill \\ \hfill & +da·\left(ch·M_{A_1\overline{H}}{{\downarrow }_{〈D,C〉}+dh·M_{A_1\overline{H}}\downarrow }_{〈D,D〉}\right))\hfill \end{array}$$

(12)

4.1.2. Determination of Gains for Human Beings

To compute the Nash equilibrium, it is necessary to estimate the utility of each human participant $H_j$, depending on the strategies of the assistant agents and on their own strategy. A similar reasoning, considering the particular case where the coefficients ${\mu }_{i,j}$ are identical, leads to a simpler rewriting. Based on the notations $ca$, $da$, $ch$, and $dh$, we obviously obtain a dual formulation of Equation (12):

$$\begin{array}{cc}{u}^{\prime }(s_j^{\prime },\overline{s_j^{\prime }})\hfill & =\frac{1}{n·m}·(ch·\left(ca·M_{\overline{A}\overline{H}}{{\downarrow }_{〈C,C〉}+da·M_{\overline{A}\overline{H}}\downarrow }_{〈D,C〉}\right)\hfill \\ \hfill & +dh·\left(ca·M_{\overline{A}\overline{H}}{{\downarrow }_{〈C,D〉}+da·M_{\overline{A}\overline{H}}\downarrow }_{〈D,D〉}\right))\hfill \end{array}$$

(13)

4.2. Distributed Approach for the Decision of Each Agent

A distributed approach consists in studying the construction of the matrix from the point of view of a single assistant agent $A_i$, which could be expressed as a weighted sum combining the coefficients of the initial two-player matrix. In this distributed approach, we determine the gains obtained for assistant agents (Section 4.2.1) and human beings (Section 4.2.2).

4.2.1. Determination of Gains for Assistant Agents

For n agents and m humans, let us denote the utility associated with agent $A_i$ by $u^{A_i}(s_i,\overline{s_i})=u^{A_i}(s_1,\cdots ,s_i,\cdots s_n,s_1^{\prime },\cdots s_m^{\prime })$. Moreover, we have previously seen that the initial matrices are identical for all assistant agents ( Cf. Equation (11)). Using similar reasoning, we can also simplify the computation of utilities for the assistant agents, if the different weights are equal. Thus, we can make the parameters $ch$ and $dh$ be such that $ch+dh=m$:

$$u^{A_i}(s_i,\overline{s_i})=\frac{1}{m}·\left(ch·M_{A_i\overline{H}}{{\downarrow }_{〈s_i,C〉}+dh·M_{A_i\overline{H}}\downarrow }_{〈s_i,D〉}\right)$$

(14)

Note that this formulation is the same as the general equation defined previously (Cf. Equation (12)). Indeed, It is only necessary to take $ca=1$, $da=0$ if the agent $A_i$ chooses Strategy C (similarly for Strategy D, $ca=0$, $da=1$). This calculation is, in fact, the outcome of an analysis from the point of view of a single agent $A_i$, and not of all the assistant agents (note that $n=ca+da=1$). Moreover, if the parameter $m=1$ (only one human being), the formulation allows us to find the initial matrix by playing with the values of $ch$ and $dh$.

4.2.2. Determination of Gains for Human Beings

As expected, the utility of $H_j$ only depends on its own descriptors, whatever the agent $A_i$. We can thus simplify the calculation of utilities for the assistant agents, if the different coefficients are equal. Thus, we can introduce the parameters $ca$ and $da$, such that $ca+da=n$. The formulation becomes:

$$u^{{}^{\prime }i}(s_j^{\prime },\overline{s_j^{\prime }})=\frac{1}{n}·\left(ca·M_{\overline{A}{H}_j}{{\downarrow }_{〈C,s_j^{\prime }〉}+da·M_{\overline{A}{H}_j}\downarrow }_{〈D,s_j^{\prime }〉}\right)$$

(15)

Note that this formulation is similar to the general equation defined previously (Cf. Equation (13)). Indeed, it is sufficient to take $ch=1$, $dh=0$ if $H_j$ chooses Strategy C (similarly for Strategy D, $ch=0$, $dh=1$). This parameter setting is the consequence of an analysis from the point of view of a single participant $H_j$, and not on all humans ($m=ch+dh=1$). Let us also underline that, if the parameter $n=1$ (only one agent), this formulation allows us to obtain the initial matrix by playing with the values of $ca$ and $da$.

In the studies proposed above, we are able to construct corresponding matrices, and thus determine their Nash equilibria. We will now illustrate this analysis using a scenario.

5. Case Study: Scenario Based on Two Agents and Two Humans

Let us illustrate our proposal for a team of two agents and two humans by describing a scenario. We present a description of the scenario (Section 5.1). It is necessary to determine and build the initial two-player matrix (Section 5.2). Then, we determine the Nash equilibria for the two approaches, a centralized approach with a single matrix (Section 5.3) and a distributed approach according to the point of view of each participant (Section 5.4).

5.1. Description of Scenario

We propose a scenario with four configurations by assuming $\delta =0$ for these illustrations, i.e., the agents’ decision is very sensitive to the minimum valuation (

Table 2. Description of typical configurations for $A_2{H}_2$.

Configuration	Workload		Experience Level		Criticality
	${\mathit{wl}}_{\mathbf{1}}$	${\mathit{wl}}_{\mathbf{2}}$	${\mathit{expl}}_{\mathbf{1}}$	${\mathit{expl}}_{\mathbf{2}}$	$\mathit{crt}$
$c_1$	1	1	1	1	1
$c_2$	2	4	3	1	5
$c_3$	2	4	3	2	1
$c_4$	2	2	3	2	3

). For each configuration, we present the workload, the experience, and the criticality levels.

Initially, the descriptors are positioned at low levels (Configuration $c_1$). As a result of their action/inaction, probably due to their average experience, the environment has deteriorated and the criticality level becomes very high, as well as the workload (Configuration $c_2$). Following the actions of the participants, the situation returns to a normal mode (Configuration $c_3$), assuming an increased experience level for one of the two human operators. Their action depending on their experiences allows decreasing the danger of the environment; one of the two humans, thus, increases his/her experience level. Finally, the last situation (Configuration $c_4$) is again degraded.

5.2. Determination of the Initial Two-Player Matrix According to the Three Predefined Descriptors

We calculate the initial matrix based on the three descriptors, using the previous formulation and the various usual assumptions: $v^{{}^{\prime }expl}=v_{fixed}^{{}^{\prime }expl}$, identical weightings (${\lambda }_{crt}={\lambda }_{wl}={\lambda }_{expl}=1$) for these three descriptors (with $v^{\prime }=v_{fixed}^{\prime }$), $v_{min}=1$, $v_{max}=5$, $v_{fixed}^{crt}=\delta ·v_{max}$, $v_{fixed}^{wl}=\delta ·v_{max}$, and $v_{fixed}^{{}^{\prime }expl}=\delta ·v_{max}$, where $\delta =0$), and the valuations of the agents equal to those of the human being, $v_A^{crt}=v_H^{crt}=crt$, $v_A^{wl}=v_H^{wl}$ and $v_A^{{}^{\prime }expl}=v_H^{{}^{\prime }expl}$. The consideration of these assumptions defines the initial matrix (Equation (16)):

$$M_{A_x{H}_y}=\left(\begin{array}{cc}(\frac{v_H^{crt}+v_H^{wl}}{3},\frac{v_H^{crt}+v_H^{wl}}{3})& (\frac{v_H^{crt}+v_H^{wl}}{3},\frac{v_H^{{}^{\prime }expl}}{3})\\ (\frac{v_H^{{}^{\prime }expl}}{3},\frac{v_H^{crt}+v_H^{wl}}{3})& (\frac{2+v_H^{{}^{\prime }expl}}{3},\frac{2+v_H^{{}^{\prime }expl}}{3})\end{array}\right)$$

(16)

5.3. Building the Centralized Matrix for Two Agents Two Humans

For this particular case, a team of two agents and two humans ($x=1$ or $x=2$; $y=1$ or $y=2$) is considered. The resulting matrix would be defined by ${\overline{M}}_{A2H2}=M_{A_1{H}_1}\overline{+}{M}_{A_1{H}_2}\overline{+}{M}_{A_2{H}_1}\overline{+}{M}_{A_2{H}_2}$.

Table 3. Determination of equilibria for typical configurations in the $A_2{H}_2$ case (centralized approach).

Configuration	Workload		Experience Level		Criticality	Nash Equilibria
	${\mathit{wl}}_{\mathbf{1}}$	${\mathit{wl}}_{\mathbf{2}}$	${\mathit{expl}}_{\mathbf{1}}$	${\mathit{expl}}_{\mathbf{2}}$	$\mathit{crt}$
$c_1$	1	1	1	1	1	$CCCC$, $CDCD$, $CDDC$, $DCCD$, $DCDC$, $DDDD$
$c_2$	2	4	3	1	5	$CCCC$
$c_3$	2	4	3	2	1	$CCCC$, $DDDD$
$c_4$	2	2	3	2	3	$CCCC$

presents a summary of the Nash equilibria for the four studied configurations. Recall, for example, that $CDDC$ means the following distribution of strategies: $s_{A_1}=\left\{C\right\}$, $s_{A_2}=\left\{D\right\}$, $s_{H_1}=\left\{D\right\}$, and $s_{H_2}=\left\{C\right\}$.

Configuration $c_1$ gives different Nash equilibria, since the intervention of one or more agents is not necessary. The matrix is as follows:

$$\begin{array}{lllllllll}& \mathrm{CCC}& \mathrm{CCD}& \mathrm{CDC}& \mathrm{CDD}& \mathrm{DCC}& \mathrm{DCD}& \mathrm{DDC}& \mathrm{DDD}\\ C& (0.67,0.67,0.67,0.67)& (0.67,0.67,0.5,0.5)& (0.67,0.67,0.5,0.5)& (0.67,0.67,0.33,0.33)& (0.5,0.5,0.67,0.67)& (0.67,0.67,0.67,0.67)& (0.67,0.67,0.67,0.67)& (0.83,0.83,0.67,0.67)\\ D& (0.5,0.5,0.67,0.67)& (0.67,0.67,0.67,0.67)& (0.67,0.67,0.67,0.67)& (0.83,0.83,0.67,0.67)& (0.33,0.33,0.67,0.67)& (0.67,0.67,0.83,0.83)& (0.67,0.67,0.83,0.83)& (1,1,1,1)\end{array}$$

Configuration $c_2$ changes the utility calculations (expressed by Equations (12) and (13) and gives the final matrix:

$$\begin{array}{lllllllll}& \mathrm{CCC}& \mathrm{CCD}& \mathrm{CDC}& \mathrm{CDD}& \mathrm{DCC}& \mathrm{DCD}& \mathrm{DDC}& \mathrm{DDD}\\ C& (2.67,2.67,2.67,2.67)& (2.67,2.67,1.67,1.67)& (2.67,2.67,1.67,1.67)& (2.67,2.67,0.67,0.67)& (1.67,1.67,2.67,2.67)& (1.83,1.83,1.83,1.83)& (1.83,1.83,1.83,1.83)& (2,2,1,1)\\ D& (1.67,1.67,2.67,2.67)& (1.83,1.83,1.83,1.83)& (1.83,1.83,1.83,1.83)& (2,2,1,1)& (0.67,0.67,2.67,2.67)& (1,1,2,2)& (1,1,2,2)& (1.33,1.33,1.33,1.33)\end{array}$$

Configuration $c_3$ gives only two Nash equilibria, either both agents intervene or do not intervene. Configuration $c_4$ leads to only one Nash equilibrium: the cooperation of both agents is then necessary.

5.4. Building the Distributed Matrix According to the Point of View of Each Agent

In the context of our illustration, two agents and two humans, the resulting matrix would then be (with the exception of weighting) reduced to the following calculation: ${\overline{M}}_{A2H2}^i=M_{A_i{H}_1}^i\overline{+}{M}_{A_i{H}_2}^i$ for Agent $A_i$. This result implicitly leads to a specific matrix for each agent. Indeed, for Agent $A_1$, its calculation is defined by $M_{A_1{H}_1}+M_{A_1{H}_2}$ (the same for $A_2$).

Table 4. Description of typical configurations for $A_2{H}_2$ (distributed view).

Configuration	Workload		Experience Level		Criticality	Nash Equilibria
	${\mathit{wl}}_{\mathbf{1}}$	${\mathit{wl}}_{\mathbf{2}}$	${\mathit{expl}}_{\mathbf{1}}$	${\mathit{expl}}_{\mathbf{2}}$	$\mathit{crt}$
$c_1$	1	1	1	1	1	$CCCC$, $CDCD$, $CDDC$, $DCCD$, $DCDC$, $DDDD$
$c_2$	2	4	3	1	5	$CCCC$
$c_3$	2	4	3	2	1	$CCCC$, $CCDC$
$c_4$	2	2	3	2	3	$CCCC$, $DDDD$

presents the results in terms of the Nash equilibria for the four configurations considered.

Configuration $c_1$ corresponding to the initial state of our scenario assumes all the descriptors at the minimum value. This is described by the following matrix, whose Nash equilibria require the intervention or not of the assistant agents. Given this situation, the analysis corresponds to the intuitive interpretation (no real reason for the agents to intervene).

$$\begin{array}{lllllllll}& \mathrm{CCC}& \mathrm{CCD}& \mathrm{CDC}& \mathrm{CDD}& \mathrm{DCC}& \mathrm{DCD}& \mathrm{DDC}& \mathrm{DDD}\\ C& (0.67,0.67,0.67,0.67)& (0.67,0.67,0.67,0.33)& (0.67,0.67,0.33,0.67)& (0.67,0.67,0.33,0.33)& (0.67,0.33,0.67,0.67)& (0.67,0.67,0.67,0.67)& (0.67,0.67,0.67,0.67)& (0.67,1,0.67,0.67)\\ D& (0.33,0.67,0.67,0.67)& (0.67,0.67,0.67,0.67)& (0.67,0.67,0.67,0.67)& (0.67,0.67,0.67,0.67)& (1,0.67,0.67,0.67)& (0.33,0.33,0.67,0.67)& (0.67,0.67,0.67,1)& (1,1,1,1)\end{array}$$

For Configuration $c_2$, the environment evolves relatively quickly to a critical situation. This differs from the previous one by having a criticality at the maximum value. Following the experience level and despite a low workload, the assisting agents must intervene (the selected action, as the Nash equilibrium is $CCCC$). Intuitively, related to the urgency of the situation, the intervention of the two agents becomes necessary. Thus, we obtain the following matrix:

$$\begin{array}{lllllllll}& \mathrm{CCC}& \mathrm{CCD}& \mathrm{CDC}& \mathrm{CDD}& \mathrm{DCC}& \mathrm{DCD}& \mathrm{DDC}& \mathrm{DDD}\\ C& (2.67,2.67,2.33,3)& (2.67,2.67,2.33,0.33)& (2.67,2.67,1,3)& (2.67,2.67,1,0.33)& (2.67,0.67,2.33,3)& (2.67,1,2.33,0.67)& (2.67,1,1.33,3)& (2.67,1.33,1.33,0.67)\\ D& (0.67,2.67,2.33,3)& (1,2.67,2.33,0.67)& (1,2.67,1.33,3)& (1.33,2.67,1.33,0.67)& (0.67,0.67,2.33,3)& (1,1,2.33,1)& (1,1,1.67,3)& (1.33,1.33,1.67,1)\end{array}$$

Over time, let us imagine that one of the two humans has an experience level that has increased (as a result of learning from their mistakes or following training, for example). This would not change the intuitive interpretation of the actions of the assistant agents. Configuration $c_3$ regarding a low criticality level requires the intervention of one or both assistant agents. Configuration $c_4$ is a slight modification of the previous one, in which the criticality level is increased from 1 to 3. The assistant agents are told to choose between their interventions (or not) (Actions $CCCC$ and $DDDD$).

6. Discussion

Theoretically, we assume that the descriptors are defined for all the agents. We propose evaluations by showing their respective influence on the agents’ strategic behaviors. In this study, we considered three descriptors (workload, experience level, and criticality), which we combined and weighted. The illustrations used to evaluate our approach considered these three descriptors. We have therefore introduced three examples for the building of two-player matrices (agent–human), illustrating our proposal for a given descriptor. The first example used two descriptors from Category 1 that can be considered in many application domains: criticality and workload. The second example involved a descriptor from Category 2 that is also widely used in many fields: experience level.

However, the state of the art allows us to highlight a set of descriptors that we could take into account. We also showed that our model remains valid for any number of descriptors. Thus, we have completed our validation by analyzing the results for combinations of descriptors: a combination of two descriptors from the same category (criticality and workload), two descriptors from different categories (criticality and experience level), and finally a combination of three descriptors mentioned above. Currently, in the context of several descriptors (e.g., workload and criticality), the same importance is given to each one, in order to simplify the presented examples. It is of course entirely possible that each descriptor has its own weighting. It would be possible to develop a method to define the priorities to be given to descriptors according to the desired objective. Thus, in the case of high criticality, the intervention of the agent could be the priority, whether the human workload is medium or low (not just high). In this case, each agent could consider only a restricted set of considered descriptors. The concerned coefficients ${\lambda }_i$ and the choice of these descriptors would explicitly guide the agents’ decisions. In the same way, the threshold set modifying the behavior of the agents for a particular descriptor could also change the agents’s strategic behavior. Indeed, the sensitivity of the assistant agents has been defined according to this fixed threshold (depending on a parameter, $\delta$). The value of this parameter (varying from 0 to 1) makes it possible to determine the thresholds for their interventions. In our model, we assume that this parameter is unique for all descriptors; but it would be possible to associate a specific value for each descriptor (or even for each agent).

Generalization for any number of participants has been also investigated. We proposed a weighting of coefficients ${\mu }_{i,j}$ for these different initial matrices. These coefficients make it possible to define a level of trust for the other interacting participants [60,61,62]. We could then propose a dynamic evolution of these coefficients according to the level of trust for the different interactions.

7. Conclusions

Interactions between humans and agents have been proposed in different research domains (human–computer interaction, artificial intelligence and multi-agent systems) and for different applications (for example, autonomous cooperative robotics, workflow modeling). Different approaches have been proposed to model teams of participants (agents and humans). The models described in these studies are often challenged by the specificities of humans (workload and experience level, for example) and software agents (autonomy, for example). Recently, studies on AI and MAS have considered human-compatible agents. We think that the game theory approach is a well-suited method for studying social interactions in such teams. The idea is based on the equilibrium concept, to find the strategy which helps a group of participants to maximize their own benefits (utilities). This mathematical approach has also been investigated for various real applications based on interactions between agents (e.g., traffic management, resource allocation, social networks, urban planning). These approaches are often based on the empirical construction of a two-player matrix, and studies underline the good behavior of the built matrix. The human intervention is usually not directly considered, nor included in the analyses of these matrices. We assume that their characteristics should be initially defined in the decision processes, and the agents should also be allowed to decide to cooperate or not. The assistance of one or more humans by one or more agents was the subject of this paper.

The main objective was to achieve a model of the agents’ interventions, in order to assist humans in the context of task realization. This modeling can be qualified as generic, regarding the intervention criteria used by the agents. Thus, to determine the decision of the agents (and thus their intervention or not), we proposed considering matrix games. These depend on the descriptors (acting as criteria) defined by the designer for a given application. These descriptors are classified in two categories. We have seen that the first category means that the minimum value of the scale does not require any particular attention; on the other hand, the maximum value generally requires an intervention by the agent(s). The second category of descriptors, on the other hand, considers a dual interpretation. The set of descriptors we propose to use is thus based on an explicit combination of all interactions between agents and humans (represented by two-player matrices). We also proposed a model based on a weighted sum of the matrices (associated with an agent–human interaction) corresponding to the predefined descriptors. Finally, we described a general model (n agents and m humans) allowing the definition of any number of participants and any number of descriptors. The exploitation of the model was illustrated using a typical scenario, functioning as a proof of concept. This scenario considered a matrix with two agents and two humans, using the same three descriptors. We considered the obtained equilibria according to two different approaches (centralized or distributed) depending on the point of view: global or relative to a given agent.

The proposed model is generic but reveals two possible alternatives, leading to complementary research. On the one hand, it would be possible to take into account the feedback of an agent’s action and to interpret the result with respect to the decision determined by the selected Nash equilibrium. The action (with or without cooperation) can in this case be judged irrelevant (for example, instead of intervening to help the human, it would have been better for the agent to wait). On the other hand, instead of each agent intervening individually, it would be possible for multi-agent coalitions [63,64,65] to be formed to assist a human or group of humans in the achievement of a task. This would allow the other agents to focus on other tasks. We plan to implement and test such configurations in different application domains.