Socio-Economic Decision Making and Emotion Elicitation with a Serious Game in the Wild

Abstract

There is a growing body of research in the literature that investigates the relationship between emotions and decision-making in socio-economic contexts. Previous research has used Serious Games (SGs) based on game theory paradigms with socio-economic contexts to explore this relationship in controlled settings, but it is unclear whether such SGs can be applied in the wild, which would improve their usability in natural settings. The emergence and widespread availability of mobile computing technologies have enabled the creation of diverse SG implementations. We developed a mobile SG that combines four socio-economic interactions (trust game, ultimatum game, dictator game, prisoner’s dilemma game) into one narrative and conducted an initial experiment with eight participants. This study discusses the challenges of deploying SGs in the wild and proposes solutions to overcome them. The results from the preliminary user experience experiment indicate that the SG is a suitable tool for emotion elicitation in the wild and that the results are similar to those observed in previous studies under controlled environments. We argue that these findings suggest that mobile SGs have potential for studying the relationship between emotions and decision-making in socio-economic contexts in the wild, improving the reach of these methods beyond laboratory settings.

1. Introduction

Emotions are essential factors that influence our decision-making processes and affect our daily lives, including our behavior, clothing choices, dietary habits, and investment decisions [1,2,3,4]. It is known that both incidental and integral emotions are critical in establishing our default preferences when making decisions in our everyday lives [5,6,7,8,9]. The incidental emotional state, also known as mood, reflects the general state of our minds and is not dependant on conscious decisions or judgments made by ourselves at any given instant [9]. The integral emotional state is the emotional state of a person when a decision is being made [9]. Currently, automatic emotion detection (for both incidental and integral emotions) is being used in several avenues for mitigating the damaging effects of mental illnesses and neurological conditions, i.e., for teaching social interaction to children with Autism Spectrum Disorder (ASD) using robotics [10], for using emotion recognition as a tool for detecting depression [11], for promoting positive emotions [12], and so on. The advancements and miniaturisation of modern technology means that these automatic emotion detection methods now have unprecedented accuracy and portability [13,14].

Applications of automatic emotion detection have been possible due to the existence of certain patterns in emotional responses, most notably, of individuals with social anxiety [15], depression [11], borderline personality disorder [16,17,18] and post-traumatic stress disorder [19]. Moreover, gamified activities have shown to produce specific emotional responses both in individuals with affective disorders [20,21,22] as well as mentally healthy individuals under socio-economic scenarios [23,24,25]. Similarly, decision making patterns from individuals with and without affective disorders under socio-economic scenarios have been reported in literature (e.g., [8,20,22,25,26,27,28,29]). When making a decision in a socio-economic scenario, the incidental emotional state of a person influences the decision one will make and this, in turn, gives rise to the integral emotional state of the person [9]. In the case of sequential decision making tasks, the integral emotion of the first task becomes the incidental emotion to the subsequent task and hence, each decision made by an individual has an impact on the subsequent decisions in the sequence through the changes in the emotional states of the individual [9,30,31]. Hence, studying decision-making patterns over a sequence of socio-economic tasks can be useful in understanding whether the elicited emotional responses conform with the ones observed in the literature. This would help in detecting affective disorders such as depression and anxiety [32,33], especially for ASD individuals with communication disabilities [34] and for acceptance testing [35].

On a different note, studies investigating the affective state of participants in experiments under controlled settings often tend to induce some level of anxiety [36,37,38]. The factors that are often controlled in such experiments include elimination of external stimuli in a laboratory setup to minimise distractions; controlling environmental factors such as humidity, air temperature, quality of air to minimise participant discomfort are controlled; and having experimenters who have properly planned set of activities to minimise uncertainties. Such setups are often foreign to the participants, and may induce stress. These drawbacks are usually not encountered in experiments that are setup ‘in the wild’, which is a general term indicating that data is collected in uncontrolled settings, not in a laboratory and typically using devices owned by consumers [39]. More specifically, emotions elicited in the wild have an inherent advantage of situated perspective [40], which is particularly important in socio-economic contexts [41]. Essentially, situated perspective refers to the consideration of emotional responses of people in the context of their environments. Hence, emotional responses elicited under environments that an individual is comfortable in (e.g., their own home) may produce emotional responses that are more realistic [36,40,42,43,44,45]. A potential compromise could be the adoption of “ecologically valid” settings, similar to real contexts [46,47]. Implementing ecologically valid settings in laboratory or in clinical environments (for instance in occupational therapy contexts) [48] can be quite useful for making tasks and conditions quite meaningful for the subject without detrimental effects on the experimental control of the investigations. In all settings presented above, subjective and objective assessments can be performed according to quantitative and qualitative data: each perspective and each type of information can provide an investigation with impactful advantages [49,50]. However, a preliminary study can focus on a subset of such methods for exploring a hypothesis before proposing a wider methodology encompassing different solutions for evaluating the individual behaviours. In terms of socio-economic decision making, the literature shows that socio-economic state of a person can have a significant impact on a person’s mental and emotional health in the wild [42,51,52,53,54,55,56,57]. This indicates that, not only decisions made in interactions based on socio-economic scenarios in the wild can have a deep rooted relationship with one’s emotional state, but also studying them could give us a an understanding of a person’s mental state, which is the basis of understanding their social functioning in the modern society [58]. Since mobile serious games (SGs) can be designed to be engaging and enjoyable [54,59,60,61], their application for capturing the affective states of individuals may enable collection of emotion data without the negative affect bias that experiments under controlled settings might induce. As the literature also reports significant differences in the user experience (UX) of users of smart devices between lab settings and in the wild settings for ubiquitous applications (e.g., [43,62]), we argue that investigating the feasibility of a tool for emotion elicitation in the wild, and verifying its UX is a significant preliminary research challenge in the area of emotion and decision making.

The wide popularity of Role Playing Games (RPG) on mobile platforms [63,64] has enabled the usage of decision-making games in several different real-world contexts. However, to the best of our knowledge, none of these games have been specifically designed to port ‘in the wild’ the aims of controlled experiments on emotions and decision making in socio-economic contexts [8,20,22,25,26,27,28,29], which severely limits the applicability of such games to mostly clinical contexts. A possible reasoning behind this lack of application in the wild could be the fact that there are several challenges, such as difficulty in reliable emotion measurement [38,65,66] and in measuring player engagement [54,67,68,69,70] as well as minimising contextual variability (consisting of external factors, such as the environment and internal factors, such the mood of the player) [54,66,68,71], ensuring cultural sensitivity [72,73] and finally, addressing ethical considerations [66,74] and technical limitations [75].

The main goal of this study is to investigate the possibility of producing decision-making patterns and emotional responses under socio-economic scenarios in the wild, that are seen under controlled environments in the literature. Thus, our main focus is on the design of a SG that is able to bring into the wild the interactions which, until now, were designed, deployed and verified in controlled settings only. Cohesive storylines may lead to greater immersion in SGs and, thus, greater engagement [76,77]. Coherent narratives also tend to establish an emotional connection with the characters in the game [78,79] and make the player feel s/he is making meaningful choices in the game interactions [80,81,82]. Since the games developed for controlled settings concern only single scenarios, we are interested in trying to design a SG that combines multiple socio-economic game paradigms into one cohesive storyline. We also tested the developed SG in a preliminary user experience experiment checking feasibility and potential of the approach before a necessary comprehensive study [83].

This paper is structured as follows. Section 2 discusses the state-of-the-art in the application of mobile as well as computer-based SGs and in person SGs in general for emotion elicitation. Section 3 describes our game design elements and choices. Section 4 explains the ‘in the wild’ experiment and our core methodologies that are employed in this study. Section 5 expands on our analysis and Section 6 discusses the results obtained from them. Finally, we provide our conclusions, the limitations of this study as well as some directions in which future research on emotion elicitation using mobile SGs ‘in the wild’ could take in Section 7.

2. Related Works

There have been several socio-economic game paradigms utilised for in-person RPGs, such as Ultimatum Game (UG) [8,20,22,25,26,29], Trust Game (TG) [8,28], Dictator Game (DG) [27,84] and Prisoner’s Dilemma Game (PDG) [8,27,28,29] for a number of studies over the years for various purposes including emotion elicitation. These studies were conducted without the involvement of computerised implementations and under controlled environments. Hence, methodologically, most of them had a similarity, which was the approach towards how the the games were played. More specifically, either the participants tookup roles in the games to play with each other or the participants were playing with the experimenters. Since these were played in person, an obvious limitation of such games for in the wild settings was the fact that it would not be possible to control the non-protagonist characters’ playing behaviour as each person will play a role differently, which could introduce some contextual variability in their studies [85].

Table 1. Decision making and emotional response patterns in socio-economic scenarios.

Socio-Economic Scenario	Type of Pattern	Pattern Observed
Ultimatum Game *	Decision making	Favours accepting any offer as responder and makes fair but lower offer to maximize profit
	Emotion Elicitation	Induces sadness when unfair offer is presented and happiness when fair offer is presented
Trust Game *	Decision making	Favours investing smaller amounts in the beginning and defect more often as trustee
	Emotion Elicitation	Induces sadness and anger when trustee does not return profit shares and happiness for  the contrary
Dictator Game *	Decision making	Favours making lower allocations to recipients
	Emotion Elicitation	Induces happiness in being able to provide any amount of resource to the responder
Prisoner’s Dilemma Game **	Decision making	Favours cooperative over selfish behaviour
	Emotion Elicitation	Induces anger, sadness and sometimes disgust when betrayed and happiness for the contrary

* [8,20,22,25,26,27,28,84], ** [27,28,29].

shows the decision-making and emotional response patterns for the different socio-economic scenarios employed by the aforementioned studies. For UG, the observed decision-making pattern observed in the literature was that the participants favoured accepting fair and unfair offers as a responder and make fair but smaller offers as proposers [8,20,22,25,26,27]. In terms of emotional responses the dominant emotion was sadness when a participant was presented with an unfair offer and happiness when a fair offer was presented for this socio-economic scenario [8,20,22,25,26,27]. For TG, the observed decision-making pattern observed in the literature was that the participants favoured investing smaller amounts as the trustor and defecting more often as trustee [8,28]. In terms of emotional responses, the dominant emotion was sadness and anger when no profit returned from an investment and happiness for the opposite [8,28]. For TG, participants favoured lower allocations of resources to the recipients and induced happiness at being able to help someone else [8,84]. For the PDG, participants favoured being cooperative over selfish behaviour and in terms of emotional responses, anger, sadness and sometimes disgust was reported by participants after being betrayed and happiness when the other prisoner cooperated as well [27,28,29].

Recently, there have been several applications of computerised SGs for academic performance enhancements [86,87,88,89], developing emotional intelligence skills [90,91], emotion regulation [92,93] and affective disorder therapies [94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129]. All of these studies were carried out under controlled settings. For studies in the wild, most of the SGs applied were for enhancing typing speed [61], emotional training of children with ASD [130], promoting eco-driving [131,132] and enhancing programming skills [60]. However, none of the aforementioned SGs had been applied for emotion elicitation. At the time of completing this article, we are not aware of any implementation of a SG that incorporates multiple socio-economic game theory paradigms for the purpose of elicitation of emotional response and decision-making patterns, especially in the wild and hence, there are no studies that are directly comparable to ours.

In a previous study, we presented the application of a mobile SG design, the player engagement metrics and a general overview of the emotions elicited in the players during the socio-economic interactions in our game [133]. However, the study lacked a comprehensive analysis of the emotional responses as well as an analysis of the decision-making patterns observed during the socio-economic interactions. This is important as it would add another point of comparison with the player patterns observed in the literature in addition to the emotion elicitation patterns. The literature indicates a relationship between decision-making patterns and elicited emotions under socio-economic scenarios [1,2,3,4,7,134,135,136]. Since there can be significant implications of the socio-economic state of a person on their mental and emotional health in the wild [42,51,52,53,54,55,56,57], it is important to investigate whether the decision-making patterns and elicited emotions under socio-economic scenarios observed in controlled settings can be replicated in the wild. For applications of this nature, it is essential to test the user experience of a game developed utilising socio-economic scenarios and deployed in the wild in order to determine its applicability in such contexts.

3. Game Design

3.1. Game Requirements

The first step for the design of the SG concerned the definition and analysis of a set of requirements, which are stated and explained as follows:

(R1)

The game should be able to elicit specific emotions and decision-making patterns in its players as seen in the literature for relevant socio-economic scenarios.

As by definition, SGs are “video games developed for a primary purpose other than pure entertainment” [137,138], the primary purpose of our game is the elicitation of emotions and decision-making patterns and hence, this requirement.

(R2)

The game should be immersive and engaging enough to minimise the contextual variability around a player.

Emotions can be influenced by many different factors in the wild, mostly the environment and the player’s pre-existing mood. These factors are known as contextual variability [54,66,68,71]. It is essential for a game to be immersive in order to engage the player well enough for reducing the impact of contextual variability [139].

(R3)

The game must be supported on popular platforms so as to make it possible for a larger population to play the game.

There are obvious technological constraints of the technological nature for implementing a SG for emotion elicitation, for instance, the availability of hardware, software, and compatibility of the game [75]. This requirement refers tothe fact that a game which is usable in the wild would need to run on widely available platforms.

(R4)

The game should have a mechanism for emotion measurement that does not interrupt the progression of the storyline or gaming experience.

While emotion measurement can be taken relatively easily and reliably under controlled settings, this becomes much more difficult in the wild, especially when the emotion data needs to be collected from physiological sources [38,65,66].

(R5)

The storyline, dynamics and NPC interactions should be designed keeping the cultural influence in mind for the intended players.

This requirement is related to the challenge of cultural sensitivity, meaning that the mobile SG to be used for emotion elicitation needs to seem culturally relevant to the players. Mobile SGs for emotion elicitation must be designed to keep cultural influences in consideration, as emotions can vary greatly between cultures and it is important to be aware of cultural sensitivities and to design the game in a way that is culturally appropriate for the intended players [72,73].

3.2. Design Decisions

As we are investigating whether socio-economic scenarios studied in controlled settings can produce the emotional responses and decision-making patterns in the wild, we implemented four different NPC interactions (further explained in Section 3.4) modelled on those socio-economic scenarios and this addresses requirement R1. For addressing the contextual variability, indicated by requirement R2, our game was designed to be a 3D RPG game, as several studies have shown that 3D [71] and RPG games [13,14,63,64] can immerse the player into the game environment. It is also known that immersive games can disengage players from their environments and pre-existing mood state [139,140,141], making them the most engaging and popular type of video games [142,143]. Since the main focus of this game was to keep the player engaged in the game environment enough for them to complete the NPC interactions, no reward mechanisms (such as badges or leaderboards) were implemented with any aspect of the gameplay, as incorporating these might shift the motivation of the player from attentively completing the tasks to rushing them to be at a higher position on a leaderboard [144,145] and this also stems from requirement R2.

The first prototype of the game was developed for the Android platform due to it having the second largest user base of game users [63] and the straightforward deployment process. This addresses requirement R3. Furthermore, requirement R4 was addressed by implementing an emotion self-reporting mechanic within the game’s storyline as it has been extensively used as a reliable data collection method under controlled settings [13,19,133,146,147,148,149,150,151] and has shown be to be reliable for in the wild settings as well [152,153]. This has been further discussed in Section 3.5. Finally, the game was designed to have conversational NPCs that use English as the language of communication. This is significant as players feel a game’s content to be relatable when they are conveyed in their native language [154,155] especially when this is paired with conversational NPCs in RPGs [156,157]. This addresses requirement R5 mentioned in Section 3.1.

Another fundamental design choice concerns the storyline [158]. In our SG, the player will live a day in the life of “Joe”, an average boy in the hilly, small town of Eridanus in the island nation of Laniakea. Joe wakes up in the morning everyday and has to complete a certain set of tasks each day, interacting with the locals. Each level in the game consists of all the activities Joe carries out in a day with limited resources (money, food and health) that Joe must perform in order to keep his Overlord satisfied. The proposed game has one such level and the player will explore one of those days in this level when Joe visits a new part of the town called Caldwell, where he is required to meet with a number of people and make socio-economic decisions during the interactions.

In order to build a single coherent narrative joining several specific socio-economic scenarios, the scenarios are sequenced such that they will induce emotional responses of high valence and to make the NPC interactions engaging. The first interaction will be about borrowing money from someone rude, immediately followed by an interaction where the player is awarded for placing his trust in a stranger. These will instil a sense of positive uncertainty in the player which will keep the player guessing what might happen in the next interaction. This is beneficial towards increasing player engagement as it is known that people are particularly receptive to situations involving uncertainty [21,24,82,159].

The game significantly emphasizes engaging with NPCs during interactions. As the player navigates between sites of interaction, the camera perspective will be in third-person. However, during all interactions in the game, the camera will switch to a first-person point-of-view from Joe’s perspective. This allows the player to view the NPCs from Joe’s perspective, further enhancing the immersive nature of the game.

Other important mechanics that have been implemented in the SG are the following:

• After interacting with an NPC, the player is informed of their next objective.
• Interacting with an NPC can either positively or negatively impact one of the game’s resources, which is represented by a change in the corresponding player status bar on the game’s interface.
• To progress to the next level, the player must maintain at least two out of three resources at a level above “low” or in the “green” range on the player status bar.
• The player must navigate between sites of interaction while staying within the road bump barriers. Colliding with these barriers will reduce the player’s health bar, adding an element of risk and engagement to the gameplay experience, and keeping the player in the “flow” [21,24]. Additionally, the base mechanic regarding the movement of the player character has been kept flexible and straightforward, meaning the player character can freely move around the 3D world (on paved and unpaved roads, on the grass, inside the buildings, etc., if the player chooses to jump over the bump barriers and create his own shortcuts through the terrain to the next task location, as free-form movement is beneficial to make task-based games enjoyable and motivating [160]).
• The game includes a navigation arrow that directs the player to the next NPC and a map that displays the locations of all NPCs in the scene, as well as the player’s current location.
• The game design ensures that socio-economic interactions cannot directly lead to a failure of game progression, as this could cause unexpected emotional responses from players [21,24,27,161]. Therefore, the mechanic of requiring two resources to be maintained at a level above “low” is essential for successful completion of the level, as it guarantees that there is no direct connection between socio-economic interactions and failure to progress in the game.

3.3. User-Interface

Figure 1 shows a snapshot of the game UI in between interactions. There are only four UI elements that are constantly displayed to the player, the player status panel (top left corner of the screen), NPC location pointing arrow (the free to pivot white arrow near the center top of the screen), the navigation map button (the map with a location pin icon at the bottom left corner of the screen), the multi-directional on-screen joystick to control the player character (on the center of the screen just under the player character) and a jump button the player character jump (on the bottom left corner of the screen). The player status panel itself contains three status bars for the limited resources that the player has, namely health at the top, followed by money and then food. These change in length horizontally according to the changes for the respective resources for the player throughout the gameplay. For instance, whenever the player character hits a bump barrier, the health bar decreases by a unit or when the player decides to donate or accept money (or food) to an NPC, the money (or food) bar will reduce or increase by an amount that is proportionate to the money (or food) donated or accepted by the player. The status bar changes are range-colour coded (100% to 70% is green, 70% to 19% is yellow and the rest until 0 is red) and the distribution of the colour ranges is set such that the player will not easily fall into two red status bars which would stop the progression of the game. The changes in the player status bar components are only feedback provided to the player for events that change health, money or food resource for the player as distractions from haptics and other visual feedback might make the player lose focus in the game, possibly diverting the player’s attention from completing the main tasks of the SG [162]. The direction of the NPC location pointing arrow changes according to the position of the NPC in the scene relative to the direction Joe is facing, hence it always points to the absolute position of the NPC and the player has to determine how to get there by navigating the scene. At any time during the gameplay, clicking on the navigation map icon displays a top-angled view of the entire scene with markers for locations of the tasks ordered according to their sequence of occurrence in the game, as well as the real-time position of Joe in the scene, as shown in Figure 2. This enables the player to quickly find the playing character’s position in the scene at any point in the game and where they might need to go next.

3.4. Socio-Economic Decision Making Interactions

We have implemented four different socio-economic scenarios or game paradigms as known in game theory [163,164,165], in the form of player-NPC interactions in our game. Combining multiple socio-economic game paradigms into player-NPC interactions in one cohesive storyline and gameplay is a novel factor of our implementation. These interactions are explained below, written in the order they appear in the game [8,20,22,25,26,27,28]:

• Ultimatum Game
  In this game, there are two players, a proposer and a responder. The proposer is given some money and is asked to propose a split amount with the responder. The responder can accept or reject the offer. If the responder accepts the offer, the two players split the amount according to the agreement, if the responder rejects the offer, the responder and the proposer do not get any money from the game. In our game, the NPC plays the role of the ‘proposer’ and the player character is the ‘responder’. Furthermore, the interaction is designed as such that the NPC makes an unfair offer and the polarity of the interaction is negative, meaning that the NPC dialogues are condescending and slightly rude but not offensive, as such interactions in UG settings have been shown to produce more distinct negative emotional responses with higher valence [26,151,166]. Figure 3 shows a part of the dialogue between the NPC for this interaction and the player character.
• Trust Game
  In this game, there are two players, a trustor and a trustee. The trustor is given some resource, e.g., money and is asked to propose a split amount with the trustee. The trustee can accept or reject the offer. Whatever amount is given to the trustee is double/tripled by the game. However, the trustee has the freedom to choose whether or not to repay the trustor any amount from the income made from the game. In our game, the NPC plays the role of the ‘trustee’ and the player character is the ‘trustor’. Here, the interaction is designed as such that if the player chooses to trust the NPC with its resources (which is food for this interaction), the NPC chooses to reward the player for it and returns the player’s share on the profit to the player. The polarity of the interaction is positive, meaning that the NPC dialogues are uplifting and indicating gratefulness of the NPC. This is due to two reasons, firstly to balance the negative approach of the NPC in the previous UG interaction, as prolonged negativity from NPCs can effect the decisions of the player [167] and secondly as positive polarity interactions in TG settings promote co-operation [168]. Figure 4 shows a part of the dialogue between the NPC for this interaction and the player’s character.
• Dictator Game
  In this game, there are two players, a dictator and a responder. The dictator is given some money and is asked to donate a fraction (or the whole amount) with the recipient. The responder has to accept the donation and has no other choice. The dictator also has the option of not donating any amount at all to the recipient and the recipient has no say in this game. In our game, the player character plays the role of the ‘dictator’ and the NPC is the ‘responder’. This interaction is designed as such that if the player chooses to donate some of its resources (which is money for this interaction) to the NPC then the NPC expresses its gratitude for the generosity of the player. Otherwise, the NPC is still polite and the interaction has a positive polarity. This interaction having a positive polarity is important as it will ensure that the prior TG interaction having a positive polarity will not seem like a one-off phenomenon in the game and will solidify the notion that not all NPCs in our game’s world are rude. This will also provide a better impact for the next interaction being of a negative polarity again as any changes in the narrative of a game elicits stronger emotional responses in the players [169]. Figure 5 shows a part of the dialogue between the NPC for this interaction and the player’s character.
• Prisoner’s Dilemma Game [27,28,29]
  In this game, there are three players, a jailer, prisoner A and prisoner B. Both the prisoners are interrogated by the jailer regarding a crime they are accused of, if both the prisoners accuse each other of being guilty, they both get one year of prison/low damage or punishment of some arbitrary type. If both the prisoners do not accuse each other for being guilty, they both get five years of prison/moderate damage or punishment of some arbitrary type. If one of them, for instance prisoner A, accuses prisoner B of being guilty while prisoner B does not accuse prisoner A, then prisoner A will walk free and prisoner B will get ten years of prison/the highest damage or punishment of some arbitrary type. The flipped situation of prisoner B accuses A while A does not accuse B, prisoner A will receive the highest damage or punishment of some arbitrary type. In our game, the NPCs play the role of the ‘jailer’ and the ‘prisoner B’ while the player’s character is ‘prisoner A’. Lastly, the interaction is designed as such that the player chooses to help an NPC (let us call it prisoner-NPC) to complete the final task’s objective and by being associated with the prisoner-NPC, the player becomes an accomplice in the eyes of the second NPC (let us call it the jailer-NPC). This interaction is specifically designed to orchestrate a betrayal from the prisoner-NPC, which is supposed to elicit a high valence of negative emotional response [170]. Figure 6 shows a part of the dialogue between the prisoner NPC for this interaction and the player character, while Figure 7 shows a part of the dialogue between the jailer NPC for this interaction and the player’s character, explaining the rules of the interaction.

3.5. Emotion Self-Reporting

A standard practice that can be seen in the literature [13,19,133,146,147,148,149,150,151] for capturing emotional responses of individuals after an event is self-reporting and we have incorporated this action into the gameplay itself. After each socio-economic interaction, the player perspective in the game changes to a ‘self-reflection’ mode, where the player’s character is displayed on the screen and the UI shown in Figure 8 is displayed. This is so that the player feels that at that moment, the player is ‘talking to themselves’ to self-report their emotional state at that very instant. This implementation of self-reflection dynamic within the gameplay ensures the immersion of the player is not broken from the game world [171] and also satisfies requirement R3 discussed in Section 3.1. The options for self-reporting emotional state includes the six basic emotions (Happiness, Surprise, Sadness, Anger, Fear and Disgust) that are considered to be the basic building blocks of all of our emotion expression range [172,173]. The player has the freedom to choose multiple emotion types and provide individual valences for each of the emotion types on a scale of 0 to 10. This is so that we may be able to derive more complex emotions from the data collected in the future [174].

4. Experiment

A small scale experiment with eight participants was conducted to investigate the user experience of the game. The player engagement and also the elicit emotional response as well as the decision-making patterns obtained in the wild were analysed to assess the game. Pilot studies such as ours are important in quickly validating research areas that have been unexplored, and can help identify potential research areas for further exploration [83], which, in our case is the application of a mobile SG in the wild for emotion and decision-making pattern elicitation. The ‘in the wild’ application of our game is the main novelty factor of our study, as the game paradigms we implemented as our NPC interactions had previously only been used for in person, controlled experiments in the literature. In this section, we discuss the core methodologies that were employed to obtain the data for analysis for this study.

4.1. Decisions of Interest and Emotion Data

In our game, the Decisions of Interest (DoI) are the socio-economic interactions the player character has with the NPCs. Brief descriptions of expected decision-making and emotion elicitation patterns from the literature for studies that used SGs under controlled settings for each type of SG we utilised as DoI interactions in our game are given in Table 1.

The decision-making data was composed of several features of each DoI interaction. These features, their descriptions, value ranges and their types are given in

Table 2. Interaction features and their details.

Feature	Description	Value Type	Value Range
Decision Type	These will be scenarios that will be supplied to our player during the gameplay and will consist of the different games we identified from the literature to be important for depressed individuals	Categorical	Ultimatum Game Dictator Game Trust Game Prisoner’s Dilemma
Decision	The decision taken by the player in the game	Categorical	Yes/No/ Made_offer/No_offer
Resource	The type of an item of finite quantity that is exchanged during an interaction in the game	Categorical	Food/Money
Integral Emotional Response	The emotion valence for the six basic emotions that is self-reported by the player	Continuous	Positive Real Numbers
Reaction Time	Amount of time taken by the player to make a decision of interest in the game	Continuous	Positive Real Numbers
Interaction Speech Polarity	A measure of how harshly the NPC speaks with the player	Discrete	Integer Numbers (−5 = n = 5)
Interaction Expression Polarity	A measure of how negative or positive the NPC shows facial expressions to the player	Discrete	Integer Numbers (−5 = n = 5)
Prior Decision	The decision activity the player was involved in prior to the current one, i.e., its value will be NULL for the first Decision-of-Interest (DoI) interaction	Categorical	Ultimatum Game Dictator Game Trust Game Prisoner’s Dilemma NULL Non-DoI
Incidental Emotional Response	It is the integral emotional response for the emotional state of the player prior to entering the current DoI interaction	Emotion Vector	Positive Real Numbers
Interaction Fairness	A measure of how fair the DoI interaction is	Discrete	Integer Numbers (1 = n = 5)
Player Role	Whether the player is the initiator or the recipient of a decision in a DoI interaction	Categorical	Ultimatum Game ∘ Responder ∘ Proposer Dictator Game ∘ Dictator ∘ Responder Trust Game ∘ Trustor ∘ Trustee Prisoner’s Dilemma ∘ Prisoner ∘ Jailer

. The following quantities are necessary to understand the context of the emotional responses of the players: decision type, decision, interaction fairness and player role in the interaction [8,20,22,25,26,27,28,29,175]. Both resource and reaction time quantities are important in understanding the motivations of the players behind a certain decision [176]. The quantity ’reaction time’ was measured from the time a player was asked a question by a NPC and to the time when the player chose an answer on screen. It has been seen that the quantities ’interaction speech polarity’ and ’interaction expression polarity’ play an important role in not only setting up the tone of the game but also have an impact on the emotional responses from the players [26,151,166,168,169]. Integral emotional response contains the emotional self-report response of the player during each interaction and holds the six basic emotions’ valences. Since the internal processes that are responsible for elicitation of emotion are impacted by the actions we take in an instant and by our current emotional state, it is important to keep track of the previous emotional states and the context of that emotional state in order to understand the current emotional state of a person [1,2,3,4,5,6,8]. Prior decision and incidental emotional response are the immediate previous interaction and emotional response of the player during that interaction. These quantities are captured in real time utilising custom scripts that run in the background of the game and record the interaction data whenever a DoI interaction has taken place. For ease of analysis of the emotional responses to be elicited, we have kept the interaction features, which include interaction speech polarity, player role, interaction expression polarity and interaction fairness, fixed for each socio-economic interaction. Since this is a preliminary analysis of the user experience of such a game in the wild, there will be different versions of this game where, for instance, the player roles can be changed and so, the data collection pipeline for these features are established in this study. The emotion data had six dimensions, each a basic emotion, along with a valence for each type of basic emotion from the self-reports by the players. The emotion data self-reported after each DoI interaction would be the integral emotion of the player for that interaction and will later become the incidental emotion of the player for the next DoI interaction.

4.2. Methodology

We addressed the challenges mentioned in Section 1 mostly in the designing stage of this study. In order to ensure that the emotion and decision-making data collected during our experiment were primarily due to the players’ interactions with our game, the player engagement was measured using the Game Engagement Questionnaire (GEQ) [67]. The full GEQ used for this study is provided in Figure A1 in Appendix A. A game designed for emotion elicitation will involve several ethical considerations, primarily as player privacy and informed consent [74] and potential psychological harm [66]. In order to ensure no ethical considerations were overlooked, a formal ethics review was conducted by Queen Mary Ethics of Research Committee (QMERC) and they deemed the study to be low risk in nature, subsequently providing their approval for the experiment. More specifics for the said ethics review can be found under the “Institutional Review Board Statement” at the end of this paper.

For our experiment, we used a “within-subjects” experimental design, which meant we did not have a control group as this approach has been found to be effective for conducting experiments in real-world settings, which was not controlled, hence the term ‘in the wild’ [177]. Most of the previous studies utilised socio-economic game paradigms in person gameplay arrangements, which can introduce some unavoidable contextual variability in the their studies (as discussed in Section 2). ‘In the wild’ settings overcome these hurdles by letting the participants experience the experiment in the comfort of their own choice of place and time, that greatly improves the probability of capturing patterns of behaviour and affect that can only be seen in the real-world [177,178].

The small-scale data collection experiment was conducted with ten individuals for a preliminary study in the wild, out of which eight agreed to share their emotional response data captured by the game software. Participants were selected based on not having any neurological conditions or affective disorders, such as autism or depression, and on having access to an Android device. The participants were from different age groups and academic backgrounds, including two teenagers, two people in their 20s, and four people in their 30s. The participants played the game on their own Android devices in the wild. The number of participants was chosen based on the recommendation for preliminary user experience assessment, which suggests that about five participants are sufficient to identify 90% of usability issues [179]. While the sample size of this pilot study was too small to yield statistically significant results, it is crucial to conduct exploratory studies similar to ours, as such studies can play a significant role in promptly confirming the viability of research areas that have not been thoroughly explored before, as well as in pinpointing potential areas for further investigation [83]. The study outcomes could shed light on the effectiveness and feasibility of using mobile SGs for eliciting emotional responses in diverse situations and settings, including in the wild, making it a valuable area of research to explore [133].

5. Analysis

This section presents the analysis that was performed on the emotion and decision making data from the pilot experiment, and for the sake of the ease of describing the results obtained from the analysis, Figure 9, Figure 10, Figure 11, Figure 12 and Figure 13 along with their discussions will use the acronyms UG, TG, DG and PDG (ultimatum game, trust game, dictator game and prisoner’s dilemma game, respectively).

Figure 9 shows the emotional responses of the players for all of the four DoI interactions (UG, TG, DG and PDG) in our game. From Figure 9a, we can see that the dominant emotional response for the UG interaction was Anger (average of 6.75) followed by Surprise (average of 3.13). Similarly, Figure 9b,c show that the dominant emotional response for the TG and DG interactions were Happiness (average of 7.88 for TG and 7.25 for DG) followed by Fear (average of 1.38 for TG and 1.5 for DG). Finally, Figure 9d shows that the dominant emotions were Anger (average of 9), followed by Sadness (average of 8.25) and Disgust (average of 5.88).

To understand the emotional responses better, they were aggregated into ‘positive’ and ‘negative’ emotional response categories. Figure 10 shows the aggregate emotional responses of the participants for each of the four DoI interactions. We grouped Happiness and Surprise as ‘positive emotions’ and Sadness, Anger, Fear and Disgust were grouped together as ‘negative emotions’. This was performed by averaging the emotional responses of each participant and then adding the averages up for the positive emotions and since there are twice the number of negative emotions in the six basic emotions in our emotion spectrum, in order to keep the negative and positive emotion quantities comparable, the total of averages for negative emotions were halved, essentially making it a weighted average of negative emotions. The interactions with negative polarity in speech and expression of the NPC as well as having negative fairness (UG and PDG, as mentioned in Section 3.4) elicited higher total negative emotions (Sadness, Anger, Fear and Disgust) and for the interactions with positive polarity in speech and expression of the NPC as well as having positive fairness (TG and DG, as mentioned in Section 3.4) elicited higher total positive emotions (Happiness, Surprise).

Figure 11 shows the reaction times for each player during the DoI interactions. The reaction times for almost all of the participants for PDG was higher than the other DoI interactions and as a result the average reaction time for PDG is higher as well, which can be seen in Figure 12. Visualising the average of the reaction times enables observing the general trends in the reaction times of the players when they made a decision in the DoI interactions.

In regards to the decision making patterns, Figure 13 shows the percentage of participants that either accepted an offer, made an offer or chose to cooperate with the NPCs of the DoI interactions, in other words, had a positive impact on the interaction from the player’s perspective. For all of the interactions, more than half the participants’ interaction decisions resulted in a positive impact on the interactions. A positive impact on an interaction was considered when the decision had an increase in the resource associated with that interaction (money for UG and DG, food for TG and PDG) [8,20,22,25,26,27,28,29].

An important aspect to be taken care of for games involving emotion elicitation is player engagement, as without proper engagement [67,68,69,70], it is not possible to establish whether the emotional responses were indeed resulting from the game interactions or from elsewhere [54,67] and the difficulty in measuring this is especially notable for in the wild settings. The GEQ used in this study included 19 questions that were categorized into four aspects of player engagement: Immersion (question 1), Presence (questions 2 to 5), Flow (questions 6 to 15), and Absorption (questions 16 to 19) [67]. Each question had three possible answers, “No”, “Maybe”, and “Yes”, which were assigned a score of 1, 2, and 3, respectively, in a Likert scale. The responses for the questions in each aspect were averaged for each participant, and the resulting scores are shown in Figure 14. The average score for Immersion was 2.4, for Presence it was 1.8, for Flow it was 2.3, and for Absorption it was 2.2. The average trend indicated that players tended to respond closer to “Yes” for Immersion, Flow, and Absorption compared to Presence. This was expected since the questions for Presence included statements such as “My thoughts go fast”, “Things seem to happen automatically”, “I played longer than I meant to”, etc. (a full list of all GEQ questions can be found in Figure A1 in Appendix A) [67].

6. Results and Discussion

We believe that the conducted experimental analysis provides significant insights into the main goal of our study. By developing a mobile SG designed around socio-economic interactions with the associated design decisions explained in Section 3.2, we were able to replicate the emotion response and decision-making patterns in players in the wild which were previously observed in the literature under controlled settings. The decision-making patterns for each of the DoI interactions were in line with the expected decision-making behaviour presented in the literature under controlled settings [8,20,22,25,26,27,28,29] based on how the DoI interactions were designed. More importantly, the emotional response patterns observed for the different DoI are also in line with the emotion elicitation patterns that were seen in the literature, for instance, UG interaction was designed with negative speech polarity and an unfair offer from the NPC, which was expected to elicit high valence of negative integral emotions, more specifically, Anger in this case, but we also observed Surprise as the second most dominant emotion, indicating that the players were caught off guard by the unfair offer at the very beginning of the game. The integral emotion of UG became the incidental emotion of the players for the subsequent TG and DG interactions, which were designed with positive speech polarity and positive outcomes of the interaction which were expected to induce positive integral emotions in the players and in this case, it the most dominant emotion was Happiness, however, Fear was the second most dominant emotional response. This, for TG, could be attributed to the fact that the interaction polarity prior to that (UG) was negative and so the player was weary of trusting another NPC in the game and for DG, since it involved forgoing some in-game resources from the player, such interactions are bound to produce some levels of stress in player [180]. Finally, the dominant integral emotional responses for the PDG interactions were also expected as betrayal in PDG was expected to elicit high levels of negative emotions, which in this case were Anger, Sadness and Disgust. These were further supported by the aggregated emotional response patterns which were in line with the literature.

Due to the elicitation of emotional responses and decision-making patterns with our game in the wild that are similar to such patterns observed under controlled settings, it can be argued that internal mental processes that led to the emotional elicitation patterns observed in the literature are similar to the ones in the players of our mobile SG in our in the wild experiment. As discussed earlier, emotional responses and decision-making patterns are intertwined, by the virtue of the integral emotion of one decision serving as the incidental emotion of the next decision in the game, our game’s ability to induce similar decision-making patterns as well as emotional responses in its players in the wild as compared to the controlled experiments further solidifies the notion that our game could be suitable for deployments in the wild for eliciting emotional responses in a predictable manner.

In terms of player engagement, data collected from GEQ responses indicate that players were sufficiently engaged for us to assume that their reported emotions were resulting solely from the game interactions. However, the average for the ‘presence’ aspect of the GEQ was lower than the rest and a possible reason could be that the game required a conscious effort to interact and take decisions [181]. Therefore, it is likely that the players would feel a higher cognitive load while playing our game than in other games they played before, naturally leading to more frequent ’No’ and ’Maybe’ for questions relevant to the presence aspect of engagement. It is also known that third person games typical induce lower feeling of presence than first person games [182] and that presence in the virtual game environment often decreases with increasing cognitive loads [183,184]. Another reason for the low average for presence could be the fact that all NPC interactions were socio-economic scenario based and there were no other obstacles for the player to overcome, thus rendering a form of monotonicity in the interactions [85,185,186]. Other than these, there were no significant differences between the responses of different age groups, meaning that the game was nearly equally engaging for different ages.

In addition to the above, the reaction time being higher for PDG than any other DoI interactions for almost all of the participants could be explained by he fact that the PDG interaction was the most cognitively taxing interaction amongst the rest [181] and also, that being the last interaction in the game could mean that the players could have become tired by the end of game play. The steady increase in the reaction time also indicated that the players were focusing during the DoI interactions at greater levels as they progressed in the game [176].

7. Conclusions and Future Works

The study of emotions in decision-making in socio-economic scenarios is a well-established research field. However, to the best of our knowledge, the literature has presented studies in controlled environments only, particularly because of the difficulty of obtaining accurate measurements in the wild. The recent spread of cellphones and advances in mobile computing technologies opens new perspectives that we have explored in our research work developing a mobile SG for emotion elicitation in the wild. This article has presented the requirements and consequent design decision choices, for instance in terms of extremely limited exploitation of gamification features, giving priority to pleasantness of interaction and supporting the player to focus on them, without being distracted by extrinsic factors (e.g., score, leaderboards). Particular attention was devoted to develop a proper, engaging narrative, and we succeeded in integrating in a single narrative multiple socio-economic-scenario themed NPC interactions, which we consider a significant novelty in the research landscape. Results of a preliminary experiment dedicated to assess user experience are promising, since data about elicited emotions are similar to those observed in previous studies under controlled settings. The GEQ responses ensure that the engagement of the player with the game was good enough for us to argue that the effect produced was mostly, if not fully, due to the players interacting with the SG.

The sample size for the experiment is small as the study was a user experience analysis, which we deemed as a necessary preliminary investigation. By no means do we intend to validate any causal relationship between the emotional responses and the decision-making patterns observed. Our main contribution to the research is the characterization and assessment of a tool that is able to bring into the wild interactions so far employed only in lab, reporting promising initial results. A full-fledged quantitative study for the effects measured with more subjects is the fundamental next step for this research field.

This study also opens various other research perspectives. For instance, it would be interesting to understand if a different sequence of socio-economic scenarios, and different interaction parameters would elicit emotional response and decision-making patterns that still conform to patterns observed in the literature for the in person experiments that were conducted under controlled settings. It would be interesting also to compare the emotional responses of the players for different roles in the DoI interactions. Additionally, it will also be intriguing to explore emotion elicitation under ecologically valid settings in a lab to verify if the patterns observed are similar to the ones obtained in controlled settings. Research is needed to identify more scenarios, in order to increase the diversity of the interactions, which is a key factor for a game. The GEQ responses of the players indicated that they were not feeling being ’present’ in the game world, as much as they were immersed in the flow. We attribute this to the game interactions being cognitively taxing and some effort can be taken to design DoI interactions that may reduce cognitive load on the player, to ensure greater engagement. The introduction of gamification and playful elements (e.g., obstacles in the narrative sequence) could be carefully considered, in order to increase playability and the enjoyment factor of the game.