Towards a Framework for Designing and Assessing Game-Based Approaches for Sustainable Water Governance

Abstract

Most of the literature on serious games and gamification calls for a shift from evaluating practices to using theories to assess them. While the former is necessary to justify using game-based approaches, the latter enables understanding “why” game-based approaches are beneficial (or not). Based on earlier review papers and the papers in this special issue of Water entitled “Understanding game-based approaches for improving sustainable water governance: the potential of serious games to solve water problems”, we show that game-based approaches in a water governance context are relatively diverse. In particular, the expected aims, targeted audience, and spatial and temporal scales are factors that differentiate game-based approaches. These factors also strongly influence the design of game-based approaches and the research developed to assess them. We developed a framework to guide and reflect on the design and assessment of game-based approaches, and we suggest opportunities for future research. In particular, we highlight the lack of game-based approaches that can support “society-driven” sustainable water governance.

1. Game-Based Approaches for Water Governance Lack Systematic Assessments

Using game-based approaches in a water governance context is not new. For instance, the role-playing games “Water on the West Bank” [1] and “the river Wadu” [2] date to 1988 and 1989, respectively, while the simulation game “FishBanks” dates to 1993 [3]. At the moment, interest in game-based approaches in a water governance context is growing. Illustrative of this interest is the recent general assembly of the European Geosciences Union that featured many serious games on water [4]. In this section, we first define what we mean by “game-based approaches”. We then define the dimensions and variables that characterize them based on a review of the literature on water governance, decision analysis, and game design (both in gray (e.g., [5]) and peer-reviewed e.g., [6,7,8,9] literature). We then address the current state of assessment of game-based approaches.

1.1. Characterizing Game-Based Approaches for Governance

A wide range of games for water governance currently exists, such as serious games, role-playing games, simulation games, applied games, and educational games. To them are added the inter-related phenomena of “gamification” and “gamified applications”. Given the various definitions and interpretations of “games” and “gaming”, it is useful to provide a more detailed outline of the emerging landscape of game-based approaches.

A dictionary definition of a “game” is an activity solely aimed at leisure or fun, defined by rules, and unsolicited (intrinsically motivating) [10]. Games often offer a chance to win or lose. They also use various amounts of physical and/or mental skills, and luck [10]. The term “serious game” refers to games developed for a purpose beyond entertainment alone [6]. Serious games address societal challenges (e.g., ecological, social, economic, environmental, or a combination thereof) [6,11,12,13,14,15]. Serious games tend to combine elements of entertainment (e.g., fun, suspense, mystery, inquiry) with elements of learning (e.g., developing knowledge, insights, skills). Gamification or gamified applications, on the other hand, are applications or services that include some game elements as motivational affordances [16] to motivate desired psychological processes [11,16,17] to prompt particular desired behaviors [18]. The following elements are usually considered: Points, achievements/badges, leaderboards, clear goals, challenge, progression, feedback, status, levels, rewards, roles, story/theme [16,19]. Designing gamified application and services implies the intention that the game elements create engagement, as a game does [17,18,20].

The boundaries between “serious games” and “gamification” are inevitably somewhat blurry. For instance, a stakeholder workshop can be “gamified” by introducing sessions dedicated to playing a serious game focusing on particular desired learning outcomes. The serious game thus represents the game element designed to help achieve the workshop’s desired outcomes. The workshop remains a solicited activity, with introduction and debriefing sessions that are not part of the serious game. In addition, some serious games used during stakeholder workshops are not fully fledged, in the sense that they are not stand-alone or fully self-engaging. In this paper, we use the term “game-based approaches” to reflect the blurred boundary between the two terms and to recognize the multiple interpretations of them, all having some merit, that can be found in the literature. To provide more clarity, we now focus on the dimensions and variables that characterize game-based approaches for governance.

Based on recent and established literature on games and game-based approaches, we defined these dimensions and several associated variables (

Table 1. Variables of the dimensions of game-based approaches.

Variable	Description	Example References
GAME DESIGN & TECHNICAL ASPECTS
Immersive experience	Extent to which a game makes players feel that they are an intrinsic part of the game world through visualization and interactive stories	Dede 2009 [27], Burke 2014 [23], Zhou 2014 [25]
Levity	Extent to which a game draws target players into playing it and interacting with each other through an effective balance of seriousness and playfulness	Lankford and Watson 2007 [21], Burke 2014 [23], Hertzog et al. 2014 [24], Zhou 2014 [25]
Complexity	Extent to which the game design (i.e., data, models, game interface) resembles real-world complexity	Lankford and Watson 2007 [21], Zhou 2014 [25], Wesselow and Kleemann 2018 [26], Aubert et al. 2018 [6]
Game/motivational affordances	Extent to which game/motivational affordances are included (e.g., points, leaderboards, achievements/badges, levels, theme, clear goals, rewards, progress, challenge)	Burke 2014 [23], Hamari et al. 2014 [16], Seaborn and Fels 2015 [19], Aubert et al. 2018 [6]
Action- consequence feedback	The evaluation of actions taken to assess their effectiveness and to determine future action	Mendler de Suarez et al. 2012 [28], Plass et al. 2015 [29], Aubert et al. 2018 [6]
Flow experience	Amount of deep absorption that the game facilitates for players (i.e., symbiosis between challenges and the skills needed to meet them, skills neither outmatched nor under-used)	Csikszentmihalyi 1990 [30], 1997, Nakamura and Csikszentmihalyi 2002 [31], Sweetser and Wyeth 2005 [32]
PEOPLE & PROCESSES
Representation	Extent of diversity/heterogeneity of the viewpoints and interests of those who play the game	Simon and Etienne 2010 [33], Barreteau et al. 2014 [34], Hertzog et al. 2014 [24], Medema et al. 2017 [35], Wesselow and Kleemann 2018 [26]
Level playing field	The relative absence of hierarchy and power inequities among the participants	Zhou 2014 [25], Ubbels and Verhallen 2000 [36]
Open-endedness	The space for emergence where participants accept that goals cannot be pre-determined in advance	Zhou 2014 [25], Ubbels and Verhallen 2000 [36]
Internalization of motivation	Extent of players’ engagement to continue the game until the challenge is met	Rigby 2014 [17], Zhou 2014 [25]
Reflection/transformative learning	Amount of individual/collective reflection of players’ game experiences that is facilitated during the game	Lim et al. 2006 [37], Jean et al. 2018 [38]
Emotional involvement	Extent to which game play affects players’ bodily responses to, emotions about, and motivation for the outcome of the game	Zhou 2014 [25]
PURPOSE & OUTCOMES
Shared understanding of facts	Extent to which the game enhances players’ understanding of the ecological system in which they operate	Medema et al. 2017 [35], Jean et al. 2018 [38]
Shared understanding of values	Extent to which the game enhances players’ understanding of different perspectives/interests in the socio-ecological system in which they operate	Medema et al. 2017 [35], Jean et al. 2018 [38]
Commitment to solution finding	Extent to which the game enhances intrinsic motivation of players to find a solution to real-world issues	Csikszentmihalyi 1990 [30], 1997, Nakamura and Csikszentmihalyi 2002 [31], Hamari et al. 2014 [16], Aubert et al. 2018 [6]
Social capital	Extent to which the shared game-experience can form a collective memory, create a sense of togetherness/trust, and strengthen social ties	Lankford and Watson 2007 [21], Zhou 2014 [25]
Sense of ownership/self-organization	Extent to which the game creates a sense of individual and collective ownership in players through the game experience and outcomes	Zhou 2014 [25], Jean et al. 2018 [38]
Transference of learning	Extent to which a game enables players to capture the emerging complexity of game play, help make sense of individual and isolated experiences, and fill the transfer gap between the game and the world outside	Zhou 2014 [25], Wesselow and Kleemann 2018 [26]

). The three dimensions are (a) game design and technical aspects, (b) the diversity of people who participate and how game-based processes are facilitated, and (c) the intended purpose and expected outcomes of game-based approaches. Some variables could be associated with more than one dimension. The framework we propose in Section 2 influenced the present classification. An example of an associated variable is the degree of complexity, which refers to the extent to which the game design (i.e., data, models, and game interface) resembles real-world complexity [6,9,21,22]. Other variables are levity and transference of learning. Levity is the extent to which a game is designed to draw targeted participants into playing it and to collaborate with each other [23,24,25]. Transference of learning refers to the extent to which a game enables participants to capture the emerging complexity of game play, to make sense of individual and isolated experiences, and to transfer what they learn in the game to the real-world context in which they need to act [25,26].

Table 1 summarizes the key variables characterizing game-based approaches. This overview of variables is not meant to be exhaustive or definitive, but rather to indicate the wide range of possibilities that currently exist.

1.2. Assessment of Game-Based Approaches

The literature (see Section 2) reveals that game-based approaches in water governance tend to focus mostly on their potential to advance Multi-Criteria Decision Analysis [6], watershed governance [7,8], and water systems planning and management [9]. A common conclusion in papers studying game-based approaches is that research should go beyond suggesting potentials, and, instead, systematically assess how players experience game-based approaches and the degree to which they realize intended and unintended outcomes. In this section, we review the assessment of game-based approaches for water governance.

A review of game-based approaches for water management and planning [9] reported varying degrees of assessment of serious games among several studies. It noted that some studies did not clearly indicate how players had played and experienced games, or what the game outcomes were. Other studies, however, did describe game sessions that were successfully completed, with clear evidence from participants who provided feedback using, for instance, pre- and post-game questionnaires. A more recent review [6] indicated that many games developed and played in the water sector have not been adequately studied. Another review [39], focusing on the use of serious games to address sustainability issues by increased social learning, agreed: Few systematic assessments of game-based approaches exist.

Likewise, Soekarjo and Oostendorp [40] conclude that none of the 15 studies they assessed used control treatments, or if they did, they provided no information about them. In the closely related field of climate change, assessment of game-based approaches also varies widely [41]. Reviews report either a lack of proper assessment (e.g., small sample sizes, lack of control treatments) or equivocal assessment results (e.g., supporting the benefits of game-based approaches only under specific conditions or for specific user groups).

A recent review of game literature [18] calls for better scientific design in the emerging field of gamification science. In particular, it calls for more reliable constructs, internal validity studies (i.e., control treatments to manipulate study variables, systematic experimental design), and external validity studies to replicate and cross-validate the findings (e.g., using a heterogeneous sample or multiple samples). The general insight is that systematic scientific assessment of game-based approaches is essential to determine whether they fulfill the purposes and objectives for which they were developed. Systematic assessment would also contribute to the development of gamification theories, which hardly exist at present.

Hence, even though the “did it work?” question needs to be asked and researched, this question is insufficient in itself [7]. Following the trend in economics and operations research [42], research on game-based approaches should also contribute to a deeper understanding of people’s actual behavior and help unravel the reasons for, or processes leading to, that behavior. Along these lines, Seaborn et al. [19] distinguish gamification-in-action (i.e., did it work?) from gamification-in-theory (i.e., why/how was the observed outcome reached?). Developing a deeper and more detailed understanding of how and why certain game-based approaches and processes lead to specific outcomes is receiving growing attention e.g., [18,43,44]. This will likely contribute to further development of gamification theories. This will also inform frameworks that not only will help understand how gamification works but also help improve game design, which will ultimately lead to (more) effective serious games. In short, there is a need for systematic assessment of game-based approaches to (a) properly assess their benefits, (b) develop theories, and in turn (c) design effective approaches.

In the next section, we develop a framework to reflect on, in a structured way, the diversity of existing game-based approaches used for sustainable water governance. Using this framework should help researchers to systematically assess (and design) game-based approaches. Thus, the framework aims to help address the limits identified above. We then present examples, structured according to this framework, using the papers in this special issue (SI) of Water entitled “Understanding game-based approaches for improving sustainable water governance: the potential of serious games to solve water problems” [45]. We conclude by discussing key findings of the assessment of game-based approaches for water governance as well as windows of opportunity for further research.

2. Framework to Map the Diversity of Game-based Approaches

2.1. Methodology

Our literature review focused on water governance and/or decision-analysis, and game design. We searched the Web of Science for recent (2015–2018) reviews at the intersection of these topics and consulted some of the references that these reviews cited. We supplemented this thematic literature review with a review of recent literature that focuses on the interface between society and the environment (e.g., the Open Traceable Accountable Policy [46] or the related Record of Engagement [47]). Identifying similarities within this extensive body of literature, we developed a framework that helps inform and structure the design and assessment of game-based approaches.

Much of research in this vein provides good-practice guidelines for participation in policy development. A recent theory of participation [48] proposes moving from Arnstein’s ladder of participation [49] to a “wheel of participation”, defined as having two major dimensions: (a) Hierarchy (e.g., bottom-up participation vs. top-down instruction) and (b) interaction (e.g., one-way communication vs. two-way deliberation or co-engaged co-production). The theory’s authors suggest that the type of participation itself does not predict its success. Instead, the success of any participatory process will depend on (a) the context (i.e., previous experiences and failures or successes with participation), (b) the design (i.e., representation of the parties, transparency and structure), (c) the power (i.e., management of power relations), and (d) the scalar fit (i.e., spatial and temporal scales) [48].

This wheel of participation resembles a heuristic (i.e., a tool to promote discussion) developed in the context of education for sustainability by Jickling and Wals [50], who distinguish processes by what we consider to be the same dimensions. They refer to the “bottom-up/top-down” dimension as the degree of authority vs. participation, and the “one-way/two-way” dimension as the extent to which outcomes are fixed (i.e., predetermined to a transmissive end) vs. open (i.e., emergent, jointly established to a socio-constructivist end) (Figure 1). They refer to the former as more instrumental and the latter as more emancipatory. In other terms, one can define “society-driven” processes as processes in which multiple perspectives are represented (implicitly involving many stakeholder groups and citizens) and that are co-designed.

This literature from various disciplines shared similarities, which helped us identify four important aspects for classifying and assessing game-based approaches (Figure 2):

• “What?”, i.e., the topic of the game-based approach (here, water governance)
• “Why?”, i.e., the purpose of the approach
• “Who?”, i.e., the stakeholders participating in the approach
• “When and where?”, i.e., the temporal and spatial contexts of the process

These aspects are in line with current game and gamification literature [12,20]. They also match the essential questions used to develop theories in the social sciences [51]. We argue that they inform (a) the three dimensions of game-based approaches and their characteristics (Table 1) and (b) the assessment of game-based approaches. The remainder of this section presents each aspect individually and ends by integrating them into a framework.

2.2. Why Are Game-based Approaches Used for Water Governance?

Water governance can be defined as “the set of rules, practices, and processes through which decisions for the management of water resources and services are taken and implemented, and decision-makers are held accountable” [52]. The Organization for Economic Co-operation and Development further specifies that water governance engages not only governments but also other stakeholders. Based on two popular management concepts for water resource planning and management—Integrated Water Resource Management and Adaptive Management [53,54]—water governance can be roughly captured by the iteration of seven phases, each with defined purposes (Figure 2, left side).

Many of these water governance purposes aim for some kind of learning (as defined in Reference [55]): From discovering, explaining, clarifying, changing (e.g., views) to creating (e.g., new alternatives). These types of learning can result from specific actions that lead directly to clear and immediate effects (“single-loop” learning) or from reflecting on the assumptions underlying these specific actions (“double-loop” learning) [56]. Sometimes, because of single- and double-loop learning, values and norms underpinning assumptions and actions are challenged, triggering “triple-loop” learning [57]. To some extent, water governance always aims to enable multi-loop social learning [58].

While learning is a common purpose of game-based approaches, engagement of participants is equally important. In their own review, Den Haan and Woort [39] report at least six other recent reviews about learning with and within serious games stemming from the educational sciences. Such educational serious games on societal challenges are also important for water governance because they help build awareness (phase 2 of water governance). On the other hand, Aubert et al. [6] suggest that game-based approaches are used mainly to create engagement in learning and other related spin-off activities. Creating or enhancing engagement is a goal for the entire spectrum of game-based approaches. The game elements included are usually affordances to make playing the game engaging and worthwhile [10,59]. Aubert et al. discuss how the engagement promoted by game-based approaches can benefit each step of a structured decision-making process ([6], Figure 1). Engagement can also be a means to achieve other outcomes, such as enhancing commitment to solution finding, creating social capital, and creating a sense of ownership of the decision (Table 1, purpose & outcomes).

2.3. Who is Involved in Game-Based Approaches Used for Water Governance?

By definition, a game is played by individuals or one or more groups of players. In the classification of Djaouti et al. [12], player characteristics are contained by the “scope” of the game. Recent reviews of game-based approaches for water governance [6,9] summarize several characteristics. For instance, game-based approaches can be single-player or multi-player. Some game-based approaches target a specific age group, but game-based approaches on water can be found for all age groups [6]. Players can vary from laypeople to experts. These two player characteristics (i.e., age and expertise) strongly influence the choice of the game-based approach. Most often, the targeted players come from a specific geographic area (e.g., due to the language used in the game, contextualization of the game to increase engagement). It is not necessarily exclusive, however; players from other regions can also use such regionalized game-based approaches. Some reviews e.g., [6,9,41] also highlight the involvement of a facilitator (sometimes assisted by other people). Flood et al. [41] point out that the experience of the facilitator and her/his knowledge on the topic strongly influences the success of the process.

Depending on specific water-governance phases and related objectives (Figure 2, left side), the following stakeholders can be targeted: Experts with a political mandate, politicians (with varying degrees of knowledge on the topic), technical experts (e.g., engineers, scientists), citizens enrolled in opinion groups (with some knowledge about the topic), and citizens as affected laypeople/end-users. In a multi-scale or multi-sector context, stakeholders can come from different sectors (e.g., agriculture, water supply). In multi-level contexts, stakeholders can come from different levels of decision-making or governance (e.g., local, regional, national) [60]. From a water governance perspective, the extent of representation (i.e., who participates and who does not) is important. In addition, consideration of previous relationships between participants can be critical, especially if there have been conflicts [48].

Considering this “who” question, several foci for studying the assessment of game-based approaches appear. One can study the game itself, independent of players (e.g., whether it is playable, how it works). One can study the individual, such as whether and how s/he reacts to the affordance of the game-based approach, or whether there are consequences for her/him (e.g., learning, engagement). One can focus on group dynamics, since interactions between individuals in a group often become complex and can lead to issues of power, conflict management, group thinking, etc. Depending on the purpose of the game and the stakeholder groups involved, a fourth focus would be the societal level, or the impact in the real world [61] that transcends the game-based approach itself.

2.4. What Are the Spatial and Temporal Contexts of the Game-Based Approach Used for Water Governance?

Games are characterized by spatial and temporal contexts, which, together, Huizinga calls the “magic circle” (i.e., the space and time of play) [62]. Within this magic circle created by the game, specific rules prevail. Game-based approaches last from a few minutes to several days [6]. Some game-based approaches enable players to take breaks, or they are developed as a sequential process, defined by the number of sequences (e.g., different scenarios), their frequency (i.e., time between two sequences), and whether there is a difficulty gradient (i.e., levels within a game). Regarding the spatial aspect, two types of spaces are reported [6]: Physical (e.g., meeting room, classroom) or virtual (e.g., online). The latter creates an additional option for the temporality of multi-player approaches: Contributions can be asynchronous.

In a water governance context, one-time meetings occur according to the phases previously described (Figure 2). Some of their outcomes should last long after the meeting. Water governance is sequential and iterative. In vertical governance, some of the stakeholders involved may be detached from the local water issue. In other cases, administrative boundaries do not match watershed boundaries [8]. These aspects could complicate physical meetings. Some patterns of scalar fit between space and time can be observed: a mediation process would most likely involve stakeholders within a local network, over a short period [48]. In contrast, deliberative democratic processes most likely involve stakeholders at a large scale (e.g., national) over a long period [48]. A final temporal aspect to consider is whether deadlines are set, which influences the time available to prepare meetings.

2.5. Framework to Define the Most Appropriate Game-Based Approach and Assess It

We argue that combining answers to the questions “why”, “who”, and “where and when” will help to (a) design the most effective game-based approach to reach the targeted outcomes and (b) assess the game-based approach, adequately monitoring the achievement of the targeted outcomes (Figure 3). In particular, answers to the “why” question guide the choice of the theoretical background to use to design and assess the game-based approach. Answers to the “who” question influence the focus of study: The game itself, the individuals involved, the groups involved, or the organization/society. Combining the answers to the “why” and “who” questions fully determines the focus. Combining answers to the “when and where” and “why” questions determines the research design (i.e., the methodological plan to collect assessment data). In certain spatial and temporal contexts, qualitative inquiry may be more appropriate than quantitative assessment.

Finally, design and assessment can focus on the outcome (i.e., variance approach) and/or on understanding the process (i.e., process approach), as is defined in the assessment of decision-making [63]. Combining both approaches, researchers assess not only whether the expected outcomes have been achieved (i.e., variance approach) but also how and why they were achieved (if at all) (i.e., process approach). These two complementary approaches involve different scientific methods, and thus imply a different research design and/or focus.

As mentioned, this framework encompasses the three dimensions of game-based approaches and their variables (Table 1). In particular, the “why” question addresses the purpose and outcomes, the “who” question partly addresses the people and processes, and the “where and when” questions address the context and strongly influence the design and technical choices (“how” question). In the following section, we use this framework to review the game-based approaches featured in the SI. Based on the combinations of “why, who, when and where” in them, we identify similarities and differences in the use and assessment of their game-based approaches.

3. Results

3.1. The Context

3.1.1. Reasons for Using Game-Based Approaches and Topics Addressed

Predominantly, the game-based approaches for water governance described in this SI aim to (a) facilitate understanding of the overall complexity of the real world; (b) foster stakeholder collaboration, cross-sectoral integration, and/or multi-stakeholder dialogue; (c) experiment with multiple scenarios in a safe trial environment; (d) facilitate capacity building; and (e) overall, enable single- to triple-loop learning [64,65,66,67,68,69,70,71,72]. These aims correspond mostly to phase 2 of the water governance process (i.e., create stakeholder awareness and trigger multi-stakeholder dialogue), as well as phase 4 (i.e., creating an enabling environment) (Figure 2). Rodela et al. [70] call these game-based approaches “learning interventions”. In addition, Galván-Pérez et al. [73] reviewed 20 game-based approaches (“educational videogames”) that aimed to educate players in ecology, to raise awareness. Although not explicitly mentioned, the connection to water governance is that the younger generation will govern in the future. Presumably, the game authors aimed to prepare an enabling environment for sustainable future water governance.

The game-based approaches described in the SI papers focused on different water topics: Water supply safety plans at the watershed and household scales [64], water supply in peri-urban areas [66], local to transnational maritime spatial planning [65,67], flood mitigation in urban-rural watersheds [68], water resource management at the watershed scale [69], mangrove shrimp farming [70], the water-energy-food-land-climate nexus [71], how water managers perceive “models and interactive interfaces (serious games)” [72], and a review of games on the water cycle, water and aquatic ecosystem management, and the human right to water [73].

3.1.2. Stakeholders Involved

The game-based approaches described in the SI papers targeted diverse stakeholders. Overall, each of the studies included the key stakeholders concerned with the issue, from citizens (as end-users) directly impacted (e.g., [66,68]) to those involved in the decision-making process (i.e., stakeholders with a political mandate, such as water authorities) (e.g., [69,71,72]). Thus, the degree of expertise of those involved varied from being a layperson (e.g., [66,68]) to being an expert in the field (e.g., [65,72]). In the case of role-playing games, some of the participants sometimes played the role of another stakeholder not involved in the game (yet being represented in it) (e.g., playing the role of farmers [64,69]).

This diversity of stakeholders encompassed different levels of governance (e.g., from local coastal organizations to national and international organizations in charge of maritime spatial planning [65,67]). Stakeholder diversity also involved different sectors (e.g., fisheries, energy, trade [65,67]). Sometimes, a game-based approach tried but failed to include participants from all relevant sectors (e.g., private companies and land development sectors did not join [66]).

All of these game-based approaches required at least one facilitator and often more than one. In addition to guiding the game-based process, facilitators sometimes played a role in the game itself (e.g., the Game Overall Director (G.O.D.) [67]). In most cases (e.g., [68,69,70]), however, facilitators played no active role in the game but rather helped participants understand the rules and the technology used. The targeted players described in the review of Galván-Pérez et al. [73] differed significantly from an adult audience: They were teenagers (around 12). Most of the games reviewed in this paper were single-player games, and none of the games involved facilitators.

3.1.3. When and Where Game-Based Approaches Were Used

The game-based sessions described in the SI papers occurred in recent years, all over the world, from Bangladesh [66] and Vietnam [70] to Europe (e.g., [69,71]). Zhou and Mayer [72] reported on experts’ perceptions of game-based approaches in China and the Netherlands. Jean et al. [67] focused primarily on a Canadian context, although they also included findings from Europe. Game-based approaches reviewed by Galván-Pérez et al. [73] were developed by European, American, and Australian organizations, as well as international committees.

Regardless of differences in the general setting, the spatial and temporal contexts of the game-based sessions were similar. Each of the game-based approaches discussed required interactive sessions (i.e., workshops that included specific game elements) (discussed in Section 3.2). These workshops required participants to meet in person; thus, they occurred in a physical space, usually a conference room or classroom. Sometimes additional arrangements were required to organize the room according to the required and available paraphernalia.

The workshops were also defined in time. They lasted from a few hours (e.g., [68]), to an entire day (e.g., [66]), to a few days (e.g., [67]). In one case, participants would have been willing to participate for even longer [66]. Workshops were divided into sessions. Most workshops started with a welcoming and game-introduction session. They then continued with the game experience itself, the longest session, which could be divided into a sequence of several rounds. The number of rounds depended on the total length of the workshop. Finally, the workshop ended with a debriefing session. Between the sessions and rounds, participants could usually take breaks.

The spatial and temporal contexts of the games reviewed by Galván-Pérez et al. [73] ranged from virtual space (e.g., online) to the classroom, with combinations being possible (e.g., players in a computer room playing an online game). The game-based approaches usually lasted from a few minutes to several hours. These game sessions were part of the school curriculum.

3.2. Design of Game-Based Approaches

Given their similar aims, targeted stakeholders, and settings, the game-based approaches described in the SI papers were somewhat alike. Primarily, they were simulation games: Based on models representing the real world, the stakeholders could “experience” consequences of their choices under different scenarios. Motivation to explore the scenarios came from a challenge given to workshop participants or to students in a course. The simulation game was sometimes combined with role-playing (e.g., [69]) or not (e.g., [68]). Simulation games described in the SI papers were multi-player games in which participants mainly had to perform various water governance steps in a safe trial environment to achieve the challenge at hand (e.g., [65,66,67,70,71]). The games created environments in which mistakes could be made without consequences besides the ability to learn from them, to reduce the chance of repeating them in the real world. In most cases, the games had well-defined rules. In a few cases, participants needed only to discover answers to guiding questions (e.g., [68]), and the rules of the game consisted of answering those questions.

We also observed some differences among the games. The technology varied from a board game (e.g., [66]), to a hybrid game (using a mix of physical and virtual elements) (e.g., [65]), to a virtual game (e.g., [68]). Partly due to the choice of technology, the degree of complexity represented varied. Several of the game-based approaches described in the SI papers used simplified models of stakeholder interactions that fit with the local regional context (e.g., [66]), but others were more complex 3D GIS simulations (e.g., [68]). Simplified representations of the complex real world were less realistic than complex representations of the real world. As the representation of the world became simpler, however, generality increased (e.g., representing no specific location [69]). The opposite was found as well: Complex, and thus more accurate, models were associated with specific geographic contexts (e.g., [71]). In turn, these differences led to varying degrees of play vs. seriousness. In the papers from this SI, we observed simulation games at the serious end of the continuum (e.g., [68]), while others were designed to offer a more playful experience (e.g., [70]).

The educational video games reviewed by Gálvan-Pérez et al. [73] were single-player computer games. In addition to simulation games, there were also adventure games, platform or question games, and tile-based games. They differed in the game elements that they included, for instance, in adventure games, the narrative was salient. The level of complexity fit the audience: the complexity was often simplified, and thus the realism somewhat low. The play dimension tended to be as important as the seriousness.

3.3. How Game-Based Approaches Were Assessed

The papers in the SI report various ways of assessing game-based approaches. As mentioned (Section 3.1.1), the aim of most game-based approaches was related to some extent to a learning outcome. Consequently, we found assessments based on Kolb’s experiential learning cycle [64], the knowledge co-creation cycle [67], the relational theory of multi-party collaboration processes ([69], though the authors did not refer to a specific seminal theory), and multi-level social learning (e.g., [67,69,70]). These various theories stress that learning is usually an iterative process, occurring when acquiring new knowledge, which in turn may create, change, or strengthen the belief of the individual. The last three theories listed suggest that knowledge is acquired through dialogical exchanges among individuals in a social context.

In some cases, theories were mentioned in reference to the design and conceptualization phase of the game. Among them, we found the constructivist learning theory [65], the Socratic method [68], and game theory [66]. The first two connect to the learning theories highlighted in the previous paragraph, but in these cases, they guided design of the game-based approach. The latter theorizes the behaviors of agents, in particular when they share a common and limited resource. Finally, Marini et al. [74] propose using Schwartz’s socio-psychological theory of basic human values to design game-based approaches in which participants could move beyond their self-interest to more transcendental values. Some papers did not refer to any theories (e.g., [71]). Galván-Pérez et al. [73] assessed game-based approaches by developing an integrated quality indicator based on an ISO standard applied to video games and inspired by the Social Discourse of Video Games Analysis Model.

We observed different foci of assessment. Most studies had two to three foci, including the game itself (e.g., playability [70], an integrated quality indicator [73]), the individual (e.g., individual learning [68]), and/or the group involved (e.g., interaction analyses [67]). Only one paper [68] reported an outcome of a game session with the focus on society. Nonetheless, others (e.g., [66]) also highlighted the importance of follow-up studies to assess the outcomes of game-based approaches for society.

We observed a variety of research designs. Some assessments were based on case studies involving the targeted stakeholders (e.g., [69,71]). Others were based on single to multiple experimental game sessions (e.g., [64,65,67]). In these cases, the participants were not necessarily the targeted audience but a sub-sample of it (e.g., students). Finally, two studies stood apart from the previous assessments. The study of Zhou and Mayer [72] consisted of expert interviews based on the comparative Q-method, independent of a specific game session. Comparing the Netherlands and China, it assessed how experts perceived this common type of game-based approach, including a cultural dimension. The review of Galván-Pérez et al. [73] included the building of expert consensus on the rating of 20 games using Delphi expert consultation. The assessment intended to explain the process of increasing educational potential using narrative and gameplay.

In the papers of this SI, we encountered both the variance (i.e., outcome) and process approaches. Most of the papers, however, focused on the variance approach (e.g., did the individual learn? e.g., [68]). One paper that did focus on the process approach [67] monitored the game session while trying to explain the interactive processes among players during the session. We found a relatively rich diversity of assessment studies.

4. Discussion and Conclusion

4.1. Reflecting on the Diversity of Game-Based Approaches

4.1.1. Strong Points

As discussed by Barreteau ([75], and references therein), a dominant game-based approach for water governance is simulation role-playing, during which workshop participants can make decisions in a safe trial environment. This approach is justified by the enhanced potential to communicate about the complexity of the real world, to enhance collaboration and/or sharing of worldviews, and thus to promote multi-level learning. Another common game-based approach consists of games to engage in learning and create awareness about environmental issues. Thus, overall, game-based approaches for water governance aim to engage stakeholders in some learning activity about the complexity of water issues.

Of note, all papers in the SI include terms relating to stakeholder participation, such as “engagement”, “collaboration”, “participatory modelling”, and “co-operation”. This indicates that studying game-based approaches through the lens of a theory of participation might be promising [48]. Most game-based approaches described in the papers of SI appear to be located in the “expert-driven sustainability” corner (Figure 1, upper left). In this corner, there is strong consensus among experts with a policy and/or science background about what needs to happen or what is the right approach or solution to address the issue in a sustainable way. This is particularly true for educational games, which promote a “correct” solution. Most simulation games described in the papers of the SI, with or without role-playing, were designed by experts.

Despite the clearly dominant aim of using game-based approaches for a water governance, we observed finer variations in the characteristics of the game-based approaches identified (Table 1). Although the list of variables is not exhaustive, and no measurable attributes exist yet to characterize game-based approaches objectively, we used another heuristic to help design and assess game-based approaches: A radar chart (Figure 4 and Figure 5). The radar chart shows key variables identified as continuums that range from low (center) to high (edge). Depending on the key challenge and main purpose, the starting situation for each of these variables is likely to differ, as are the expected changes in them resulting from using a game-based approach. Certain questions will need to be asked early in the process, such as (a) what needs to change (e.g., increasing knowledge, awareness, or understanding; optimizing current knowledge; engaging, co-creating or transitioning)? and (b) what is the nature of the problem (i.e., simple, complex, wicked)?

For instance, one can consider three hypothetical examples, each of which has a different starting points and different desired end points for each of several variables (Figure 4). Positions on the graph are meant to indicate, in a qualitative manner, results of discussion among the relevant stakeholders designing the game. Using the heuristic, they need to answer three questions: (a) Given the current water governance context, where are we in terms of each of these variables? (benchmarking); (b) given our overall purpose, what should we strive for, for each of these variables? (forecasting); and (c) given our starting situation and purpose, what game-based approach will work best to achieve it? (designing). The purpose of a heuristic is not to be precise or accurate but rather to get people to discuss and dialogue.

4.1.2. Blind Spots or Hotspots

First, reviewing the papers of the SI revealed that few if any game-based approaches currently exist in the “society-driven sustainability” corner (Figure 1, lower right). We found no game-based approaches with either representation of a wide range of worldviews (implicitly involving many citizens) or co-design of the approach. At best, the game-based approach featured different roles and worldviews [67,69], but the number of participants was limited and did not include all affected citizens. One explanation could be that the papers did not focus on the design itself, with one exception which reported that the design of the game-based approach included some local knowledge [70], as a step towards co-design; however, a third party initiated and led the design. The companion modelling approach, which sometimes also includes role-playing games, has a long history of co-design [34].

Second, the game-based approaches reviewed rarely addressed phase 7 of the water governance process (i.e., monitor and evaluate progress). This might indicate windows of opportunity to develop future research and explore the design of effective game-based approaches to support this phase and move towards more “society-driven sustainability” approaches. Two paths are possible to move toward this end, which addresses wicked, ill structured and ill-defined issues, for which no consensus on the sustainable solution exist. One can increase the representation by involving as many participants, and hence worldviews, as possible, and/or follow user-driven development (co-design). Addressing such wicked issues, or moving towards “society-driven sustainability”, will require boundary crossing and building new connections. The resulting game-based approaches designed to tackle such issues may differ from the existing, reported approaches.

4.1.3. Windows of Opportunity

Game-based approaches in the “society-driven sustainability” corner (Figure 1, lower right) deal with relatively poorly structured issues that have relatively high dissonances and distances physically, normatively, or both. Few precedents or examples of how a game-based approach might support these cases exist. In “wicked” contexts, issues are often nexus issues: They go well beyond technical water management issues, including phenomena such as boundary-crossing, co-creation of knowledge, and dealing with emergence and associated uncertainty. The question is whether game-based approaches can help within such a “messy” environment, and if so, what kind of game-based approaches and characteristics might work in such contexts. In particular, how do designers transfer the variables of purpose and outcomes (“why” in the variance approach) and of people and processes (“why” in the process approach) (Figure 4) into the game’s design and technical aspects (“how”) (Figure 5).

Similar to the goal of Dillon et al. [76], the point of positioning the SI papers in a heuristic (Figure 2, Figure 4, and Figure 5) is not to describe them precisely. Instead, it is meant to invite deeper and more thorough discussion about the parameters and dimensions that underlie water governance and help determine the types of game-based approach that may best address or overcome the given water governance challenge. For instance, designers can use a radar chart (Figure 4) to map the starting position of all variables of purpose and outcomes in a given context and then use another radar chart (Figure 5) to translate their agreed-upon configuration into a specific corresponding game design. Game designers may need to find a design (Figure 5) that builds trust between stakeholders who may not trust each other, find a way to arrive at a common language to discuss the issue at hand, or develop game mechanisms that can help participants deconstruct and reconstruct their worldviews and frame the issue–together. Future research could focus on how stakeholders can agree upon how to position or assess each variable represented in the heuristic. The game-based approach will also need to find a way to (a) address emergence and the iterative nature of reflection and co-creation, and (b) treat goals and knowledge as tentative, being subject to revision based on ongoing critical and collaborative dialogue, inquiry, and action.

4.2. Reflecting on Assessment of Game-Based Approaches

As observed in previous studies, assessment studies are diverse. The aim of this paper is to provide a framework to support future research related to game-based approaches. Assessing game-based approaches for water governance would mean assessing (a) whether the specific objectives of the water governance processes are met, (b) whether the game design actually helped reach the specific objectives that were set, and (c) how the game design influenced the process of achieving these objectives. In particular, we recommend developing—simultaneously—game-based approaches and the studies used to assess them, to develop research that enables the former’s goals and objectives to be “measured”. In addition, we suggest connecting the assessment to existing theories (e.g., participation theory, theories of trust building or behavioral changes) or, if no theories are available, justifying why that is so and helping to develop new ones. Referring to theories is valid not only for the assessment phase but also for the design phase [74].

We hope that this framework for assessing studies of game-based approaches will help develop rigorous and systematic assessment that matches the objectives advertised for a given approach. Alternately, we recommend designing research along the lines described by Landers et al. [18]. Such research is essentially trans- and inter-disciplinary (i.e., involving stakeholders outside academia across sectors, and involving scientists across academic disciplines, respectively [77])—for two reasons. First, gamification researchers are often “scientist-practitioners” [18], since they develop, assess, theorize, and implement their work. Second, game-based approaches for water governance also require anchoring the playfulness in reality and giving it a meaning, to follow the wording of Harteveld about triadic (serious) game design [6,78]. Performing a complete assessment would lead to many smaller assessment studies (e.g., for each focus, for the two approaches). Covering the entire range of questions means covering many disciplines, which seems unrealistic for a single researcher, as highlighted recently in socio-hydrology [79]. Thus, we recommend that future projects gather game designers, psychologists (for the individual focus), sociologists (for the group and society foci), experts from water governance, and experts from the water issue at stake.

Since water governance aims at decision-making for the real world, we call for longer-term assessment of governance processes conducted with game-based approaches in the real world. For instance, a tracer study was able to demonstrate a long-term effect on water resource sharing in rural areas of Tanzania [80]. This point relates to difficulties in transposing outcomes of the game-based approach to the real world, as reported in papers on the use of role-playing games for resource management [81]. Lankford and Watson [21] discuss it using the terminology of cognitive metaphors: The experience in the source domain (the gamified workshop) should be rich enough to trigger actions in the target domain (the complex real world). Others suggest that outcomes of game-based approaches are not reached in the spatial and temporal contexts of the game itself, but in debriefing sessions during and immediately after a game [82]. Thus, assessing game-based approaches for water governance would require assessing whether one-time game-based approaches have (a) a short-term effect or (b) a longer-term effect, and (c) whether and how longer game-based approaches have a long-term effect. In addition, one could assess whether and how spatial context has an effect or study the connection between the safe trial environment created in the game and the real world. As mentioned, how—if at all—do participants transpose their game experience to the real world? How does game design transcribe real-world complexity into a game setting?

Finally, assessing game-based approaches for water governance raises normative considerations. In particular, does the design of game-based approaches strengthen some characteristics of society? If so, which ones? Alternately, does it challenge society, emphasizing transgression? To what end are game-based approaches used?

4.3. Conclusions

Future studies could consist of developing questionnaires to rate characteristics and variables of game-based approaches as objectively as possible. We recommend starting by assessing the purposes, people, and processes of the situation, and then deciding on the game design and technical aspects. Doing so would allow an underlying governance question to drive game design, rather than a given technology. This would help ensure that the most context-relevant game-based approaches are developed. This would also increase the potential effectiveness of game-based approaches, which should support water governance rather than drive it. Clearly, using game-based approaches to address wicked water governance issues—in which knowledge often is unstable and contested, and multiple stakeholders hold multiple value positions, interests, and constructions of reality—poses challenges for game designers. The specific context, understood through talking to stakeholders and immersion, should drive the design in a trans- and inter-disciplinary process.

Again in line with Dillon at al. [76], the point of our framework is not that game-based approaches need to move from instrumental or outcome-based uses (e.g., implementing a preconceived outcome) to more emancipatory or process-oriented uses (e.g., developing certain capacities of players). Instead, the point is that before considering the use of any game-based approach, designers need to reflect on the nature of the water governance problem at stake. Doing so will influence the design and assessment of an appropriate game-based approach. We hope our framework will nourish these thoughts as well as further discussions about the design and assessment of game-based approaches.