Developing a Serious Game for Rail Services: Improving Passenger Information During Disruption (PIDD)

Abstract

Managing passenger information during disruption (PIDD) is a significant factor in running effective and quick-to-recover rail operations. Disruptions are unpredictable, and their timely resolution is ultimately dependent on the expert knowledge of experienced frontline staff. The development of frontline employees by their employers usually takes the form of practice reviews and ‘on-the-job’ learning, while academic education majors on theoretical approaches and classroom-based teaching. This paper reports on a novel industry-funded project that has developed a serious game (the ‘Rail Disruption Game’) that combines theory and practice to better manage PIDD for frontline staff in a UK train operating company (TOC). It defines challenges and the development method for the Rail Disruption Game; it also incorporates developer and user feedback. This paper provides insight into how to design, make and deploy a serious game as part of a gamified management process.

1. Introduction

Passenger journeys in the UK reached over 359 million in the 3rd quarter of 2022 [1]; the rail sector is a significant part of the UK economy, and is used by millions of passengers every day. In order to meet government targets as part of the Rail Technical Strategy (2012) [2], key elements of rail operation such as the management of information during disruption and resulting delays had to be addressed, as their effects negatively impact on the train operating companies (TOCs), their customers, their employers and the wider economy.

It is important to remember that while disruptions to the rail system can be mitigated, they can neither be predicted with any high degree of certainty, nor can they be eliminated. Consequently, to limit the impact of disruptions on customers, the effective management and communication of information on disruptions is vital.

Thus ‘Passenger Information During Disruption’ (PIDD) management is a critical issue for the UK rail industry, as it greatly affects customer satisfaction. For example, the Rail Delivery Group (RDG) (2017) found the following [3]:

• Only 26% of passengers consider PIDD is effectively managed;
• As much as 77% of passengers became aware of disruptions at the departure station during the journey, from other passengers or from the customer information system (CIS);
• PIDD was believed to be handled ‘fairly poorly’ or ‘very poorly’ by 54% of passengers;
• Disruptions made up to 75% of passengers feel frustrated, and up to 38% resigned and angry.

Overall, the RDG’s report concluded that passenger-rated information provision was ‘poor’, and identified the four areas most in need of attention to be (i) the ease of understanding the information provided; (ii) the relevance of the information provided; (iii) the delivery style; and (iv) the consistency of the information provided.

Therefore, it follows that improving frontline staff’s capabilities to manage timely, accurate and efficient provision of information during disruptive incidents would, in part, address these issues. This project, therefore, rather than being driven by a traditional academic-based research question, addresses, as its objective, the practical PIDD management challenge—manifesting itself as the development of a serious game for PIDD frontline staff in a gamified process of PIDD improvement.

The management of disruptions in rail operations (i.e., PIDD) is besieged with uncertainty regarding the current status of any incident, the likely outcomes of any interventions taken, and the tradeoffs between providing good customer service quality and meeting operational requirements. Further complications arise as TOC practices will often differ between regions, different TOCs and their internal operating functions (e.g., signaling, ticketing, on-train staff and platform staff).

Impactful learning approaches are needed to educate staff on the management of PIDD by providing opportunities to practice real-world scenarios in a risk-free environment, share practice and tacit knowledge, and identify opportunities for improvement. This is often difficult to achieve in traditional classroom approaches, which are inherently divorced from context, or through ‘on-the-job’ training, which during busy and complex disruptions can create extra stress, confusion and delay. In contrast, problem-based, social learning and ‘trial-and-error’ experimentation approaches can provide opportunities for learners to practise desirable behaviours, such as sharing experiences and developing effective team-working skills [4,5,6]. Thus, the use of a gamified learning approach in this context (e.g., PIDD) has the potential to provide a synthetic low-risk problem-based learning environment that is both authentic and realistic enough to engage and educate subject matter experts (SMEs).

Furthermore, it has been shown that serious games can improve business decision-making by engaging and motivating workforces, improving the efficacy of training outcomes and influencing the behaviours of new and existing customers [7,8,9,10]. Thus, it is apparent that developing a ‘game-based learning capability’ for a TOC would allow its user staff to develop a deeper understanding about PIDD, share contextual experiences and develop new ideas on how to further improve PIDD to benefit passengers.

This paper is provided in eight sections, including this Section 1. The next section (Section 2) provides background approaches and principles to serious game design. Section 3 details the successful methodology used in this project. Section 4 outlines the results of the project, namely the ‘Rail Disruption Game’ per se. Section 5 provides developers’ feedback on the methodology used. Section 6 provides developer-players’ feedback on an early prototype of the game. Section 7 provides end-user players’ feedback on the end-user versions of the game. The final section, Section 8, concludes this paper.

2. Background to Developing Serious Games and Gamification

The study of designing ‘serious games’ is a relatively new field that combines learning design principles with game mechanics and game logic [8,9,11]. The term ‘serious games’ denotes a dramatic convergence of games, e-learning technologies and pedagogical models to provide a rich, immersive virtualised or synthetic environment. By combining sophisticated education theories with new technology, serious games can tackle a broad spectrum of needs, ranging from corporate training and education through to emergency medical response [9,11,12]. Educational computer games (e.g., serious games) founded on pedagogical goals with appropriate game mechanics can effectively teach their player users through their intrinsic abilities to engage, motivate and influence behaviours. Specifically, use of serious games through storytelling, quests, rewards and competitions can create an environment in which serious learning can take place. Large organisations such as IBM, Microsoft, Dell and Cisco use serious games to train their workforces in business and management skills, ranging from basic compliance testing to the development of advanced leadership behaviours [9,10].

Well-designed serious games can also make entertaining learning that is both challenging and rewarding, since designers strive to create games that are both fun and have overt pedagogical elements [13]. Therefore, any serious game design methodology must seamlessly deliver engaging game characteristics and pedagogy into game topics, and into the mechanical rules of play. Hence, formal design methods for serious games are becoming defined [13]. However, an ongoing aim remains to provide evidence and experience for more specific contexts, since proven approaches for one context may not be applicable to other given contexts. Since there is such a diverse range of subjects, industry sectors, demographic and neural diversity in users of serious games, more empirical research is needed in this field. This project provides such evidence for frontline staff training in the UK rail sector. Furthermore, since existing e-learning development methodologies have had limited success when transposed to serious gaming, as they emphasise instruction and content with little gamification of learning, future serious game and e-learning pedagogy needs to be able to attract, engage and retain users [9,10] if it is to usurp more traditional methods.

To create useable serious games, there is a persistent need to involve stakeholders (e.g., to champion development projects and their potential end users) during the development process. Thus, all reasonable steps need to be taken to encourage stakeholder involvement regardless of context, supporting where possible co-location and open channels of communication across all stakeholders [9]. Furthermore, the multiple tenable perspectives of stakeholders must be imbibed through an objective research and development (R&D) process rather than via directive input, say, from ‘training departments’, or else a serious game can risk duplicating existing learning problems or fail to address root causes of problems [10,14]—which may have originated in training departments themselves. Similarly, serious games often fail to match learners’ expectations by adopting an approach that is too simulation-like and/or too deterministic, and other routes associated with more conventional educational material rather than probabilistic game-based learning. In some respects, simulation is almost paradigmatically opposite to gaming, as simulation builders strive to faithfully recreate reality, whereas serious game designers may readily sacrifice reality if reality itself becomes a barrier to user experience and enjoyment—particularly where deeper learning can be achieved through simile, metaphor and experiential peer-to-peer learning [10].

According to relevant literature, there are several useful frameworks focusing on the development of serious games [9,11,12]. These frameworks’ key pedagogical elements are summarised below [11]:

• Competition and collaboration: against, between or with other players, time or the game itself to achieve specific tasks and/or goals;
• Rules: mechanics defining how a game must be played;
• Choices and goals: users are given context-based decisions to make and take action;
• Challenges and tasks: game users are presented with aims to address;
• Assessment and feedback: to reinforce lessons learned.

Table 1. Pedagogical game elements with their attributes and mechanisms.

Pedagogical Game Elements	Attributes and Mechanisms
Competition and collaboration	Community collaboration, role-playing, contest, bonuses, score, timers, in-game rewards (e.g., game currency), inventories, leader boards.
Rules	Moving, physics, progress bars, levels, timers, scoring.
Choices and goals	Storytelling, missions, NPCs, dialogues, branching, puzzles.
Challenges and tasks	NPCs’ quests, progress bars, branching, puzzles.
Assessment and feedback	Game hints, non-playing-characters (NCPs) or facilitators, levels, lives, progress bars, dashboards, virtual currencies, progress trees.

classifies these pedagogical elements for serious games, with their attributes and mechanisms to guide developers [11,12]; these key pedagogical elements are used in the quantitative questionnaire-based evaluation of final end user versions of the game described at the end of this paper.

All of the elements in Table 1 are imbued into the new Rail Disruption Game to effectively manage disruption to rail operations (i.e., PIDD). Some attributes and mechanisms are also used where suitable. The design and play of effective serious games should ultimately be an appropriate mix of these pedagogical elements, attributes and mechanisms.

There are many challenges when porting serious game requirements from manual (paper or board) environments to digital environments, and it would be wrong to automatically assume that digital computer-based game environments are always the best solution. For this purpose, future research should focus on the effects of different learning platforms on specific learner groups [10].

Furthermore, it has been demonstrated that serious games need to be effective at learning transfer and engagement for users [10,15]. However, some research has focused on assessing learning transfer in game play in isolation without taking into consideration the end user engagement [16]. This is problematic, because if a serious game cannot engage its learners, then finding an adequate sample of experienced players with whom to assess learning transfer outcomes becomes almost unachievable [17]. This conundrum comes back to the earlier point made on the importance of involving potential users early on in any game-development methodology. Therefore, we go on to describe evaluation criteria (key pedagogical elements, Table 1, and PIDD criteria) in regard to two key factors: engagement and learning transfer [10].

Engagement has been measured in medical fields in applications such as stroke rehabilitation [18]. Burke et al. [18] defined game design principles for upper-limb stroke rehabilitation, and developed several games using video-capture technology. The evaluation approach used randomised control trials monitoring different playing behaviours between healthy subjects and stroke victims; the study showed positive early results. Other heuristic evaluation methods are also shown to be beneficial [10,19]. For instance, Pinelle et al. analysed reviews of 108 games, and identified twelve common classes associated with usability and engagement, which led them to develop ten usability and engagement heuristics [20].

Other studies focused on the development of innovative approaches for learning transfer in operations management [21] that question the effectiveness of knowledge transfer via traditional lecture and classroom-based approaches.

In applied fields such as rail operations and managing disruptions, non-traditional approaches such as virtual learning environments, live cases, role plays, business simulations, experimental teaching methods and group exercises have been used to good effect to increase effective transfer of knowledge [10,21,22]. Other gamified approaches for learning employ ‘co-production of knowledge’ and ‘experiential learning’ techniques [23] to achieve similar outcomes.

In operations management pedagogy, it is important to not only acquire technical knowledge or to recollect procedures, but to understand the role, importance and impact of operations within any organisation. Education of any type should, therefore, prepare students and/or users for the complexities of real-world business problems they will face, and provide realistic experiences of multi-criteria, heuristics decision-making carried out ‘on-the-fly’ that is familiar to managers’ normal practices and use everyday vernacular. Problem-based learning approaches (e.g., the planning and scheduling of PIDD) have shown some success when students/learners apply their existing knowledge to find newly acquired knowledge and to solve challenge-based problems. In operation management, these often tend to be strategic or process-based problems [23] where there is not one single best solution. In situations where existing knowledge is held in a group of learners, (e.g., practitioners), social learning theory suggests that learners can gain as much useful knowledge and understanding from their experienced peers as they can from subject matter experts (SMEs) [24] such as a trainer, coach or instructor hosting a serious game.

Often, when applying problem-based learning, the key issues are the authenticity of the problems and tasks being played, and the effective transfer of related outcomes into desirable learning skills; in turn, if conducted purposefully, this can create positive practical impact [25,26]. Directly addressing these issues makes experiential learning cycles more effective, and provides a shared “concrete experience” (Kolb, 1984). Experiential learning approaches are, hence, very relevant to managing disruptions (i.e., ‘the problem’) and for shared team-based learning outcomes involving the use of tacit organisational knowledge exchange via game-based social learning activities. This is because rewarding, engaging and memorable learning experiences can be created through ‘play as experimentation’ based on serious events [27] (e.g., a train collision, breakdown or points failure).

The application of serious games in the scenarios outlined above shows improvement in soft and cognitive skills, knowledge acquisition, content understanding, problem-solving and behavioural change to create understanding, empathy and motivation in participants [28,29]. For instance, for PIDD management, the Rail Disruption Game has led to a clearer understanding of disruptions, and promotes greater customer-focused behaviours among frontline staff.

An evaluation methodology is crucial to demonstrate whether a serious game is achieving its purpose. There are several generic evaluation methodologies that can be used in serious games, such as the 4DF Framework [30,31], the Technology Acceptance Model (TAM) [32], and Kriz and Hense’s framework for theory-based evaluation in serious games [33,34]. These, however, need adapting to the domain context for which the game has been designed and the specific evaluation method [10,35]. Possible evaluation methods include randomised control trials [36], focus groups [37], user narratives, interviews, qualitative inquiries, post-use questionnaires and quantitative analyses of game-engine data [38].

Even though serious games are an emerging and increasingly accepted form of training, sceptics will contend that serious games should be evaluated in ways comparable to any other educational medium, and affording them specific consideration with respect to evaluation diminishes the usefulness of any results [9,10]. Therefore, the methodological toolkit of a serious game evaluator needs to be innovative to be able to measure a novel game’s engageability, and knowledge transfer capabilities, and, to placate sceptics by approximating traditional training evaluation methods, be both broad and relatable. Often, the sector, problem and context dictate the most appropriate overall development, assessment and evaluation methodology. In this research, this was a customised new product development (NPD) process, with built-in assessment as part of development, and a post-completion evaluation.

It is shown from the literature that qualitative methodologies have been used extensively to assess and evaluate serious games—although one may argue their selection is probably more grounded in pragmatism than suitability [9,10]. Qualitative research can, however, be crucial in providing insight into learner behaviours and understanding, and, when conducted rigorously, form a basis to build quantitative structural models of assessment and evaluation [9,10]. However, qualitative findings alone, particularly with a limited sample size, are often one of the central criticisms of inadequate serious game evaluations. For these reasons, the Rail Disruption Game was assessed by discursive qualitative focus groups and evaluated by quantitative questionnaire methods.

The Rail Disruption Game (see Section 4) was assessed qualitatively by small groups of subject matter experts in focus groups in its early developmental stages, and evaluated by larger quantitative questionnaires completed by players (frontline staff end users were not involved in the development of the game) in latter (final) versions of the game.

3. Methodology: Developing a Serious Game for a Gamified Process

Although serious games have been around for a significant period of time, the literature regarding development methodologies remains diverse, and there is no definitive methodology for the development of serious games or contextual gamification [39]. Thus, to develop a gamified learning capability to improve PIDD management, a new product development (NPD) process was used, as shown in Figure 1, based on Tidd and Bessant’s methodology (2013) [40]; this was because, at the outset of the project, it was not certain that a serious game or gamified processes were necessarily the best solutions to the given industry-based challenge, as other learning, training, policy, practices or technology solutions may have possibly been more applicable.

Once the ‘ideas’ stage of the NPD process (Figure 1) was complete, it became clear that gamification of PIDD management and the development of a serious game was feasible and desirable. Hence, in order to define the content for subsequent phases of development (e.g., gamification and serious games), it was appropriate to consider frameworks and methodologies more specific to gamification and serious games. Of the known frameworks available, the 6D framework (Figure 2) [41,42] was considered to most closely align with and complement the NPD approach (Figure 1) initiated, and was therefore applied from that point onwards.

Stage 3 of the 6D Framework (Figure 2), and stage 3 of the NPD methodology (Figure 1), both involved the iterative prototyping of the initial and evolving gamification concept and serious game product to make sure that the ‘don’t forget the fun’ and ‘deploy the appropriate tools for the job’ aspects were achieved. Thus, Simonsen and Hertzum’s [43,44,45] ideas were incorporated into this creative process by piloting evolving ideas with subject matter experts (SMEs), obtaining feedback from players and the facilitator, and combining these with observations from the design team. Amendments to designs for the next pilot were included as appropriate. This merged and expanded stage 3 (in Figure 2) is shown in Figure 3.

The iterations of stage 3 (in Figure 3) were concerned with the following design factors:

• Mechanisms by which participants learned from the game given the nature of the PIDD scenario and the knowledge in the player cohort;
• Maximizing benefits gained by sharing tacit knowledge in risk-free discussions of PIDD scenarios;
• Identification of specific, representative and frequently occurring PIDD scenarios;
• Development of a game board and mechanisms capable of realistically representing the chosen scenario;
• Further interviews with supervisory management, social media teams, customer service staff and frontline staff at the TOC to gain more detailed insight into the interaction and local occurrences that are characteristic of PIDD scenarios;
• Development of questions to prompt discussion relevant to identified learning objectives;
• Identification and recruitment of a facilitator to manage the game;
• Designing and testing through ‘dry-runs’ of the game with the project team and subject matter experts to establish game playability and potential to stimulate desirable discussion.

The above frameworks were combined with relevant knowledge from other educational game developers [46,47,48] to provide the overall methodological approach used in this project, which is shown in Figure 4.

Literature on educational games and game design advocates a process such as that shown in Figure 4 because it is a user-centred design approach [39] that can achieve meaningful user immersion [46] and motivate users to engage with development methodologies and pride themselves on the end output—that being the gamified process and the serious game [41,42], per se. User-centred and participatory design approaches are closely aligned, and both attempt to capture and align designs to end users’ needs [47,48,49]. This is analogous to implementing a prototype, exposing the prototype to real use, evaluating use, and identifying desired change [45] in iterative prototyping cycles [49].

Members of the project team were as follows: a senior member of a commercial serious games developer, a user (the train operating company) and university researchers. The three perspectives provided creative tension in the team: pedagogical (the university), user (the TOC), and practitioner (the games company)—each having to be incorporated into the game design. Specific actions taken by the project team in each stage of this project (Figure 4) are outlined in

Table 2. Specific purpose and actions in this project with respect to methodological stages in Figure 4.

Stage	Purpose	Actions
0: Ideas (or ideation)
Define your business objectives. At this stage the business outcomes are clearly defined, which could be the basis for the development of gamification or a serious game	(1) Identification of broad objectives	Project framing focus group (executive and sponsors)
1: Prototype definition
Delineate your target behaviours. At this stage, specific actions and behaviours that players exhibit during serious games are defined. Describe your players. At this stage, characteristics of target end users of the serious game are defined	(1) Identification of end-user requirements and learning outcomes (2) Scope definition for a literature review of commercial game research (3) Identification of and acquisition of appropriate data for game development	Interviews and focus groups with director, management, and supervisory level stakeholders
2: Conceptual design
Devise your activity cycles: at this stage, preliminary mechanisms to be used in the serious game are developed. These are designed to engage players in the player journey through the game, thereby motivating actions and behaviours identified in the prototype definition	(1) Recruitment of appropriate development support (2) Refinement of requirements from the Learning and Development Dept./Manager	Recruitment of specialist game development company Development of learning objectives Validation of learning objectives by the Learning and Development Manager Identification of additional data needs and acquisition including interviews, supervisory management, customer-facing staff and social media teams Iterative design and testing with three subject matter experts within the project team Approval from the industry sponsor
3: Prototype and pilot
‘Don’t forget the fun’. At this stage, the serious game is assessed as to its playability by end users ‘Deploy the appropriate tools for the job’. At this stage, the mechanics, elements and structures within the serious game are assessed for their effectiveness in achieving the specified outcomes and modifications made as appropriate	(1) Validation and refinement of the initial design through pilot testing of the serious game with the subject matter expert end-user group	Iterative design testing and modification through a series of pilot studies with TOC front-line staff Learning objectives assessment through questionnaire
4: Refine, rework and review
At this stage, the serious game is validated in practice, and implementation plans are developed for an end-user organisation	(1) Further validation testing with the subject matter expert end-user group with minimal intervention from the project team	Live game workshops with non-participatory observation. Collection of additional participant feedback Live game use with TOC staff independent from the project team Integration of the game into the current learning and development and improvement processes within the TOC Commercial development activities

:

This design process (Figure 4) concluded with a design that was considered valid by subject matter experts in the end-user TOC, including the head of customer service quality and the learning and development manager. In addition, the game concept was reviewed and approved by members of the Rail Safety and Standards Board (RSSB) who are national experts in PIDD management.

The Rail Disruption Game is outlined below; however, due to brevity, it is not intended for this paper to describe the entire game play in great detail. Instead, this paper focuses on the design, development assessment and evaluation methodology of the game. Sufficient details about the Rail Disruption Game for readers to appreciate what was produced are provided, so subsequent evaluation sections are understandable.

4. The Result: The Rail Disruption Game

The result or outcome of this methodology is the Rail Disruption Game; it is designed for between six to eight players (e.g., TOC frontline staff from different areas) to work together as a single team. The aim of the game is for players to assist specific passengers (e.g., child, vulnerable adult, etc.) to complete their journeys in the best possible way within a defined time for a given disruption. A disruption scenario is presented in a video (on a digital tablet) which is placed in the centre of the playing board. A typical session will last between three to four hours. During the game, players are given a realistic scenario (two different scenarios were created: ‘points failure’ and ‘broken down train blocking the line’). In each turn of the game, players must collectively respond to a commonly occurring customer situation and request that occurs during such a disruption (e.g., a child with no phone or money and who may be lost, confused and panicking). The players must work together as a team, against the clock, to identify the best way to resolve specific customer problems, because the same advice cannot be given to every passenger. Therefore, experiential learning, engagement in the game, and transfer of tacit expert knowledge is vital to players’ success in the Rail Disruption Game. In each turn, players are also provided with realistic information updates on the disruptive incident (that may change and even contradict one another over successive turns and rounds of the game), and must provide, under time pressure, the best possible customer service advice in the current situation with the available information. Players must simultaneously consider how they could support work colleagues to get the rail network to return, as quickly and as safely as possible, to normal passenger service. An image of the Rail Disruption Game is shown in Figure 5.

Four turns of the game form a round, and there are four rounds (incident beginning, incident worsening, incident improving and incident after-effects) per game, with sixteen turns in total per game. After each round, the players are guided by a facilitator to reflect on the advice they gave to customers, and how it might have been executed better. Time is also provided to reflect on the advice players gave to other colleagues to help them return the network to normal service.

As the game progresses through rounds of play, the challenges presented in each turn become more complex—just as in the real world. When players have successfully completed the game, they are provided time to reflect on their overall collective experience and suggest ways in which PIDD management and day-to-day service operations can be improved, thus empowering them to make a difference to their roles and to their company’s operational processes and practices. These suggestions are collected by the learning and development manager and fed back to the operations director. Thus, the Rail Disruption Game becomes an important idea generating activity to improve the PIDD management process. In other words, the Rail Disruption Game has gamified the PIDD improvement process.

5. Developers’ Reflection on Methodology Used

This section presents a stage-by-stage reflection from the project development team on the methodology used.

The Ideas stage was part of earlier research in the organisation, which motivated the management team to think about more engaging ways to develop frontline employees. Thus, the Ideas (or ideation) stage identified potential approaches to the problem that aligned with the organisation’s strategic objectives. This was achieved by the engagement of the executive and sponsors/funders through focus groups. This stage defined the project’s scope and identified the need for a gamified learning capability (i.e., some form of employee engagement that would increase their knowledge and performance) to improve PIDD management. This stage did not specifically focus on the development of a serious game, but rather on a more general open solution to an organisational problem. Subsequently, through abductive reasoning, the serious game became a viable proposal, but ‘serious gaming’ or ‘gamification’ was not a preconceived solution looking for a problem; this open-mindedness at the beginning of the project was seen as a critical success factor to innovation.

The Prototype Definition stage developed a broad scope to clearly define the requirements of a gamified learning capability for both the organisation and the end users. Interviews were conducted with stakeholders at the director, managerial and supervisory levels to construct a holistic view of the organisational and behavioural requirements for effective PIDD management, and the characterisation of the target user group: namely, frontline employees. Potential approaches, informed by abductive reasoning from relevant literature and stakeholder views, were considered by the design team who concluded that a serious game would indeed be the most effective way of meeting known requirements. Subsequently, further operational information and frontline employee inputs were gained. Maximising stakeholder involvement and use of abductive reasoning were seen as critical success factors at this stage.

The Conceptual Design stage focused on defining activity cycles users would complete—leading to desirable outcomes. At this stage, experienced game designers were recruited to the design team to provide expert input to support educational professionals and subject matter experts (the end users). This stage involved the identification of specific learning objectives and the choice of a representative scenario to support activity cycles. The overall aim of the conceptual design was to enable frontline employees to discuss, practise, challenge and develop new organisational practices, guided by a facilitator, through game-based scenarios, in order to focus on managing customer interactions as part of PIDD management. Maximising interactions between players was seen to be a critical success factor for the game in this stage, which resulted in the conscious decision to make a board game rather than an online game.

The Prototype and Pilot stage developed the serious game concept into a useable prototype board game, which was initially tested by the development team to see if the game mechanisms and content had sufficient potential to generate appropriate discussions between players. At this stage, a facilitator, who was also a subject matter expert, was recruited into the design team, and thus able to provide feedback on the design’s usability. This was, in hindsight, highlighted by the facilitator as a critical success factor of the overall methodology. This stage resulted in a partially user-validated implementable prototype that was taken through to pilot testing with end users. The Prototype and Pilot stage involved iterations of assessment and design modifications, resulting in the final game design.

The ‘Refine, rework and review’ stage used ‘live’ assessment testing of the serious game with subject matter experts. Each game play was conducted under the control of the facilitator, as would be the case when in use, and passively observed by members of the development team. This stage required only minor changes to the design and branding, and resulted in a subject matter expert user-validated final design. The development team considered rapid iteration cycles to be another critical success factor.

6. Users’ Evaluation of Early Versions of the Game

The first use of the ‘final design’ was with six players from a pool of frontline employees who had no prior knowledge of the game or its development. The use of the game was considered to be a success by these players in as much that the players were actively engaged in discussion of the management of disruption for a period exceeding two hours. Players demonstrated ‘spontaneous empathy and insight to the customers’ states of mind’, and the pilot was considered ‘a positive and productive experience’ by all participants. The game was played to specification (the rules), with some flexibility to allow for constructive feedback as issues arose; this was supplemented by specific feedback from participants on completion. A structured evaluation questionnaire was completed, where players were asked to ‘agree’ or ‘disagree’ with a series of statements relating to engageability (questions 1–8) and knowledge transfer (questions 9–19). There was also an option to provide additional unstructured feedback: the results are summarised in

Table 3. Feedback from objective subject matter experts on how well the game addressed key engageability and knowledge transfer criteria (n = 32).

	Mean	Mean
Question	Agreement	Importance
(1) This is a competitive game	2.2	2.3
(2) The goals of the game are clearly defined	4.4	4.4
(3) The game rules are clearly defined	4.3	4.5
(4) The game rules are easy to apply	4.4	4.6
(5) The game realistically represents the range of choices characteristic of disruption	4.3	4.8
(6) The challenges (questions) are realistic and have a range of difficulty	4.4	4.7
(7) The coaching, debriefing, and feedback improve understanding of the management of PIDD	4.5	4.7
(8) The debriefing sessions enable an assessment of performance in terms of customer service	4.6	4.8
(9) PIDD-11 The game highlights different sources of information and when they should be used	3.9	4.4
(10) PIDD-17 The game increases understanding of TOC legal responsibilities to customers	3.4	3.8
(11) PIDD-13 A/B The game allowed me to assess and improve my customer service skills	4.3	4.6
(12) PIDD-15 The game makes it clear when additional information should be communicated to customers	4.2	4.6
(13) PIDD-33 The game increases understanding of the effect of disruption estimates on customer travel options	4.5	4.6
(14) The purpose of the workshop is clearly outlined	4.5	4.6
(15) PIDD-08 The game gives a structured approach to sense making for the customer	4.1	4.5
(16) PIDD-08 The game gives opportunities to suggest improvements to customer service	4.7	4.7
(17) PIDD-12 The game increases understanding of how to deliver the right information to the right people at the right time	4.6	4.7
(18) PIDD-13A/B The game clearly identifies the PIDD standard message descriptions	3.9	4.3
(19) PIDD-14 The game gives opportunities to suggest improvements to customer service	4.6	4.7

.

Players’ concerns focused on the specificity of game questions. This view was also shared by the facilitator and observers. Both observers and facilitator expressed concerns regarding the ‘flow of gameplay’ at some points, when it was necessary for the facilitator to drive the discussion. It was considered important to reduce the reliance on the facilitator so the facilitator could capture notes for organisational improvement discussion at the end-of-game debrief. A facilitator guide was also developed to assist facilitation of the game, including a more detailed introduction to the game focusing on potential discussion of customers’ perspectives, specific discussion topics for each scenario, a heuristic for scenario discussions, and space for note taking.

Game turn information was presented on a digital tablet, and positioned in the middle of the game board to promote a ‘huddle’ for group discussion, which was augmented with audio and visual cues (timers) to stimulate game flow independent from the facilitator. Players preferred the digital tablet in the centre of the game board rather than a computer projector beaming information on a wall. This was expressed by players when both digital tablet and computer projector were made available, as in the first uses of the final versions of game.

It was also observed that group discussion was influenced by the everyday job of a player (i.e., what they did for their everyday role), and a balance of different types of functional knowledge made for more engaging game play. The facilitator ensured that players’ discussions were kept on topic.

Changes to the final version of the game were well received by players as, through iteration, their concerns were addressed. Players’ discussions became more focused, and the facilitator was able to concentrate more effectively on in-game discussion. It was also believed that game flow would improve further as the facilitator gained more experience.

Following the positive response to early use of the game, the focus shifted to improving the depth of discussion from the game scenarios. These iterations eliminated questions and information in turns that did not generate significant discussion, or provided unnecessary repetition. Moreover, the identification and replacement of video images that did not promote sufficient consideration of a customer’s (i.e., avatar’s) emotions were replaced with more emotive images. The result of these iterations was a reconfigured set of scenario questions and images, including the replacement of ‘weak scenarios’ turns with new, more engaging topics based on disabilities, safeguarding and the inclusion of images that more clearly articulated customers’ states of mind, happiness or distress. This resulted in a revised final version of the game considered appropriate for ‘live’ testing without the game developers needing to be present: see next section.

7. End Users’ Evaluation of the Final Game

The Rail Disruption Game was then tested in four sessions/plays without any of the developers present. This section reports on the overall feedback of users on the final revised version of the game. Data were obtained from users through an evaluation questionnaire after the end of the session. The game was played by 32 subject matter experts across these sessions. Participants were experts in the area of rail disruption and were not part of the development team—and so were able to provide objective subject matter expert opinions on the game.

The Rail Disruption Game generated 199 PIDD process improvement ideas over this six-month evaluation period. This compares to only 34 ideas being produced in a previous six-month period using the original non-gamified PIDD management process involving approximately 800 employees, resulting in a 165-fold increase in improvement suggestions per full-time-equivalent (FTE) employee over the same timeframe. This fact was considered astonishing by the TOC. The authors believe this is the real validation of the Rail Disruption Game’s success, as the ultimate objective of this research was to improve the PIDD management process. For this to happen, PIDD improvement suggestions had to come from PIDD managers: in other words, those who work in PIDD management daily and are responsible for sustained change to occur had to be involved in its iterative improvement. The massive increase in the number of suggestions came through increased engageability of staff and staff-to-staff knowledge transfer during the game—which did not previously happen before this game was implemented.

To evaluate the Rail Disruption Game per se, players were asked to rate ‘agreement’ and ‘importance’ to game criteria relating to engageability (questions 1–8) and knowledge transfer (questions 9–19) that had been summarised from frameworks in relevant literature (c.f. bullet points in Section 2 for questions 1–6) and PIDD management, as set by the Rail Delivery Group (for the remaining questions). A Likert scale (1 = ‘strongly disagree’, 2 = ‘disagree’, 3 = ‘neutral’, 4 = ‘agree’, 5 = ‘strongly agree’) was used. The summary of responses is shown in Figure 6.

Each plot in Figure 6 is the mean average score for each statement with respect to the perceived ‘importance’ and perceived ‘agreement’ of the statement, as detailed in Table 3. Evaluation scores were awarded to the game experience immediately after players completed playing the game. It can be seen that the majority of outcomes, the learning objective and game design criteria were considered to have been achieved. In other words, subject matter experts on average ‘agreed’ with statements, and ‘considered it important’ in their role as managers of disruption. This result, using objective players who were not involved in the original design of the game, meant that the game had been successfully designed, was useable, and was fit-for-purpose. There were, however, two significant outliers, which were: ‘Statement 1: This is a competitive game’, and ‘Statement 10: PIDD-17 The game increases understanding of TOC legal responsibilities to customers’.

Responses to Statement 1 are interesting because literature on serious games and games, in general, states that competition is important in game design. However, the Rail Disruption Game, considered as successful overall by its users, was perceived as not being overtly ‘competitive’ but, at the same time, ‘competitiveness’ was not perceived as important in the game. Despite these perceptions by subject matter expert user players, it would be premature to question commonly accepted wisdom that competition is not important in serious games. Furthermore, although ‘competition’ in the Rail Disruption Game may not be overt, as players are not competing against each other or are obtaining a specific target, in effect, players are competing against the scenario to solve problems it presents, and do this against the clock. Therefore, implicit engageability and peer-to-peer knowledge transfer created by the game was almost more important than explicit feelings of competition. Consequently, in the future when developing competitive game elements, more consideration should be given to the underlying contextual experiences and motivations of players and how important these are in affecting perceptions of in-game competition importance versus engageability and knowledge transfer importance. This was considered under the ‘Describe your players’ section of the 6D Framework; however, it would seem from these findings that motivation in this case was more related to perceived knowledge transfer and engageability rather than explicit player-to-player competition. In this context, this is a good result, as the game was designed to have players working together, breaking down functional barriers, and solving complex challenges instead of competing against one another.

Statement 10 related to the legal responsibilities a TOC has towards its customers, which includes safeguarding vulnerable members of the public, as well as general health and safety considerations. Although the average response (3.4) is greater than neutral (3.0) this is somewhat lower than the other criteria. However, it was noted in the observations that players were considering the practical aspects of their legal responsibilities without realising it. This was because the care of vulnerable passengers (e.g., children and disabled adults) was built into the scenarios and passenger types. Therefore, players, without realising it, were addressing safeguarding issues for each passenger ‘avatar’. It is argued that this was again another success of the Rail Disruption Game, as it was able to stimulate engaging discussion on issues of legal responsibilities by transferring relevant peer-to-peer (i.e., player-to-player) knowledge, but without players even realising it. This point, which covers both engageability and knowledge transfer, was featured at the end-of-game debrief to further improve PIDD management without inhibiting the flow of the game.

Overall, very few individual question responses in the evaluation questionnaires were rated as negative, fewer than three, and the majority of those related to Statements 1 and 10 as discussed above. The remaining statements received on average less than one negative response across the responding players.

8. Conclusions

This paper presented the implementation and evaluation of the Rail Disruption Game. The aim of the project was to improve the management of passenger information during disruption (PIDD) via a safe and controlled environment, which became a gamified process by embedding a serious game [50] into it as a process improvement idea-generating instrument. These authors conclude that serious games, founded on pedagogical goals and the appropriate use of game mechanics and gamified processes, have the potential to engage, motivate, transfer knowledge and influence the behaviours of their users.

These authors also advocate that if serious game designers were able to apply increasingly user-centred design principles, as stated above, they could develop more purposeful games. One might think of this as intervention-based research (IBR), where methods (M), theories (T) and problem situations (S) (e.g., ineffective PIDD management) are addressed simultaneously [51]; where moving ‘S’ to state ‘S’’ as first envisaged at the onset of a project may not be the best route to proceed along, but something different such as ‘S*’. The authors demonstrating this as ‘M’ has been changed (revised NPD), and there are lessons learned on effective socialisation, externalisation, combination and internalisation of dynamic knowledge ‘T’ [52], and the PIDD management process has improved from ‘S’ (poor process) to ‘S’’ (recognition of some kind of learning capability) to ‘S*’ (a serious board game and gamified process) [50,53].

Throughout this project, domain experts were involved to assess and evaluate the game via a series of workshops (seven in total). The first four workshops used developers to assess and improve the game (in iteration cycles), and the final three used testers to evaluate the effectiveness of the game (in questionnaires). The assessment developers and evaluation testers were different sets of people, so that assessment and evaluation were conducted objectively.

The final game workshops were well received by the players, who believed the Rail Disruption Game was engaging, valuable and improved their ability to manage PIDD, and provided them with a greater understanding of rail operations and decision-making processes in other functional departments and UK regions. Hence, collectively, the PIDD staff had a greater ability to recommend PIDD management process improvement suggestions, which is the real value created for players and TOCs using the Rail Disruption Game.

The authors thank reviewers for their valuable feedback on earlier versions of this paper that have greatly improved it.