Participatory Modeling with Discrete-Event Simulation: A Hybrid Approach to Inform Policy Development to Reduce Emergency Department Wait Times

Abstract

We detail a case study using a participatory modeling approach in the development and use of discrete-event simulations to identify intervention strategies aimed at reducing emergency department (ED) wait times in a Canadian health policy setting. A four-stage participatory modeling approach specifically adapted to the local policy environment was developed to engage stakeholders throughout the modeling processes. The participatory approach enabled a provincial team to engage a broad range of stakeholders to examine and identify the causes and solutions to lengthy ED wait times in the studied hospitals from a whole-system perspective. Each stage of the approach was demonstrated through its application in the case study. A novel and key feature of the participatory modeling approach was the development and use of a multi-criteria framework to identify and prioritize interventions to reduce ED wait times. We conclude with a discussion on lessons learned, which provide insights into future development and applications of participatory modeling methods to facilitate policy development and build multi-stakeholder consensus.

1. Introduction

Evidence-based decision making is the foundation for health policymaking and health service planning. There remain many practical challenges to integrating the results of research to identify evidence-based interventions to implement in a given context, especially in the face of the uncertainty and complexity that characterize many healthcare delivery systems. Although research presents a multitude of acceptable evidence-based options, it is difficult to determine which interventions will have the greatest impact given the heterogeneity in the population and the health-service delivery system. Complex or uncertain disease epidemiology, including multimorbidity, can introduce additional challenges to decision making. In addition, a large number of stakeholders with differing perspectives and competing priorities are typically involved in the decision-making processes. Failure to achieve consensus among these stakeholders can hinder the effective implementation of interventions [1,2,3].

As a result of these challenges and the increasing availability of advanced computer technology, there has been a growing interest in using modeling and simulation techniques to assist in the decision-making process in the health sector [4,5]. The design and development of health services would benefit from using these techniques to systematically integrate diverse evidence sources into a computer model and validate the underlying causal mechanisms that drive complex healthcare problems. More importantly, these models allow assessment and comparison of the effects of proposed changes or resource configurations in service design through “what-if” scenarios, providing insights on potential impacts in a cost-effective and timely manner compared to real-world trials. A number of simulation studies have successfully applied computer modeling methods to inform strategic planning and health policy-making for stroke care improvement, alcohol-related harm reduction, cardiovascular disease interventions, and managing diabetes in pregnancy [6,7,8,9].

Despite this growing interest and an increasing number of simulation studies in health-related fields, serious and widespread application of systems modeling and simulation remains lacking in informing health-policy development [5,10,11,12,13,14]. Low stakeholder engagement is a major factor that hinders the translation of model findings into evidence-based policy and practice [13,15,16]. Recent studies in the context of healthcare found that relatively few simulation models addressed the needs of policymakers, and models were often constructed without the involvement of health-system managers or policymakers (who are considered the ‘end users’) in the research process [4,5,15,17]. Many factors contributed to the “implementation gap” between model findings and serious application in the health field; for instance, the communication gap between research and stakeholder groups [15], the lack of involvement of researchers when important policy decisions are made [18], and the trust issues throughout different modeling stages that involve interactions between the model, the modeler, and the stakeholders [19]. Harper et al. proposed a trust model and discussed different aspects of trust building in the life cycle of a simulation study [20].

Successful development of health interventions or policies depends on stakeholder support for the proposed improvements or actions. There is a growing recognition of the value of using a participatory modeling approach in simulation studies to bridge the communication gap and enable collaboration between stakeholders and modelers in building the simulation models [3,7,21]. The participatory modeling (also referred to as collaborative modeling, participative modeling, or facilitated modeling) approaches involve the joint creation of a computer model that reproduces a shared representation of the system in silico with the end users, stakeholders, and experts to facilitate collaborative learning, build consensus, and inform the group’s decision making [22]. In these approaches, expert modelers directly collaborate with a team of end users or stakeholders throughout the whole simulation study life cycle, as opposed to the traditional approaches where the modelers conduct the simulation studies independently and only present the findings to the stakeholders [4]. A participatory approach to simulation modeling helps strengthen relationships and improve knowledge translation when designing health services and policies for complex problems [23].

This case study describes modeling ED and acute care patient flow using a four-stage participatory modeling approach with discrete-event simulation (DES) in a Canadian health-policy setting. The paper complements the literature by reporting on the use of a participatory approach in developing DES models of patient flow in the emergency department (ED) to inform the design and planning of effective wait time reduction strategies from a whole-system perspective. The DES models, built in collaboration with a multidisciplinary group of stakeholders, served as a decision-support tool that integrated existing data, research evidence, expert knowledge, and local context to assess the potential impact of different intervention scenarios.

The modeling findings have been utilized to inform policy development and subsequent actions. We detailed our DES model’s structure and major quantitative results in previous work [24]; discussions on major simulation findings can be found therein. In the current study, we provide methodological details of the participatory approach used in model development and model use with stakeholders. We focus here on describing the following key aspects of the participatory modeling process:

• building trust in the modeling approach and seeking buy-in from the project leads for further development during the project initialization;
• leveraging existing conceptual mapping tools used in the health system for conceptual modeling;
• using co-production methods to build trust in the model and its outputs;
• identify and prioritize intervention scenarios using a multi-criteria framework.

We then discuss lessons learned and implications for adopting the participatory modeling approach in the health-policy setting.

2. Brief Review of Participatory Modeling Approaches

2.1. Motivation

In the past two decades, there has been little improvement in translating simulation findings into policy and practice [12,25,26,27,28,29,30,31], with communication gaps and a lack of understanding of policymakers’ needs contributing to the underutilization of model findings in healthcare decision making [15,31]. There is a rising awareness that more needs to be done to ensure that research results in better knowledge translation and improved health services and patient outcomes. Engaging stakeholders in research has been highlighted as a possible mechanism to increase the value, use, and relevance of research [32]. Scholars have called for collaborative engagement with health professionals, managers, decision makers, and patient representatives in simulation studies to increase the likelihood of successful implementation of simulation outputs [3,9,12,30]. In addition, a range of programs were established to support stakeholder engagement in health research, such as the development of the US Patient-Centered Outcomes Research Institute and knowledge translation activities in Canada [32].

Participatory modeling approaches have been proposed as one way for engaging stakeholders and improving stakeholders’ knowledge and understanding of a system and its dynamics under a variety of conditions in order to support shared learning or the decision-making process [3,9,21,22,33,34]. It also increases trust in and use of scientific information in decision making [35]. It is a deliberate learning process that involves a diverse range of modeling activities that draw on stakeholders’ knowledge to develop a shared and formalized understanding and representation of the reality or system [22,31]. In a participatory modeling process, stakeholders can be involved in one or more modeling stages with different levels of engagement, ranging from passive participation (e.g., being merely informed about model findings) to active participation where stakeholders might contribute to problem identification, data collection, model design, and model use [31,34].

2.2. Applied Fields and Purpose

Participatory approaches to modeling have gained recognition in a number of sectors, including natural resource management (e.g., water management) [31,36], environmental planning [34,37], and health research and policy [3,33,38,39,40]. Voinov et al. reviewed participatory modeling studies in resource management and environmental planning that involved multiple sectors and stakeholders [34]. There has been an increase in the popularity of participatory modeling in recent years, as evidenced by the rising number of papers published on the topic [34,41]. The authors presented different components for stakeholder participation within the development of specific environmental models and summarized tools and methods for participatory modeling in different modeling stages [34,41]. Participatory modeling approaches are also being used in public health and health services research. Freebairn et al. described the novel use of participatory simulation modeling combined with system science methodology in informing health policies through three case studies aimed at reducing alcohol-related harms, childhood obesity, and diabetes in pregnancy in real-world policy settings in Australia [3,42]. The authors published a series of papers on these case studies that provided valuable lessons, detailed procedures, and concrete guidance in using participatory dynamic simulation modeling methods in health policy settings, such as integrating knowledge translation and mobilization into participatory processes, converting conceptual system maps into a dynamic simulation, and building consensus with stakeholders for policy actions [3,7,9,43]. Frerichs et al. and Gerritsen et al. discussed and showcased the potential of using community-based participatory modeling approaches in public health and health equity research [44,45].

Depending on the form of participatory modeling approach used, participatory modeling processes may serve somewhat different purposes. Most participatory modeling projects focused on developing a collective understanding of the issues among the participants, and the model was viewed as a tool to support shared learning or community-based learning; while others engaged stakeholders to inform policy-making or mobilize actions [3,31,33,40,42,44,46,47]. Using the participatory modeling approach was found to have several benefits, including providing a better evidence base for policy decisions, improving the quality of the research, and increasing the dissemination and implementation of interventions [40,48,49]. A number of tools and methods can be employed to facilitate stakeholder engagement in participatory modeling processes. Voinov et al. provided a systematic overview and assessment of various participatory modeling methods and discussed their strengths and weaknesses [22]. The authors proposed a typology of methods used in participatory modeling with illustrated workflows and provided practical guidance for method selection. Depending on the modeling stage, either qualitative (e.g., rich pictures or causal loop diagrams) or quantitative methods (e.g., agent-based modeling or system dynamics) could be utilized.

2.3. Specific Participatory Modeling Approaches and Tools

2.3.1. Soft Operation Research Methodology

Soft Systems Methodology (SSM) is a broader participatory modeling methodology that emphasizes the use of rich picture tools to learn about a problem situation and start exploratory discussions with people [50]. It also embraces the systems perspective to explore problematic situations with relevant stakeholders. SSM used a sequence of steps from problem finding and rich picture building to solution identification and action mobilization [22,50]. It has been used as a problem structuring approach in a number of sectors (e.g., strategic planning and policy development concerning agriculture) [51], but its application in the healthcare setting is limited [46]. There are other “soft” problem structuring methods used for studying complex and unstructured problems involving multiple stakeholders with diverse perspectives, such as strategic options development and analysis, and hierarchical process modeling [52,53].

2.3.2. Group Model Building

Group model building (GMB) refers to a set of techniques used to engage client groups directly in the process of problem finding, model construction, and use through facilitated workshops or sessions [39,54,55]. Group model building has matured as a field, specifically as a community-based participatory approach for social learning or the development of system dynamics (SD) models [39,45,47,54,55,56]. Depending on the type of problem, the end product often results in either qualitative models (causal loop diagrams) or quantitative models (such as SD models).

2.3.3. PartiSim Framework for DES

Tako and Kotiadis developed a facilitated modeling approach named PartiSim [21,57,58,59]. PartiSim (short for participative simulation) is a multi-methodology framework that integrates DES with SSM. The PartiSim framework integrates stakeholder input and facilitation as part of the process of conducting DES studies. The framework is divided into six primary stages in the simulation study life cycle, each with its own set of activities, deliverables, and tools to help the modeling team engage stakeholder participation in the study. The authors employed the PartiSim framework while collaborating with healthcare organizations on a variety of operational issues in healthcare settings [60]. The authors reflected on practical challenges encountered in facilitating the conceptual modeling process, particularly panel composition and team roles in handling conflicts and promoting involvement among stakeholder teams and modeling teams [57]. The authors also discussed and reported on the post-model coding step, specifically scenario development and experiments [59]. The authors achieved success in implementing model findings in an obesity service to improve patient waiting times [60].

2.3.4. Other Participatory Modeling Approaches Used in Case Studies

Unlike the formal frameworks or methodologies introduced above, several studies reported the use of participatory modeling through case studies [3,7,9,42,43]. These studies elaborated on the rationale and procedure of the participatory modeling approach used [7], integration of rapid review with participatory modeling processes for knowledge mobilization [3], the conversion of conceptual mapping into a quantitative simulation model [9], and decision makers’ experience in the participatory modeling processes [42].

2.3.5. Participatory Modeling Tools

Participatory modeling can be used in either qualitative or quantitative modeling. For qualitative modeling, a number of tools can be used at different modeling stages or combined with more general frameworks or methodologies such as SSM, GMB, or PartiSim. Diagramming or graphical tools, such as rich pictures (as a part of the SSM), causal loop diagrams, and cognitive (or conceptual) mapping, were often used in participatory modeling to generate visual representations of the components of the problems [9,22]. These qualitative maps or diagrams are utilized for collaborative exploration and group understanding of a complex issue by representing diverse relationships among many interacting components, illustrating how changes in one area affect other factors, and drawing the feedback loops that are assumed to explain the dynamic behaviors [9,22]. Unlike participatory modeling approaches for qualitative modeling, which are primarily concerned with fostering trust and understanding among stakeholders, participatory modeling approaches in quantitative modeling are more focused on “solving” a specific problem and frequently produce forecasts or quantitative estimates [22]. Participatory modeling can be embedded in the development and use of SD, DES [21], and agent-based models (ABM) [22].

2.4. Hybrid Modeling and Simulation

With new developments in the simulation tools and the rising complexity of the studied systems and decision-making needs, a growing number of studies in the field of modeling and simulation (M&S) have embraced a hybrid approach [61,62,63]. This involves combining multiple simulation techniques (e.g., in the form of mixing SD, DES, and ABM) or integrating hard or soft operation research (OR) methods in one or more stages of a simulation study (e.g., combining SSM with DES) [61,62,63]. The adoption of hybrid approaches becomes increasingly appealing, especially from a practical standpoint, as hybrid approaches can overcome some limitations of using any single M&S technique and complement each other [61,62]. This approach can further facilitate the nimble evolution of model scope and formulation in light of the growing understanding of the system and changing policy evaluation needs and context. One of the main areas of application of hybrid M&S was found to be healthcare [61]. A combination of multiple simulation techniques (e.g., SD, ABM, DES, and social network analysis) has been employed to investigate a range of complex health-related issues and systems, such as disease prevention and care management [43,64,65,66], projection of disease burden [67], health workforce dynamics [68], and immunoepidemiology of infectious diseases [69]. Reflecting the diversity in the mixing of different methods in simulation studies, Mustafee et al. presented a conceptual classification of hybrid M&S and discussed innovation in the M&S field by highlighting hybrid modeling [70,71,72]. Unlike hybrid simulation, which combines two or more simulation techniques (e.g., SD, DES, or ABM) primarily in the model implementation stage of a simulation study, hybrid modeling extends hybrid simulation by emphasizing the combined application of simulation techniques together with theories, frameworks, methods, or tools in the broader OR domain or from other disciplines in stages of the modeling process of a simulation study beyond implementation [70,71,72]. Hybrid modeling can involve the synthesis of a qualitative soft OR technique (such as SSM or a participatory approach) with a simulation technique [62]. For instance, McDonald et al. used a community-engaged collaborative modeling approach to co-create an ABM with the indigenous community to better understand the water services [73]. Jiang et al. combined the use of causal loop diagrams in the conceptual modeling stage with ABM to study breastfeeding interventions [74]. Other forms of hybrid modeling include the co-application of simulation techniques with cross-disciplinary methods from other fields or domains [70]. Tian et al. proposed a hybrid approach by linking multiple techniques with an SD model to investigate stroke prevention and care [75]. Kreuger et al. discussed the combined use of agile design in software engineering with hybrid simulation design and use [76]. Hybrid modeling stands at the forefront of fostering innovations in both the theoretical and practical aspects of M&S, especially in light of new developments in conceptual modeling to better understand and depict studied systems [62], and in stakeholder engagement to capture domain knowledge from diverse stakeholders and build trust in simulation models [70,71].

3. Case Study Context

3.1. The Problem of Emergency Department Crowding and Wait Times

ED crowding, often reflected in lengthy wait times for emergency care, is a worldwide healthcare delivery issue that negatively affects patient safety, experience, and clinical outcomes, such as increased mortality rates and worse patient satisfaction [77,78,79,80,81]. The causes of ED crowding and lengthy ED wait times are multi-factorial and complex [77,82]. Among 11 developed nations, Canadians reported the longest ED waits [83,84]. Lengthy ED wait times are a major concern for Canadians [85], and several Canadian provinces have launched initiatives to address the ED crowding and patient wait times [78]. The government of Saskatchewan challenged the health system to tackle the waiting time problem in EDs [86]. In response to the challenges of improving timely access to emergency care, the Saskatchewan ED Waits and Patient Flow Initiative was launched to address ED wait times. The provincial initiative was charged with developing and implementing ED wait time reduction strategies provincially [86]. Given the complex nature of the issue, limited resources for allocation, and an extensive set of evidence-based options from both the literature and expert opinions, it is challenging to quantify the impact of each plausible contributing factor. It is also costly to pilot each evidence-based intervention individually or collectively for the studied EDs. Ideally, we would desire a safe and cost-effective approach to understand the causes and examine the likely consequences of proposed changes in advance. With the dual objectives of identifying the causes of lengthy ED wait times and comparing potential solutions to allocate limited resources, this study investigated the ED patient flow in six Saskatchewan hospitals. At the time of the research, the six hospitals were located in three health regions, which collectively served 63% of the Saskatchewan population.

3.2. Whole-System ED Patient Flow Modeling

There is a significant amount of literature on using simulation, particularly DES, to study ED crowding, streamline ED patient flow, and reduce wait times [12,28,30,87,88,89]. However, most of these ED simulation studies have high unit specificity with a focus only on care processes in a single ED [25,28,30]. Only a few studies investigated ED crowding or wait times as part of a larger system, such as a whole hospital or complete acute care system [90,91,92]. There has been limited improvement in multi-facility and whole-system modeling in the area of ED simulation studies that utilize DES [25,28,30,89].

Despite the broad agreement that the ED problem largely requires system-wide solutions and that solutions within the ED alone are insufficient, there remains a lack of whole-system multifacility modeling, as documented in the ED literature. Empirical evidence suggests that ED performance depends on adjoining systems, such as acute care, primary care, and sub-acute care to function efficiently [28,30,56,88,93]. Jun et al. concluded that future simulation results on ED crowding or wait times need to depict the interaction of major service departments and support services in a hospital to gain insights from analyzing the system as a whole, rather than in a unit-specific piecemeal fashion [25]. Gunal et al. argued that unit-specific and facility-specific ED simulation studies, which ignore the subtle linkages between the ED and other units or services within the hospital, might oversimplify the complexity of hospital activities within a simulation model and overlook side effects and unintended consequences [28]. Zhang echoed that healthcare management would benefit from DES models, which capture the intricate interactions of healthcare services rather than just limiting them to single units [89]. Salmon et al. also found high unit specificity among ED simulation studies, thus calling for the examination of system influences beyond the ED itself to better understand emergency department operations from a whole-system perspective [30].

This paper complements the literature by demonstrating how multi-facility whole-system DES models of patient flow can be used to design policies to improve ED patient flow and reduce wait times. Notably, the study embraced the systems view and studied ED wait times as a part of a larger integrated acute care system with multiple interacting subsystems or units.

4. Materials and Methods

4.1. Overview

This study described a participatory modeling approach for embedding stakeholder engagement in the development and use of DES models to support effective and coordinated policy development to reduce ED wait times. The participatory modeling approach used in this study drew on several existing frameworks for working with groups [21,56,94] and was adapted to the local context. The approach was applied through collaboration between the Saskatchewan ED Waits and Patient Flow Initiative, Saskatchewan Ministry of Health, regional health authorities, Saskatchewan Health Quality Council, clinicians, health professionals, researchers, and health policy planners. The study was exempted by the University of Saskatchewan Research Ethics Board.

The participatory modeling approach used in this study can be broken down into four major stages: (1) project initialization; (2) conceptual modeling; (3) model implementation; and (4) model use and policy co-development. Figure 1 illustrates the overall participatory modeling process with a detailed visualization of the model use and policy co-development stage.

4.2. Stage 1: Project Initialization

The initialization phase assembled the project team, defined the purpose and scope, and assessed the feasibility of using computational modeling to understand ED patient flow and inform the policy development process. The primary objective of this phase was to seek initial buy-in from the project lead (GF) and physician leads (JB and JS) of the ED Waits and Patient Flow Initiative. The goal was to help them recognize the value of computer simulation models as decision-support tools for addressing the ED patient flow challenges with a broader group of regional and provincial stakeholders. To achieve this, a proof-of-concept analysis was carried out to demonstrate the ability of computational models to simulate ED patient flow, synthesize evidence, data, and expert opinions, and allow for the evaluation and comparison of alternative wait time reduction strategies via virtual experiments.

Team Composition

The modeling project team was led by a project lead (GF) who was primarily responsible for managing the project, overseeing model development, and communicating with important stakeholders, including regional leaders and policy partners. The domain experts on the team included ED physicians, a geriatrician, family physicians, clinical experts, patient flow managers, and quality improvement consultants. These experts provided information about the current patient flow situation in the studied EDs and hospitals and offered clinical guidance for model development and use. Two key physician leads (JB and JS) shared duties with the project lead and jointly facilitated modeling meetings and stakeholder communication. One of the domain experts (JB), who was a geriatrician, also served as the project champion. Two technical advisors (NO) oversaw high-level model conceptualization and development to ensure that the model was suitably scoped, rigorously validated, and could be delivered within a policy-friendly time frame. The lead modeler and researcher (YT) was responsible for designing the participatory modeling process with the project leads and technical advisors, managing data requirements, synthesizing evidence for model conceptualization, building and validating simulation models, developing intervention scenarios with stakeholders, and conducting simulation runs. Another modeler served as an intern for five months during the project’s initialization phase to assess the feasibility of using computational modeling to inform decision making through proof-of-concept analysis. Two quality improvement consultants (AD and PF) helped facilitate meetings with stakeholders.

4.3. Stage 2: Conceptual Modeling

The first step of the study was to understand the existing ED waits situation by getting an overview of the entire ED patient flow and care processes in the studied EDs and hospitals. Conceptual modeling refers to a series of activities for abstracting a qualitative model from the problem situation and the real system with relevant domain experts and stakeholders [2,95]. The conceptual modeling process in the present study was inspired by elements from existing conceptual modeling frameworks [2,95] and comprised three key activities for knowledge discovery and information elicitation: (1) conceptualizing the problem, specifically the causes of ED wait times, and identifying a preliminary set of experimental factors (model inputs that can be changed); (2) understanding the data and care processes in the studied EDs and hospitals; and (3) defining the scope and boundaries of the model. The sequence of these activities was not linear, and they were pursued in a highly iterative fashion. We used tools that were borrowed or adapted from qualitative improvement toolkits [96] and lean management techniques [97]. Prior to the formation of the modeling team, the provincial initiative’s working groups had conducted initial conceptual mapping of the ED wait time issue. Rather than introducing new conceptual mapping tools, the modeling team opted to review and enhance the previous work to generate new insights and foster a more comprehensive understanding of the problem situation.

Problem conceptualization started with a qualitative representation of the factors contributing to the ED wait times. A comprehensive literature review was conducted at this stage to identify important contributing factors to ED wait times. The modeling team also consulted with a wide range of domain experts and regional stakeholders through informal meetings and onsite visits to acquire knowledge about the studied sites. The perspectives of stakeholders and relevant domain experts were both crucial at this stage, as they assisted the modeler in better understanding the contextual factors in the studied EDs. A driver diagram was developed to depict the causes that could influence ED wait times from multiple perspectives. A driver diagram is a quality improvement tool used to illustrate a project team’s understanding of the factors that contribute to the project objective [96]. It displays key areas (primary and secondary drivers) to influence in order to achieve the aim of an improvement project when tackling a complex problem.

To understand the data and care processes in the studied EDs and hospitals, the modeling team worked with process improvement consultants, patient flow managers, regional stakeholders, and patients and family advisors through rapid process improvement workshops to develop value stream maps (VSMs) of the current state of ED patient flow. VSM is a lean management technique that uses a flowchart to draw the current (and future) state of a specific process (or service), with the goal of identifying waste, reducing process time, and improving the flow [98]. At the time of this research, many lean tools were already in use as part of the lean implementation in the province’s healthcare system [99]. Rather than re-inventing the wheel, the modeling team leveraged these existing conceptual mapping tools and instead focused on understanding and refining the VSMs. The modeling team met with process improvement consultants and individuals who have expertise in patient flow management in the studied hospitals and conducted onsite visits. The VSMs of ED and acute patient flow illustrated the major steps in the patient’s journey from registration to discharge. They also mapped the ED activities (both necessary and non-value-added activities), current care processes, and lead times from the whole-system perspective. Additionally, input data for the model was analyzed using administrative databases based on the driver diagram and VSMs.

In the subsequent stage of the process, the model scope and level of detail were discussed and defined. The primary outcome of the study was the wait time in EDs, which made DES an appropriate choice for the modeling approach as it is well-suited for modeling queuing systems with constraints. The modeler, technical advisors, and project champion collaboratively defined major components (entities, attributes, activities, and resources) to be included in the DES models. Furthermore, assumptions and simplifications were made with input from domain experts and technical advisors.

4.4. Stage 3: Model Implementation

4.4.1. Turning Conceptual Modeling into DES Models

To operationalize the conceptual model as a computational model, the modeler worked with patient flow managers, clinical experts, and quality improvement consultants to identify critical components (e.g., units), interactions, and patient cohorts to be captured in the first iteration of the development of DES models. This was an iterative process throughout the project cycle that involved reviewing evidence and the conceptual models, analyzing administrative databases, and ongoing consultation with clinical experts to determine important processes and agents to be represented in the DES models.

In the early phase of the model development, to communicate and consult with clinical and domain experts on model logic, we added 3D visualization of the floor plan of the studied EDs to showcase the patient movement in the simulation. We conducted face validation and external validation to maintain the credibility of the model. Multiple patient cohorts (e.g., ED visits for family practice sensitive conditions, hospitalization for ambulatory care sensitive conditions, etc.) were identified in the conceptual modeling stage. We extracted these patients cohorts from administrative databases, analyzed their use of ED and acute care services, and characterized them in the DES models.

4.4.2. Capacity Building and Health System Modeling Workshop

Most project stakeholders knew very little about computational modeling and its use in healthcare. Their participation and involvement in the project were largely driven by the priority of the issue and their trusted relationship with the lead domain experts and the provincial initiative. However, many stakeholders were curious about modeling methods. A 4-day health system modeling workshop, led by the modeling team, was hosted by the Saskatchewan Health Quality Council and the provincial initiative. Thirty-five participants (including several provincial leaders) from Saskatchewan health regions, health organizations, and the Ministry of Health participated in this modeling workshop on computational modeling and system thinking in healthcare [100]. Several participants were data content experts and had worked with the modeling team on parameterizing the simulation models of patient flow for the hospitals studied. The modeling team also presented the initial DES models of patient flow to the participants.

4.5. Stage 4: Model Use and Policy Co-Development

After completing the initial model development and validation, the project moved to the model use and policy co-development stage. We formed an advisory group that included stakeholders from three health regions and ministry departments within the healthcare system. To work towards shared goals and aligned actions to reduce ED wait times, we invited the advisory group to participate in four facilitated advisory group meetings to co-develop intervention solutions for implementation to reduce ED wait times in the studied EDs. We highlighted the use of simulation models for integrating data, evidence, and stakeholder inputs to support evidence-informed policy development.

The advisory group was formed by the provincial initiative. The advisory group included all relevant senior stakeholders from the Ministry of Health branches, executive leaders from three regional health authorities, physician leads and clinical experts, researchers, and patient advisors. Figure 1 illustrates the major activities that involve interaction and engagement with stakeholders in the model use and policy co-development process. It also highlights four major advisory group meetings with the engagement of the full panel of the advisory group during this process. The first meeting introduced the use of simulation models for policy co-development with stakeholders. The second meeting reviewed the preliminary scenario results and discussed creating additional intervention scenarios based on stakeholders’ assessments of their regional priorities, readiness, and drivers for lengthy ED wait times in the studied EDs. The last two meetings were centered around discussing the model results and reaching an agreement on recommended interventions for further analyses and the eventual development of a business case for implementation.

4.5.1. The First Meeting

The first advisory group meeting introduced the use of simulation models for policy co-development. The meeting emphasized the participatory approach being undertaken to support evidence-informed policy development with the active engagement of the advisory group. We presented our data analyses on a number of patient groups that were identified as potentially contributing to the problem of lengthy ED wait times. Then we summarized our literature review and environmental scan on two topics: (1) what interventions had been tried, what worked, and what did not work; and (2) promising practices from other jurisdictions and lessons learned from other organizations experiencing similar ED crowding or flow issues. Finally, we proposed the use of simulation models as a decision-support tool to synthesize data, evidence, and input from the advisor group to explore and compare the impact of proposed interventions. The modeling team also described the collaborative model development and model validation activities with patient flow managers, physicians, and data analysts. The advisory group agreed with using the participatory modeling approach to develop policy recommendations for the budgeting cycle.

After the first meeting, the modeling team started to conduct simulation runs of interventions that emerged from evidence and data. In addition, key contacts were identified for the three health regions where the six studied hospitals were situated. Engagement with regional and smaller stakeholder groups also occurred after the first meeting. We held multiple regional advisory group meetings with leaders of specific regional health authorities to gather information about hospitals studied in their region, their local contexts, data sources, interventions of interest, readiness, and priorities. Specifically, a multi-criteria prioritization framework for scenario development was designed and used with the regional leads (seen in Figure 2 and

Table 1. Scoring Matrix. Criteria used for prioritizing requested modeling scenarios.

Component	Criteria	Description	Scoring (in the Range of 0 to 5)
Data	Regional Data Availability	Degree to which regional data is collected or available for understanding the current state or the targeted cohort for the intervention scenario	0 = no data; 1 = limited data with poor quality; 2 = partial data with poor quality; 3 = partial data with good quality; 4 = almost complete data; 5 = we have everything we need
	Administrative Data Availability	Degree to which data are readily available in administrative databases for modeling or understanding the current state or target cohort.
Evidence	Literature Support for Modeling	Degree to which evidence (e.g., meta-analysis and systematic reviews) provides relevant outcome metrics for modeling	0 = very low; 1 = low; 2 = low to moderate; 3 = moderate; 4 = moderate to high; 5 = high
Modeling	Modeling Effort	The amount of effort and time it will take for the modeler to incorporate and simulate the scenario using the DES model.	0 = extreme effort; 1 = significant effort; 2 = moderate effort; 3 = moderate to low effort; 4 = low effort; 5 = almost no effort at all
Stakeholders Inputs	Organizational Readiness	Degree to which the organization is ready and committed to implementing the idea	0 = not ready at all; 1 = not ready; 2 = neural or uncertain; 3 = moderately ready; 4 = mostly ready; 5 = ready to go immediately
	Regional Priority	Degree to which the health region agrees that this will address their ED wait times or patient flow issues	0 = not at all; 1 = unlikely; 2 = somewhat unlikely; 3 = neutral or uncertain; 4 = somewhat likely; 5 = very likely
	Provincial Priority	Degree to which the provincial stakeholders and ED waits initiative agrees that this will address ED wait times or patient flow issues
	Length of Time to Impact Drivers	The length of time it would take to see an effect on ED waits, volume, ED LOS or acute LOS.	0 = >7 years; 1 = 5–7 years 2 = 3–5 years; 3 = 1–3 years 4 = within 1 year; 5 = immediate
	Length of Time to Get Service Ready	The length of time required to alter or design service.

). The modeling team distributed this framework to key contacts in each health region before the second meeting. The key contact (a senior regional leader) discussed with managers and directors in their region about what interventions they would like to explore via simulation modeling to reduce ED wait times given their current state and local challenges. We also instructed them to review and assess each proposed intervention using the criteria in the framework. We then gathered an initial list of intervention scenarios that each region would like to explore using the models.

4.5.2. The Second Meeting

The second meeting began by presenting the early findings of five model intervention scenarios for five hospitals. For each intervention scenario, we presented the reduction in ED waits over three years and the supporting evidence for the proposed models of care or interventions. These intervention scenarios were identified by the modeling team through literature review, data analyses, and consultation with domain experts. These initial scenario runs were undertaken for demonstration purposes and exploration of “what-if” interventions, rather than for necessarily selecting the most desired scenario. The results of the scenario exploration stimulated discussions with the stakeholders.

In the wake of the above, we focused on discussing the development of additional modeling scenarios according to regional priorities and readiness. In the second meeting, we reviewed the list of interventions proposed by each health region. The regional leads explained the rationale behind each proposed scenario and reflected their regional priorities and alignments with the initiative’s goals. Ideally, it would be desirable to include all proposed scenarios from the regions in the simulation model; however, given the large number of scenarios that we received, the advisory group agreed on incorporating the top-ranked scenarios for each region based on the multi-criteria framework described in Figure 2 and Table 1. The modeling team took on the task of finalizing the list of scenarios for each health region and discussing data requirements and relevant evidence after the second meeting.

4.5.3. The Third and Fourth Meetings

The modeling team worked with each health region to refine and clarify the requested intervention scenarios using the multi-criteria prioritization framework. The third and fourth meetings consisted of an extensive review and discussion of the scenario results requested by each region, respectively. The modeling team presented the impact of each intervention scenario on ED wait times for the hospitals studied in the corresponding health region. For each scenario, we reviewed data input, supporting evidence (if any), evidence used in the model, key assumptions, and implementation ideas (e.g., how to achieve the expected wait reductions and what needs to be done in terms of implementation). To ensure a comprehensive exploration of intervention options to reduce wait times, additional intervention scenarios were requested during this time and incorporated into the simulation model.

To build trust in the modeling tool and continuously validate the model given the new changes made to incorporate different patient groups, we held a model challenge session between the third and fourth meetings. We invited clinical and data experts as well as interested parties to review the model structure, parameter values, and assumptions. Specific questions on the components being evaluated included: (1) how well the model component reflected the reality considered important by the clinical or domain experts; (2) whether the model output generated outputs that matched the real data; and (3) model assumptions, data sources, and data quality. We were able to gather feedback, suggestions, and critiques on (1) model input and data sources; (2) methodologies used; and (3) assumptions and ideas for future developments. This helped to further improve the DES models.

4.6. A Multi-Criteria Framework for Identifying and Prioritizing Policy Options with Stakeholders

It was challenging to select the most effective strategies to reduce ED wait times, given the large number of evidence-based options available that addressed different aspects of clinical practice and processes in the patient journey through the ED. Factors contributing to ED wait times might also differ between EDs due to variations in local context and organizational factors. Additionally, diverse opinions existed regarding the causes and solutions for the ED wait time issue in the studied hospitals. To address this, a multi-criteria framework was developed for the co-development of interventions with the stakeholders and the advisory group. The purpose of the framework is to promote transparent collaboration and communication with stakeholders during the policy development stage to identify feasible solution space. It is important to note that this framework is distinct from a multi-objective optimization framework and is not programmed into the DES models.

At the stage of co-development of intervention scenarios, we gathered and generated a large inventory of possible intervention scenarios from various sources, including (1) those supported by evidence; (2) those proposed by diverse stakeholder groups and domain experts; (3) those identified as likely to be promising through data analyses; (4) those currently considered for implementation; and (5) existing programs under consideration for expansion. Rather than accepting all scenarios for modeling without questioning, we screened and decided on the sets of interventions to be tested with the simulation models. The multi-criteria framework was used for identifying and prioritizing interventions or policies for modeling. The framework was developed following several knowledge translation principles and criteria to improve evidence-based decision making [101]. The multi-criteria framework has two components: (1) a modeling scenario intake process (seen in Figure 2) and (2) a scoring matrix (seen in Table 1). The framework took into account a number of criteria for managing scope, synthesizing evidence through a literature review, adapting knowledge to the local context, identifying gaps in practice, assessing feasibility, time to impact, and alignment with regional or provincial priorities.

4.6.1. The Intake Process for Modeling Scenario

To prevent “scope creep” and ensure that proposed intervention scenarios from stakeholders were appropriate and within scope [102], we created and used an intake process to screen the proposed interventions or recommended policies. The intake process is presented in Figure 2. The screening process involved evaluating each proposed modeling scenario against several criteria, including its suitability for modeling, availability of data, and supporting evidence about its potential impact on ED wait times or patient flow. It was important to note that local knowledge could not be unquestionably accepted and required assessment together with evidence and data. In addition, for each proposed intervention scenario, we identified a key contact (usually a stakeholder). We clarified the proposed intervention and data requirements with key contacts as needed. We also communicated to the stakeholders about our screening process to manage expectations.

The modeling team conducted the Initial screening and made a first decision on whether we would accept the proposed intervention scenarios for modeling. The proposed intervention scenarios for modeling were declined if one or more of the following conditions were met:

• The proposed intervention scenario is not within the scope of the project;
• Data are not available or the proposed intervention requires primary data collection that cannot be completed within the current timeline and budget cycle;
• There is a lack of evidence regarding the efficacy of the proposed intervention in reducing ED wait times or improving ED patient flow;
• If the intervention scenario is unsuitable for modeling, as determined by the modeling team, it will not be pursued;
• If the intervention is not feasible for implementation in the current local context.

Although the modeling team conducted a literature review on the causes of and solutions to ED wait times during the conceptual modeling phase, the proposed intervention scenario might not have been included in the initial review. The second phase of screening involved a rapid review of the effectiveness of the requested intervention scenarios that were not captured in the initial literature review. If the evidence base was found to be thin but the intervention was of high interest to the stakeholders, the modeling team would work with the advisory group and domain experts to determine whether assumptions should be made about program efficacy based on expert opinions and whether sensitivity analyses should be conducted.

4.6.2. The Scoring Matrix

To facilitate policy recommendations at the third and fourth advisor meetings, the modeling team evaluated a list of proposed interventions aimed at reducing ED wait times for each health region using the intake process (Figure 2). Due to the high number of intervention scenarios received from various sources, a scoring matrix was used to further shortlist the interventions for modeling, as shown in Table 1. The modeling team, along with regional leaders and stakeholders in the advisory group, collaborated in filling out the scoring matrix, which evaluated each proposed intervention scenario based on the criteria outlined in Table 1. Each scenario—whether derived from data or evidence or requested by stakeholders—received a score. The scenarios were then sorted accordingly. The scoring matrices for each health region were developed and discussed during the third and fourth advisory meetings.

5. Application of the Participatory Modeling Approach

5.1. Stage 1: Project Initialization

Planning meetings were held to define the project scope and boundaries. It was agreed that the main goal of the project was to validate current assumptions about the causes of ED waits and examine a portfolio of possible intervention options to improve patient flow in the studied EDs, with a focus on ED wait times. Interventions or activities that solely focused on improving primary care, home care, mental health, or chronic disease management, but were unrelated to emergency care or wait times, were deemed out of scope. Two primary outcomes of interest were identified for the project: (1) time waiting for physician initial assessment: time between registration (or triage) and initial physician assessment; and (2) time waiting for an inpatient bed: time between the decision to admit the patient to an inpatient bed and the patient’s departure from the ED for the inpatient unit.

Initially, the project leads were not convinced of the value of using a simulation approach to explore policy options related to ED patient flow due to their unfamiliarity with the modeling methods. To address this, a proof-of-concept analysis was conducted. The modeler consulted with several health experts, quality improvement consultants, and the provincial initiative to gather insights on the patient flow within the ED and acute care settings. Two preliminary ED patient flow simulation models were built under the guidance of the project champion and technical advisors. Figure 3 provides a flow chart of the simulated patient flow in the proof-of-concept DES model. The proof-of-concept models demonstrated how patient flow could be represented and how qualitative insights and high-level quantitative predictions on patient flow metrics could be generated. The analyses showed that computational models in principle had the potential to advance the evaluation of policy scenarios and improve the decision-making process. It is worth noting that the project champion played the role of an advocate for the simulating modeling approach in this stage and assisted in building trust and confidence with the project leads by articulating the proof-of-concept models in language familiar to the project leads. As a result, the project was sponsored, and a modeling team was formed as one of the working groups within the provincial initiative.

5.2. Stage 2: Conceptual Modeling

The initial conceptual modeling provided a theoretical basis for communicating with the domain experts, stakeholders, and the modeler. This was instrumental in guiding the modeler to develop DES models that captured the system problems with ED wait times while maintaining a reasonable scope and a moderate level of complexity.

5.2.1. Problem Conceptualization

The causes of ED crowding and lengthy ED wait times are complex [77,82]. ED crowding and long waiting times originate mainly from three interdependent components: the high volume of patients demanding ED care (input), the inability to assess and treat patients in a timely manner in the ED (throughput), and the boarding of inpatients in the ED after disposition decisions (output) [103]. Many potential contributing factors have been identified. They are present within each component of the input–throughput–output conceptual model of ED crowding [77,82,104,105,106]. Previous studies have investigated multiple input factors, such as the high volume of low-complexity patients in the ED, increased presentation with urgent and complex needs, and access to primary care [77,82,107]. Many throughput and output causes were also reported, such as ED staff shortages, limited availability of timely specialty consultation, delays in disposition decisions, and access block to inpatient units due to inadequate acute care beds or delayed discharge of inpatients [85,103,108]. Many input, throughput, and output solutions to ED crowding and wait times were trialed, modeled, and suggested with varying levels of success in different local contexts [77,109,110,111].

The modeling team jointly developed the driver diagram to depict five key areas to influence to reduce ED wait times. Figure 4 presents five primary drivers that are larger key topic areas related to the input, throughput, and output components of ED patient flow. Each primary driver was then linked with several secondary and tertiary drivers, which are less important or smaller in scale. The driver diagram was developed by incorporating diverse viewpoints from domain experts and stakeholders about the current state of the studied hospitals as well as evidence from the literature review. Domain experts and stakeholders were represented by physicians, nurse managers, operational leaders, and other health professionals from different health regions.

5.2.2. Core Patient Flow Processes Emerging from the Value Stream Mapping

To gain insights into the flow of ED patients in the studied hospitals, the modeling team made use of VSMs that were created by healthcare professionals and improvement consultants during rapid process improvement workshops. Figure 5 illustrates the core care processes that emerged from the VSMs that were developed by regional teams that worked in the studied hospitals. It is worth noting that the regional teams took a whole-hospital perspective when mapping out the current state rather than just focusing solely on ED activities. Community and primary care services were also mapped out and connected to ED inputs and hospital discharges.

5.2.3. Model Scope and Level of Details

Figure 5 further helped to determine the scope of the simulation models and identify the boundaries of the model. The need to explore the inputs to the ED (e.g., ED volumes) was an obvious entry point into the simulation model. The requirement to study the cause of prolonged waits for inpatient beds suggested that the flow of acute care patients should be included in the model in order to capture the intricate interactions between the ED and other acute care services in the hospital. Although community and primary care services were also mapped out in Figure 5, they were excluded from the model based on guidance from the initiative leads and domain experts. Activities aimed solely at improving primary care, home care, or mental health care were also deemed beyond the intended scope of the study.

We then identified entities, activities, queues, and resources that fell within the model boundary. We focused on identifying components that were connected with secondary and tertiary drivers and areas requiring improvements in Figure 4. Knowledge of the domain experts and technical advisors was vital at this stage to help decide the level of detail that required modeling.

5.3. Stage 3: Model Implementation

During the conceptual modeling stage, the identification of key components, care processes, and model scope facilitated the selection of appropriate data sources and model inputs for the DES models. Patient-related model inputs were obtained from various administrative databases, including the National Ambulatory Care Reporting System, Discharge Abstract Database, Physician Billing Data Repository, and Person Health Registration System. These databases contain individual-level data on ED visits, hospitalizations, physician billing, and covered population demographics. Staff-related model inputs include ED-specific physician shifts, ED physician assessment time, registration time, and ED triage time. Resource-related model inputs include ED beds and acute care beds. The DES models were created using AnyLogic 8.7.2 (professional version; AnyLogic North America, Chicago, IL, USA) with the Java-based Process Modeling Library on an Intel^® Core™ i7-9700 T CPU at 2.00 GHz, operating on Windows 10 Pro. Figure 6 shows the simulated patient flow from ED to acute care in the DES models.

As ED patient flow was the central focus of the model, it needed to be explicitly modeled to ensure that the core ED care processes were captured. In addition, knowing that ED wait times could be the result of delays that occurred in other parts of the healthcare system—in particular acute care—made it essential to capture inpatient flow in acute care settings. The resulting model captured patients from ED presentation to discharge from the ED or an inpatient ward (for admitted patients), as illustrated in Figure 6. The model further incorporated the volume of patients admitted via elective admissions to an inpatient ward. The model allowed for adjustment of the volume of the ED, the arrival rate by hour and day of the week, the acuity, and the volume of the ED of a specific cohort (e.g., ED visits for family practice sensitive conditions). Clinical experts also proposed reasonable assumptions to be used in the models. For instance, high-acuity patients often receive immediate treatment and interventions upon arrival; therefore, registration or triage might occur concurrently with the treatment rather than before the treatment. For parameters that were unknown due to a lack of data in the administrative database, clinical experts (e.g., nurse managers and ED physicians) provided estimates for the studied EDs, including the duration of nurse initial assessment and the duration of physician assessment.

We developed 3D visualizations of the studied EDs in the early phase of model development to aid domain experts’ understanding of the model logic and structure. Figure 7 illustrates examples of the visual representation of the studied EDs. Such visualization proved to be an effective approach in facilitating model interpretation and enabling effective communication with stakeholders without overwhelming them with detailed model logic and coding. The visualization allowed domain experts to provide valuable input and help refine model structure based on their clinical expertise and knowledge of the local contexts. For instance, this led to the incorporation of representations of chair spaces and surge stretchers in the EDs. Overall, incorporating visualization into the models improved model transparency and allowed for effective face validation of the models with domain experts.

The DES models were configured with a warm-up period of 5 months, followed by a 2-year run. Each scenario was replicated 40 times. The details on the model structure, inputs, assumptions, and experiment setup were reported in the previous study [24]. We focused on studying two ED wait time outcome measures: time waiting for physician initial assessment and time waiting for an inpatient bed. External model validation was performed by comparing the quantitative outputs of the simulation models in the baseline scenario (no intervention) with empirical data on the two outcome measures for the six studied EDs located in three health regions.

Table 2. Model validation [24].

	Mean Time Waiting for Physician Initial Assessment, Hours			Mean Time Waiting for Inpatient Bed, Hours
Site	Simulated	Actual	Δ	Simulated	Actual	Δ
Site 1	1.99	2.04	−0.05	10.17	10.18	−0.01
Site 2	1.18	1.27	−0.09	7.81	7.88	−0.07
Site 3	1.26	1.26	0.00	11.13	11.01	0.12
Site 4	2.29	2.35	−0.06	4.65	4.61	0.04
Site 5	1.36	1.43	−0.07	1.38	1.29	0.09
Site 6	0.95	0.94	0.01	0.68	0.69	−0.01

Δ: difference (Simulated—Actual). Sites 2, 3, and 6 are in region A; site 5 is in region B; sites 1 and 4 are in region C. Licensed material reproduced with permission from Springer Nature.

displays the validation results. It is important to highlight that we performed multiple validations at various stages of model development, particularly when structural modifications were incorporated or new model inputs or data were used in the DES models.

5.4. Stage 4: Model Use and Policy Co-Development

The use of the multi-criteria framework allowed for a systematic approach to identify, screen, and shortlist the intervention scenarios to reduce ED wait times in each health region. This approach was critical in achieving the goal of co-developing effective interventions for reducing ED wait times. The use of the scoring matrix, as illustrated in Table 1, enabled the modeling team, regional leaders, and stakeholders in the advisory group to collaboratively assess each proposed intervention scenario based on a set of predefined criteria. This allowed for a transparent evaluation of the proposed interventions. The resulting shortlist of interventions, as shown in

Table 3. Shortlisted intervention scenarios per health region in the third advisory group meeting.

Region	Scenario Name	Organizational Readiness	Regional Data Availability	Administrative Data Availability	Literature Support for Modeling	Regional Priority	Provincial Priority	Modeling Effort	Length of Time to Impact Drivers	Length of Time to Get Service Ready	Total Score
A	Attachment to primary care provider	3	3	5	5	5	4	4	3	3	35
	Reducing admissions for ACSCs	3	3	4	3	5	4	4	3	3	32
	Reducing ED visits for family practice sensitive conditions	3	3	4	1	5	4	4	3	3	30
	Alternate level of care reduction strategy	3	2	1	3	5	5	2	3	3	27
	High-quality care transitions (discharge planning and coordination)	3	3	4	4	4	4	1	4	3	30
	Hospital at home (early supported discharge) for surgical and neuro patients	4	4	4	5	4	4	5	3	3	36
B	Expansion of community nurse practitioner services for COPD patients	5	5	3	2	5	4	3	5	4	36
	Hospitalist model for medical and surgical units	5	3	3	3	4	4	3	3	2	30
	Additional ED physician coverage	5	5	2	1	3	2	4	4	4	30
C	Reducing ED visits for family practice sensitive conditions	4	4	4	1	4	3	4	3	3	30
	ALC reduction strategy	-	1	3	3	-	5	2	3	3	20
	COPD clinical pathway	4	2	2	3	4	3	2	3	3	26
	High-quality care transitions (discharge planning and coordination)	-	3	4	4	3	4	1	3	4	26
	Hospital at home (early supported discharge) for surgical and neuro patients	4	4	4	4	4	3	5	3	5	36

ACSCs, ambulatory care sensitive conditions; COPD, chronic obstructive pulmonary disease; ALC, alternate level of care.

, reflected the varying priorities and readiness of each health region, as well as the unique contexts and varying perspectives of the stakeholders involved. After the third advisory group meeting, we incorporated several additional intervention scenarios for modeling based on discussions with stakeholders. These additional scenarios included combinations of individual interventions presented in Table 3 and expansion of existing programs in the healthcare system.

Considering the diverse range of proposed interventions aimed at specific patient groups, we systematically identified and extracted the corresponding target patient groups for each intervention in each studied ED or hospital and represented them in the DES models.

Table 4. Effects of selected scenarios on ED wait times.

Scenario	Simulated Mean Time Waiting for Physician Initial Assessment, Hours			Simulated Mean Time Waiting for Inpatient Bed, Hours
	Pre	Post	Δ	Pre	Post	Δ
Reduce ACSC-related hospitalizations by 10% (input)
Site 1	1.99	1.95	−0.04	10.17	9.38	−0.79
Site 2	1.18	1.16	−0.02	7.81	7.43	−0.38
Site 3	1.26	1.22	−0.04	11.13	10.62	−0.51
Site 4	2.29	2.28	−0.01	4.65	4.47	−0.18
Site 5	1.36	1.36	0	1.38	1.22	−0.16
Reduce LTC-related ALC hospital days by 30% (output)
Site 1	1.99	1.94	−0.05	10.17	7.83	−2.34
Site 2	1.18	1.12	−0.06	7.81	7.08	−0.73
Site 3	1.26	1.26	0	11.13	10.99	−0.14
Site 4	2.29	2.25	−0.04	4.65	3.95	−0.7
Site 5	1.36	1.36	0	1.38	1.23	−0.15
Reduce Non-LTC-related ALC hospital days by 30% (output)
Site 1	1.99	1.93	−0.06	10.17	7.66	−2.51
Site 2	1.18	1.07	−0.11	7.81	6.21	−1.6
Site 3	1.26	1.26	0	11.13	10.91	−0.22
Site 4	2.29	2.25	−0.04	4.65	4.10	−0.55
Site 5	1.36	1.36	0	1.38	1.28	−0.1

Δ: simulated outcome postintervention minus that from the preintervention (baseline scenario). LTC: long-term care; ALC: alternate level of care; ACSC: ambulatory care sensitive conditions. Per scenario, 40 replications. We eliminated site 6 for these three selected scenarios because the ED in site 6 does not provide 24/7 service and has low ED admission rates; thus, the presented scenarios are not expected to have an impact on its ED patient flow and wait times. Sites 2 and 3 are in region A; site 5 is in region B; sites 1 and 4 are in region C.

presents the effects of three selected scenarios on ED wait times, with one targeting the input component of the ED patient flow, and the other two targeting the output component. The quantitative simulation results of other proposed intervention scenarios have been described and discussed in detail elsewhere [24,112]. The three scenarios in Table 4 studied two patient groups: (1) patients aged under 75 who were hospitalized for ambulatory care sensitive conditions (ACSCs), and (2) inpatients who no longer need acute care but experienced delayed hospital discharge, also termed as alternate-level-of-care (ALC) patients.

Hospitalizations for ACSCs are considered largely preventable through improved primary care on an outpatient basis [113]. These conditions have been well documented in the literature. We analyzed the hospitalizations for five ACSCs: asthma, heart failure and pulmonary edema, chronic obstructive pulmonary disease (COPD), diabetes, and angina. We extracted the patient group using the most responsible diagnosis codes and cases with certain cardiac procedures were excluded for certain ACSC conditions [113,114]. Individuals aged 75 years and older were also excluded. It is worth noting that COPD is one of the ACSCs, and many health regions were interested in COPD prevention and care. Hospitalizations for these potentially avoidable ACSCs account for 1.9% to 4.4% of the hospitalizations in the five studied sites in Table 4.

We used ALC hospital days to measure the delays in hospital discharge experienced by ALC patients. The reasons for these delays can differ. They can occur when ALC patients are waiting for transfer to long-term care (LTC) facilities, resulting in LTC-related ALC hospital days. Alternatively, delays can occur before patients are discharged home but are awaiting post-acute care or home care, resulting in non-LTC-related ALC hospital days. The effect sizes of these three scenarios, such as 10% and 30%, were obtained from the advisory group and domain experts based on their assessments of achievable targets.

Reducing ACSC-related hospitalizations led to a slight decrease in the mean time spent waiting for an inpatient bed across EDs. The reduction in mean time waiting for an inpatient bed was greater in site 1 (∆ 0.79 h) owing to the higher proportion of ACSC-related hospitalizations in this hospital (4.1%). Reducing either LTC-related ALC days or non-LTC-related ALC days resulted in a larger reduction in mean time waiting for an inpatient bed for sites 1 and 2. The three selected scenarios had limited impact on the mean time waiting for physician initial assessment; therefore, they will not be able to solve the ED waits associated with physician assessment.

6. Discussion

The study presented a case study that aimed to support the identification of interventions to address the complex problem of lengthy ED wait times from a whole-system perspective in a Canadian policy development setting using a participatory modeling approach. We presented and discussed the quantitative results of the model findings in a previous contribution [24]. This paper focused on describing the formulation of the participatory modeling approach used and demonstrating its application in the process of developing DES models of EDs and acute care patient flow for six studied hospitals located in three Saskatchewan health regions. The participatory modeling approach involved engaging with key stakeholders in various modeling stages and co-developing interventions with stakeholders using a multi-criteria framework to identify, screen, and shortlist modeling scenarios to support effective policy development. The participatory modeling approach enabled the provincial initiative to effectively engage a broad range of stakeholders to examine and identify the causes and solutions to the ED wait time in the studied hospitals. Our methodology also contributed to the M&S field by introducing an innovative multiparadigm hybrid modeling approach that integrates qualitative and quantitative paradigms in different modeling stages [71].

To assess the impact of experts’ and stakeholders’ involvement in improving the model, we compared the proof-of-concept DES model (Figure 3) with the DES models developed using a participatory approach (Figure 6). While both models aimed to capture ED and acute care patient flow, significant differences were observed in their structures and development processes. The proof-of-concept DES model was constructed in an ad hoc manner, with the modeler taking the lead and consulting individual experts and stakeholders for information on the ED problem. The modeler represented the ED patient flow based on the obtained knowledge with very limited and ad hoc involvement of domain experts and stakeholders in the model development process. In contrast, the subsequent DES models were developed using a multistage participatory approach through iterative deliberation with stakeholders. Using the proposed participatory approach, domain experts and stakeholders collectively discussed evidence and shared knowledge about the causal mechanisms of the ED wait problem, identifying key drivers and areas requiring improvement through value stream mapping in the conceptual modeling stage. As a result of the different model building processes, notable differences emerged in the model structures. The proof-of-concept DES model primarily focused on ED patients, considering only ED-admitted patients for the acute care component, and featured simplified ED activities. In contrast, the later DES models incorporated detailed ED activities, included elective patients, stratified inpatients based on patient service groups, and accounted for separate alternate-level-of-care lengths of stay. The active and collaborative engagement of experts and stakeholders through the participatory approach contributed to these improvements in model structure and enhanced the modeler’s understanding of the ED problem. The models were also better designed to evaluate proposed interventions by stakeholders. The use of qualitative tools and techniques in the conceptual modeling stage not only improved our shared understanding of the ED problem situation in the studied hospitals but also complemented the development of the DES models [62]. It is important to note that the ways in which the stakeholders and domain experts are engaged, either collaboratively or consultatively, also exert a significant impact on the development and quality of the models. Furthermore, the participatory approach allowed for a comprehensive system perspective. Given that ED and acute care systems are complex systems, individual experts or stakeholders may possess deep knowledge of specific aspects while lacking a holistic understanding. Engaging them collectively through the participatory approach provided a platform for multiple perspectives to be shared and fostered a broader understanding of the ED problem. In contrast, individual engagement could risk losing the systemic perspective that is crucial for addressing the ED wait times in the face of the complexity associated with the ED and acute care systems [115]. The system perspective enabled by applying a participatory modeling approach allowed us to study ED wait times as a part of a larger integrated acute care system. Our study contributed to the literature on whole-system DES models of ED patient flow for policy development.

Our study contributes to the conceptual modeling literature by demonstrating the use of the quality improvement toolkit and lean techniques as a problem structuring method to support problem conceptualization in the health policy development context [72]. Unlike other problem structuring methods which were rooted in the OR domain [53,116], we explored and used a number of existing tools that had already been used in the healthcare system, such as lean techniques and quality improvement tools for conceptual modeling. During the conceptual modeling stage, it is common for the modeling team to take the lead and utilize mapping tools that are unfamiliar to the domain experts or stakeholders. Persuading domain experts and patient flow managers to use new tools can be challenging and may require additional training and resources. Moreover, such stakeholders may not feel ownership of the project if they are only being consulted for information. In this study, instead of introducing new conceptual mapping tools, the modeling team explored and learned about existing quality improvement and mapping tools that were already in use within the healthcare system for structuring the problem. For instance, we used VSMs of ED patient flow that had already been developed by domain experts to conceptualize the problem of ED wait times. The use of these existing tools also facilitated communication with domain experts and flow managers about the ED wait time problem in the studied hospitals, given their familiarity with the lean techniques through the implementation of the lean methodology across the province’s healthcare system at the time of this research. Such findings are consistent with previous research demonstrating that combining lean techniques with discrete-event simulation is a promising approach for studying care processes in the healthcare setting [117,118,119].

Two important facets of trust that influence the implementation of simulation findings are the stakeholder–model relationship and the stakeholder–modeler relationship, as the decision to implement changes based on simulation results is held by the stakeholders [19,20]. In contrast to the existing participatory modeling approaches in quantitative modeling, which primarily focus on resolving specific problems and generating quantitative forecasts or estimates [22], the novelty of our participatory modeling approach also lies in dedicated processes and activities specifically aimed at building trust in the computer modeling approach with the stakeholders and improving stakeholders’ modeling knowledge. We conducted proof-of-concept analyses during project initialization and provided simulation modeling workshop to help stakeholders become familiar with the computer simulation approach. This helped foster trust at the beginning of the study and build confidence in the modeling approach that we used. Furthermore, our approach empowered stakeholders to develop skills to apply the modeling approach on their own. This, in turn, strengthened the stakeholder–model relationship, as they perceived the modeling approach to be more useful and credible. The stakeholder–modeler relationship is primarily centered around the stakeholders’ perceived trust in the modeler’s modeling and communication competencies, particularly whether stakeholders perceive that the modeler has a good and shared understanding of the problem situation in the early phases of a study [19,20]. The conceptual modeling component in our participatory approach played a key role in building trust in this relationship, thereby further improving trust in the stakeholder–model relationship.

Another contribution of our participatory approach lies in the development and use of the multi-criteria framework for co-developing interventions to address ED wait times. The use of the framework helped stakeholders reach an agreement regarding the feasible solution space and allowed a transparent assessment of the proposed actions. We included both the assessment of evidence and the availability of data in the framework to promote evidence-based decision making. In previous simulation studies, modeling scenarios were typically not fully designed to address the needs of stakeholders or policymakers. While stakeholders may have been involved in the development of intervention scenarios, their involvement was often limited to providing information rather than having a sense of ownership or being held “accountable” for the proposed intervention scenarios. In our study, we worked closely with stakeholders by involving them in the identification and prioritization of intervention scenarios through advisory group meetings. Stakeholders were held “accountable” for their proposed intervention scenarios, as a contact person was identified for each proposed scenario. The stakeholders and the modeling team collaborated to fill in a scoring matrix to prioritize intervention options based on criteria such as data availability, local practice, regional priority, and time to impact. This approach conferred on stakeholders a high degree of ownership and trust in the process and allowed regional leaders to view the scoring matrix of other regions. This process further aided in breaking the silos between organizations and various departments.

This case study further supported some learning regarding the application of the participatory modeling approach. We agreed with previous research findings on the importance of engaging stakeholders early on in the participatory modeling approach [42,48]. We would also like to highlight the role of the project champion in the team composition, as the project champion served as a key linkage between the modeling team and the stakeholder group and played a crucial role in building trust in the modeling methodology. Previous studies have also emphasized the importance of having a project champion in participatory modeling projects, as they can greatly contribute to the successful completion of such projects [10,57]. Our core engagement with stakeholders was in the co-development of intervention solutions to reduce ED wait times. Although we have been transparent about the model assumptions and model validations, we experienced challenges in building trust in the model outputs when the model outputs contradicted the stakeholders’ prior beliefs or expectations. This phenomenon was also discussed in previous research [3,15]. In such cases, we found that it was important to facilitate ongoing and open dialogue with stakeholders through facilitated meetings.

7. Limitation and Future Work

The study has several limitations. The participatory modeling approach was applied to the case study of modeling ED and acute care patient flow using DES in a Canadian health policy setting. Due to the context-specific nature of the case study, the participatory modeling approach may not be directly applicable in other health-policy settings due to differences in organizational structures and policymaking environments. Second, the modeling team completed the scenario runs outside of the advisory group meetings, which might have reduced model transparency with stakeholders. An ideal solution would be to build models with a visual interface that allows for real-time experimentation during the advisory group meetings. However, this would require the identification of input variables related to the scenarios prior to the meetings and it is not always feasible to explore all the possible intervention solutions. Lastly, while the advisory group meetings were facilitated, formal scripts (e.g., PartiSim scripts) were not utilized, leading to a less structured approach that lacked standardization and might have resulted in missed opportunities.

The use of participatory modeling approaches with DES is still in its early stages in healthcare [59] and future research is needed to explore and develop innovative tools and processes that encourage stakeholder participation in the DES modeling processes and improve the implementation of the model findings. This includes combining tools or processes from diverse participatory modeling approaches (e.g., combining elements of GMB with PartiSim) and drawing insights from these experiences, and enhancing model validation [120]. This study further suggests the value of investigating the adaptation of existing tools employed in the healthcare system for use in the participatory modeling processes.

8. Conclusions

This study has successfully employed a four-stage participatory modeling approach with discrete-event simulation to identify intervention strategies for reducing ED wait times in a real-world health policy setting. The use of a participatory approach has enabled a broad range of stakeholders to examine and identify the causes and solutions to ED wait times and co-develop interventions for implementation. This approach has shown to be an effective way of engaging stakeholders throughout the modeling process and building consensus. Additionally, the study embraced a system view by studying ED wait times as a part of a larger integrated acute care system with multiple interacting subsystems or units, contributing to the literature on whole-system DES models of ED patient flow for policy development.