Strengthening Participation Using Interactive Planning Support Systems: A Systematic Review

Abstract

Interactive Planning Support Systems (PSS) implemented on a maptable are deemed suitable to support participatory planning processes. They are supposed to facilitate exchange of knowledge between stakeholders, consensus building among them, and group-learning processes. In this systematic review, based on 16 case studies using interactive PSS, we analyze how these have contributed to the goal of strengthening stakeholder participation. To this end, we first elicit details of the interactive PSS and the related participatory processes. In the second step, we analyze how and what the studies report, as the impacts on participation. Results show that tools and applications have become more sophisticated over time and goals of the studies changed from collaboratively designing interventions to observing and understanding how the application of such tools contributes to improved plan outcomes and group-based learning. All interactive PSS succeeded to facilitate intensive stakeholder collaboration. However, many studies lack a proper framework for investigating its impacts on participation and therefore assess these rather incidentally based on implicit assumptions. Thus, a significant outcome of this review is an evaluation framework, which allows the structural assessment of the impacts of interactive PSS on stakeholder participation.

1. Introduction

With the communicative turn [1], planning theory moved its focus from “a preoccupation with the distribution of material resources” to a “process of working out how to coexist in shared space” [2] (p. 219). Since then, collaborative planning, understood as an interactive and interpretative process involving a dialogue between stakeholders [2] (p. 221), has become the major paradigm guiding the practice of spatial planning and strategy making. However, the growing complexity of multidimensional planning tasks [3] calls for other forms of public participation, i.e., forms that go beyond what is legally required. These should enable “a multiple-way set of interactions among citizens and other players who together produce outcomes” [4] (p. 419). Closely related, Planning Support Systems (PSS), defined as “geo-information-technology-based instruments that are dedicated to supporting those involved in planning in the performance of their specific tasks” [5], are being implemented as instruments attuned to support specific planning tasks [6].

A particular type of PSS aiming to enable the collaboration of stakeholders are PSS developed for interactive surfaces, so-called interactive PSS. These interactive PSS are hardware systems in the form of a touch-sensitive table, a maptable, combined with geospatial mapping and analytical tools to support group processes [7]. Such interactive PSS provide a platform for discussing spatial problems or evaluating planned alternatives using a shared map interface to facilitate interaction among stakeholders [8]. Through their interactive nature and user-friendly interface a maptable might facilitate exchange of knowledge and preferences between stakeholders [9], consensus building among them [10], and group learning about complex problems [11,12], among others.

Since almost 15 years, various applications of such interactive PSS have been developed and implemented in case studies, often related to urban planning. These case studies, their specific contexts, tools implemented and participatory processes supported are often well reported. However, many studies focus rather on the evaluation of the usability [13] or user friendliness [14] of the PSS, i.e., the functioning of the tool [15], then on its usefulness, i.e., the impact of the use of such tools on the planning quality, though the usability of a PSS is rather a means than a goal in itself [16]. What is missing is a systematic review of case studies applying interactive PSS regarding their impacts on stakeholder participation.

We analyze in this review, based on case studies reported in interactive PSS publications, how such PSS have contributed to the goal of strengthening stakeholder participation. To this end, we first review existing interactive PSS and elicit details of the applications and cases studies, the PSS tools developed and the participatory processes they were used in. In the second step we then systematically analyze how and what the various studies report as the impacts of the use of the interactive PSS on participation, based on a framework for evaluating participation that we develop beforehand.

Accordingly, the paper is structured as follows: In Section 2 we provide an account of interactive PSS applications, tools and definitions and their use in stakeholder workshops. In Section 3 we develop a framework for evaluating impacts of interactive PSS on stakeholder participation. Section 4 elaborates the methodology applied in the review. Section 5 provides the results of the review analysis, i.e., the review of the interactive PSS applications, tools, and participatory processes (Section 5.1), and the reported impacts of interactive PSS on the various aspects and criteria of participation (Section 5.2). Section 6 discusses the results gained in the review before Section 7 concludes with key findings and further research needs.

2. Interactive PSS Enhancing Stakeholder Participation in Planning

Conceptual and technological research on maptables dates back to the turn of the century. Coors et al. [17] presented a first prototype of a virtual table composed of an interaction interface and 3D-GIS linked to it. In 2001, the MIT presented its Luminous Planning Table (LPT), a prototype developed as a “new form of information delivery that is direct and easily understood by professionals and laypersons” [18] (p. 195). Hopkins et al. [19] presented a so-called sketch-planning workbench implemented on a large-scale touch-sensitive screen that used gestures for drawing, annotating, and manipulating ideas during a collaborative sketch planning session.

The first case study of interactive PSS implemented in stakeholders workshops dates back to 2006 [20]. Since then various applications of interactive PSS have been developed and tested in various sectors of urban and regional planning, including sustainable urban development [11], climate change adaptation [21], energy transition [9], environmental health [12], and urban redevelopment [22], among others. Next to that, a couple of studies applying maptables in the field of disaster risk management can be found [23,24].

2.1. Interactive PSS Tools

An interactive PSS tool typically consists of a large-scale horizontal, touch-sensitive screen, that stakeholders can stand around and interact with, and a geospatial tool that enables the interaction (Figure 1). Sometimes a second, smaller screen is added to the setup to visualize the results of stakeholder’s interaction or other information, e.g., a 3D model. While over the last ten years, technological innovations have immensely improved the availability and affordability of large screen touchscreen as the key hardware for interactive PSS, suitable geospatial tools particularly made for interactive PSS applications are still hardly available. The development of interactive PSS tools is typically done by researchers for specific case studies and contexts, in some cases following a socio-technical approach [25] in order to increase their usefulness [15].

Proven benefit of an interactive PSS compared to a desktop PSS is that the maptable has a positive influence on working styles and group dynamics and thereby allows an enhanced interaction between stakeholders. Therefore, such interactive PSS are particularly suited to support participatory activities within a planning process. Dias et al. [26] describe the use of such interactive PSS in participatory planning processes as a promising way to bridge the gap between the creative design process of architects and urban designers and the more analytical process of planners. Ryall et al. [27] report the use of such applications “less intimidating, and often more playful than a traditional computer desktop environment”. Doeweling et al. [28] compared the use of classical paper maps, desktop GIS and interactive maptables in the context of stakeholder collaboration during crisis management. The authors found that a maptable-based solution can provide significant advantages when multiple users with different roles collaborate in sense making and planning tasks. Finally, the horizontal setup of a maptable revealed significant benefits for group collaboration compared to a vertical setup of the same device [29,30].

2.2. Planning Processes and Interactive PSS Workshops

A typical planning process can be structured into the phases of intelligence, design and choice [31]. The intelligence phase often comprises a baseline and problem analysis, in the design phase typically alternative plans or solutions are designed, while in the choice phase a decision is made on which plan alternative to implement [32]. Theoretically, stakeholders might be involved in any or even multiple of these phases of a planning process, e.g., by contributing the problem analysis, designing alternative plans, or evaluating the different alternatives [33].

The application of interactive PSS tools is typically done in PSS workshops [34] that often relate to a specific step or task within such a planning process. Two types of workshops can be distinguished here, controlled experiments and close to real-world stakeholder workshops. In controlled experiments, tasks often involve students playing a role of stakeholders [34]. Such experiments allow to test specific questions with respect to the usability of the tool, a comparative analysis of different tools [28,35], or tools being tested under changing conditions or with different user groups [36]. Moreover, regularly larger samples for statistical analysis are achieved.

On the other hand, in close to real world stakeholder workshops, tests provide richer information on their added value for the planning process and outcomes. However, those workshops are more complex to set up and implement, and include a certain level of uncertainty. Ideally, controlled experiments with a PSS are conducted prior to using it in stakeholder workshops. Finally, both types of workshops benefit from a high degree of facilitation, including a moderator steering the discussion and a chauffeur supporting the use of the PSS [37].

Methods for the evaluation of the use of interactive PSS in stakeholder workshops range from observations, interviews with participants, surveys based on questionnaire, to audio or video recording of stakeholder sessions. Often various methods are combined in order to verify the consistency of findings based on multiple sources via triangulation. Janssen et al. [38] conducted participant’s surveys before and after the workshops in order to explore how perceptions and preferences changed throughout a planning session.

3. Impacts of Interactive PSS on Participation

In order to analyze to what extent interactive PSS tools and applications have contributed to the goal of strengthening participation during PSS workshops, a framework for evaluating participation is needed. Such framework can be built from two partly separated scientific debates: First, the evaluation of public participation, which is an essential element of planning practice; second, the evaluation of PSS in the context of planning support science [39].

The evaluation of public participation within a planning context essentially looks at the effectiveness [40] or success or failure [41] of the participatory activities and processes. Gunton and Day [42] (p. 9) identified four key evaluative criteria for assessing participatory practices, which are: “(1) Success in reaching an agreement, (2) efficiency of the collaborative process relative to alternative processes, (3) satisfaction of stakeholders with the process and outcome, and (4) achievement of other social capital benefits such as improved relationships among stakeholders and enhanced stakeholder skills and knowledge”. More elaborated evaluation frameworks were developed by Rowe and Frewer [43], Laurian and Shaw [41], Brown and Wei Chin [40], and Mannieri and Talo [44]. Commonly, these frameworks distinguish between process criteria, that refer to the effective construction and implementation of a participatory activity, and acceptance or outcome criteria that refer to the broad range of results obtained from a participatory activity [40,43]. The frameworks include a number of—partly similar—process and outcome criteria. Depending on the goals of an individual participatory activity different sets of criteria might be taken into account, because “each of these criteria suggests a different perspective on what ‘successful’ participation means” [41] (p. 296). Moreover, these criteria are not mutually exclusive, because a participatory process might “seek to promote different goals simultaneously” [41]. Finally, these criteria are developed and used independent of the individual participatory methods, e.g., public hearing, consensus conference, focus group discussion.

The evaluation of the use of PSS in participatory workshops has focused for long time rather on the usability of the PSS tools [13,45] than on its impacts on the participatory process. Only since 2010 the latter has gained attention in the scientific debate on PSS (e.g., [7,8,14,15,16,46]. However, studies that evaluate the impact of the tools on the quality of the planning process and outcomes adopt different assessment frameworks, such as performance [16], added value [7,46], or effectiveness [47], all of them being based rather on implicit assumptions than on empirically grounded outcomes [15]. These frameworks likewise distinguish process and outcome criteria, but a thorough operationalization of the criteria is often lacking. Instead, scholars investigate potential correlations between the usefulness of a PSS and certain explanatory variables manifested in the usability of the particular PSS tool in order to determine options for improvement of the PSS. For example, te Brömmelstroet [14] has investigated the correlation between the usefulness of a PSS and its user-friendliness.

The evaluation framework developed for this study (

Table 1. Framework.

Dimensions, Aspects and Criteria	Definition in a PSS Workshop Context	Sources
Process: Communication and collaboration (C)
Transparent (tr)	Information about issues and process is available, clarity on model assumptions is given	[40,41]
Fair (fa)	Dialogue is free from bias, no group or single participant is dominating the workshop, and all participants are respectful with each other	[40,41]
Inclusive (in)	All participants are given equal opportunities to actively participate in the discussion	[40,41,44]
Efficient (ef)	Same and more tasks can be conducted with lower investments and/or in less time	[7,16]
Process: Social learning (L)
Awareness raising (aw)	Participants are informed and increase their knowledge about an issue	[9,41]
Single loop learning (sl)	Participants are changing behaviour to address a challenging situation	[12,49]
Double loop learning (dl)	Participants are reflecting underlying assumptions and values	[12,49]
Outcomes: Issue related outcomes
Knowledge created/integrated (kn)	Participants tacit/experiential knowledge is added	[12,49]
Consensus achieved (co)	Acceptable solutions are found based on mutual understanding	[7,40,41]
Understanding of the problem improved (un)	Participants can understand the given information and material (un)	[14,16]
Quality of decision improved (qa)	Decision integrates broad knowledge base and public input, New aspects added	[41]
Outcomes: Social outcomes
Ownership/commitment (ow)	Participants are committed to the outcome/plan	[16,50]
Mutual understanding (mu)	Participants understand each other’s perspectives and issues	[14,16,40,41]
Community building (cb)	Development of new collaborations, improved social cohesion	[51]
Satisfaction (sa)	High satisfaction among participants, participants are satisfied with process and outcomes	[14,16,40,41]

) integrates dimensions and criteria from both scientific debates. It distinguishes criteria to evaluate the participatory process as well as those to evaluate the outcomes achieved through the participatory process. Both types of evaluation criteria are meant to evaluate participatory activities conducted in a PSS workshop context.

Process criteria include on the one hand criteria assessing the quality of the communication and collaboration between stakeholders during the PSS workshop and on the other hand whether and how social learning, understood as a learning through group activities [48], happens during the process. A participatory activity enabling a good communication and collaboration between stakeholders supports what Laurian and Shaw [41] call a democratic process. Such a process should ideally be transparent ([40,41], fair [40,41], inclusive [40,41,44], and efficient ([7,16]. The two criteria of a process being inclusive and fair are closely related to each other, though the latter is sometimes overlooked in participatory practice [41]. Inclusive means that all participants are given equal opportunities to attend a workshop [44]. Fair means that all participants are given equal voice or standing [41], the dialogue is free from bias, no group or single participant is dominating the discussions, and all participants are respectful with each other [44].

Criteria for social learning during a PSS workshop evaluate the development and increase of knowledge and understanding [44] of the participants during the participatory PSS process. Different levels of social learning are to be distinguished here. Awareness raising occurs when participants are informed about issues and stakes at hand [41] and they increase their knowledge about these [9]. Single and double loop learning occurs when repeated experiences shared in group dialogues help participants to detect a mismatch between intentions and consequences [52]. Single loop learning is defined when participants are challenging their behavior to address a challenging situation, while double loop learning refers to participants reflecting underlying assumptions and values [53].

Outcome criteria are structured into issue related outcomes and social outcomes [41]. The latter is frequently neglected in PSS workshops, probably due to their singular, often project-related character. Issue related outcome criteria evaluate the broad range of results obtained from a PSS workshop. They comprise the creation and/or integration of knowledge, such as tacit, contextual-embedded, and codified knowledge [54], whether there is an enhanced understanding of the problem [12], whether a consensus is achieved, and whether the quality of the decision is improved [41]. The last one also includes the aspect of novel ideas or issues being developed (e.g., new alternative plans or interventions), that te Brömmelstroet [14] mentions. Social outcome criteria measure whether PSS workshops result in capacity building, and increased trust and social networks [41]. They include creation of commitment and ownership among the participants [16,50], the development of a mutual understanding of each other’s perspectives and issues [14,16,40,41], the initiation of community building, e.g., new collaborations or improved social cohesion [51], and whether participants are satisfied with the outcomes of the workshop [14,16,40,41].

Though separated in the framework, process and outcome criteria are closely related, because they likely affect each other [51]. For instance, achieving a common and good understanding of the problem is a prerequisite of achieving a consensus. Likewise, a fair and inclusive process is not only a prerequisite for consensus building but to a certain degree also for achieving a common and mutual understanding.

4. Review Methodology

The review is based on the analysis of studies describing interactive PSS that are used for facilitating stakeholder collaboration in urban planning related participatory activities. Figure 2 depicts the entire review methodology.

The review includes tools and applications published in journal articles and conference proceedings. If various articles refer to the same interactive PSS tool, they are considered as one study. The search strategy for finding publications included two steps. First, we searched for publications from the year 2000 to 2018 in databases using relevant combinations of keywords plus commonly used synonyms. We then screened the obtained long list of publications and selected papers based on predefined selection criteria. Scopus was used as the search database, because it also includes publications in conference proceedings, with the following search terms:

• Keyword = maptable or one of the following synonyms: touchtable, tabletop, touch user interface, tangible user interface, tui, plus different spellings of these terms such as map table;
• AND
• Keyword = urban planning or one of the following related terms: spatial planning, strategic planning, urban design.

In total 58 papers were found in Scopus. After reading the abstract of these papers, a short list of papers to be included in the review was compiled based on the following selection criteria:

• Focus on stakeholder participation in spatial planning in general, i.e., planning related activities at any scale (national to sub-local),
• Use of the maptable to facilitate the participatory process,
• Real-world case study, i.e., applied in a real context, with real stakeholders, and addressing a real or at least realistic problem setting, i.e., a typical contemporary planning issue,
• Peer reviewed and published articles, book chapters, or conference papers,
• Interactive PSS applications, tool and workshop described in sufficient detail, so that the necessarily details for all criteria can be elicited.

The following table presents the 16 tools and papers included in the study (

Table 2. Studies to be included in the review.

ID	Interactive PSS	Year	Purpose	References
1	MapTable	2006	Collaborative design water management measures along a river	[20]
2	Touch Table	2010	Integration of sectoral climate change adaptation measures	[55]
3	Landscape Constructor System	2011	3D design and visualization of ecological corridor for forest regeneration	[56,57]
4	SimLandScape	2011	Integration of design and planning to facilitate a collaborative process	[58,59]
5	Interactive decision support tool	2012	Involvement of stakeholders in the entire planning process of analysis, design, choice	[10,60]
6	MapTable PSS	2013	Improved communication among stakeholders during different planning stages	[11,37]
7	Interactive geodesign tool	2014	Design of three spatial adaptation strategy for water management in peat meadow area	[38]
8	Geospatial Tangible User Interface (GTUI)	2015	Support of novel collaborative planning concepts	[61]
9	Adaptation Support Tool (AST)	2016	Collaborative design of green and blue adaptation measures	[21]
10	geodesign simulation game	2016	Negotiation of the supply of development sites at a regional level	[62]
11	Interactive Spatial Understanding Support System (ISUSS)	2017	Support of interactive knowledge co-production among stakeholders in an environmental health context	[12]
12	GPS Tracking and Maptable tool	2017	Exploration of spatial behaviour in the local landscape	[63]
13	Collaborative Location and Allocation Gaming Environment (COLLAGE)	2017	Collaborative allocation of renewable energy projects in a city	[9]
14	Cumulative burden assessment (CuBA)	2018	Collaborative assessment of multiple environmental burdens affecting health	[51]
15	Urban Redevelopment Financial Feasibility (URFF)	2018	Development different scenarios of urban redevelopment	[22]
16	Walkability PSS	2018	Exploration and design built environment scenarios regarding their impacts on walkability	[64,65]

). All studies describe the applications of interactive PSS in stakeholder workshops to support steps in a planning process. Some studies made use of multiple case studies. Study ID 6 included one case that was not truly supporting a group process, because stakeholders had to use the table one by one.

The analysis of the studies is structured into two parts (see Figure 2). In the first part the interactive PSS are reviewed in detail, structured into three categories that are defined above: (a) interactive PSS applications, (b) interactive PSS tools, (c) participatory process. Category (a) looks at the context of the applications, i.e., the case studies, the sector, scale and the goals of the applications, category, category (b) studies the details of the hard- and software details of the interactive PSS tools and how they were developed, category (c) finally studies the characteristics of the participatory processes the interactive PSS were applied to including the methods of evaluation. The full list of review criteria, that have been defined based on a screening of all selected papers, is given below (

Table 3. Planning Support Systems (PSS) Review criteria.

Category	Criteria	Description
Application	Case studies, location	Cities/regions and countries where the tool has been applied to
	Sector/field of application	The sector/field of application the tool is developed for and used in within the domain of spatial planning
	Scale	The planning level in which the tool has been used, from national to sub-local/neighbourhood level
	Goal of the application	The goal of the participatory activity, what the stakeholders are asked to do using the interactive PSS
Tool	Hardware setup and system architecture	Hardware components of tool, next to the maptable, e.g., second screen, and system architecture, e.g., stand alone or server client system
	Software	Software used on the maptable
	Main GIS functions	Main GIS function included in the model, e.g., drawing, annotating, multi criteria analysis, scenario analysis, 3D visualization, etc.
	External model(s) coupled	Whether an external model (e.g., simulation model) is coupled to the tool
	Stakeholders involved in tool development	Involvement of stakeholders in the development of the tool
Participatory Process	Planning phase	Phases of a typical planning process that the tool is used in: problem analysis, design, choice [31]
	Stakeholders/Participants	(Type of) stakeholders participating in the process (e.g., researchers, practitioners, citizens)
	Participatory process	What phases or steps the participants are going through during the participatory activity
	Other participatory steps	Whether other (also non-digital) steps and methods are included in the participatory activity, e.g., the drawing of a rich picture [12]
	Process facilitation	Involvement of facilitator, moderator, or chauffeur in the participatory activity [66]
	Process evaluation	Methods applied to evaluate workshop process and/or outcomes

).

The second part of the analysis aims at understanding how the studies covered in the review have addressed and measured impacts on stakeholder participation during the PSS workshops. In order to do so we analysed how the studies report on hypothesized or measured impacts on participation using the framework developed above (Section 3, Table 1). Statements of the papers regarding the impacts of the interactive PSS use on various aspects of participation were matched to the different evaluation criteria and classified into hypothesized or measured (see Appendix B,

Table A4. Communication and Collaboration.

ID	tr	fa	in	ef	C
	H = 2	H = 4	H = 2	H = 3	H = 12
	M = 1	M = 2	M = 7	M = 4	M = 12
	M = 1	M = 2	M = 2	M = 2	M = 3
1 [20]	“The design [of the maptable interface] should address [...] efficiency [...] properly.” (p. 265)				ef	H
	“... Very suitable in brainstorming, quick and easy to present ideas, empowers collaboration (p. 270)				C	M
	“Designing is equally possible for all participants.” (p. 270)				in	M
2 [55]	“...the table setting was expected to better facilitate face-to-face communication...” (p. 7)				C	H
	“...the [...] table was expected to entice participants to interact and share on the table.” (p. 7)				C	H
	The table setting allowed four to five people at a time to view the work and communicate comfortably. (p. 7)				C	M
	“The simultaneous projection of intermediate steps and resulting maps on a wall enabled larger groups to follow the work in progress...” (p. 7)				in	M
	“This led to a moderate degree of group participation.” (p. 7)				C	M
3a [57]	“...such a suite of geo-visualisation tools will enable active participation and exploration of decision spaces by a number of end users.” (p. 331)				C	H
	“enables and engages participants in an interactive session bringing together researcher, landscape planners and policy-makers in a participatory way” (p. 331)				C	H
	“...using this approach helps the user to make changes in their 2D plan […] and view them interactively in the 3D environment immediately...” (p. 338)				ef	H
3b [56]	“All participants can view the designed biolink…”				fa	H
	“...interactive participatory decision making approach to involve multi-stakeholders in landscape.”				C	H
4a [58]	“...enable experts and citizens to exchange ideas in an efficient and accessible way.” (p. 8)				ef	H
	“...invites participants to engage actively in a process of communication ...” (p. 8)				C	H
	“The MapTable environment stimulated discussion...” (p. 13)				C	M
	“...the user who is holding the pen dominates the session...” (p. 14)				fa	M
	“The use of the MapTable keeps all the participants active.” (p. 14)				in	M
	“...leading to relevant and well-informed discussions.” (p. 16)				C	M
4b [59]	“Each stakeholder [...] has the opportunity [...] to contribute its own ideas...”				in	H
5a [10]	“...maps are used as the means of communication to facilitate stakeholder dialog”. (p. 334)				C	H
	“Stakeholders are invited to work together [...] and share their views...” (p. 334)				C	H
	“To communicate information about land use suitability” (p. 334)				C	H
	“...tools were effective for communicating large amounts of information...” (p. 337)				ef	M
	“prompted input from all participants...” (p. 337)				in	M
5b [60]	“During the ‘local-knowledge’ workshop all contributors participated enhancing interactivity and communication across sectors.” (p. 7)				in	M
6a [11]	“...urban designers from the external firm were less actively involved...” (p. 178)				in	M
	“...otherwise the momentum would have been lost.” (p. 179)				ef	M
	“…leading to sensitivity to technical errors and lengthy calculation times.” (p. 179)				ef	M
	“...of the environmental analysts found it difficult to think in terms of scores of 1–10...” (p. 180)				in	M
	“...would be successful if the urban designers would stay around MapTable until the end...” (p. 180)				in	M
	“...intensive discussions about the incorporation of environmental issues in the spatial plan...” (p. 180)				C	M
	“...measures couldn’t be presented right away, people lost their concentration...” (p. 181)				ef	M
	“...leads to an open and constructive group dynamics...” (p. 182)				fa	M
6b [37]	“...the PSS was utilized to support this collaboration...”				C	H
	“...the MapTable PSS clearly involved [...] communication support...”				C	M
	“It enabled the different professionals to explore together the sustainability Consequences...”				C	M
	“...it had a broader aim of supporting a constructive dialogue with increased collaboration and communication...”				C	H
	“...were enthusiastic about how this information on consumption patterns was visualized and communicated...”				C	M
	“...is its ability to keep participants active around it, their attentions focused on the information...”				ef	M
	“...the MapTable PSS [...] led to a fair, clear and transparent selection process...”				fa, tr	M
	“...participants could directly be addressed by other participants and convinced to participate in the collaboration process...”				C	M
7 [38]	“...provided a common platform for discussion.” (p. 104)				C	H
	“...did well in facilitating exchange of information...” (p. 113)				C	M
8 [61]	“...Users should be able to talk to each other face-to-face...” (p. 189)				C	H
	“...provide a new, common language for inter-discussion.” (p. 189)				C	H
	“...each of them playing an equal part in the discussion.” (p. 190)				fa	H
	“...communicate differently [...] and will collaborate more...” (p. 196)				C	H
	“...stakeholders to discuss longer...” (p. 196)				ef	M
	“...the discussion process will get shorter and that a solution will be found faster...” (p. 197)				ef	M
	“...allowing stakeholders to express themselves in a clear way... “(p 197)				in	M
	“...allows everybody to participate...” (p. 198)				in	M
	“...encourage communication...” (p. 198)				in	M
	“...some citizens will have problems to understand everything.” (p 198)				in	M
	“...helpful for brainstorming with other people or to convince a politician of a strategy...” (p. 201)				C	M
	“...multiple participants actively interacted...” (p. 202)				in	M
9 [83]	“...discuss alternative measures...” (p. 428)				C	H
	“...facilitates reasoning together...” (p. 429)				C	H
	“...effectively supported climate-proof planning...” (p. 433)				ef	M
	“...details of the plan were discussed among participants at the design table...” (p. 434)				C	M
10 [62]	“...facilitated a focused form of communication...” (p. 84)				C	M
	“All participants pointed to relevant features on the map...” (p. 84)				in	M
	“...in order for the negotiation to work, insight in the figures is essential...” (p. 85)				tr	M
	“...helps to structure the discussion...” (p. 85)				ef	M
	“...used the Maptable more as a discussion tool...” (p. 86)				C	M
11 [12]	“...to enable communication...” (p. 4)				C	H
	“...listening to each other, treating each other’s interests with respect...” (p. 5)				fa	H
	“...discussing their own perspectives and knowledge...” (p. 10)				C	M
	“...information and knowledge are communicated...” (p. 11)				C	M
	“...tensions and resistances also surfaced...” (p. 11)				C	M
12 [63]	“We observed that interaction with the touch table can be dominated by one or two individuals...” (p. 13)				fa	M
	“... can make it easier for more reserved participants to contribute.” (p. 13)				fa	M
13 [9]	“...encourages stakeholders to discuss jointly...” (p. 3)				C	H
	“...various groups focused [...] more on achieving the given goals rather than on an eventually conflicting discussion of potential locations...” (p. 10)				C	M
	“...allows better comunication...” (p. 12)				C	M
14 [51]	“...active dialogue [...] and an exchange of each other’s perspectives...” (p. 5)				C	H
	“...transparency in the process...” (p. 5)				tr	H
	“...an active dialogue among the participants...” (p. 13)				C	M
	“...exchange of perspectives among				C	M
	Participants...” (p. 13)				C	M
	“...Collaborative relationships were also evident...” (p. 17)
16a [64]	“...two-hour participatory workshop to facilitate dialog [...] among researchers and practitioners...” (p. 131)				C	H
16b [65]	“...degree of transparency.” (p. 157)				tr	H
	“...sketch and test ideas together...” (p. 158)				C	H
	“...supporting communication...” (p.159)				C	H
	“...effective tool to support communication...” (p. 160)				C	M
	“...support consultation with other stakeholders and public...” (p. 161)				C	M
	“...unclear whether the operation of the Walkability PSS on the MapTable itself was essential to support participation and engagement...” (p. 164)				C	M

Communication and collaboration (C); transparent (tr); fair (fa); inclusive (in); efficient (ef); H = hypothesized claims; M = positive claims measured; M = negative claims measured.

,

Table A5. Social learning.

ID	aw	sl	dl	L
	H = 2	H = 4	H = 2	H = 3
	M = 1	M = 2	M = 7	M = 4
	M = 1	M = 2	M = 2	M = 2
5a [10]	“...helped increasing their awareness of new aspects of the region’s problems” (p. 337)			aw	M
6a [11]	“...is clearly capable of improving the learning processes of actors...” (p. 181)			L	M
	“...this interaction did not necessarily lead to learning processes...” (p. 181)			L	M
6b [37]	“...analysts indicated to have learned a lot from each other...”			L	M
9 [83]	“...should raise awareness...” (p. 428)			aw	H
	“...impacts and vulnerable spots were recognized by the participants...” (p. 433)			sl	M
	“...supported a learning process...” (p. 433)			L	M
10 [62]	“...learning (individual and group learning) as the most important quality criterion...” (p. 77)			L	H
11 [12]	“...to enable [...] social learning...” (p. 4)			L	H
	“...leads to enriched insights and social learning...” (p. 14)			L	M
12 [63]	“…had become more aware of other people she encountered on her walk...” (p. 11)			aw	M
13 [9]	“...to raise awareness and facilitate social learning of stakeholders...” (p. 3)			aw	H
	“...achieving the goal of social learning...” (p. 11)			L	M
	“...learned by the participants in the sense of social learning...” (p. 11)			L	M
	“...learned something about renewable energy during the workshops...” (p. 12)			L	M
14 [51]	“...intended outcomes of knowledge co-production and social learning...” (p. 3)			L	H
	“...detected mismatch between expected and observed outcomes is corrected by revisiting the assumptions...” (p. 6)			dl	H
	“...mismatch between expected and observed outcomes is corrected...” (p. 6)			sl	H
	“...Critical questioning of underlying assumptions of the model...” (p. 13)			dl	M
	“Evident instances of single-loop learning...” (p. 15)			sl	M
	“Instances of double-loop learning occurred...” (p. 16)			dl	M
16a [64]	“...two-hour participatory workshop to facilitate [...] co-learning among researchers and practitioners...” (p. 131)			L	H

Social learning (L); awareness raising (aw); single loop learning (sl); double loop learning (dl); H = hypothesized claims; M = positive claims measured; M = negative claims measured.

,

Table A6. Issue related outcomes.

ID	kn	co	un	qa
	H = 2	H = 4	H = 2	H = 3
	M = 1	M = 2	M = 7	M = 4
	M = 1	M = 2	M = 2	M = 2
1 [20]	“After an agreement has been reached on a common design...” (p. 267)			co	M
	“In the discussion [...] ideas for changing in the design in the next cycle were indicated. (p. 269)			qa	M
2 [55]	“The resulting, integrated map was accepted by all participants...” (p. 8)			co	M
3a [57]	“…reason for bringing together these spatial data layers is to enhance participants’ understanding and thinking about the study area.” (p.335)			un	H
3b [56]	“…this gives users the opportunity to consider these factors in their biolink delineation”			kn	H
	“improved biodiversity and habitat plans can be created between…”			qa	H
	“a more informed way of understanding environmental processes”			un	H
4a [58]	“...invites participants to engage actively in a process of [...] sharing of knowledge.” (p. 8)			kn	H
	“landowners and formal organizations [...] try to come to a consensus.” (p. 9)			co	H
	“In an iterative working process, ‘hotspots’ were continuously evaluated, discussed and changed.” (p. 15)			kn	M
5a [10]	“To improve stakeholder understanding of the problem” (p. 334)			un	H
	“exchange of local knowledge and stakeholder dialog” (p. 334)			kn	H
	“To use this information to improve the quality of the plans” (p. 335)			qa	H
	“To negotiate a land use plan that is perceived as acceptable by all Participants” (p. 335)			co	H
	“… increased the credibility of the information presented...” (p. 338)			qa	M
5b [60]	“... aim was to gather spatially explicit data from local sea-users...” (p. 3)			kn	H
	“...aim [...] was to build a consensus on the best locations...” (p. 3)			co	H
	“Conflict [...] emerged throughout the workshop process.” (p. 3)			co	M
	“...new stakeholder maps were generated {...] based on the knowledge of the [...] stakeholders...” (p. 4)			kn	M
	“Participants added a considerable amount of data ...” (p. 4)			kn	M
	Discussion [...] regarding the quality of the data used led to suggestions that the published tidal data were inaccurate.” (p. 5)			qa	M
	Results [...] indicated that 80% of participants found the combination of knowledge from researchers and stakeholders to be most helpful” (p. 7)			kn	M
	“...contributed to a broader understanding of the complexity of problems...” (p. 7)			un	M
6a [11]	“...the structuration of local knowledge...” (p. 174)			kn	H
	“...it helped them to envision the diverse topics of a planning problem in an interconnected way...” (p. 180)			un	M
	“...If you emphasize integrality too much, it hampers focus and creativity.” (p. 180)			qa	M
	“...lead to group dynamics in which it is easier to come up with ideas...” (p. 180)			kn	M
	“...understanding of the issues of relevance to other disciplines and the interrelatedness of problems...” (p. 180)			un	M
6b [37]	“...which both provided insight in the planning issue...”			un	M
	“...gained more insight into the plans of other municipalities and into the future economic consequences of these plans...”			un	M
	“...the increased transparency through [...] the PSS did not always support reaching agreements...”)			co	M
7 [38]	“...increased understanding of these relations...” (p. 113)			un	M
8 [61]	“...better understand the maps...” (p. 196)			un	M
	...new type of overview [...], which is more understandable...”			un	M
	“Stakeholders will discuss longer before reaching a consensus.” (p. 196)			co	H
9 [83]	“...supports them in how to share their knowledge...” (p. 428)			kn	H
	“Two alternative plans emerged from this discussion...” (p. 433)			qa	M
	“...forced participants to be explicit about their proposed interventions...” (p. 434)			qa	M
	“...to find a common preference with all participants...” (p. 434)			co	M
10 [62]	“...used in-depth knowledge...” (p. 84)			kn	M
	“...identify and discuss locations [...] that otherwise might be overlooked.” (p. 86)			qa	M
	“...successfully brings information together in interactive fashion...” (p. 87)			kn	M
	“...Learning about the issue...” (p. 87)			un	M
11 [12]	“...to develop a shared understanding about the problem situation...” (p. 4)			un	H
	“...enable integration of explicit knowledge and stakeholders’ tacit knowledge...” (p. 4)			kn	H
	“...willing to put their “half baked” ideas forward...” (p. 5)			kn	H
	“...knowledge is shared easily...” (p. 11)			kn	M
	“...externalize and capture tacit knowledge...” (p. 12)			kn	M
	“...Consensus was reached...” (p. 14)			co	M
12 [63]	“...method to facilitate discussions about people’s perceptions and knowledge of local landscapes.” (p. 1)			kn	H
	“…the touch table facilitated participants in gaining an understanding...” (p. 7)			un	M
	“...the session around the touch table had helped to change the understanding [...] people had...” (p. 11)			un	M
13 [9]	“...discussions towards consensual options and locations...” (p. 3)			co	H
	“...high level of agreement between the participants...” (p. 11)			co	M
	“...mapping results were perceived as less valuable and reliable...” (p. 13)			qa	M
14 [51]	“...engaged in cooperative endeavors of knowledge production...” (p. 3)			kn	H
	“...participants may also come to understand...” (p. 4)			un	H
	“...elicitation of stakeholders’ tacit knowledge...” (p. 5)			kn	H
	“...participants’ suggestions and knowledge were also considered.” (p. 12)			kn	M
	“...co-creation of knowledge among the groups...” (p. 16)			kn	M
	“...participants discovered common linkages between various interests...” (p. 17)			co	M
	“...were able to agree on a set of indicators/indices...” (p. 17)			co	M
	“...lead to collective action by the group...” (p. 17)			co	M
15 [22]	“...combination of models in the tool facilitates the understanding of the role of different variables...” (p. 1389)			un	M
16a [64]	enabled a group of spatial planners to [...] understand how this increased the probability of transport walking			un	M

Knowledge created/integrated (kn); consensus achieved (co); understanding of the problem improved (un); quality of decision improved (qa); H = hypothesized claims; M = positive claims measured; M = negative claims measured.

and

Table A7. Social outcomes.

ID	Ow	mu	cb	sa
	H = 2	H = 4	H = 2	H = 3
	M = 1	M = 2	M = 7	M = 4
	M = 1	M = 2	M = 2	M = 2
1 [20]	The design should address [...] satisfaction properly (p. 265)			sa	H
5a [10]	“the Touch table helped increasing their awareness of [...] different standpoints as well as scopes of other stakeholders (p. 337)			mu	M
6a [11]	“...lead to group dynamics in which it is easier [...] to think beyond professional roles...” (p. 180)			mu	M
6b [37]	“...the PSS gets people off their islands, they come closer....”			mu	M
7 [38]	“...raised awareness of the perspectives of other Participants...” (p. 113)			mu	M
9 [83]	“...satisfaction with the way the planning process was structured...” (p. 433)			sa	M
	“...stimulates thinking beyond the own professional roles...” (p. 434)			mu	M
10 [62]	“...tool effectively brings motives to the surface...” (p. 87)			mu	M
	“...insight in each other’s current situation and future plans...” (p. 87)			mu	M
11 [12]	“...participants identify commonalities in views...” (p. 7)			mu	H
	“...construction of shared meanings was evidenced...” (p. 14)			mu	M
12 [63]	“...how what the other person said made them remember other aspects of their walk...” (p. 11)			mu	M
13 [9]	“...to increase community engagement...” (p. 3)			ow	H
	“...to get to know the views of others...” (p. 11)			mu	M
	“...their views on renewable energy with the other participants...” (p. 12)			mu	M
14 [51]	“...shared understanding of the issues under investigation...” (p. 5)			mu	H
	“...insights into normative aspects and each other’s assumptions...” (p. 5)			mu	H
	“...participants were able to work through their interests linking these with others...” (p. 17)			mu	M

Ownership/commitment (ow); mutual understanding (mu); community building (cb); satisfaction (sa); H = hypothesized claims; M = positive claims measured; M = negative claims measured.

), depending on whether the authors aim to achieve the effects (i.e., hypothesized) or provide empirical evidence of having achieved them (i.e., measured). Where multiple statements in a study regarding the same dimension or criteria were found, these were counted as one in the final table (

Table 4. Hypothesised and measured impacts on participation in the reviewed studies (N = 16).

Evaluation Dimension and Criteria	Hypothesized Impacts	Positive Measured Impacts	Negative Measured Impacts
Process: Communication/collaboration
Transparent (tr)	2 1	1	1
Fair (fa)	4	2	2
Inclusive (in)	2	7	2
Efficient (ef)	3	4	2
Communication/collaboration unspecific (C)	12	12	3
Process: Social learning
Awareness raising (aw)	2	2	0
Single loop learning (sl)	1	2	0
Double loop learning (dl)	1	1	0
Social learning, unspecific (L)	4	5	1
Outcome: Issue related outcomes
Knowledge created/integrated (kn)	9	6	0
Consensus achieved (co)	3	6	2
Problem understanding improved (un)	4	8	0
Quality of decision improved (qa)	2	4	3
Outcome: Social outcomes
Ownership/commitment (ow)	1	0	0
Mutual understanding (mu)	2	9	0
Community building (cb)	0	0	0
Satisfaction (sa)	2	3	0

¹ numbers of studies (N = 16) that have hypothesized or measured a certain or dimension of participation.

), because it is not relevant for the analysis how often a certain aspect of participation was mentioned. Negative statements indicating that certain dimensions of participation were not achieved were particular highlighted, because also these provide insights into the value of interactive PSS for facilitating participation. In cases where statements in the studies were not explicit but rather claimed general achieved impacts like social learning or better communication, these statements were classified in the appropriate dimension of participation.

The extraction of statements from the papers and their categorization into the different aspects and criteria of the framework was conducted by the first author. In order to minimize potential misinterpretation or omissions and to increase the reliability of such an intersubjective endeavour [14] the second author reviewed the results independently and added eventual missing statements. Disagreement between the two authors regarding the categorization of certain statements into the framework were resolved in a discussion of the entire team of authors.

5. Review Results

The presentation of the results is divided into two parts. First, the details of the applications and cases studies, tools developed and participatory processes they were used in are presented. Second, we analyse how and what the studies reported on the impacts on participation, based on the framework developed in Section 3.

5.1. Interactive PSS Applications, Tools, and Participatory Processes

The development of interactive PSS included in this review ranges from 2006 to 2018. Nonetheless, half of the applications have been developed and applied in PSS workshops between 2016 and 2018 (see Table 2). This shows that the use of such application has become more widespread and common in recent years. Detailed results of the review of the various interactive PSS are given in Appendix A,

Table A1. Interactive PSS applications.

ID	Case Studies	Sector/Field of Application	Scale	Goal of the Application
1	Ijssel river clos to Deventer, Netherlands	Water management	Regional	Designing spatial measures for controlling water levels of the river
2	Groningen province, Netherlands	Climate change adaptation	Regional	Integrating sectoral adaptation measures
3	South West Victoria, Australia	Forest regeneration planning	Regional	Designing biolinks (ecological corridors) for forest regeneration
4	Three case studies: (a) Stedendriehoek area, (b) hypothetical area, (c) Brabantse Delta (all areas in the Netherlands)	Water related land use planning (a, c), energy transition (b)	Regional	Exploring available information, exchanging ideas about the current and future situation, sketching potential interventions
5	Two case studies: (a) Bodegraven polder, Netherlands, (b) Mull of Kintyre, Scotland	(a) Rural land use planning, (b) offshore renewable energy planning	Regional	Involving stakeholders in entire planning process of analysis of the situation, design of potential interventions, choice of agreed upon interventions
6	Four case studies: (a) Rijnenburg, Utrecht, (b) Arnhem, (c) Deventer, (d) Achterhoek (all areas in the Netherlands)	(a) Sustainable urban development, (b) energy transition, (c) rban design, (d) adjustment of urban development plans	Urban, city to neighbourhood level	Facilitating communication among stakeholders during different planning stages
7	Three case studies: (a) Hommerts, (b) Veenpolder, (c) Buitenveld (all areas in the Netherlands)	Rural land use planning	Regional	Designing a spatial adaptation strategy for land use related water management in a peat meadow area
8	Two case studies, each conducted in multiple areas: (a) Esch-sur-Alzette, Luxembourg; Ludwigsburg, Germany, (b) London, UK; Brussels, Belgium; Luxembourg	(a) Climate change mitigation, (b) urban logistics	Urban	Translating policy visions into concrete projects
9	Six case studies: Beira, Mozambique; London, UK; Oaxaca, Mexico; Utrecht, Dordrecht, Tilburg, all Netherlands	Climate change adaptation	Urban	Defining green and blue adaptation measures
10	Liemers Corridor, Netherlands	Regional development planning	Regional	Negotiating distribution of commercial and industrial development locations at regional level
11	Dortmund, Germany	Environmental health	Urban, city to neighbourhood level	Mapping stakeholder knowledge on environmental health issues
12	North-East Scotland, Scotland	Landscape planning	Urban and rural, regional scale	Exploring spatial behaviour in the local landscapes
13	Enschede, Netherlands	Energy transition	City	Discussing locations of renewable energy projects
14	Two case studies: Dortmund and Munich, Germany	Environmental health	City district	assessing and mapping multiple environmental burdens
15	Sydney, Australia	Urban redevelopment	Urban, precinct level	Developing different scenarios of urban redevelopment
16	City of Hume, Melbourne, Australia	Health related urban planning	Urban, city level	exploring built Environment scenarios regarding impacts on walkability and designing interventions

,

Table A2. Interactive PSS tools.

ID	Hardware Setup/System Architecture	Software	Main GIS Functions	External Model Coupled	Stakeholders Involved
1	Maptable used with stylus pens, stand alone system	ArcGIS plus interface panel for maptable (Woosh)	Drawing features, sketching, annotating, queries, calculation of effects	Hydrologic model (Waqua), spatial schematization tool (Baseline)	No
2	Maptable, stand alone system	ArcGIS	Drawing, visualisation	No	No
3	Maptable, stand alone system	ArcGIS plus landscape constructor extension and visualisation tool (SIEVE)	Drawing polygons, assigning vegetation types, 3D visualization	Landscape object library	No
4	Maptable, clinet-server structure	ArcObjects technology, various libraries	Sketching, indicator calculation	4b) Energy performance model	Lab based study beforehand, (see [25])
5	Maptable (Diamond Touch™) 3, stand alone system	CommunityViz Scenario 360 1, DT collaborate (ArcGIS plugin)	Drawing, MCA, allocation	No	No
6	Mapsup MapTable®, stand alone system	CommunityViz Scenario 360 1	Drawing, indicator calculation, visualisation	Sustainability Profile of the Location (SPL) score calculated based on indicators	No
7	Maptable, SUR 40 Samsung, stand alone system	CommunityViz Scenario 360 1	Geodesign tools, indicator calculation,	No	No
8	Maptable (GTUI) used with physical objects (tags), web-based	PostgreSQL1 server with geospatial extension; client: Java application (open source)	Place tagged objects on the screen for calling certain functions such as split screen, data layers	No	No
9	Maptable, stand alone system	Not given	Allocate predefined interventions, ranking of interventions, performance indicator calculation	No	Test cases of the tool run in Rotterdam and Delft (see [21])
10	MapsUp MapTable®, stand alone system	CommunityViz Scenario 3601	(Re)-allocating pre-configured building blocks of development space, indicator calculations	No	No
11	Maptable plus extra screen, stand alone system	Phoenix 2	Sketching, annotating with symbols 3D visualization	No	No
12	Maptable, Samsung SUR40 with Microsoft Pixelsense, web-based	Bing maps	Pan and zoom, no particular GIS functions used	No	No
13	Maptable plus extra screen, stand alone system	CommunityViz Scenario 360 1	Sketching, indicator calculations	No	Discussions of the model with the municipality
14	Maptable plus extra screen, stand alone system	CommunityViz Scenario 360 1	Sketching, multi criteria analysis (MCA)	No	No
15	Maptable, stand alone system	ArcGIS toolbox	Input of model parameters, scenario calculation, 3D visualization	No	No
16	Maptable, stand alone system	CommunityViz Scenario 360 1	Sketching, changing model assumptions, indicator calculation	Custom regression models built into CommunityViz	No

¹ ArcGIS Extension [68], ² by Geodan [67]. ³ This maptable system allows user recognition.

and

Table A3. Participatory process.

ID	Planning Phase	Stakeholders/Participants	PSS-Supported Steps	Other Participatory Steps	Process Facilitation	Process Evaluation
1	Design	Practitioners from city, ministry, forestry, infrastructure agency, architect	Three stages: define measures, calculate effects, evaluate results	Evaluation of design done verbally based on calculated effects	Process and technical facilitator	Not given
2	Choice	Different sectoral experts	Three stages: map sectoral measures, integrate spatial measures into one coherent map, agree on integration map	Measures first sketched on paper maps	Moderator to operate the GIS on the maptable	Not given
3	Design	Landscape planners, scientists	Design and then visualisation in 3D	No	Not given	5 point likert scale questionnaire on usability of tool
4	Problem analysis and design	Practitioners (landscape architects, planners) and academics	Three stages: area exploration, selection of suitable locations, design of new locations	No	No	Observers (development team and experts) filling evaluation questionnaires, discussion of findings with participants; stated experiences of participants
5	Problem analysis, design, and choice	Design workshop: 10 stakeholders (municipality, waterboard, NGO, etc.); choice workshop: local stakeholders, citizens, and municipality	Two workshops, local knowledge and negotiation workshop, each workshop having 3 sessions: intro to research problem, familiarise with tools, participants use of tools to fulfil the objectives	Design on paper maps in parallel groups, use of DEFINTE MCA in design workshop	no	Survey at the end of each workshop, plenary discussion
6	Problem analysis, design, and choice; partly addressed in separate case studies	Different practitioners: environmentalist analysts (noise, air quality, water management), urban designers	Various number of workshops in different cases workshops;	No	Technical facilitation on using the PSS	Survey questionnaire in negotiations workshop
7	Problem analysis and design	Farmers, interest groups, NGOs	Three assignment in the workshops, related to three strategies	No	No	Survey done before and after the workshop
8	Design	Local stakeholders: urban planners, GIS technicians, social scientists, engineers, energy experts, and companies; municipalities, public transportation	Translate vision into concrete projects); understanding the context	No	No	Questionnaire
9	Design	Various local experts: municipal staff members, planners, local experts, NGOs, partly also citizens	First maps and 3D visualization were shown	Loosely linked to climate adaptation app [86]	Not given	Questionnaire
10	Not strictly related to a typical planning process	regional urban development practitioners from province and municipalities	Two workshops: various rounds of playing with different tasks, foci	No	No	Video recording, three questionnaires during the workshop regarding expectations
11	Problem analysis	Practitioners from planning and health (municipality, agencies, NGOs)	Three phases: explore existing information, add tacit knowledge, integrate knowledge	Rich picture drawing	Facilitator for substantial and procedural issues	observations, questionnaire, oral feedback, screen and voice recording
12	Not given	22 participants, individuals, most of them being researcher on land based issues	Joint discussion self-recorded GPS walking routes visualised on the maptable	Combined with smart phones GPS tracking of walking routes	No	Individual interviews 4-6 weeks after workshop, video recording of the sessions
13	Problem analysis	Citizens, policy makers	Two stages, focussing on different types of renewable energy	No	Moderator and chauffeur	Questionnaire, screen and voice recording
14	Problem analysis	Researchers and practitioners (municipality, housing association, NGO)	Part of a one day planning game	Planning game on identifying economically and socially deprived districts, and communities	Moderator and chauffeur	Questionnaire, observations, screen and voice recording
15	Problem analysis and design	Urban planners	Session structured into five steps related to five different questions	No	Technical facilitator	Pre and post workshop questionnaire
16	Design	Urban planners and one health expert	Sketching interventions after intro to the tool, compare scenarios	No	Technical facilitator	Questionnaire, group discussion

.

The applications taken into account in this review cover a wide range of topics within the domain of spatial planning, from rural land use planning to urban redevelopment at neighbourhood level (Appendix A, Table A1). Next to common topics such as sustainable urban development, the sectors of urban health (3 applications), energy transition (4 applications) and climate change (3 applications) are predominantly addressed, which reflects typical foci of contemporary planning debates of the last 10 years. Half of the applications include at least one Dutch case study, which has to do with the fact that various research groups in the Netherlands work on the topic of interactive PSS. The goals of the applications are either related to designing spatial interventions such as adaptation measures, or to analysing the current situation and developing e.g., a good understanding of the problem and exchanging knowledge about it. Hardly any study covers the final phase of a decision-making process in which typically one option among a set of alternative designs is chosen, based on a proper evaluation.

All interactive PSS tools make use of a similar hardware setup of a maptable most often as a stand-alone system, plus in some cases an additional second screen e.g., for showing results of spatial interventions on the table in the forms of charts (Appendix A, Table A2). Maptables are operated either with special stylus pens or with fingers, only one study (ID 8) explores the use of physical objects (tags) to interact with the maptable. One PSS tool (ID 5) used a maptable that allowed distinction of inputs from different users. However, this was obviously not relevant for the results, because it was not made use of in the analysis of the workshop, nor has it prevailed in later studies.

The use of software for the tools reveals the fact that so far, hardly any software particularly made for maptables is available, with the exception of Phoenix, a participatory mapping tool [67], used in study ID 11. Most of the tools, therefore, use various ESRI ArcGIS components or an extension for the same, often CommunityViz Scenario 360 [68], which increases possibilities of dynamic user interaction with the geospatial content. Only one of the 16 tools considered here is developed based on an open source platform.

The GIS functionality mainly used by stakeholders during the interactive sessions comprises both functions for sketching and drawing features, as well as functions for calculating outcomes of certain interventions in the form of indicators, often visualized in charts. Since the limited capacity of the used GIS platforms for advanced modelling, some tools couple the GIS with a dynamic simulation model, such as a hydrologic model for calculating outcomes of interventions (ID1, ID 4b). Only three tools make use of 3D visualization for presenting spatial data. Also the inclusion of stakeholders in the development of the tools, as recommended by several scholars [13,15] in order to increase the usefulness of the tools, is hardly practised up to now, with the exception of studies IDs 4, 9, 13 (Appendix A, Table A2).

The planning phase most often addressed in the participatory processes supported by the tools is the design phase (Appendix A, Table A3). In the design phase, different stakeholders develop together measures or interventions in order to address given problems or planning challenges. In more recent applications, the problem analysis phase is also aimed at, in which the interactive PSS typically supports in gaining a thorough understanding of the given problem or an integration of knowledge prior to the design of interventions. Almost all processes involved expert stakeholders, either academics or practitioners from different domains. Only four processes (IDs 5, 7, 9, 13) aim at including also layperson, i.e., ordinary citizens, in the participatory processes.

In all processes, the participants are guided through more or less clearly defined steps or assignments that build up upon each other. Six studies combine the use of the maptable with other participatory activities such as rich picture drawing (ID 11) or GPS tracking (ID 12). The more concrete the steps are that the stakeholders have to go through, the less seems to be a need for extensive moderation. In tool ID 5 e.g., the participants first had to design suitable bicycle paths and then to evaluate land use scenarios [10]. On the other hand, for rather open processes a moderator has a relevant function to guide the process and engage the stakeholders. The use of a technical facilitator or chauffeur, who helps in handling the PSS, is common in recent participatory processes.

The early studies are not very explicit about the methods applied for evaluating the results and outcomes of the workshops or do not discuss these at all. Later studies applied a variety of methods for evaluating workshop processes and outcomes. Most common are post workshop questionnaires in which participants assessed various factors of usability of the tool as well as usefulness of the workshop process and outcomes. Seven studies combined various methods for evaluation in a sort of triangulation in order to answer various research questions or to crosscheck plausibility of findings. Only two studies applied questionnaires prior to and after the workshop in order to determine behavioural changes or learning outcomes (IDs 7 and 15).

5.2. Dimensions and Aspects of Participation Reported in the Studies

In this section, we analyse how and what the studies reported as impacts of the interactive PSS implementation on participation. The overall results are presented in Table 4 (for details see Appendix B, Table A4, Table A5, Table A6 and Table A7). The table shows how many of the in total 16 studies have explicitly reported statements related to the dimensions and criteria of participation as elaborated in the framework (Section 3, Table 1). Hypothesized and measured impacts are separated. Measured impacts are distinguished in positive and negative claims, the latter indicating aspects of participation that have not been achieved. Multiple claims that are collected for the same criteria or dimension are counted as one impact, as explained above (Section 4). Claims that could not be related to specific criteria of participation were associated with the relevant dimension of participation as an unspecific impact.

Table 4 shows that for some dimensions a huge difference between hypothesized and measured criteria of participation exists. In many studies, authors measured more detailed impacts on participation than they hypothesized upfront. In fact, several studies lack a proper framework for evaluating impacts on participation. While in some studies this was according to the authors also not the aim of the study (IDs 1, 2, 3), other studies failed to develop upfront a clear framework. Leao et al. [22] e.g., claim to test the performance and usefulness of the tool, but their study lacks an operationalization towards this goal. In other studies, the applied framework for evaluation towards participation impacts is introduced rather implicitly while explaining the questionnaire for the evaluation [61]. While the study ID 16 [65] has not stated an explicit focus on strengthening participation by using the PSS, some implicit references towards supporting participation are given, e.g., strengthening communication. On the other hand, another group of studies (IDs 6, 11, 14) developed upfront a dedicated framework for evaluating impacts on participation, which is often focusing on a specific dimension such as social learning.

Particularly in the process dimension of participation, a number of studies is making no distinction between the different criteria of participation. Both hypothesized and measured impacts are reported to support communication and collaboration (e.g., [9,64]) or trigger learning (e.g., [11,62]), without distinguishing what aspects of communication and collaboration or learning are addressed. In contrast to that, in the outcome dimension of participation impacts are more clearly distinguished both, while being hypothesized or measured. Different outcomes of a PSS workshops such as achieving a consensus is obviously much more concrete to grasp the rather vague notions of learning [69].

In total, the reviewed studies have mainly aimed at communication and collaboration as the process dimension and issue related outcomes as the outcome related dimension (Table 4). Those studies that have investigated communication and collaboration in more detail hypothesize mainly fairness (4 studies) and efficiency (3 studies) being the main goals of the interactive PSS workshops and applications. Interestingly, the criteria of inclusiveness and efficiency of the process are more often measured than hypothesized upfront. This supports the findings of the limited structured evaluation frameworks developed prior to the application. Learning during PSS workshops as an impact of participation is taken into account explicitly only in three newer studies (IDs 6, 11 14). These studies have hypothesised to investigate impacts of interactive PSS workshops on social learning and have also measured these.

Issue related outcomes are the by far most often hypothesized as well as measured dimension of participation. Nine studies aim at creating or integrating knowledge by means of interactive PSS workshop, while only six studies measure these. Three respectively four studies aim at achieving a consensus and/or increasing the problem understanding, but in six resp. 8 studies authors have measured these impacts. Interestingly, the criteria of improving the quality of the decision is only hypothesized in two studies, but measured as an issue related outcome in seven studies, four of them finding mainly positive indications, and three studies also reporting negative statements.

Finally, social outcomes are hardly addressed as hypothesized criteria in the studies. Increased ownership and commitment are hypothesized only in one study and both mutual understanding and satisfaction are only hypothesized in two studies. This shows that authors do not expect much on that dimension or do not consider that to be relevant in a PSS workshop, probably because it is often a one-time activity. On the other hand, in particular an increased mutual understanding among participants of a workshop is measured as a social outcome in nine of the studies, showing that such workshops in fact do have an effect on social aspects, at least at short term, i.e., at the time when the evaluation was done. With the same argument the creation of ownership or community building might not be considered, because these kind of impacts in fact require much more frequent and an intense working on an issue over a longer duration.

6. Discussion

6.1. Interactive PSS Applications, Tools and Processes

The interactive PSS analysed in this review reveal a large variety of applications, goals, contexts and case studies for different planning topics, illustrating the broad applicability of such applications. The more recent applications have a clear focus on issues at an urban scale [9,12,21,51,64], which on the one hand might reflect the increasing importance of urban topics in international planning debates [70]. On the other hand, this might also be due to the fact, that more small-scaled data is available nowadays enriching such urban studies and applications. Different case studies have been applied in different stages of a planning process, illustrating the potential value of these in various phases of planning. Some studies even tested the application of interactive PSS throughout different phases of a planning process [10,60], thereby demonstrating the value of such tools for supporting stakeholder participation in planning.

Nevertheless, most of the interactive PSS studies are developed and implemented in a research context, while only a minority of studies (IDs 6, 13) have been initiated through a municipality or planning agency. In the case of Pelzer et al. [11], the municipality “brought together an urban design firm, several environmental research agencies and a geo-communication firm to concretize its vision of the new neighbourhood” [11] (p. 177). The stakeholder workshops using the interactive PSS were perceived useful by all participants, but “the commercial developers (who owned most of the land) […] had not been involved in the MapTable sessions […] which led to a pause in the project execution.” [11] (p. 178). Moreover, no study discusses how the results from the PSS workshops are further used in the planning process. In the study of Flacke and de Boer [9] “the workshop results […] were summarized in a policy “document [71]. However, no follow up steps have been taken so far” [9] (p. 14). One can conclude that while the tools and applications presented in the studies might have the potential to overcome the widely discussed PSS implementation gap [6,72] any proof for that is still lacking.

The analysis of PSS tools shows that different types of hardware setups and infrastructure systems are being implemented successfully. In contrast, there is an obvious demand for the development of dedicated software tools supporting a group process on maptables. In most cases, generic and off-the-shelf GIS software augmented with certain extensions or extra functions is being used, which is not designed for tangible user interfaces, resulting in limited user friendliness of the tools. No open source tools such as QGIS, holding the promise of permanent refinement and tailor-made modification through a large user community [73], are being used. Moreover, only one study (ID 11, [12]) uses a tool that is explicitly developed for tangible user interfaces, but that tool supports basically participatory mapping studies lacking analytical functions of a PSS. In conjunction with the findings that the tool development processes hardly involve any stakeholders [14,15], we speculate that the development and implementation of such maptable based PSSs are still the job of experts and consequently the tools are far from being used in every day planning practice.

Finally, only very few tools reviewed make use of 3D visualization techniques, though this is reported to enhance participation, particularly of layperson, in planning significantly [74]. In line with that, PSS workshops reviewed here are mainly conducted involving planning experts as stakeholders, either researchers or planning practitioners, who are likely to be experienced in working with maps and 2D visualizations of spatial data [8]. Laypersons, such as ordinary citizens, are involved only in 4 of the 16 studies. From our own experience [9] this might be explained by the non-availability of user-friendly PSS tools and interfaces, e.g., tools made explicitly for tangible user interfaces, which would be easier to use also for non-experts. Vonk et al. [75] found in a study on the bottlenecks in the uptake of PSS that using such tools […] depends, among other things, upon previous experiences in planning practice. The vast potential offered by user friendly maptable applications and attractive (3D) visualization for involving particular non experts is hardly explored yet [8].

6.2. Interactive PSS Contribution to Participation

This review shows that all interactive PSS applications and tools succeeded to facilitate intensive, in some cases, enthusiastic [37] stakeholder collaboration and workshops participants reacted predominantly positive to the possibilities provided by the PSS tools. Thus, we can conclude that in all cases studies the interactive PSS has contributed to strengthening participation, as it can be seen from the results in Table 4. However, the most apparent finding is that many interactive PSS case studies do not make use of a very distinct and explicit framework in order to evaluate impacts of the interactive PSS tool on participation. Thus, impacts on participation are often being assessed rather incidentally based on implicit assumptions [15], without being underpinned by a proper conceptual or theoretical discussion. Having said this, we can conclude that many of the studies reviewed here can be characterized as studies testing the usability of the tools rather than their usefulness for the planning practice [14]. However, given the obvious strength of many of the interactive PSS for improving aspects of participation, detailed frameworks of usefulness or performance of PSS as initiated in contemporary scholarly debates [8,14,15], are needed in order to equip practical and applied studies with a proper basis and to make studies better comparable.

Given the different orientations of reviewed studies regarding the goals of participation, foci on different aspects vary accordingly. However, the focus of participation in general is mainly on how interactive PSS help facilitating the communication and collaboration and on issues related outcomes that can be achieved. While a few newer studies focus explicitly on the impacts of interactive PSS on aspects of learning [11,12,51], no studies have yet focused on analysing the social outcomes of PSS workshops. This matches with the observation of Goodspeed and Pelzer [34] that it is more likely that process attributes are measured, because outcomes are often hard to see, because they develop over a longer time period. Nevertheless, given the increasing need to engage with stakeholders under current planning regimes [76] to engage with stakeholders for the co-production of services [77], it is interesting for future studies to investigate what kind of social outcomes can be supported by means of interactive PSS applications. Therefore, PSS tools would have to be applied over a longer period, e.g., a series of workshops.

Last main finding worth to discuss here is that obviously scholars tend to report what works well, rather than what does not work. In all papers related to the different studies in total 176 positive quotes (including multiple quotes in one paper referring to the same criteria of participation) were found relating to one of the different aspects and criteria of participation, and only 20 negative statements (see Appendix B). It seems that authors are predominantly looking for evidences that prove their hypotheses and thereby devaluing negative findings, as it is rather unlikely that in only a bit more than 10 % of all instances, they did not succeed in doing so. This phenomenon known as the publication bias is reported by other scholars and occurring in various disciplines [78,79]. Fanelli [80] (p. 892) observes “various forms of conscious and unconscious biases that affect all stages of research […] producing positive findings when there should be none, thus creating distortions that are difficult to correct a posteriori”, particular “in fields where theories and methods are less clearly defined, and true replication is rare or impossible” Having tested 4600 papers from different disciplines over a period of 17 years (1990–2007) he further hypothesizes that research is becoming less pioneering and that the objectivity with which results are published is decreasing [80]. Nevertheless, as the exact number of positive or negative statements is not relevant for this study, because multiple quotes are counted as one per study (see Section 4), this observation does not significantly affect the reliability and relevance of our findings.

6.3. Reflection of the Methodology Applied in this Review

The methodology applied in this review includes various aspects that might limit the validity of the obtained results. First, the characteristics of the various interactive PSS presented in Appendix A, Table A1, Table A2 and Table A3 (applications, tools, processes) result from the interpretation of the authors if explicit details were not given. Moreover, when a ‘no’ is given in the tables it means it was not given in the paper, which does not necessarily mean that it was not done. It can of course be that the authors of the study simply did not explain it. Second, the hypothesized and measured impacts of the various studies on the different aspects and criteria of participation as analysed in Section 5.2 (see also Appendix B, Table A4, Table A5, Table A6 and Table A7) is an intersubjective endeavour, particularly when hypotheses are not explicitly spelled out by the authors. In order to minimize this subjectivity and to increase inter-rater reliability [81] all statements were cross-checked by the co-authors. Third, certain impacts of the use of interactive PSS in stakeholder workshops that were reported by the authors were not taken into account, due to the theory driven analysis method applied [16]. For instance, some studies reported that the use of the maptables created enthusiasm among participants [20,37]. Enthusiasm is for sure something the use of a maptable might create, but in our view, it does not necessarily lead to better or more participation. Therefore, it is also not included in any of the studies used here to develop the framework in Section 3. On the contrary, in our own studies we have observed enthusiasm rather limiting a serious and productive use of the interactive PSs, and therefore would have to be overcome in order to institutionalize such tools in everyday practice.

6.4. Future Research Needs

Findings from the review allow identifying the following future research needs in relation to interactive PSS. First, most tools reviewed here are developed using proprietary GIS software which is in most cases not explicitly developed for tangible user interfaces. Open source tools offering potentials to further refine applications by others or in future case studies are so far very much neglected. Future research studies should explore impacts of interactive PSS tools made explicitly for tangible user interfaces on their user friendliness and its correlation with the usefulness of the tool [14,82].

Second, only a minority of reviewed studies reported the inclusion of users and/or stakeholders in the PSS tool development process. A stronger integration of them in the development process is recommended by scholars as being integral for enhancing uptake and implementation of PSS [15,83]. Thus, future interactive PSS studies shall research the challenges as well as added values of including users and stakeholders in the tool development process, e.g., following a human centred design (HCD) approach [84].

Third, the review revealed that social outcomes of the use of interactive PSS in stakeholder workshops are hardly researched yet. Thus, in suitable long-term case studies potential social outcomes such as community building or increased ownership through interactive PSS workshops series might be investigated. In this context te Brömmelstroet [82] suggests to research how more flexible PSS that that can be quickly adapted and contextualized [85] allows the actors to develop a sense of ownership and adapt the PSS to their specific demands.

Finally, the adoption of results from interactive PSS workshops into ongoing planning processes needs to researched. To this end Geertman [6] recommends to study successful or best practice studies and to learn from them. Te Brömmelstroet ([83] p. 80) proposes a research programme that combines “context-rich and control-rich [PSS] research environments” in order to examine the uptake of PSS research in planning practice.

7. Conclusions

This review analysed how interactive PSS have contributed to the goal of strengthening stakeholder participation. Findings from the 16 studies reviewed here show that the implementation of maptable-based interactive PSS in planning related studies has increased significantly since the first case study in 2006 in numbers of tools developed, geographic regions covered, and in the variety of topics and tasks addressed. Over the years, not only the tools and applications have become more sophisticated and complex, also the goals of the studies using such tools have changed from involving stakeholders in designing interventions [20,55] to involving stakeholders in earlier planning stages, e.g., in order to develop a joint problem understanding or integrating knowledge [12,51]. In line with that, the interest of researchers in such studies has evolved from basically exploring the usability of interactive PSS for stakeholder participation [57] to more explicit research questions related to stakeholder participation such as the added value of interactive PSS on group-based learning [37,55].

Based on the second part of the analysis we can conclude that in all case studies the interactive PSS has contributed to strengthening participation, because all studies reported positive statements regarding the different aspect sand criteria of participation. Nevertheless, a key finding is that many case studies of interactive PSS reviewed here lack a proper framework and operationalization for investigating the effects of the tools and applications on the different aspects and criteria of participation. Though studies usually hypothesize certain impacts of their tools on participation and apply suitable methods for measuring and evaluating such impacts in PSS workshops, in many studies a certain mismatch was detected. Many studies are much more detailed in measuring the impacts on participation, e.g., by evidencing impacts on the transparency or inclusiveness of the public dialogue, than hypothesizing them before. Other studies provide detailed hypotheses of impacts but fall short in evidencing them by means of evaluation. Only three of the reviewed 16 studies, come up with a proper framework upfront and systematically measure impacts of those aspects of participation hypothesized upfront. In conclusion, further theoretical studies conceptualizing impacts of interactive PSS on participation are needed as well as more empirical studies, testing these impacts in real world case contexts with various groups of stakeholders.

Moreover, most studies analysed result from a more or less strict research context. Consequently, the adoption of the tools and applications in planning practice, e.g., through the involved case study partners, is not reported. Thus, no straight conclusions can be drawn from the review regarding whether or not the implementation of interactive PSS might have the potential to mitigate the often-stated PSS implementation gap. On the contrary, findings from the review help identifying issues that needs to be addressed in future studies in order to mitigate the implementation gap, such as the use of open source software explicitly made for maptables, the inclusion of users in the PSS development process, and the in depth analysis of results from PSS workshops in planning processes (see Section 6.4).