A Good Practice for Integrating Stakeholders through Standardization—The Case of the Smart Mature Resilience Project

Abstract

A priority must be made on making cities more resilient against crises such as the COVID-19 pandemic to help plan for an uncertain future. However, due to the insufficient transfer of knowledge from, among others, research projects to cities, they are often unaware of the resilience tools available as well as possible standardization activities to foster the integration of relevant stakeholders. To address this issue, this paper analyzes the use of standards and the contribution to standardization in a multi-case study of nine European Framework Program projects and with the Smart Mature Resilience (SMR) project, a more in-depth case study. SMR integrated several European cities in its co-creative approach for developing city resilience tools and the related standards series CWA 17300 on ‘City Resilience Development’. Furthermore, the paper defines five steps for integrating standardization in research projects with a focus on (city) resilience and shows the benefits of the standardization process for fostering the engagement of relevant stakeholders.

1. Introduction

Standardization was described as an instrument to support mass production during the era of industrialization [1], while more recent literature confirms that standardization has a significant link with innovation [2,3,4]. It is further recognized as beneficial for society in general and for research projects in particular [5].

As collaborative research projects face several challenges and their success depends on an implicit knowledge transfer between industry and academia [6], respective supporting activities are needed. However, related actions to disseminate and exploit project outcomes have mostly led to the development of scientific publications or patents rather than the transfer to practice [7,8]. Discussions on the ownership of results of European Framework Program (FP) projects show that academics want to publish as much as they can and industry partners mostly prefer to fill time-consuming patents [9]. Therefore, why not bring both actors together to develop a standard for the projects’ tangible results? Thus, contributing to standards helps researchers to widely disseminate their research results and gain specific knowledge [10] and the standard itself can support the diffusion of the innovations [4].

The need to integrate standardization properly in FP projects to boost innovation has already been addressed in the FP6 [11]. Within FP7 and Horizon 2020, as well as Horizon Europe, the European Commission proposed standardization as a tool in FP projects to support exploitation and dissemination [12,13,14]. In order to increase the projects’ impact, as well as its dissemination and exploitation, it is essential to integrate the relevant stakeholders and transfer the project results to them. Nevertheless, the implementation phase of FP projects often lacks enhanced dissemination and exploitation activities aimed at the target group [15] and the role of standardization and standards in this regard has been recently described as underrated [16].

As the stakeholder engagement for topics such as city resilience is crucial, FPs such as Horizon 2020 have focused on and even requested standardization in some of their calls on resilience [17]. Nowadays, over half of the world’s population lives in cities, and according to the United Nations, this percentage is expected to reach 68% by the year 2050 [18]. Due to this rapid urbanization, cities are becoming more exposed and vulnerable to the effects of a wide spectrum of disasters [19]. In this context, there is a pressing need for cities to improve their resilience [20,21].

The still limited resilience maturity of cities can be explained by the lack of awareness of resilience-enhancing tools developed in, for example, research projects and by the poor interaction with relevant stakeholders. The uptake of resilience tools of research projects to the city’s portfolio as well as the participation in these projects could overcome these gaps. The external stakeholder engagement is especially challenging but crucial for bringing significant value to all stakeholders and thus to the projects’ success [22]. In order to transfer research project results and bring stakeholders together, so-called Workshop Agreements, such as the CEN Workshop Agreement (CWA), are a fast-track possibility of the standardization system to develop a standard. These agreements are developed in an open workshop environment in a temporary committee, thus enabling project external stakeholder engagement outside the classical more time-consuming technical committee system [23].

However, there is little information available on how standardization can be addressed in the funded projects. Examples of previous studies refer to the use and application of a standard for sustainable cities [24], assessing already developed resilience-related standards [25] or the development of a new country-specific project management standard [26]. Nevertheless, in order to address the increasing demand for standardization in projects and to specify successful approaches to integrating standardization and stakeholders, more research on projects that have integrated standardization are needed [5].

This gap in transferring resilience-related project results to relevant stakeholders such as cities and in engaging relevant stakeholders actively in the project raises two questions, which this research addresses: what are the different standardization activities for the transfer of results of a research project on city resilience? How does standardization support (external) stakeholder engagement?

To respond to these questions, this paper reviews nine FP projects focused on (city) resilience. The projects have been initially assessed regarding their use of standards and integration of standardization. One project that successfully integrated standardization throughout the entire duration was analyzed more deeply as a case study. The SMR project, whose objective was to develop resilience tools for and together with cities in a co-creative approach and circle of learning [27], created several standards from its results with the participating cities and other relevant stakeholders. As an output of these cases, five steps for integrating standardization in research projects on (city) resilience are proposed, which will help to achieve a significant impact in the dissemination and exploitation of project results. The research is novel, and as there is currently no analysis of how research projects have integrated standardization, this research will certainly contribute to further research.

This paper is organized as follows: Section 2 presents a review of standardization activities for projects in general and with a resilience focus. Section 3 outlines the research method used to analyze the different FP projects and describes the SMR case study and the methods used in its standardization activities. The findings of the research are presented in Section 4, which leads to the proposal of five steps needed for successful integration of standardization in resilience-related research projects (Section 5). Section 6 discusses the results obtained and Section 7 highlights the main conclusions drawn from this research.

2. Status Quo of Standardization Activities in Research Projects

2.1. Standardization Activities for Projects in General

Several studies have shown the general relevance of standards for project management (e.g., [28]), the standardization of project management processes itself (e.g., [29]), and the challenges of applying these project management standards (e.g., [30]). Furthermore, Hällgren et al. [30] stated that standards can have an enormous impact for interested parties, initially seeking to adopt the standard and later on for requiring it for organizational issues such as tendering. They also refer to standardization as a possibility to create standards as knowledge domains that support further research. Additionally, the improvement of (technical) standards has been identified as an item to give value to project management with the society as beneficiary [31]. However, the successful implementation of a project relates to using project management standards rather than using standardization as a tool within the project (e.g., [29,32]).

The use of standards and standardization for research projects and programmes has been validated in, for example, the area of construction [5]. Moreover, Sanjuán et al. [5] declared that standardization should be considered through all stages of a research project, as standards provide a path for the dissemination of project results and for reaching the market faster. A study from the European standardization organizations CEN and CENELEC [33] confirms this and concluded that the main drivers and needs for including standardization in FP projects are the facilitation of market access and exploitation of the results. Equally important is the consideration of and alignment with existing standards during the project to, for example, ensure the interoperability of project tools. The study also found that by working towards standardization, projects are more efficient and effective, and create results of enhanced quality with a better chance for being accepted by scientific and industrial communities [33].

Two topics that are linked to the standardization activities within research projects are the required technology transfer of project results and the engagement of relevant stakeholders in the standards development process. Strategies for successful technology transfer have been provided in the literature [34], but direct references to standardization are missing. With regard to stakeholder engagement, studies have shown the reasons for researchers and industry player to engage in standardization ([10,35]). Furthermore, approaches for project stakeholder management have been only partly included in project management standards [36]. However, examples of using the process of standardization for engaging stakeholders effectively are rare (e.g., [37]).

This engagement depends also on the mode of standardization chosen, which is different in the case of technical committee-based standardization resulting to full consensus standards [38] and open workshops resulting in the above-mentioned Workshop Agreements [23]. In general, the open process and the following different phases of a CEN Workshop provide and foster the engagement of relevant stakeholders:

• Initiation phase: public availability for commenting of the CEN Workshop project plan and active involvement of relevant stakeholders before the kick-off meeting;
• Development phase: optional commenting phase to include the public; possibility to join standards development anytime with prior request to developers; and
• Publication phase: distribution of the CWA to all interested stakeholders, including members of standardization committees to support a potential uptake into, for example, ISO standards ([39,40]).

A study from CEN-CENELEC [41] shows that when including a standardization organization in the research project, with mostly CWAs and further standardization activities as a result, more focused and advanced standardization outcomes can be expected. Sanjuán et al. [5] confirm the value of standardization organizations for research projects and specify their role in integrating standardization activities in these projects as well as in engaging the relevant stakeholders.

However, as the standardization of research project outcomes is still quite a young topic, the current literature landscape provides very limited information on how standardization activities are conducted in the different research activities; thus, research on best practices to tackle this issue of exploiting formal standardization in research projects is needed. Furthermore, there is often a misperception about how standardization is implemented and who should facilitate the standardization activities [41]. In this regard, it is important to illustrate the differences between standards and standardization. Existing definitions for a standard refer, among others, to research results [42] and for standardization to the relevance of integrating different parties [43]. In this paper, ‘standardization’ refers to a process of using research project results for developing new standards or as inputs to existing standards and the term ‘standards’ refers to existing standardization deliverables that need to be considered for the implementation of research project activities.

2.2. Standardization Activities for Resilience-Related Research Projects

Existing studies related to (city) resilience refer, for example, to the importance of using respective standards and suggest developing new standards that can be used for knowledge-sharing activities [44] or for increasing the application of tools on a large scale [45]. Moreover, Fathani et al. [46] describe the proposal and promotion of a new standard on early warning systems and Poustourli [39] provides information on the process of developing Workshop Agreements in relation to initiatives in European security research. However, a lack of standardization was identified in the field of (city) resilience [47]; this is consistent with what Zuccaro et al. [48] pointed out as a research priority.

The topic of city resilience was also taken up in formal standardization work within the standardization committees ISO/TC 268 and CEN/TC 465 on ‘Sustainable Cities and Communities’, as well as ISO/TC 292 on ‘Security and resilience’ [49,50,51]. The European standardization activities are also linked to the European Innovation Partnership on Smart Cities and Communities, which states that standardization can provide additional confidence in the market, as it supports the industrialization of solutions, aligns approaches between city systems, speeds replicability and helps to create scaling up processes [52]. In general, standardization not only reduces the costs associated with the implementation of the solutions identified by Smart Cities and Communities, but also enhances the inter-cooperation, interoperability, and social acceptance of these solutions.

As for the topic of resilience, the importance of stakeholder engagement and interaction was stated in several papers and studies (e.g., [44,47]). Standardization in research projects is one option of fostering engagement and interaction with relevant stakeholders in order, for example, to fulfil the need to include all interested parties for framing future research activities [5]. Therefore, aligning different initiatives and reframing resilience practices through standardization will enhance the sharing of knowledge and experiences among the different stakeholders, especially within urban areas. Additionally, the topic of city resilience has several needs that can be addressed via standardization, such as facilitating discussion and thus creating a link between researchers as resilience tool developers and cities as end-users of these resilience tools [53]. Thus, standardization can be a means to overcome existing city resilience gaps.

Several FP projects emphasized (city) resilience and considered standards or even included standardization activities (e.g., [27,54,55]), but proceeded differently due to the lack of a common approach (see Section 4). Good practices are needed to define a common approach and thus to benefit most from project-based standardization work.

3. Methodology

To answer the research questions on which activities are needed to standardize results of a research project on (city) resilience and how the respective engagement of relevant stakeholders can be ensured, the case study methodology was used. As stated in the previous section, the literature on the integration of standardization in research projects is quite limited. Thus, the case study methodology used will contribute to bridging the gap in the existing literature landscape.

A case study is defined as ‘a study in which (a) one case (single case study) or a small number of cases (comparative case study) in their real-life context are selected and (b) scores obtained from these cases are analyzed in a qualitative manner’ [56]. A case study includes data from direct observation and systematic interviewing, as well as from public and private archives. This methodology has also been used previously in cases with reference to standardization. Larsson and Jakobsson [57] use the case study approach to examine the role of standardization in a professional service firm, and Blind et al. [10] use the case of a federal research institute to explain the motives and barriers for participation in standardization.

In particular, an initial multi-case study was conducted to analyze different (city) resilience-related FP projects regarding their integration of standardization and standards. Then, one city resilience-related FP project was assessed in detail, as a case study, in order to gain deeper and more valuable insights into how standardization can be integrated successfully in a project. We have chosen the European project SMR as a case study as it has integrated standardization as an essential part of the project and reflects the three strengths of case research [58] owing to the direct involvement of the research team in the project. In fact, SMR was deemed a success story by the European Commission based on the project outcomes, among which the effective and fruitful integration of standardization activities within the project lifetime were lauded [59].

Figure 1 shows the three steps of the research, as well as the sources from which the case studies information was obtained.

The objective of the multi-case study was to obtain a general analysis of resilience-related FP projects. Therefore, the following nine projects with a focus on (city) resilience, which were carried out between 2015 and 2019, were assessed for their integration of standardization and standards: DARWIN, DRIVER+, IMPROVER, RESCCUE, RESILENS, RESIN, RESOLUTE, Smart Resilience and SMR (step A in Figure 1). As the research projects and the organizations involved have different priorities (e.g., [7]), the selected projects have addressed standardization in various ways. In order to identify similarities and differences, the projects were assessed for their consideration of existing project-relevant standards and their contribution to standardization. Therefore, in total, 24 sources including project deliverables and websites as well as further publications with references to the projects were analyzed in depth in this part of the study, looking for evidence of activities related to standardization. Information concerning the use of standards and their inclusion in the project was sought in each project. This information was searched in the respective state-of-the-art deliverables that are usually written at the beginning of each project. The contribution to standardization—which could be the identification of gaps in standardization, providing input to existing or draft standards or the development of new standards out of the project outcomes—was explored through information gathering of the above-listed sources that relate to the projects’ dissemination activities.

A similar approach was used for the SMR case study analysis (step B in Figure 1). However, due to the research team’s direct participation in the project, insights and know-how that are not completely displayed in the 13 assessed sources of project deliverables, websites and further project-related publications are provided. Nevertheless, also due to the involvement of a standardization organization and in total five standardization-focused project deliverables, more in-depth information could be gathered. A lesson learned survey, furthermore, aimed to reflect the completed standardization activities and to support future FP projects by providing recommendations on the integration of standardization in research projects [60]. The online survey was sent to all CEN Workshop members. In total 21 responses, equal to a response rate of 35%, were collected anonymously. From the background information collected, participants represented at least 13 cities, six researchers, and two consultancies. This response pattern was not surprising since it was possible for an organization to represent all its members in a single response. The survey itself consisted of four different question blocks, with a total of 24 questions. These question blocks included general questions, such as knowledge of standardization and reasons for participating in the standardization activities, project-related questions on the methodology and criteria used for the standardization activities, questions on the experiences of taking part in the CWA developments, and general feedback on the participation in the standardization activities.

The analysis of the multi-case study and SMR case study provides the basis for defining five steps for integrating standardization in research projects (step C in Figure 1).

4. Analysis of Cases and Presentation of Results

The analysis of the multi-case study and the SMR case study in this section focuses on the revision of the conducted standardization activities and on the respective stakeholder engagement; both support answering the research questions.

4.1. Analysis of the Multi-Case Study

From the nine FP projects assessed, seven made reference to existing resilience-related standards and brought them in line with the project. Two projects [54,61] made references to standards in their state-of-the-art analysis, two projects [62,63] assessed the identified standards in more detail, and three projects [55,64,65] completed an extensive assessment of the standardization landscape within the topic of resilience. Activities for identifying the projects’ standardization potential have only been carried out by four projects. One project [66] organized workshops which included presentations on standardization from other projects, one project [67] collected the cities’ needs for standardization in a survey, and two projects [68,69] identified seven and six standardization potentials respectively in project internal workshops or surveys. With regard to the standards development, only two projects [70,71] developed resilience-related standards, but three organizations of other projects have joined the standards development of one project [72]. Additionally, four of nine projects [67,70,71,73] provided or suggested input to draft or existing ISO standards via a liaison and/or direct participation of project members in the standardization committees.

Table 1. Comparison of standardization activities within resilience-related R&I projects.

Project (Call Topic)	Standards Analysis	Standardization Potential	Standardization Outcomes
SMR (DRS-07, H2020)	64 relevant standards assessed	6 standardization potentials identified	3 CWAs: 17300, 17301, 17302 on ‘City Resilience Development’, input to one ISO standard
DARWIN (DRS-07, H2020)	Reference to 2 standards	-	Input to one ISO and EN standard
RESOLUTE (DRS-07, H2020)	40 relevant standards assessed	-	Support to CWA 17302
IMPROVER (DRS-07, H2020)	2 relevant standards assessed	Workshop with presentations on standardization	-
RESILENS (DRS-07, H2020)	Reference to 3 standards	-	-
DRIVER+ (SEC-2013, FP7)	70 relevant standards assessed	7 standardization potentials identified	4 CWAs: 17335, 17513-15, input to one ISO and EN standard
RESIN (DRS-09, H2020)	9 relevant standards listed	Survey on cities’ needs for standardization	Proposals to 7 ISO standards
RESCCUE (DRS-09, H2020)	-	-	Support to CWA 17301 und 17302
SmartResilience (DRS-14, H2020)	-	-	Support to CWA 17301

summarizes the project analysis.

Furthermore, only the projects SMR, DRIVER+ (both DIN), and RESIN (NEN) had a standardization organization as a partner and a dedicated standardization work package.

4.2. Analysis of SMR Case Study

This section provides an overview of the case study itself, the relation of standardization and the projects’ tool development, as well as an analysis of the projects’ standardization activities.

4.2.1. Description of SMR and the Relation of Tool Development to Standardization

The aim of the EU-funded Horizon 2020 project Smart Mature Resilience (SMR) was to develop a European Resilience Management Guideline (ERMG) for building local resilience. The ERMG consists of five different tools that provide guidance to cities and their most relevant stakeholders in implementing an integrated management system that enhances city resilience [71]. In total, seven project partner cities have been proactively involved in a co-creation method [74] for developing, testing, and validating the following five tools:

• The Resilience Maturity Model (RMM) helps cities to assess their resilience status and to identify the ideal path for the resilience building process.
• The Risk Systemicity Questionnaire addresses the risk assessment aspect of increasing the resilience level of cities and prioritizes risk scenarios.
• The Resilience Information Portal (RP) supports the building of a web-based environment for facilitating awareness and engagement among key partners in resilience building.
• The City Resilience Dynamics Tool helps cities to explore and simulate different strategies for implementing resilience policies.
• The Resilience Building Policies Tool combines custom ways to view policies of the RMM with examples from case studies for policy implementation [27].

The co-creation approach in SMR was supported by using a circle of sharing and learning with three different tiers of cities, various methods, such as surveys, Delphi, and Group Model Building, and the integration of standardization activities to involve project external stakeholders [71]. In the first phase of the project, the general requirements, concerns, and needs for resilience tools were gathered from the cities involved in the project and project-relevant standards were identified and analyzed to support the initial development of the tools. Afterwards, the three tier 1 pilot cities (Glasgow, Kristiansand, San Sebastian) tested each draft tool with the aim of improving and validating their usefulness, reliability, and trustworthiness. The four tier 2 cities (Bristol, Riga, Rome, Vejle) peer-reviewed these pilots. These project cities also helped to identify the standardization potential of the project. With the support of the seven project-external tier 3 cities (Athens, Amman, Manchester, Malaga, Malmö, Reykjavik, and Thessaloniki), each tool was validated for its flexibility to be adapted to any European city. This final step was conducted in conjunction with the development of standards derived from the project results. Therefore, all three tiers were invited and participated in the three CEN Workshops established for the three tools chosen to be transferred into a standard. Furthermore, other cities and resilience-focused projects were involved within the standardization activities to further validate and enhance the quality of the envisaged standards [71].

In summary, the following three tool development phases were supported by standardization activities in the project: (1) gathering requirements for the development of project tools (analysis of existing standards), (2) assessing the draft project tools (identification of the projects’ standardization potential), and (3) validating and promoting the tools (developing and promoting standards).

4.2.2. Analysis of Existing Standards

The standards search using 97 search terms was carried out primarily with the support of the PERINORM database [75] to identify an initial set of standards. Additionally, websites of technical committees that were identified as relevant from project partners were assessed for existing standards and standards under development. Each identified standard was evaluated by three SMR research project partners by agreeing or disagreeing to pre-defined criteria (project relevance, connection to sustainability, impact, and effectiveness). Thus, the list was shortened from an initial 276 to 95 standards considered to have significant importance for the SMR project. They were then categorized according to the three main contexts of the project (crisis, resilience, and smart city) and its three subsequent focus areas (critical infrastructures, climate change, societal aspects) [65]. An update of the standards analysis was carried out after about a year to consider newly published standards during this period and relate the identified relations to SMR tools. This ultimately resulted in 64 standards identified as relevant for the SMR tools.

Table 2. Overview of identified standards.

Category	Total vs. Relevant Standards	Example of Relevant Standard and Possible Benefit for the Project
Crisis	52 vs. 22	ISO 31000 ‘Risk Management-Principles and guidelines’ → Consider risk management as basis for resilience activities.
Resilience	18 vs. 10	ISO 22316 ‘Security and resilience-Organizational resilience-Principles and attributes’ → Consider resilience for organization aspects for city resilience.
Critical infrastructures	75 vs. 7	DIN SPEC 91330 ‘Terminology relating to events in pipeline- and cable-based infrastructures’ → Awareness and application of relevant terminologies.
Climate Change	28 vs. 9	BIP 2178 ‘Climate change adaptation’ → Link management standards that cities use to climate change.
Societal aspects	29 vs. 5	ISO 22398 ‘Societal security-Guidelines for exercises’ → Support cities’ application of SMR tools in exercises.
Smart City	74 vs. 11	ISO 37100 ‘Sustainable cities and communities–Vocabulary’ → Awareness and application of relevant terminologies.

provides an overview of the standards identified per category, including the total amount of standards initially found vs. number of relevant standards after final assessment, as well as an example of a relevant standard and its possible benefit for the project.

4.2.3. Identification of Standardization Potentials

By comparing the demand and supply side, new standardization potentials were identified [76]. For the demand side, the needs of the cities were initially identified with the support of an offline questionnaire, as well as using working sessions during SMR project workshops ([69,71]). The questionnaire was answered by all seven project cities (tier 1 and 2) and included 10 open questions on the city’s challenges and needs, their proposals for new standards to address these needs and their implementation plan for the SMR resilience tools [69]). The resulting preliminary list of needs was complemented and assessed against available resilience tools in the European Workshop on Resilience in Cities and Communities [53]. In total, 44 people from 28 organizations and 11 countries attended, including all project partners, as well as additional relevant stakeholders, such as other FP projects, cities (i.e., Prague, Udine, and Thessaloniki), consultants, and standardization experts. A total of four main needs were identified: (1) to network with other cities facing a similar variety of risks; (2) to have a common terminology for a shared understanding on city resilience; (3) to simplify cross-sectoral cooperation and integrate all relevant stakeholders of a city; and (4) to have one set of resilience tools, including good practices and clear resilience action plans, and a description of how to best use them.

Finally, to assess the SMR tools and the ERMG for the supply side, the research project partners defined the following five assessment criteria [69]):

• necessity (cities’ needs for having the solution implemented or taken up);
• transferability (solution has high potential to be transferred into a standard, i.e., at least 90% of the envisaged standard should consist of requirements, and recommendations should make up no more than 10%);
• feasibility (current status of the solution—not ready vs. finalized—for deciding if it is possible to develop it within the project lifetime);
• complementation of existing standardization landscape (gap in existing standardization);
• further input needed (integration of project externals for enhancing the quality and uptake of the solution).

The first indicator, ‘Necessity’, was answered by all seven cities involved in the project using a five-point Likert scale (very low to very high). The research partners answered the other four indicators with either agreeing or disagreeing. The assessment resulted that the RP, the RMM, and the ERMG meet four or five of the assessment criteria and thus were deemed as appropriate for being transferred into a standards series called ‘City Resilience Development’ ([69,72]).

4.2.4. Development and Promotion of Standards

In order to develop the envisaged standards series, three CEN Workshops on the above selected SMR tools were initiated by the project partners. In total, 59 individuals from 11 research and consulting organizations and 14 cities or city associations contributed to the three CEN Workshops. With six cities and city associations as well as seven research and consulting organizations, more than half of the participating institutions were not from the SMR project [72]. As three of the assessed projects funded under the same call have joined the standards development, it can be acknowledged that SMR had successfully engaged with the relevant stakeholders through the standardization activities [72]. For each of the CEN Workshops, a series of meetings were conducted in conjunction with the projects’ co-creation workshops to further validate and promote the SMR tools. The results of the CEN Workshops were published in the CWA 17300 series on ‘City Resilience Development’ [77], consisting of CWA 17300 ‘Operational Guidance’, CWA 17301 ‘Maturity Model’, and CWA 17302 ‘Information Portal’. The Operational Guidance CWA is based on the ERMG, which consists of five operational steps that have to be repeated in its full cycle regularly: baseline review, risk awareness, resilience strategy, implementation and monitoring, and evaluation and reporting. It aims to enhance city resilience and includes different SMR tools in each of the steps [78]. The Maturity Model CWA takes its content from the RMM and contains five maturity stages (starting, moderate, advanced, robust, and vertebrate), each with a description of the relevant stakeholders and policies that should be implemented in order to achieve the objectives defined in each maturity stage [79]. Both CWAs provide an extensive overview of terminology and are mainly targeted at policy- and decision-makers at the city level and any city stakeholders working on resilience. CWA 17300 was seen as helpful for implementing and assessing effective and measurable resilience policies, whereby CWA 17301 was confirmed to support the prioritization of focus areas for resilience and the investment of respective resources [80]. Finally, CWA 17302 provides information technology professionals and information technology decision-makers with a list of requirements for how municipalities can develop an information system that facilitates resilience-building through collaboration, communication, and engagement. This CWA is connected to the RP SMR tool [81]. It was recognized as a useful tool to improve communication with citizens and stakeholders [80]. Additionally, the standardization activities were promoted in several conferences, and through a liaison with ISO/TC 268 [27], the standards series were discussed on the international level for consideration [71].

4.3. Results of All Cases

The analysis of the cases revealed that the FP projects that completed an analysis of existing standards could also refer them in their project work or apply them to their tool development. Furthermore, FP projects having a standardization organization as a partner have mostly conducted a more detailed assessment of existing standards. However, the findings show also that the analysis of existing standards is not always of high priority for the project in their state-of-the-art analysis. This was even the case in the assessed RESIN project, in which a standardization organization was involved. Furthermore, and as a result of the SMR lessons learned survey, the SMR consortium understood the crucial benefits of starting the standardization activities as early as possible in the project and that their awareness of standards relevant to the city’s interest or the project research increased due to the analysis of existing standards.

With regard to the identification of standardization potential, it can be summarized that standardization-focused events such as the European Workshop in SMR are not common in FP projects. However, the findings and lessons learned survey of SMR show that these events support sharing project results with other projects and relevant stakeholders as well as acquiring them for the standardization activities. Furthermore, standardization potential activities are mostly conducted if a standardization organization is part of the project. In terms of the most appropriate standardization potential criteria, ‘filling the gap in existing standardization’, ‘need for further input’ to the project results, and ‘transferability of the project outcome into a standard’ were highlighted in the SMR project [60]. Further input through exchange with project externals was needed for fine-tuning the SMR resilience tools and the development of new standards out of these tools was a major benefit for the cities to disseminate and further exploit the SMR results.

The analysis of cases showed that only FP projects in which a standardization organization was involved initiated new standards. However, as the process for developing standards is quite open (e.g., possibility of initiating or joining CEN Workshops or commenting on draft standards), a standardization organization is not always needed as a partner in the project. Furthermore, the process for the standards development ensured the active and targeted engagement with relevant stakeholders [71]. The meeting methods used with joint workshops for engaging stakeholders and for transferring new knowledge into the standards development was confirmed by more than 75% of the respondents of the SMR lessons learned survey. The CEN Workshop members with the three tiers as key stakeholders and the main target group of the standards saw a great benefit to enlarging their network and sharing each other’s experiences and good practices of using city resilience approaches [60]. In addition, the active dissemination via conferences or standardization committees addresses the projects’ needs and builds the basis for wider dissemination and exploitation of the project results in a sustainable way.

In summary, it can be said that these results are new, that they provide relevant insights into the use of standardization in research projects and that they support the implementation of future research projects that would like to take standards and standardization into account.

5. Proposal for Steps to Integrate Standardization in Resilience-Related Projects

From the analysis of the SMR case study and the other resilience-related FP projects, it can be summarized that five activities are usually needed to integrate standardization and to gain impact within and beyond a research project on (city) resilience. When integrating standardization in city resilience-related research projects, the following steps are especially crucial for the effectiveness of these standardization activities to ensure the sustainability of the project results and their uptake by other cities. To better understand the general approach,

Table 3. Five steps with tools and outcomes for integrating standardization in a research project.

Steps	Examples of Tools	Expected Outcomes
1. Analyze the state of the art in standardization	Databases for standards search, such as PERINORM or the Online Browsing Platform (OBP) of ISO	List of existing standards classified according to the projects content (see, e.g., [55,64,65]).
2. Identify end-user needs and standardization gaps	Open survey and/or workshop to identify end-user/cities’ needs; internal workshop to define individual assessment criteria	Overview of standardization potentials based on project results (see, e.g., [68,69]).
3. Define project standardization strategy	Project internal meetings and formal exchanges with relevant standardization committees	Standardization strategy with implementation plan (see, e.g., [68,72]).
4. Initiate end-user focused standardization activities	(CEN) Workshop with project and external partners on standardizing project results during the project	Standardization activity like CWA or input to existing/draft standard (see, e.g., [63,70,73]).
5. Promote and exploit the standardization activities	Presentations at (scientific) conferences, summarizing promotional material, etc.	Increased visibility and uptake of standardization deliverables beyond the project (see, e.g., [63,80]).

provides an overview of the different steps with examples of tools that can be used and the expected outcome. Furthermore, the characteristics of each step, such as required sub-activities, stakeholder engagement, common issues, and relation to project management theories are described afterwards.

At first, an analysis of existing standards needs to be conducted in order to link relevant existing standards with the project and to provide project partners with information on these identified standards. This can be carried out in a set of activities: (a) define keywords for the search of standards based on the main objectives and envisaged outputs of the project (see, e.g., [82]), (b) identify relevant standardization committees, (c) search for and analyze existing standards and ongoing standardization activities regarding their relevance for the project, and (d) cluster the results in order to be usable for the project partners (e.g., according to the defined keywords or main project contents). Project partners with different backgrounds (e.g., researcher, end-user) should support the analysis of identified standards. As the number of existing standards in complex topics such as city resilience is usually high, it is important to identify the standards that are really relevant for the project. Possible issues at this first step are to interest the project partners in this analysis exercise. Therefore, it is important to explain the benefits they obtain from this for the project to work, and thus confirm the relevance of standards for project management (see [28]).

In step 2, this overview of standards together with the (envisaged) results of the project needs to be compared with the requirements of the end-users to identify the projects’ standardization potential. In order to do so, the following activities are suggested: (a) identify challenges on the project topic and the needs of end-users (demand side), (b) summarize the main project results and relevant existing standards (supply side), (c) identify standardization gaps by matching demand and supply side, and (d) assess the identified standardization gaps by using individual assessment criteria. For the identification of needs, mainly the end-user project partners as well as project external end-users should be consulted. The research project partners should support the definition and application of the individual assessment criteria of the identified gaps. In the case of city resilience, it is very important to address the city needs directly by active interaction such as surveys or workshops. However, this approach is also challenging for a project, as cities do not always share their challenges and needs publicly and transparently.

Based on the identification of standardization potentials, a standardization strategy needs to be set up to decide on how to integrate the project results into standardization. Therefore, various activities should be conducted: (a) agree on identified standardization gaps the project would like to bridge, (b) assess contribution to standardization for these gaps, (c) plan these contribution possibilities, and (d) liaise with relevant standardization committees for their input and support. Having such a strategy supports visibility and ensures the integration of the standardization system before the activities start. On the topic of (city) resilience, the diversity of stakeholders affected makes it crucial to take on board the relevant standardization committee as early as possible. Thus, besides project partners the standardization organizations and/or relevant standardization committees (if not already in the project as partner) should be consulted or leading this identification. A formal liaison with a standardization committee supports the project management work in this regard.

Step 4 results in the initiation of end-user focused standardization activities based on project results, which usually leads to providing input to draft or existing standards, or to developing new standards. This can be achieved with the support of a standardization organization as project partner, by the following activities: (a) refine the contribution to standardization (e.g., title and scope of new standard), (b) acquire contributors to the standardization activities with different backgrounds and from inside and outside the project, (c) develop the content for standardization, and (d) propose the input to relevant standardization committees or publish a standard within the standardization system. For the projects’ success (see [22]), it is important to actively involve end-users, e.g., different cities working on city resilience in and outside the project, in the whole standards development process to increase the impact. Additionally, their role as potential multipliers and the open standardization process should be exploited for engaging and integrating further relevant stakeholders. Moreover, if possible, the standardization activities should be aligned with other events to benefit from this interaction (e.g., engaging further stakeholders and direct transfer of new knowledge within tool development into the standards development). However, when integrating too many stakeholders, the consensus-making process might take more time, which should be taken into account due to the limited project duration.

In the final step, the standardization contributions made need to be promoted and exploited beyond the project to ensure their uptake and to create impact for the resulting standardization deliverables. Various activities can support this step: (a) summarize the standardization activities carried out for promotional issues, (b) promote the developed or contributed standards to the target groups of these standards, including project internal relevant stakeholders, (c) present the contribution to standardization to further relevant standardization committees for their potential uptake (e.g., as full national, EN or ISO standard). All project partners that were involved in the standardization activities should support their promotion and exploitation to ensure the envisaged impact of the standards. Communication and dissemination of the standardization deliverables to their potential end-users is important. For example, other European cities need to be aware of the developed standards initiated from the SMR project to support the adoption of these standards at their national levels. However, sometimes it seems difficult to choose the best promotional means due to timing of relevant conferences or meetings of relevant standardization committees. Furthermore, standardization organizations should foster these activities, as they are recognized and acknowledged bodies.

6. Discussion

This aim of this paper is to describe possible standardization activities for the transfer of results from a research project on city resilience, and to show how standardization supports the involvement of relevant (external) stakeholders in the project. This is the first study in this regard that illustrates the appropriateness of using standards and standardization for gathering knowledge and for significantly enhancing the dissemination and exploitation of project outcomes.

For this purpose, nine FP projects on (city) resilience were initially analyzed regarding their use of standards and contribution to standardization. One of them, the SMR project, was further assessed, as this project is a successful case in terms of the standardization of its results. Out of this analysis of different cases, a five-step approach is proposed for integrating standardization successfully in resilience-related research projects.

In the early phase of research projects, there is usually an initial lack of awareness about existing standards and the standardization process. This is in line with what has been identified as a limited knowledge of standardization at the beginning of projects; in the case of SMR, this knowledge increased significantly throughout the project [60]. As the integration of standardization in research projects has only increased in the last years, this result was somehow predictable. Therefore, analyzing existing standards related to the project’s work builds a basis and provides benefits for implementing the project. For example, the list of resilience-related standards in SMR increased the awareness on resilience-related standards of project partners, built the basis for agreeing on project-relevant terms and their definitions, and thus provided them with a strong definition of resilience policies. Additionally, it supported city partners to identify their standards of interest. These findings confirm the role of standards as knowledge carrier for further research, as stated by Hällgren et al. [30]. It also shows that assessing standards should be an integral part of every project, even for those that did not plan contributions for standardization. Integrating a standardization organization or a partner actively involved in standardization committees within the project consortia can support raising awareness on existing standards.

For the analysis of standardization potentials, it is challenging to choose the appropriate method for identifying standardization needs, including relevant stakeholders and especially the potential end-users of the project results. An analysis of the end-users’ needs is therefore crucial and relates directly to the aim of a standard, which should promote optimal community benefits. In this regard, the use of a workshop, such as in SMR, responded to the identified need of end-users such as cities to share experiences and discuss resilience tools with relevant stakeholders [53]; workshops of this type also support the acquisition of relevant stakeholders for the planned standardization activities. Furthermore, the criteria for the assessment of the identified standardization potentials should be chosen wisely and according to the project’s objectives. In SMR, one of the objectives was to transfer the project tools into standards, and thus the assessment criteria related to this fact. This confirms the appropriateness of standardization as a tool for dissemination indicated in different studies (see [5,33]). Nevertheless, the question of whether a standardization organization is necessary to analyze standardization potentials cannot be finally answered through the findings of the assessed cases.

Once the standardization potentials are identified, it can be difficult to decide how these contributions to standardization can be undertaken within the project duration. A standardization strategy should rely on the identified gap within the standardization landscape on the projects’ topic. Thus, in the case of city resilience, the standardization strategy of SMR concluded with the transfer of the project tools into a series of standards to fill the gap in standardization. The process of standardization in the cases required the engagement of relevant stakeholders and therefore gave it a beneficial role for project management, which confirms existing studies (see [31,43]). The use of fast and flexible tools for standardization, such as CEN Workshops, has been confirmed in the different cases as they fit perfectly within the scope and timeframe of a project and support the engagement of relevant stakeholders. The integration of project externals, especially in city resilience projects such as SMR, is important to ensure validation and further uptake of the project results. Co-creation methods of developing tools and conducting standardization activities at the same time can support the engagement of these relevant stakeholders. The number of participants in the standards developed by SMR show the wider interest in the topic of city resilience and in contributing to related standardization activities. This confirms the identified reasons for participating in standardization activities such as the relevance for disseminating and exploiting the project results, the individual interest in the topic of city resilience and the wish to take part in creating the standards [60]. Standardization organizations are required for conducting standardization activities such as Workshop Agreements. It can be further confirmed that due to their network, they also have a dedicated role in engaging relevant stakeholders (see [5]). However, standardization organizations that are involved in projects should assess the different possibilities to contribute to standardization (see, e.g., [83]).

When developing a new standard, the difficulties are choosing an appropriate meeting culture, addressing the different needs, and consensus building. Meetings should be conducted in an interactive and co-creative manner to ensure the involvement of all participants, such as in SMR. Due to the open process of CEN Workshops, certain city stakeholder needs can be directly addressed, such as agreeing on a common terminology related to city resilience as a basis for communication, and a shared understanding between cities and an exchange of experiences with other cities on resilience challenges and tools. As cities and the project partners play an essential role in creating the standards, they acknowledge and are committed to the standards. By liaising with relevant standardization committees, such as ISO/TC 268 in the SMR case, an adoption of the project’s standardization activities and thus a further validation of the project tools can be triggered. The analysis of the cases confirms the study of CEN-CENELEC [41] in which more tangible and formal standardization outcomes such as Workshop Agreements can be reached by involving standardization organizations in projects. For the constitution of a project consortium, this implies the close examination for involving standardization organizations.

Although current studies ([16,41]) show the benefits of integrating standardization organizations, the active dissemination and exploitation of the standardization deliverables at the end of the project are often lacking. However, active promotion, as was carried out in SMR with the CWA 17300 series, to other cities is needed to support the uptake and wider application of the standards developed. Enhancing city resilience is not a matter of just one city. If cities nearby are not resilient, then the city itself may not be as mature in resilience. An assessment of the standardization deliverables to current application cases, such as analyzing the developed CWAs of SMR with regard to their relevance to the COVID-19 pandemic, could support addressing the identified needs of stakeholders and thus the consideration and uptake of these standards. Among others, such an activity would also support the dissemination and exploitation of the project results for future research, once again confirming Hällgren et al. [30]. Furthermore, the participating cities are used to working with standards and, as they are already partially applying the developed CWAs, the project tools will be sustainably used. However, how much impact CWAs are having on a projects’ topic has not yet been assessed.

A limitation of the research presented here lies in the fact that only a few cases could be assessed regarding their use of standards and contribution to standardization. This relies on the currently rather low consideration of standardization activities in research projects. Thus, only limited sources are available for justifying the proposed steps for integrating standardization in resilience-related research projects. Due to its very successful integration of standardization, the SMR case study provided mostly positive criticism. On the other hand, the further assessed cases provided very limited information and research in the literature. Therefore, future research and application of the steps defined in other research projects are needed for validating the approach and gathering more critical views. With the presented approach, this paper provides a basis for a future line of research on standardization in resilience-related projects, while taking into account that the city resilience topic is not yet fully exploited.

7. Conclusions

The topic of city resilience is an important one and has been brought to the forefront due to the current pandemic (e.g., [84]). To support research on city resilience, this paper investigated various resilience-related projects and with SMR one case study more deeply. The result is a five-step approach for integrating standardization in research projects. The outcomes of this research confirm the hypotheses and assumptions of the limited resilience-related studies. Furthermore, the standardization activities within SMR have been acknowledged by the project partners [60] and, therefore, the steps defined seem to be appropriate for the purpose of transferring research results to cities for their resilience enhancement and to foster the engagement of stakeholders; thus, the research questions are answered.

This research validates the importance of standards and standardization for resilience-related projects and thus complements the study of Sanjuán et al. [5], who confirm the relationship in the area of construction and the role of the standardization organizations. From the case analyses, it can be summarized that the FP projects having involved a standardization organization and having conducted a more detailed standards analysis also achieved relevant standardization outcomes. Moreover, with the SMR case study, the research provides an additional example to Paredes Muse et al. [37] on how stakeholders can be integrated in the open standardization process by addressing the needs of cities to strengthen their resilience capabilities. The case study analysis was also relevant for further reasons. It suggests assessment criteria for the identification of standardization potential, which were rated as suitable [60]. However, this might differ from project to project, as other criteria might be identified for different contexts. For this reason, additional insights and experiences from other research projects are needed. The study further validates the use of co-creation approaches for developing the project tools and standards content, such as for the CWA series on ‘City Resilience Development’. Furthermore, the case study shows that the exchange with relevant stakeholders and standardization committees, as well as the promotion and exploitation of the achieved standardization work, are crucial for a successful integration of standardization in research projects. Therefore, it confirms the view of CEN-CENELEC [85] that standardization supports the dissemination and exploitation of research results and the inclusion of and networking between interested parties.

With the presented means of standardization, a further strategy for supporting the technology transfer is available and complements the literature review of Cedano and Hernández-Granados [34]. It also partly confirms Velásquez et al. [45] with regard to the use of standards to increase the implementation of tools on a large scale. However, the application of standards developed out of research projects requires more research. Additionally, the identified lack of standardization of (city) resilience approaches by Linkov and Palma-Oliveira [47] could be partly overcome by the provision of the standards series on ‘City Resilience Development‘. However, the value of city resilience for further research and the organization of related standardization activities are very great due to the current crisis and the limited standards focusing on the resilience of cities. The drawbacks of this research are the limited information and literature available on the integration of standardization in research projects, an issue that offers a variety of potential research activities and thus needs to be further explored.

We believe that currently less known dissemination and exploitation tools such as standardization will take a more prominent role in future projects to increase the impact of their results and to address the needs of end-users such as cities. Especially due to the increasing number of challenges cities and society are facing, the high impact of project results is expected to address these issues. Therefore, the projects funded by the FPs will particularly need to consider standards and standardization as an integral part and an essential element with regard to the dissemination and exploitation of their results. Practical examples of projects such as SMR, in which standardization was successfully integrated, are therefore key to raising awareness and supporting this approach of effectively linking standardization with research projects. Further studies are needed to provide insights from other case studies to validate and enhance the proposed approach.