Assessing the Impact of Technology Assessment, Responsible Research and Innovation and Sustainability Research: Towards a Common Methodological Approach

Abstract

The missions of Technology Assessment (TA), Responsible Research and Innovation (RRI) and Sustainability Research (SR) are similar in their relationship to Science, Technology and Innovation (STI). Although adopting different terminology, the three research fields show great conceptual and methodological overlaps, as all three react to societal demands and policy needs via common tools. The impact of TA, RRI and SR is linked directly, but its assessment requires new thinking in terms of common conceptual and methodological approaches. The paper provides an overview of the current discussion on societal impact assessment and identifies areas of particular interest for the three fields. It describes the current discussions on impact assessment and impact indicators in each field and undertakes a bibliometric analysis that shows clear inter-relationships in terms of thematic focus and a common emphasis on impact in all three fields. Following this analysis, the paper argues for a common impact assessment methodology for TA, RRI and SR, under the notion of resonance and based on the concepts of anticipation, reflection and inclusion.

1. Introduction

Transformation research is essentially a comparative approach. It compares concepts, systems, processes, etc. that have the potential to confer change to society and as such, are legitimate candidates for societal applications. Science, Technology and Innovation (STI) research by default includes transformative research processes since it aims at improving the quality of life for every citizen. However, the actual effect of such research in society is far from clear as it usually involves complex, interconnected disciplinary approaches that are not easy to deconstruct into single cause-and-effect processes. Nevertheless, there have been significant efforts to identify the dynamics of STI impact in society, and this is nowhere more evident than in the debate on impact indicators.

The application of methods to evaluate STI is relatively new, with the first use of scientometric indicators taking place at the beginning of the 1980s [1], although the debate on the use of quantitative measurements to assess the impact of STI is evident since the early 1970s with viewpoints to evaluate scientific journals [2], scientific disciplines [3] or even the macro-level impact of science (We use the overall umbrella term of STI in this paper, although there has been an evolution in relevant terminology from Scientific research, to Science and Technology research, to the present day’s Science, Technology and Innovation research. We do not wish to discuss the merits of each term, but instead assume that STI covers the subject matter of TA, RRI and SR adequately in this paper on impact assessment) [4]. We are now in the position to have a standard global approach to STI indicators that include over one thousand indicators on research and development, science, business innovation, patents, education and the economy (See OECD https://www.oecd.org/sti/ for full details, accessed on 15 November 2021). However, we still lack a common approach to evaluating the impact of interdisciplinary research with a societal and policy focus, such as that of TA, RRI and SR. This is despite the fact that the OECD itself has recognized such impacts as central to the whole STI field. The standard eleven dimensions of the impact of STI that have been identified are informative in this debate [5]:

Science impacts:new knowledge creation and instigation of new research.

Technology impacts:new products, processes and services as well as new innovation activities.

Economy impacts:effect on costs, profit, productivity and market development.

Culture impacts: public understanding of science and development of intellectual skills.

Society impacts: effect on well-being and quality of life.

Policy impacts: effect on decision making and public engagement.

Organization impacts: effects on administration, planning and organization of work.

Health impacts:effects on life expectancy, prevention of illnesses and the health-care system.

Environment impacts:effects on the environment and natural resources.

Symbolic impacts: effects on credibility and reputation.

Training impacts:effects on curricula, pedagogical tools and qualifications.

The categories in italics (Culture, Society, Policy, Organization and Symbolic impacts) are directly relevant to TA, RRI and SR, since they deal with their core aims and processes. At the same time, these categories are the least developed and standardized on the OECD list. This is also evident in research that focuses on researcher perceptions of the societal impact dimensions of their research, which measures the problematic extent of this issue. A global survey conducted by the publisher Springer Nature of 9000 researchers across many disciplines, on the issue of societal impact of research, is very revealing in this respect (The term “societal” here denotes all four types of impacts in terms of cultural, policy, societal, organizational and reputational aspects) [6]. The great majority of respondents (68%) believed it is important that their research should have societal impact beyond academia, but only a quarter felt that they receive adequate support to achieve this, and research funder requests is given as the main reason for dealing with the issue (44% of respondents). In terms of identification of societal impact targets, the survey used the Sustainable Development Goals (SDGs) as examples, but found no preference for any target SDG amongst respondents. The main manner of achieving societal impact was via open access publications, conference presentations and social media discussions. Finally, 76% of respondents believed they should be doing more to measure the societal impact of their work but the great majority said that there was a lack of a comprehensive approach or methodology to do so. As such, the transformative potential of their research is hampered by the absence of means to facilitate a societal uptake of the research results.

It is evident that we are faced with a situation whereby there is little development in promoting societal impact assessment in STI. When it comes to specialized research such as TA, RRI and SR, we can identify certain trends in the discussion that allow us to draw positive conclusions as to the possibility of developing common impact assessment metrics. However, first, it is important to provide an explanation of our own perspective on impact in this approach. We draw from the conceptual discussion on what constitutes impact that has taken place in the realm of TA, as a unique occurrence amongst the three concepts that can directly apply to all of them. From this perspective, the term “impact” is interchangeable with the terms “success” and “resonance”. Whereas “impact” is the formal term connoting a rational and measurable decision-making process, “success” denotes a mainly subjective evaluation independent of specific measurable indicators and “resonance” describes any observable effect of the process or project under consideration. In this respect, although all three terms have a direct use in identifying what practitioners see in the value of their work, when one attempts to assess the effect of TA, SR or RRI in society, the term “resonance” is equivalent to that of impact. We will therefore include this term in our perspective of impact, and also adapt the definition of impact from Hennen et al (2004) as:

“Impact of TA, RRI and SR is defined as any change with regard to the state of knowledge, opinions held and actions taken by relevant actors in the process of societal debates on STI issues”

This definition includes the key elements that any research on impact should contain. Knowledge, opinions and actions are three interconnected and necessary aspects of impact, and each can be measured independently, as we will see later on. The plurality of actors denotes the inclusiveness of the research process, that is also a necessary ingredient in all three fields. Our main research question on how to develop a common impact methodology for TA, RRI and SR, draws from these discussions, while we also provide an argument for the significant overlaps between the three research fields. This is evident both at the conceptual and at the research output levels. The analysis of the common ingredients in the research process, in addition to the bibliometric analysis of relevant journal publications, clearly shows the case for a common impact assessment methodology.

2. Conceptual Linkages in TA, RRI and SR

The development, subject matter and aims of TA, RRI and SR show considerable conceptual overlaps. Their development was a direct reaction to societal demands and policy needs, their subject matter is the so-called “post-normal science”, and their aims point to enhancement of reflexivity in STI [7]. Accented social debates on STI developments in the 1960s triggered the development of TA as a policy tool [8] while recent debates on ethically dubious research practices and innovations have initiated the development of RRI with its particular focus on ethical innovation practices [9]. Finally, the enormous societal challenges deriving from dealing with climate change have instigated SR as the main research and policy tool [10]. All three research approaches deal with a new reality in scientific research and practice, whereby factual uncertainty is accepted, contestation of values is prominent, and stakeholder involvement is demanded [11]. Finally, all three research approaches aim at developing a reflexive attitude to STI anticipatory analysis via inclusive methodologies.

Based on the conceptual antecedents and current discussions, similarities between TA, RRI and SR are therefore evident. These do not constitute a single disciplinary theme but are especially clear at the normative level of assessing and shaping STI developments. Differences between the three are to be found in their history of development and some of their focal activities. While RRI has been operationalized according to the five research fields (Open Access, Science Education, Gender, Public Engagement and Ethics), it has also gained academic attention, especially regarding its conceptual shaping. We find several conceptualizations and descriptions of RRI [12,13] that bring it closer to the concepts of TA and SR. Especially relevant here are the dimensions of anticipation, reflexivity, inclusion and responsiveness that have been developed in the RRI academic discourse and present approaches more attuned to societal impact than the current one. There have been discussions on whether RRI is a critique of TA or if TA should rather be a “light-house” for developments such as RRI [14]. Overall, RRI can be regarded as a response to a continuous need to improve the interactions between STI and society, towards finding approaches to and new forms of, for example, public engagement, as is the case with TA and SR.

Overall, all three research approaches have a clear focus on societal sustainability, inclusion and policy acceptance. Their commonalities are moreover evident in the adoption of the key concept of anticipation, the use of engagement methodologies and the management of complex systems [7]. Anticipation of unintended consequences and specific impacts of STI developments are common in the three fields. The achievement of successful anticipatory analysis includes extra-scientific input via stakeholder engagement methodologies. Management of system complexity includes social mapping, actor identification and options development. The need for an interactive form is assessment is essential in all three research fields, that is expressed in the promotion and implementation of public debates and engagement. Part of this is effort is to function as bridge-builders between various stakeholders in order to achieve socially sustainable policy decisions.

3. A Bibliometric Analysis of TA, RRI and SR

As we have seen, the conceptual relevance of TA, RRI and SR is evident in their inclusive approaches, policy applications and aims in relation to society at large. The transformative potential of their research is directly related to their application focus and the relationship that the different focuses develop over time. In order to identify the areas in which the three research fields concentrate their research, the main themes they focus their research on and the potential overlaps between them, we have employed a bibliometric analysis methodology. Bibliometrics analysis is undertaken in order to identify strengths and weaknesses in the research domains under investigation, as well as conceptual overlaps between research fields. It is an efficient method to describe the knowledge structure and research trends in specific domains. It involves publication metadata from key international publication databases. It provides us with the possibility to review a very big corpus of publications and allows for a certain amount of quantitative analysis to back up our argumentation. To our knowledge, there is no bibliometric analysis done in these fields with the exception of comparisons between the concepts of Responsible Research and Innovation and Responsible Innovation [15].

For the bibliometric analysis we decided on a relatively recent time span that covers the last decade of research (time span: 2011–2020) as we believe that this was by far the most important development phase whereby all three research fields have been research active. (Although raw data sets were defined with the time span of 2011–2020, the actual results include the time span of 2011–2021 since some publications were already published online in 2020 before formally being published in 2021.) We applied the $Bibliometrix$ package in $R$ environment to operate the qualitative analysis [16]. The raw data for the analysis was sourced from the core collection database that includes the Social Sciences Citation Index (SSCI), the Arts & Humanities Citation Index (A&HCI) and the Conference Proceedings Citation Index—Social Science & Humanities (CPCI-SSH) of the Web of Science (WOS). The key words used and the number of publications analyzed are shown in Box 1. (The raw data was downloaded from www.webofknowledge.com/, accessed on 28 May 2021).

Box 1. Search Script in WOS.

# (1) Technology Assessment

# TS = (“Technology Assessment” NOT “Health Technology Assessment” NOT “Health Technology Assessments”) AND AK = (“Technology Assessment” NOT “Health Technology Assessment” NOT “Health Technology Assessments”) OR KP = (“Technology Assessment” NOT “Health Technology Assessment” NOT “Health Technology Assessments”) AND PY = (2011–2020)

# Indexes = SSCI, A&HCI, CPCI-SSH Timespan = 2011–2020

#Result = 502 records

# (2) Responsible Research & Innovation

# TS = (“Responsible Research & Innovation”) OR AK = (“Responsible Research & Innovation”) OR KP = (“Responsible Research & Innovation”) AND PY = (2011–2020)

#OR

# TS = (“Responsible Research”) OR AK = (“Responsible Research”) OR KP = (“Responsible Research”) AND PY = (2011–2020)

#OR

# TS = (“Responsible Innovation”) OR AK = (“Responsible Innovation”) OR KP = (“Responsible Innovation”) AND PY = (2011–2020)

# Indexes = SSCI, A&HCI, CPCI-SSH Timespan = 2011–2020

# Result of RRI = 10 records, result of RR = 481 records, result of RI = 308 records

# (3) Sustainability Research

# TS = (“Sustainability Research”) OR AK = (“Sustainability Research”) OR KP = (“Sustainability Research”) AND PY = (2011–2020)

# Indexes = SSCI, A&HCI, CPCI-SSH Timespan = 2011–2020

# Result = 421 records

# Total result = 1597 records

Raw dataset size ≈ 22.6 MB

Legend:

TS: Topic; AK: Author Keyword; KP: Keyword Plus combine keywords in the title of publication and author); PY: Year Published.

3.1. Publication Metrics

The parameters surrounding publication metrics reveal some crucial information on the overall dynamics of scientific research development in the three fields.

Table 1. Scientific publications of TA, RRI and SR (2011–2020).

Year	TA	RRI	SR	Yearly Amount	Merged	Difference
2011	41	9	20	70	68	2
2012	38	15	16	69	68	1
2013	31	18	22	71	68	3
2014	44	28	25	97	96	1
2015	57	53	28	138	136	2
2016	54	74	30	158	154	4
2017	63	91	56	210	198	12
2018	61	119	64	244	235	9
2019	49	133	63	245	240	5
2020	57	162	83	302	297	5

shows that scientific publications of TA, RRI and SR have increased overall, albeit with different ratios, from 2011 to 2020. The average annual increase ratio of publications in RRI is much higher (39.83%) than that in TA (5.63%) and SR (20.16%). This is particularly evident since 2016, whereby the annual number of scientific publications in RRI is higher than that for TA or SR, even surpassing their cumulative number in 2019/2020. This is of course a snapshot that does not account for the much longer publication record in TA (since the 1970s) than RRI or SR, which are much newer research fields. However, it shows the strength of RRI in reaching a wider audience and perhaps activating more interest in the subject matter of STI and society. Based on these findings, one can therefore draw the conclusion that, if we consider TA, RRI and SR as an integrated research domain, then RRI represents an important development dynamic particularly evident in the last ten years.

Another result of significance is the difference between the total number of publications and the merged ones (i.e., those that refer to only one of the three fields). The number of publications that deal with a combination of any of the three research processes is very small. That allows us to draw the conclusion that there is little research dedicated to comparisons and/or integration of TA, RRI and SR.

3.2. Keyword Co-Occurrences Network

The next level of analysis is that of the co-occurrence network (as shown as Figure 1), a crucial knowledge mapping device used to reveal the relationship between different knowledge units [17]. In our case, we analyzed the keyword co-occurrences network in the merged dataset of TA, RRI and SR publications. The TA subnetwork, marked with blue color, shows that the main theme of “technology assessment” is connected with the themes of “policy”, “decision making” and “challenges”. The RRI subnetwork, marked with green color, shows that the main theme of “science” is connected with the themes of “framework”, “knowledge”, “society”, “participation” and “governance”. The SR subnetwork, marked with red color, shows that the main theme of “management” is connected with the themes of “system”, “performance”, “climate-change” and “impact”. Fifty-one records were derived directly from journals in the domain of “Management”, for instance, Academy of Management Annals, Supply Chain Management—an international journal, Research Policy and Journal of Operations Management were all represented in the sample. This denotes that SR is related to supply chain management, marketing research and user engagement, which are all important research topics in the domain of management science [18,19,20,21,22]. Further analysis of the subnetworks could provide a detailed view of the thematic development in each discipline. For instance, it would be interesting to research further the thematic priority of “management” in SR and its relationship to similar themes in TA and RRI, although one could guess that this refers to managing natural systems (and hence, its relationship to Systems Analysis in the TA tradition). However, for the purposes of our current focus, we can observe the importance of science governance and science policymaking in the three fields directly connected to discussions on impact assessment.

3.3. Thematic Map

The final analysis level using bibliometrics is that of thematic map, considered a convincing method to reveal keywords as scientific research development indicators [23]. The thematic map is based on the categories of basic, motor, niche and emerging or declining themes. It shows correlated thematic focuses and their movement (or lack of) through time. (Themes in the upper-left quadrant (Niche Themes) have well developed internal ties but unimportant external ties and are therefore of marginal importance to the development of the field. Themes in the upper-right quadrant (Motor Themes) are both well developed and important for the structuring of the research field. Themes in the lower-right quadrant (Basic Themes) are not well developed but are important for the field. Themes in the lower-left quadrant (Emerging or Declining Themes) present both weak development and are marginal for the selected field.) In our case, we are particularly interested in “basic themes” that represent a constant focus and “motor themes” that represent a developing focus.

Figure 2 shows that most focus themes are also basic themes. There are themes that are either unique to each discipline or overlapping, but of particular interest to our investigation is that impact and performance are evident as basic themes in all three fields. For instance, “technology assessment”,” performance” and “decision-making” are basic themes in TA(as shown as Figure 2a); “acceptance”,” impact” and “big data” are basic themes of RRI(as shown as Figure 2b), while “vulnerability”,” climate-change adaption” and “indicators” are basic themes in SR (as shown as Figure 2c). On the other hand, “science”, “innovation”,” governance” and “policy” are present in all quadrants of the basic themes, meaning that these themes are a focus in all three fields. They also appear as basic themes in the (low) number of merged publications (as shown as Figure 2d). For instance, Putting the sharing economy into perspective (by Koen Frenken and Juliet Schor), one of the most highly cited articles in the bibliometrics dataset of TA, is also present in the dataset of SR.

In terms of motor themes, we witness an interesting overlap between TA and SR in relation to the theme of life cycle assessment. Sustainability is also a motor theme in RRI, while “corporate social responsibility”, which is a basic theme of SR, is also an important theme in RRI research [24]. Interestingly, TA appears as a motor theme in the merged publications, a fact that could be interpreted as TA being a key unifying discipline amongst the three, in the development of common research focus [25].

Overall, the thematic map analysis supports our argument that there is significant research relevance amongst TA, RRI and SR, together with a common focus on achieving impact in society and policy. It also supports our view that there is very little, if any, research on indicators of success in each of the three fields.

4. Indicators of Impact in TA

As we have seen in the previous section, TA is a central concept in the comparative/merged publication corpus, and it can offer a more advanced discussion on impact assessment than RRI or SR. Modern TA has developed a much wider scope of functions in the past four decades than in its initial conception, which is also reflected in the plurality and diversity of the methodologies it currently employs. Subsequently, TA’s impact dimensions are multivariant, and cover its functions not only as a service providing policy options assessment, but also as a process of instigating public debates, based on analyses of dominant values and the inclusion of a wide array of stakeholder input [26]. TA’s outreach has been discussed widely as part of its task and focus of activities [27]. The discussion of TA’s impact has been standardized via the categorization of the impact roles of TA that were developed in a major project publication [28]. The impact matrix below (adapted from [28]) presents two major dimensions of the TA process: the dimension of impact that the specific process is attempting to achieve, and the dimension of the issue that is under scrutiny in the project. Any combination of the two dimensions provides a set of goals that can be adopted in the project design with specific methodologies. Possible indicators of success are added in each impact category.

The dimension “Raising Knowledge” represents the classical type of TA and assumes a knowledge shortfall in STI developments as the main cause of uncertainty and social conflict. The methods employed to deal with the issue of focus are analysis of scientific knowledge on paths of technology development, risks, chances and unintended consequences (risk assessment), analysis of interests or perspectives of relevant actors (social mapping) and analysis of policymaking options (policy analysis). The main aim of this dimension is to raise the relevant actors’ awareness and provide them with new perspectives. The suggested indicators of success are the level of scientific knowledge amongst actors before and after the TA process (quantitative or qualitative comparison), direct feedback by stakeholders in the process (quantitative or qualitative measurement) and direct feedback by the policymaking community on the process (qualitative measurement).

The dimension “Forming Attitudes or Opinions” views TA as a process that goes beyond raising the level of knowledge of the particular STI development, to instigate social debates and transform stakeholder attitudes and opinions about it. The methods employed focus on the creation of thematically concrete policy and public debates with regard to new scientific perspectives (agenda setting), the resolution of policy or social conflicts via inclusive deliberation practices (mediation), the analysis of options for policymaking that are based on wider social values thus providing new perspectives in policymaking procedures (restructuring the policy debate). The suggested indicators of success are level of public awareness of the issue before and after the TA process (quantitative or qualitative comparison), resolution of conflict resulting from the TA process (qualitative measurement), references made to the TA process in policy debate (quantitative measurement).

The dimension “Initializing Actions” refers to the TA process that directly influences policymaking in the area of focus. The methods employed are analysis of STI policies and strategies, and development of R&D roadmaps (reframing of debate), analysis of public attitudes and the design and instigation of new public debates (new decision-making processes), analysis of legal status quo and identification of gaps, followed by development of legislative options (decisions taken). The suggested indicators of success are references to the TA process in the legislative debate (quantitative measurement), feedback by the policymaking community (quantitative or qualitative measurement), references to the TA process in the STI strategy documentation (quantitative measurement).

Overall, impact assessment in TA is based on its attempted resonance in social and policy debates that can be further detailed in terms of impact roles (and the corresponding methods), and can be measured with standard quantitative and qualitative indicators. In terms of the OECD classification, TA’s roles cover all five societal impact dimensions (culture, society, policy, organization, symbolic) while in terms of the dimensions of anticipation, reflexivity and inclusion, they epitomize their main methodologies and aims. As such, TA is a paradigmatic discipline on which to base a common impact assessment approach.

5. Indicators of Impact in RRI

RRI is a particular case when it comes to discussions on impact assessment as it lacks even a standard definition or standard methodologies. One can refer to RRI dimensions (reflexivity, inclusion and anticipation), which aim to give a conceptual and procedural level to RRI activities. However, since RRI is also promoted within the research programmatic approach of the European Commission in terms of the research themes of gender, open access, science education, ethics and public engagement, the development of impact indicators has been guided by these categories. So far, the main input in the effort to achieve consensus in the research community on RRI indicators is reflected in the work of the MoRRI project [29]. The approach of MoRRI is to categorise the indicators in the following: context indicators, that provide information on the environment and overall situation in a country or across countries; input indicators, that concern the activities carried out, measures taken, structures created and resources allocated to promote RRI; and output indicators, that address both the immediate and direct results of RRI activities, and the more far-reaching and long-term achievements and perceived benefits of the RRI work.

Based on this indicator framework, the project RRI-PRACTICE developed a comprehensive list of impact indicators that can be applied in the incorporation of RRI programs in research-funding and research-conducting organizations (The project RRI-Practice dealt with the incorporation of RRI in research-funding and research-conducting organizations in 12 countries: Australia, Brazil, Bulgaria, China, France, Germany, India, Italy, Netherlands, Norway, UK and the USA. An additional report on RRI indicators, based on a number of interviews and focus groups in a number of countries, forms the basis of this section. This material, along with the national comparative reports on RRI, is available at www.rri-practice.eu, accessed on 10 October 2021) [9]. To date, this is a unique case whereby potential RRI indicators have been discussed and accepted by the wider research community, providing a standard upon which RRI societal impact assessment can be based. Application of RRI indicators in the business sector has also been attempted at a limited scale with positive results [30]. Following the standard thematic categorization of RRI, these are:

Science education: This area includes student training, teacher training and continuous education (according to type of audience) and focuses on either science (staff professional research competence) or RRI itself (courses on science-society topics). The accepted indicators in this area deal with the:

• Number of dedicated educational projects (annual evolution);
• Number of projects with an educational component (annual evolution);
• Number of students, teachers and other personnel reached;
• Financial resources dedicated to educational actions;
• Human resources for educational activities (in person-months, annual evolution).

Ethics: This area includes both research integrity and deontology of researcher conduct, as well as ethical reflections within research projects. It can involve the creation of ethics committees; development of ethical codes; ethics policy statements; creation of ethics or compliance officers; ethics evaluation, survey and audit; reporting or advice hotlines; investigation and sanctioning panels; ethics training and ethics assessment. The accepted indicators are:

• Fraud, including plagiarism (number of cases per year, annual evolution);
• Conflict of interest or favoritism (number of cases per year, annual evolution);
• Research projects raising ethical issues (percent of all research projects);
• Research projects referred to external committees (percent of all research projects);
• Percentage of external members in ethics committees (annual evolution);
• Perception indicators to be measured through questionnaires:

  ○

  the extent to which individuals observe ethical/unethical behavior;

  ○

  whether ethical/unethical behavior is perceived as punished;

  ○

  whether ethical/unethical behavior is perceived as rewarded by management;

  ○

  whether staff knows where to turn for ethical advice;

  ○

  whether staff know about the existence of ethics committees/officers in their organization.

Open access: This area covers a direct function (open access publications), whereby impact assessment is very narrow in dealing with cases of either no open access policy or cases with well-established open access policy. Accepted indicators are:

• Date by which there is a mandatory OA policy in place;
• Percentage increase in total annual allocation of funds to OA publishing;
• Date by which there is an established IT infrastructure to support OA and open data archiving;
• Percentage of researchers that comply with OA requirements.

Public Engagement: This refers to a wide range of activities, including popular media publications, exhibitions and open days, but also collaborations with NGOs, citizen science initiatives, deliberative democracy exercises, etc. Impact assessment approaches deal exclusively with quantitative measurements since there is no standardization in this area to allow for external feedback. The commonly accepted indicators are:

• Number of researchers involved in activities;
• Number of participants in training initiatives on public engagement;
• Percentage of researchers with positive attitude towards public engagement.

Gender: This area also involves a wide range of activities that include training, awareness raising, gender quotas, career policies, family friendly policies, etc. with the purpose of promoting the standing of women in research. The discussion on impact assessment has concentrated on direct quantitative measurements denoting program success. The relevant indicators include:

• Percentage of female researchers by research/academic field;
• Percentage of female members of research/academic staff by grade;
• Average salary of women and men (annual evaluation);
• Percentage of men/women that are principal investigators on a project;
• Percentage of research projects including gender analysis or considering gender dimensions/gender and/or sex differences.

In summary, RRI has developed clear sets of indicators of success, despite the lack of conceptual clarity in the field. This is due to the standard thematic categorization of RRI within recognizable fields of activities that can (mostly) offer a direct impact assessment methodology. However, a possible conceptualization of RRI as a single field of activity (equivalent to TA or SR) would bring up new discussions in terms of societal impact assessment. The original concept of responsible innovation based on the dimensions of anticipation, reflexivity and inclusion, would constitute a more conceptually coherent approach to impact or resonance, but it has not been taken on board by the research community. For the purposes of our analysis, we will use the consensus reached on indicators in each RRI field to further our perspective for a common approach, but we will also attempt to offer an alternative view, based on a common conceptualization of RRI, TA and SR.

6. Indicators of Impact in SR

The debate on the impact assessment of SR is different from that of TA or RRI as it is tightly related to wider global challenges. The Sustainable Development Goals (SDGs) present the main identified international challenges regarding sustainability as defined by the United Nations (UN). (https://sustainabledevelopment.un.org/?menu=1300, accessed on 10 October 2021). These 17 goals range from environmental issues such as oceans and marine resources or ecosystems and biodiversity to societal aspects such as gender equality or peace and justice. The goals are related to impact since they provide a normative frame of what would constitute a change (resonance) regarding knowledge or actions towards sustainable development. They name certain aspects (e.g., gender equality or biodiversity) that are desirable and for which impact is needed. Further, they are useful, as they address all countries and are not specific to developed or developing nations. The agenda “Transforming our World: the 2030 Agenda for Sustainable Development” defines the SDGs and the targets needed for their accomplishment. (https://sustainabledevelopment.un.org/post2015/transformingourworld) These goals have been criticized on many fronts, for instance regarding the ambition of increasing GDP growth while at the same time aiming for sustainability objectives. The issue of containing too many goals and targets (169 in total) has also been raised. However, this process of defining SDGs and attempting to implement processes towards achieving them is relevant here, as it presents a global action (almost all actors involved in negotiations agreed on the SDGs), which has then taken shape in national and local contexts. All countries must adapt the SDGs to their national legislation and decision-making processes, design action plans, appoint budgets and coordinate their activities with other countries (For a detailed description on the goals as well as their history and implementation see: https://en.wikipedia.org/wiki/Sustainable_Development_Goals, accessed on 28 October 2021). In the context of the SDGs and their implementation, an instrument was developed and launched in 2015, especially focused on STI and mutual exchange.

Regarding indicators in SR that are relevant for a common methodological approach, one important project is LeNa (The project LeNa (Sustainability Management for non-university research organizations) was funded by the German Ministry for Education and Research (2013–2016)), which uniquely combined sustainability efforts in several main research organizations in Germany (Fraunhofer, Helmholtz, Leibnitz) and developed guidelines for various fields of action. These included sustainable staff management, sustainable infrastructure in planning and building, as well as conducting research according to societal responsibility. Even though the project mainly created options and management approaches and best practices for use within certain research organizations and did not explicitly address impact methodology as such, it can still serve as a crucial example of more concrete options and guidelines regarding desired outcomes that denote the achievement of specific resonance. Regarding connections to TA and RRI, the options developed in LeNa as orientation for research practices are especially relevant. These can be seen as ways forward in changing the state of knowledge (for instance of researchers and the inclusion of societal or sustainability aspects in their work), creating resonance or as suggested concrete actions that can be taken by organizations. The LeNa guidelines included questions to be addressed during the research process such as: Which disciplines are involved in the project and should others be included? Are there stakeholders outside of research who may be able to contribute relevant knowledge? What relevant elements should be integrated such as spatial and time aspects? Have the potential impacts for society or the environment been assessed during and after the project? Have possible uncertainties been identified?

Based on the findings of the project there have already been discussions on how to connect sustainability and responsibility in research to existing RRI approaches by developing relevant criteria. These include ethics, integrative approaches, interdisciplinarity, user orientation, reflection, transdisciplinarity, transparency and handling of complexity and uncertainties. These criteria aim to address questions of how research is conducted and how it resonates, as well as with and for whom it is done, throughout the entire research process. (For each criterion, fact sheets were developed (in German) which highlight the main aspects, relevance and good practice examples. https://nachhaltig-forschen.de/fact-sheets/forschung/, accessed on 23 October 2021). Helming et al. describe these different criteria in relation to sustainability and research in societal responsibility [31]. Ethics is concerned with questions of quality of life and action and ensures that ethical reflection is included in research processes by engaging with and discussing different norms and values. Integrative approaches refer to the systematic inclusion of relevant aspects and interdependencies in research, especially the interrelation between different sub-systems on a spatial, temporal, analytical and methodological level. Interdisciplinarity means the combination of diverse approaches and methods and focuses on research of complex problems. User orientation involves the various needs and expectations of users, especially focused on results of research. Reflection of effects is the ex-ante and ex-post view of direct and indirect, intended and unintended outcomes for society and the environment. Transdisciplinarity entails the integration of practical knowledge coming from actors outside of research. Transparency ensures the opening up of the research process, for instance regarding the normative basis, effects or funding. Handling of complexity and uncertainties is concerned with presenting risks and unsure knowledge in complex systems.

These developed criteria help frame sustainability and responsibility in research and can therefore be useful to connect with indicators and impact dimensions as described above. Regarding the OECD societal impact dimensions, these criteria correspond to culture, society, policy, organization, symbolic as well as environment. Further, we can associate the described impact roles of TA and RRI with these criteria, and through this enable the development of possible indicators. Our preliminary ideas for possible indicators for SR are based on the MoRRI and TA impact roles (

Table 2. The impact roles of TA.

Impact Dimension/Issue Dimension	I. RAISING Knowledge	II. FORMING Attitudes/ Opinions	III. INITIALISING Actions
Technological/Scientific Aspects	Scientific Assessment (a) Technical options assessed and made visible (b) Comprehensive overview on consequences given	Agenda Setting (f) Setting the agenda in the political debate (g) Stimulating public debate (h) Introducing visions or scenarios	Reframing of Debate (o) New action plan or initiative to further scrutinize the problem at stake (p) New orientation in policies established
Societal Aspects	Social Mapping (c) Structure of conflicts made transparent	Mediation (i) Self-reflecting among actors (j) Blockade running (k) Bridge building	New Decision-Making Processes (q) New ways of governance introduced (r) Initiative to intensify public debate taken
Policy Aspects	Policy Analysis (d) Policy objectives explored (e) Existing policies assessed	Re-structuring policy debate (l) Comprehensiveness in policies increased (m) Policies evaluated through debate (n) Democratic legitimization perceived	Decision Taken (s) Policy alternatives filtered (t) Innovations implemented (u) New legislation is passed
	Indicator: Knowledge level Stakeholder feedback Policymaking feedback	Indicator: Public awareness of issue Conflict resolution achieved Policymaking references	Indicator: Legislative debate references Policymaking feedback Innovation policy references

):

Ethics:

• From RRI: all indictors described in Ethics Key (e.g., how have ethical issues been considered early on?)
• From TA: Stakeholder feedback, public awareness (e.g., how many stakeholders were involved?)

Integrative approaches:

• From RRI: Engagement Key (e.g., what number of researchers were involved in activities?);
• From TA: Stakeholder feedback (e.g., when have stakeholders in research been included?).

Interdisciplinarity and Transdisciplinarity:

• From RRI: Open Access Key, Engagement Key (e.g., how openly was data shared to enable exchange between disciplines and interested communities? How many different researchers have been involved in the process?);
• From TA: Stakeholder feedback, public awareness, policymaking feedback (e.g., how many different actors are included in agenda setting, data collection, feedback?).

User orientation:

• From RRI: Engagement Key (e.g., how many researchers are involved in activities to provide feedback on expectations and needs?);
• From TA: stakeholder feedback, policymaking feedback (e.g., how many research questions were framed according to potential needs of different users?).

Reflection:

• From RRI: Ethics Key (e.g., how many ethical issues have been addressed before, during and long term in appropriate processes?);
• From TA: stakeholder and policy feedback, awareness, debate references (e.g., how many potential outcomes for society and environment have been reflected within the research process, as well as publicly and on policy levels?).

Transparency:

• From RRI: Ethics Key, Engagement Key (e.g., have conflicts of interest or favoritism been addressed?);
• From TA: public awareness, conflict resolution, policy and legislative debates references (e.g., through which processes have normative assumptions, possible effects or funding basis been communicated?).

Complexity and uncertainty:

• From RRI: Engagement Key (e.g., how many engagement processes have been conducted to better handle complexity and uncertainty?);
• From TA: stakeholder feedback, public awareness, policymaking feedback (e.g., how many processes have taken place to ensure feedback on complexity and uncertainty?).

These indicators are well-suited to create a common methodological approach on impact assessment, but it should also be stressed that they have not been discussed as such in SR. More research is needed in this field to provide a common understanding of impact/resonance, and this will be undertaken within the second LeNa project that is currently under development. Our suggestions here should be seen as input to this conceptual development in SR.

7. Discussion

As described above, there is not much conceptual work undertaken in the area of impact assessment of either TA, RRI or SR. There is even less empirical work on any type of impact in any of the three fields. That allows us for certain flexibility in approaching the subject but also creates pitfalls in finding common approaches. We have taken the step of combining different methods of measuring impact that focus on either societal, organizational or institutional aims. There are natural nuances amongst the different foci but for our purposes, the concept of impact is the overarching idea that underpins every approach. Qualitatively, the different impact foci are similar and therefore, their measurement is fully compatible. For instance, the societal impact of raising wellbeing is identical to the organizational or institutional one, requiring the same type of measurement but different methodologic approaches. As such, we are able to reach the aim of comparing and contrasting the three fields in terms of impact measurement and suggest a roadmap of action.

Overall, taking into consideration the different indicators and criteria described above, we find it useful to employ the RRI dimensions as a possible way forward towards the development of common indicators in TA, RRI and SR. The conceptual adaptation that such an approach would require from the three fields is minimal. For instance, Grunwald describes the enhancement of reflexivity of debates and decision-making regarding the shaping of technology developments as a main task of TA [32]. For SR on the other hand, reflexivity elements can be found in its transdisciplinary process and user orientation. Each RRI dimension is clearly adaptable to the impact discussions in the other fields of research:

Anticipation plays an important role in TA, also in the role dimensions of raising knowledge (scientific assessment, social mapping) and forming attitudes and opinions (agenda setting). In SR, anticipation is to be found in the context of research framework criteria as well as in the process of handling complexity and uncertainties.

Inclusion is a key element of TA, which aims to enhance debates, foster public dialogue and shape technology development. This can be seen with approaches such as participatory TA and in the role dimensions (raising knowledge, forming attitudes, initializing actions). For SR, user orientation and transdisciplinarity aim to ensure the incorporation of different values and expectations.

Reflexivity also corresponds deeply with TA, as it deals with challenges emerging in the "real world", which requires a continuous reflection for providing orientational knowledge aimed at problem-solving. Raising knowledge (policy analysis, scientific assessment, social mapping) is especially concerned with assessing and responding to relevant questions and issues. Dealing with complexity and uncertainties, user orientation, integrative approaches, inter- and transdisciplinarity and ethics corresponds with reflexivity in SR.

8. Conclusions: Towards a Common Assessment of TA, RRI and SR

The aim of the paper was not to analyze in detail the conceptual or methodological commonalities/differences in TA, RRI and SR, although it is clear that these have an effect on how a common impact assessment can develop. The further conceptual development of the three research approaches and their applications in STI analysis and policy options, will be crucial in developing a standardized impact assessment methodology. At present, any work in this field is by necessity limited in identifying potentialities and suggesting the building blocks of such undertaking. In itself, the wider area of societal impact assessment is far from being standardized, and despite the efforts of the academic community to provide a standard framework, we have seen that most researchers are not aware of how to achieve such impact or how to measure its success.

We have further seen that there are differences in the publication trajectory of the three research fields, and we have identified that impact, in terms of society at large and policy in particular, is an essential part of how they are discussed. When it comes to developing specific indicators of success, RRI and TA have somewhat more concrete approaches at hand than SR. On the other hand, SR has a much wider global remit that would necessitate the development of concrete impact assessment methodologies.

However, a common conceptualization of impact emerges with the application of anticipation, reflection and inclusion. As such, we believe that a common impact assessment methodology is possible when based on these dimensions. In terms of specific indicators of success, one can draw from the existing TA matrix and the MoRRI framework. These are a combination of quantitative and qualitative measurements, based on textual analysis, surveys and interviews. The list of indicators could include further possibilities in terms of policies adopted, raising of public awareness, providing stakeholder feedback or account for collaborative activities. Each can contain a set of measurements and feedback loops drawing from the existing pool of accepted indicators. This is perfectly manageable within our suggested approach based on the three dimensions, and can form the focus of the next step in developing a common impact assessment methodology for TA, RRI and SR alike.