Gender Equality Indicators for Research and Innovation from a Responsible Perspective: The Case of Spain

Abstract

This article offers a Spanish national perspective that contributes to European Responsible Research and Innovation (RRI) monitoring initiatives. National experts involved in gender and science and technology issues, such as policymakers, gender experts, research institutions, and equality associations, among others, have proposed indicators based on participatory decision-making techniques. The results include a complete set of 52 indicators and a reduced panel of 23 indicators—the highest-ranked ones—to monitor relevant aspects that should be measured in gender dimension from an RRI perspective: differential and asymmetric socialization and education, organizational culture, substantive representation, vertical segregation, work relations, visibility of women researchers, gender perspective in research contents, gender expertise enhancement, and resources. The results offer new indicators that differ from previous indicator panels at the European and Spanish levels in relation to those aspects that should be measured and the typology of indicators preferred. Differences suggest the need for a more nuanced debate on the purpose of indicators, and the need for national contributions to RRI and to the debate on gender perspective in EU policy. Finally, the article suggests some specific traits observed in Spain that might add to the debate on the content of an RRI gender perspective in an already developed gender policy.

1. Introduction

Responsible Research and Innovation (RRI) is an emerging framework in Science and Technology Studies (STS), which includes a gender dimension [1]. RRI is a priority policy launched by the European Union (EU) in 2010 aimed at introducing ethics and reflective insights in the field of Science and Technology (S&T), in the context of processes, organizations, results, and impacts. Efforts have been made to engage European society in its application [2]. The roots of RRI can be found in the growing connection between risks and the unexpected consequences of science, which have been discussed since the appearance of technical devices in the 19th century and, more recently, as the result of challenges such as climate change, overpopulation, increasing inequality, and ageing populations in Europe and other developed regions [3].

The EU RRI policy framework is still under construction; six intervention areas have been identified so far: governance, public engagement, ethics, science education, open access, and gender equality [4]. On this basis, an RRI Expert Group has been launched to propose indicators aimed at promoting and monitoring RRI [5]. Other initiatives, such as the MoRRI project (MoRRI’s main objective is to support directly Directorate General for Research and Innovation (DG-RTD) in relation with RRI), address the construction of robust indicators to measure RRI implementation in European S&T policies. However, these proposals are at the European level, so more national-level discussions are required given national specificities and RRI must be adapted from the European framework to national S&T systems.

The present study is a first attempt to fill this gap in RRI measurement in relation to European policy. We analyse the Spanish national S&T system from a multi-stakeholder perspective using participatory methods to establish a set of indicators. We use a multicriteria method based on the Analytical Hierarchy Process (AHP). This method has been tested and its usefulness in prioritizing indicators in participatory environments has been demonstrated [6].

In this paper, we focus on the gender dimension and consider the gender perspective in the EU and Spain. European gender policies first emerged in the 1980s, and were strengthened by the launch of the 1998 gender mainstreaming strategy that focused on evaluation and measurement. Therefore, before the introduction of the RRI approach, the gender dimension had been developed in its theoretical approaches and possible practices and aspects related to its measurement. Already existing gender indicator reports in STS, which are greatly consolidated, such as She Figures at the European level and the periodical Científicas en Cifras report, which adopts a similar approach at the Spanish level have raised questions about the place and possible contributions of an RRI perspective in the measurement of a gender dimension. Note that the gender dimension has received little attention in RRI research [7] and that national RRI policy in Spain is at an earlier stage of development as is true in other European countries [8].

Considering this early stage a participatory approach is required to explore the national Spanish RRI reality. Gender equality initiatives depend crucially on an open perspective toward conceptions that shape the priorities of the different actors participating in equality policies [9]. The RRI perspective encourages co-creation and public engagement research practices, and a participatory methodology allows discussion of coherent internal processes for RRI practice. Finally, a participatory approach can lead to new indicators if European-level indicators are not suited to the national reality.

We first describe the European policy framework, participatory issues related to gender equality, and some measurement considerations related to gender in Section 2. Second, we review some previous indicator sets and introduce the proposed method and study design in Section 3 and Section 4. We describe the results of the participation of different stakeholders and the panel of indicators derived in Section 5, and we present limitations of the study and the discussion in Section 6. Finally in Section 7 and Section 8 we present contributions to policy making, the conclusions, and further research questions.

2. Gender Dimension Review: Policies and Measurement

2.1. EU Gender Policy: Measurement, Governance and Participation

European gender equality policies evolved in three main phases. In the 1980s, a committee was set up to formulate gender policies and the Council of Europe introduced measures to integrate equality, understood as equality before the law, and equal opportunities. Later, the focus was on positive action and women’s empowerment in employment and institutional decision-making. In the 1990s, the visions were most closely linked to postmodern feminism, proposing mainstreaming, a transformative vision focused on the systematic integration of a gender perspective and the redesign of gender relations [10]. Mainstreaming has been presented as both a theoretical approach and a practice [11]. The EU proposed combining strategies to achieve gender equality including equality in terms of equal opportunities and positive action, with “gender mainstreaming actions” aimed at changing gender relations. Gender mainstreaming strategies included measures taken from previous approaches. However, note that isolated measures cannot be considered gender mainstreaming. One of the major differences in the EU political mainstreaming practice from previous approaches was its procedural nature. Although this focus on process has been highlighted, it has not been accompanied by a vision of mainstreaming as the transformation and redesign of gendered systems and structures [10,11].

The Expert Group set up in 1998 did not propose any concrete actions or priorities, but instead called for a different approach within a gender perspective to all policymaking initiatives in which evaluation and monitoring have privileged positions:

“Gender mainstreaming is the (re)organisation, improvement, development and evaluation of policy processes, so that a gender equality perspective is incorporated in all policies at all levels and at all stages, by the actors normally involved in policy-making.”

[12] (p. 15)

The systematic objective of evaluation has led to the measurement of equality advances and problems that differ among countries and sectorial policies, which suggest the need for critical reflection on the measurement of EU gender policy. First, it should be noted that the changes and positive outputs of gender policies are difficult to sustain if they are not constantly promoted and monitored [13]. However, the EU’s “obsession” with measurement has been questioned [14] because the connection to management logic could distort policy. Inequality can be considered “inefficiency management” [15] but equality is a principle that rules the EU and is a right that implies a deeper and essential debate about public priorities. Second, the EU gender equality vision is almost synonymous with women’s participation in the labor market and pays little attention to its implications in the private sphere or how women should be incorporated in masculinized environments [16,17,18]. Third, the market-oriented vision has been described as challenging scientific and innovation gender policies and RRI policies [19,20].

In addition, measuring equality is not straightforward and a quantitative approach using indicators cannot capture some of its subtleties or its pervasive and frequently hidden nature. Gender differences and biases are complex and go beyond the mere presence of women. Innovation processes, overall, have been signaled as gendered [21], as have institutions and relationships [22]. For instance, gender differences in professional networks have been identified in many studies as leading to biases in access to scientific information derived from collaborative relationships [23]. However, the presence of women is the main focus of gender monitoring in science and innovation as shown in the succeeding sections. Monitoring how widespread actions or equality measures are in science and innovation is recent and, regardless of the more or less transformative nature of the measures monitored, is an aspect that requires more in-depth attention.

These reflections lead to the consideration that although monitoring is necessary to observe equality evolution and useful for implementing better policies, the use of indicators has a great impact that should be carefully considered. Measurement often involves data collection, which is not always accompanied by reflection on why these data collection efforts are necessary or why institutions are keen to construct indicators. Equality as an objective, in the absence of discourse in organizations implementing equality practices, can hamper efforts to achieve greater gender equality [24]. Moreover, insights from work in S&T on the development of indicators [25] highlights the need to consider whether their construction could be used as a symbolic terrain in stakeholder debates over policy priorities, especially if the policy objectives are poorly defined or very general, such as “to promote university participation in society”. This suggests the need to reflect on the objective of indicators and whether they are designed to help the management of science institutions or are focused more on general policy monitoring. Policy monitoring, the more common reason for developing indicators, could be jeopardized if their development is a part of the policy process and not a method to observe its evolution.

Based on the above considerations, in the following sections, we discuss the state of equality and measurement initiatives in the areas of innovation and S&T research.

2.2. Equality Policies in S&T in the EU and Spain

Since the EC Fifth Framework Programme, various projects have been launched that include gender equality aspects. However, their impact has been questioned given the resistance to the implementation of the measures identified [24]. The scientific system relies on evidence, but that essential logic does not seem to facilitate the incorporation of a gender perspective, and perceptions continue to be dominated by strong beliefs and stereotypes [26]. However, some positive effects of the evaluation of research and innovation (R&I) policies has been observed in gender issues [27].

In 2012, the European gender strategy in R&I identified the following aspects: inclusion of gender requirements in EU and member states’ funding programs and creation of specific programs to promote institutional changes through measures such as gender training, best practice identification, adoption of equality plans, and incorporation of a gender perspective in international mobility of researchers. Also, the ERA Roadmap (2015–2020), establishing the European Research Area, defines some gender priorities, addressed mainly to the effective application of gender issues in national environments. Following the European mandate, we can identify three main aspects of the R&I policies: women’s integration—horizontally as well as hierarchically—in all organizations working on R&I, integration of a gender perspective in policies and funding initiatives for the promotion of a structural change to identify implicit and explicit barriers, and the integration of a gender perspective in research. These issues are identified in the design of the She Figures report [28].

In Spain, and considering the Europeanization of Spanish gender policies [29], we can identify similar patterns in R&I gender policies even if there is evidence that punctual equality actions are more common than structured and systematic approaches in this sector [30,31]. The basis established in the ERA have been incorporated in the Spanish roadmap. The Spanish ERA Roadmap 2016–2020 establishes monitoring and accountability as a primary objective in relation to gender equality, and the report Científicas en cifras [32], developed by the Women and Science Unit of the Minister of Economy, Industry, and Competitiveness, provides several sets of indicators.

Similarly, the RRI EU policy referred to in the introduction, has promoted two measurement initiatives that include a gender dimension: the RRI Expert Group indicators and the MoRRI project indicator set.

More details on these different indicator initiatives are provided below.

3. Gender Indicator Proposals

3.1. General Gender Indicators in R&I: She Figures and Científicas en Cifras

The most relevant document on the monitoring of gender issues in the R&I sector is the She Figures Report, which has been published every three years since 2003. The most recent version at the time of writing (2015), presents data related to more than 130 indicators and provides comparable data for EU member states and European aggregate data.

The main aspects being measured are: the pool of graduate talent, participation in S&T occupations, labor market participation as researchers, working conditions of researchers, career advancement, participation in decision-making, and R&I outputs.

The report provides an exhaustive mapping of the presence of men and women in S&T and data about their evolution. Some indicators are related to equality measures in R&I organizations, such as average adoption of equality plans by national organizations, or ratio of the implementation of such measures as mentoring and career promotion, recruitment equality, and work-family balance. The last chapter related to R&I outputs proposes indicators for gender differences in scientific publishing and patenting, and inclusion of a gender dimension in scientific articles.

In Spain, with a similar focus and number of indicators, the last report in the series Científicas en Cifras in 2015 [32] was aimed at harmonizing Spanish data with She Figures data and including additional process indicators. Previous editions focused on monitoring female presence. The most recent edition measures equality plan adoption and observes gender differences in success in science programs and calls for proposals. The report provides detailed data on decision-making positions, but does not include indicators for scientific outputs or content.

3.2. Gender Indicators within a Responsible Research and Innovation Perspective: EU RRI Expert Group and MoRRI Project

Regarding previous work on gender monitoring, the report of the EU Expert Group (EU EG) on RRI indicators states that:

“While the indicators used provide a good overview of the participation of women and men in different sectors and at different levels, they do not seem to provide insight into the cultural issues associated with gender inequality. Similarly, they do not go into depth with regard to factors influencing women’s promotion and progression in research careers, nor do they offer much insight into institutional arrangements and mechanisms for promoting gender balance. And since our main priority is the development of RRI policies in institutions and programs, we choose to focus our indicators differently”.

[5] (p. 26)

This leads to the conclusion that the RRI gender focus should be on institutional changes and on gender dimension in research content. In this context, the report refers to the qualitative debate on what the gender dimension is and how it should be included,

“Such procedures of judgement are in themselves opportunities for reflection and responsibility, and we do not recommend subjective attempts at standardisation to try to eliminate them”

[5] (p. 27)

The document describes other aspects of this vision. First, the EU EG considers it important to contextualize its proposal in the different EU countries. It refers to the need to evaluate the intrinsic risks associated with the use of indicators and highlights that some aspects require a qualitative approach that entails both reflexivity and responsibility. The indicators panel is presented as a “toolbox” not a “tick box” [5] (p. 16), that can be used in different environments. The report proposes a reduced number of indicators and a list of 16 gender indicators classified either as performance indicators or perception indicators. Performance indicators are broken down into process and outcome. Process indicators refer to institutions, and outcome indicators mainly refer to monitoring the increased presence of women in different positions, with the exception of one indicator for participation in research projects. The aspects that the EG considers fundamental to measure are not discussed explicitly in the document, but can be deduced from the indicators used to measure funding requirements and equality plans, and institutional equality measures development, women’s presence in key positions (expert groups, committees, mobility, as Principal Investigator-PI), gender issues in research content, and young people’s and parents’ perceptions of equality and career choices. It is possible to consult the table with the gender indicators proposed by the EU Expert Group in pp. 28 [5].

Regarding other measuring RRI initiatives, the MoRRI project is an ambitious initiative aimed at proposing indicators to monitor RRI, supporting the Directorate General for R&I in 2014 to 2018. The project uses EU RRI area descriptions to research possible indicators. It focuses on robustness and includes detailed information on probable periodicity, source, type, and level of implementation. The project also addresses monitoring RRI benefits and includes some possible traits related to measuring gender equality benefits. MoRRI proposal of indicators focus on the national level but remarking the importance of organizations for implementing RRI. The gender indicators of the project are related with share of female researchers among different scientific areas and sectors, gender wage gap, shares of women heads of research organizations, women´s presence in recruitment committees and, related to policies, share of equality plans in research organizations and shares of research-funding organizations promoting gender content in research. It is possible to consult the tables of indicators proposed in the MoRRI project on the website http://www.technopolis-group.com/morri/.

4. Study Design and Method

Gender equality is a broad policy objective that means different things to different stakeholders in the policy process. Gender experts define gender policies as clusters of visions of gender problematics [9]. In this context, we consider that, “Since the interpretation of a problem will determine what changes and what stays the same, social research is directed at identifying and comparing competing interpretations of problems” [33]. This highlights the need to be aware of the basis of these visions and to avoid working according to predefined assumptions about gender equality/inequality.

In light of the diverse notions involved in the measurement of gender issues, participation appears important. The difficulties related to gender policies include lack of resources, resistance, and lack of clarity [34] on the integration of different actors and dependence on gender experts’ knowledge in the development of a gender perspective in policy. Participatory dynamics have been suggested to overcome these risks and achieve the implementation of successful policies [35]. Participation is a first step toward generating deliberative democratic models. These models can foster more transformative versions of gender policies, and in particular, transformative gender mainstreaming [36]. Use of participatory techniques is appropriate, also, since this study is the first approach in Spain on gender and RRI. Debates at the European level on what a gender approach means from an RRI perspective are scarce. Some suggest measuring the results of innovation, others monitor equality policy measures (such as equality plans), or the presence of women in monitoring. In this environment, our study is exploratory and the objective was to scan those traits that define Spanish monitoring from a RRI perspective. We address the following research questions:

-

What are the gender-relevant aspects for Spanish experts from an RRI perspective?

-

Which indicators, in the opinion of Spanish experts, measure the selected relevant aspects?

-

What are the differential traits that define the Spanish RRI approach compared to previous gender measurement initiatives?

The exploratory nature of our study suggests the need for a qualitative approach and the development of a participatory initiative based on the opinions of relevant actors. We employ an Analytic Hierarchy Process (AHP) participatory approach to establish and prioritize a list of indicators. The AHP proposed by Saaty measures intangible criteria [37]. AHP theory is based on the inherent complexity of a multiple criteria decision making problem that can be solved through the construction of hierarchic structures consisting of a goal, criteria, and alternatives. At each hierarchical level, comparisons and judgments are made using numerical values taken from the AHP absolute fundamental 1–9 scale.

The synthesis of AHP combines multidimensional measurement scales into a single one-dimensional scale of priorities. The method is one of the most extended multicriteria decision making techniques that have been used in the areas of Corporate Social Responsibility (CSR) [38,39] and RRI [6]. It adapts very well to the hierarchy of indicators proposed by the EU and has the advantage of being easy to explain to experts assessing the indicators.

In order to carry out a participatory session based on AHP, a group of experts has to be selected with care because the technique is based in expert knowledge and qualitative judgements. In this study, we relied upon 14 experts who took part in one participatory session where they were asked to identify and weight the indicators. To avoid the need for more face-to-face meetings, agreement on the results was conducted via email. To tackle these research questions the methodology in Figure 1 is proposed. A detailed description of the methodology will be presented along the case study.

4.1. Lists of Gender Indicators

The four-abovementioned lists of indicators, European Union Expert Group on RRI Indicators 2015, MoRRI, She Figures, and Científicas en Cifras, were the starting point for the debate.

4.2. Identification of Experts

When working in participatory face-to-phase environments, the number of participants has to be limited. The number of participants in the prioritization process is usually unclear. It is suggested that 4–15 people can well represent society in general [40]. To ensure good representation, they should be people with strong interests in the outcomes. Also, when using the AHP technique, the experts have to have a deep knowledge on the matter being analysed. AHP participatory sessions are very time consuming. So they cannot be selected with statistical procedures.

Our list of experts was carefully selected. Our criteria for selecting them were:

-

Diverse expertise: in gender, indicators, and S&T policy.

-

Diverse actors: civil equality associations in S&T, S&T and equality policymakers, gender consultants in universities and local administration, institutes of applied research, trade unions and firm associations.

-

Diverse territories and levels: representatives of the local, regional, and national levels, and regions with low and high development of gender aspects in S&T policies [30].

-

Diverse scientific areas: humanities, science, technology, engineering and mathematics, social sciences, natural science, and medical sciences.

We considered representatives of all the stakeholder groups proposed by the EU Expert Group on Policy indicators for RRI (Expert Group 2015).

Table 1. Recruited experts.

Group	Organization	No. of Experts Recruited
Civil society (Equality Associations)	Association of women researchers and technologists (AMIT)	2
Academia Experts (Gender and Indicators)	Universidad del País Vasco	1
	Universidad de Cantabria	1
	Universidad de Valencia	2
Research Centers Management	Health Research Institute (INCLIVA)	1
Government (Science and Technology)	National level. Fundación Española de Ciencia y Tecnología (FECYT)	2
	Regional level. Science and Research Department, Generalitat Valenciana	1
Equality Units	Equality Unit of the Universitat Politècnica de València.	1
	Department of Equality and Inclusive Policies Ajuntament de València	1
Unions	UGT	1
	CCOO	1

Source: Elaborated by the authors.

lists the experts consulted.

4.3. Meeting with Experts to Consolidate the List of Indicators

The experts met in October 2017 in a participatory workshop. They proposed a list of nine relevant aspects and a set of indicators to monitor each of them (52 indicators in total). The questions addressed were: which gender issues might be more relevant in a specific Spanish RRI context, and how could these aspects be measured. They were encouraged to use both existing indicators and/or create new ones. No further information was provided and the aim was not to produce a specific typology of indicators (e.g., aimed at organizations or policies, results, or perceptions). Our interest was in observing the preferences in the Spanish case, given the differences already observed at the European level.

Once the nine relevant aspects and the panel of indicators were arranged, the experts were asked to weight the indicators, employing the AHP technique. More information on the AHP method can be found in Saaty [37].

4.4. Individual Weighting of Indicators

A questionnaire was devised based on the list of indicators; it was based on AHP principles and was distributed to each expert. The experts were asked to compare pairs of indicators and rank them, based on Saaty’s 1–9 fundamental ratio scale. Checks were carried out if the inconsistency ratios were higher than 0.1.

An example from the questionnaire is shown below:

Which of the two indicators in comparison do you consider more relevant in order to measure Aspect 4. Vertical Segregation?
I. 4.1% of women/men who are PI	9	7	5	3	1	3	5X	7	9	I.4.3 Glass ceiling index

In this case, the expert considered that, in order to measure Vertical Segregation in Science and Innovation, it was more relevant to consider Glass ceiling Index than % of women/men who are project PIs (Principal Investigator of a project).

4.5. Feedback from Experts and Data Analysis

After processing the responses using Superdecisions software, we collated the results for the individual experts and for the whole group.

The experts received a copy of the results and were encouraged to provide feedback. Six respondents offered information on differences between their individual results and those of the whole group; no disagreement was detected in relation to the results and conclusions presented in the following sections.

5. Results

Table 2. Global list of indicators.

Relevant Aspects	Indicators	Panel with Similar Indicator	Group Average	Group SD
1. Differential and asymmetric socialization and education	I1.1 % of pre-university centers with programs to promote equality in the career selection	EG	0.098	0.103
	I1.2 Perception of gender roles in science among young people and their parents	EG	0.116	0.153
	I1.3 % in the different knowledge areas in the university access stage (GDD)	CF	0.195	0.126
	I1.4 Adequacy between knowledge areas (Social Sciences, STEM, etc.) of the selected university degree and the previous knowledge area selected in the preuniversity stages (GDD)		0.108	0.093
	I1.5 % of finished PhD over the university graduates (GDD)	SH, CF	0.148	0.077
	I1.6 Time average to reach academic career milestones (GDD)	SH, CF	0.146	0.098
	I1.7 % of researchers with care responsibilities segregated by sector and age (GDD)		0.189	0.124
2.Organizational culture (including time-use issues, transparency, democratic deficit)	I2.1 % of Research Institutions ( RI, including universities) with equality plans (EP)	SH, CF, EG, MO	0.067	0.025
	I2.2 % of RI that provide documents about EP implementation	EG	0.157	0.148
	I2.3 % of RI that evaluate EP impact		0.387	0.179
	I2.4 % of RI with Transparency Plans		0.103	0.099
	I2.5 % of RI with systems of transparent distribution of workload		0.287	0.151
3. Substantive representation, informal and formal networks, intersectionality and science	I3.1 Distribution of men and women in the corporate bodies of RI by age and knowledge area	SH, CF,	0.167	0.141
	I3.2 % of RI with regulations that include gender issues		0.071	0.074
	I3.3 Adequacy grade of the researchers in the project to the project subject (GDD)		0.149	0.105
	I3.4 % of men and women in evaluation panels	SH, CF, MO	0.268	0.180
	I3.5 % of full-time working schedules during the project cycle (GDD)	SH, EG	0.136	0.093
	I3.6 Degree of proximity of researchers to the Principal Investigator (PI) in the results of the project (network analysis)		0.208	0.084
4. Vertical segregation	I.4.1 % of men and women that are PI	EG	0.167	0.163
	I.4.2 % of researchers that participates in mobility programs (GDD)	SH, CF, EG	0.075	0.059
	I.4.3 Glass Ceiling Index	SH, MO	0.147	0.154
	I.4.4 Average age of the people occupying the different academic positions (GDD)	SH, CF	0.095	0.067
	I.4.5 Average age in the different government positions of RI (GDD)	SH, CF	0.128	0.094
	I.4.6 % of men and women that are leaders of RI	SH, CF, MO	0.230	0.099
	I.4.7 % of men and women in Experts Groups	EG	0.158	0.081
5. Work relations (time-uses, working conditions, labor distribution…)	I.5.1 Gender pay gap disaggregated by academic and organic status		0.139	0.108
	I.5.2 Gender pay gap (global)	SH, MO	0.103	0.132
	I.5.3 % of technical workers by the research workers by sector (GDD)		0.123	0.082
	I.5.4 Types of employment contracts and stability (GDD)	SH	0.169	0.116
	I.5.5 % parental leaves (GDD)		0.126	0.062
	I.5.6 % of care leaves (GDD)		0.117	0.057
	I.5.7 % of paid and unpaid leaves (GDD)		0.117	0.080
	I.5.8 % of voluntary reduction of working time (GDD)		0.140	0.111
6. Visibility of women researchers as references	I.6.1 Presence of women researchers as literature referents		0.111	0.077
	I.6.2 Women’s presence in conferences and workshops		0.172	0.194
	I.6.3 Women’s presence in calls for proposals and awards		0.200	0.100
	I.6.4 Public acknowledge to women scientists (streets names, building names, other)		0.232	0.143
	I.6.5 % of scientific exhibitions with women scientist or its work in its contents		0.259	0.126
7. Research contents	I.7.1 % of funding programs that refers explicitly to gender issues	SH, CF	0.151	0.156
	I.7.2 % of research projects that include gender diversity in its samples (with human or animals)		0.226	0.167
	I.7.3 % of research references in the project that include gender perspective		0.112	0.086
	I.7.4 % of research projects with gender impact (publications, conferences, contracts…)	SH, GE, MO	0.264	0.119
	I.7.5 % of funded programs that included gender issues in its contents	CF	0.248	0.121
8. Gender expertise enhancement	I.8.1 Gender training hours of research team participants		0.164	0.157
	I.8.2 Number of publications of the research team with gender perspective		0.289	0.132
	I.8.3 % of projects that include gender experts contracts		0.392	0.150
	I.8.4 % of projects that include gender training of the research team		0.155	0.099
9. Resources (economic, human resources…)	I.9.1 % of funding programs that include explicitly gender requirements		0.240	0.217
	I.9.2 % of funding programs that includes explicitly gender as an assessment criteria		0.298	0.156
	I.9.3% of RI with specific budget aspects to gender issues		0.153	0.147
	I.9.4% of resources (human and material) that research firms allocate to researchers (GDD)		0.157	0.117
	I.9.5 % of call of proposals that consider aspects related to career interruptions		0.153	0.102

presents the preliminary results of the participatory session. The debate was conducted in the Spanish language and then their results were translated. In order to avoid presenting redundant information, the indicators are accompanied by expression Gender-Disaggregated Data (GDD) when not explicitly mentioned in their formulation. Table 2 shows whether one indicator is used to measure the same item in the previously cited panels such as: She Figures (SH), Científicas en Cifras (CF), MoRRi panel (MO), and RRI Expert Group panel (EG). Table 2 also presents indicator weights assessed by the experts in collaboration. It shows the group average and group standard deviation.

When analysing, we can highlight differences from other indicator panels in relation to the aspects measured. In our case, the emphasis was on pre-university stage asymmetries and differences (Relevant Aspect RA.1), resources as a separate area (RA.9), and organizational culture (RA.2) or work conditions (RA.5) Also, time-use and work-balance issues were strongly represented in the indicators. Transparency appears to be a new item that is related to responsibility in the context of gender measurement.

Indicators for women’s presence do not identify the distribution of men and women in the scientific system. The focus is on substantive representation (RA.3) and vertical representation (RA.4). Another related issue is the visibility of women’s participation in scientific activities (RA.6), which is a new measurement aspect. Existing measures recognize the existence of women in the system, but are hidden in the public and scientific acknowledgment processes. Gender expertise enhancement (RA.8) is another new item. According to the specific and differential focus of the relevant aspects proposed in the participatory session, a number of new indicators are proposed to measure relevant RRI aspects in Spain.

Some of the nine relevant aspects and 52 indicators identified by the experts are quite similar, for example, I2.1, I2.2, and I2.3 related to equality plans, and I2.4 and I2.5 in relation to transparency, but with different implications. I2.1 measures the existence of an equality plan, I2.2 measures progress in equality plans, and I2.3 focuses on the impact of equality plans. The panel includes different types of indicators that can be classified as in the RRI Expert Group panel into process, outcome, or perception indicators. A new category, impact indicators, could be added. Impact is observed in I7.4 (gender impact in projects) and I7.5, which require data on which funding programs supported proposals with gender content. What impact means (gender impact or science impact) requires a separate discussion. However, the idea of impact suggests the evaluative use of indicators related to equality measures, that is, not just a focus on processes or female presence, but a focus on the final transformations produced by the equality measures.

In relation to level, most of the indicators focus on the organizational and institutional levels, which also applies to the EU Expert Group indicators and MoRRI, although most can be aggregated to the national or regional levels. This applies indicators of the average of equality plans and those related working conditions and holiday entitlements. Other indicators are related to the program and project level, for example, I3.3, I3.4, I3.5, I3.6, I4.1, I4.2, I9.1, I9.2, and all the indicators in RA.7 (research content) and RA.8 (gender expertise enhancement). Therefore, the dominance of institutional/organizational level indicators is similar to the EU Expert Group indicators and MoRRI, but the focus on project-specific indicators is different.

Group average measures the percentage importance of each indicator within an aspect, assigned by the whole group. The group standard deviation measures differences among the importance assigned by individual experts. This provides an idea of the level of dispersion of individual results within the group for each indicator. Based on the standard deviation results, we conclude that the indicators on which agreement was lowest were I9.1, I6.2, I3.4, and I2.3, and the ones that achieved the highest consensus were I2.1, I5.6, and I4.2.

The number of indicators, the similarities among certain indicators, as well as the interest in detecting possible different envisions that not arose explicitly in the participatory session, suggest the indicators should be prioritized.

Reduced List of Indicators

The indicators presented in the previous section were prioritized according to their importance based on the views of the expert group. The complete list of 52 indicators could be used as a starting point to identify what indicators should be. However, for various reasons, time constraints, lack of sources of information, it may not always be possible for policy makers to collect data for all the indicators. Therefore, we proposed reducing the set of indicators in order to select the most relevant ones. In this case, policy makers should focus on gathering data to measure this shorter list without losing information relevant to the monitoring process.

Our aim was to identify those that best fit specific policies. We used the “Best in class” criterion, that is, selecting the most relevant indicator in each area and category [6]. In this case, the reduced list of indicators would include nine indicators from the original list of 52.

Alternatively, we could choose a cut-off percentage and for each aspect, select only those indicators that exceed to the cut-off percentage, starting from first positions. We chose a 50% cut off, which resulted in the indicators presented in the second column of

Table 3. Prioritized and reduced panel of indicators.

	Group Prioritization
	Best in Class	50% Criteria	Indicators
1. Differential and asymmetric socialization and education	I1.3	I1.3	% in the different knowledge areas in the university access stage (GDD)
		I1.7	% of researchers with care responsibilities segregated by sector and age (GDD)
		I1.5	% of finished PhD over the university graduates (GDD)
2. Organizational culture (including time-use issues, transparency, democratic deficit)	I2.3	I2.3	% of RI that evaluate equality plans (EP) impact
		I2.5	% of RI with systems of transparent distribution of workload
3. Substantive representation, informal and formal networks, intersectionality, and science	I3.4	I3.4	% of men and women in evaluation panels
		I3.6	Degree of proximity of researchers to the Principal Investigator (PI) in the results of the project (network analysis)
		I3.1	Distribution of men and women in the corporate bodies of RI by age and knowledge area
4. Vertical segregation	I4.6	I4.6	% of men and women that are leaders of RI
		I4.1	% of men and women that are PI
		I4.3	Glass Ceiling Index
5. Work relations (time-use, working conditions, labor distribution)	I5.4	I5.4	Types of employment contracts and stability (GDD)
		I5.8	% of voluntary reduction of working time (GDD)
		I5.1	Gender pay gap disaggregated by academic and organic status
		I5.5	% parental leaves (GDD)
6. Visibility of women researchers as references	I6.5	I6.5	% of scientific exhibitions with women scientist or its work in its contents
		I6.4	Public acknowledge to women scientists (streets names, building names, other)
7. Research contents	I7.4	I7.4	% of research projects with gender impact (publications, conferences, contracts…)
		I7.5	% of funded programs that had included gender issues in its contents
8. Gender expertise enhancement	I8.3	I8.3	% of projects that include gender experts contracts
		I8.2	Number of publications of the research team with gender perspective
9. Resources (economic, human resources, etc.)	I9.2	I9.2	% of funding programs that includes explicitly gender as an assessment criteria
		I9.1	% of funding programs that include explicitly gender requirements

.

Table 3 presents the indicators prioritized by expert preferences. These indicators focus, on the one hand, on the institution and project levels (such as I4.6, I7.4, or I8.3) and on the other hand, on the impact and outcome labels (such as I2.3, or I7.5). The latter is preferred to other similar indicators measuring process or perceptions. The items for work-life balance, time use, and transparency were retained on the list, which can be interpreted as differentiating this Spanish indicators panel. An important share of the selected indicators (12 out of 23) were new, the remainder were taken from previous panels, mainly She Figures and Científicas en Cifras reports. Only two indicators from the RRI European EEG were included (I4.1 and I7.4). Although our panel was based on the RRI vision, its focus differed substantially from She Figures and Científicas en Cifras.

The Best in Class prioritization includes the nine most important indicators in the overall group. However, the 50% cut-off criterion was considered to provide a better approximation of the prioritized results. First, the intrinsic complexity of gender aspects requires more than one approximation to measure a specific relevant aspect. Second, a reduced set of indicators is easier to manage and applying the 50% cut-off provided a panel of 23 indicators. This was almost half of the full set of indicators and included several of the indicators in other panels that proxy for RRI. Third, the 50% cut-off list is not a ranking, but is the result of consensus among all the indicators that achieve a minimum weight. This approach better reflects the diversity in the visions in the participatory sessions. These differences are presented in Figure 2.

The patterns of some of the aspects analysed were similar, for example, A2 and A8. However, A4, A3, and A6 showed great differences in actors’ preferences, related to substantive female representation, vertical presence, and visibility of women, which are important in the Spanish panel. Regarding the preferences of individual actors that differ from the average group, we observed that government actors preferred indicators related to funding and gender experts contracts (I9.1, I8.3). Research centers focused on project-related and time use indicators related to career interruptions (I9.5), full-time schedules (I3.5), and research age and scientific career milestones (I1.6). Equality units weight those indicators related to perceptions (I1.2), glass ceiling (I4.3), and increased visibility of women in street names and scientific exhibitions (I6.5, I6.4). Academic experts focus on funding requirements I7.1 and industry indicators (I9.4). Finally, the Equality Association weighted indicators emphasizing their preference for transparency and work time issues, especially at the project level.

6. Discussion and Limitations of the Study

The feedback provided by the experts on the group prioritization results confirm their agreement with the main trends observed, especially those that differed from other indicator panels. The differences in prioritization among Spanish actors was confirmed by the feedback. This suggests that policymakers in Spain should engage in a nuanced debate informed by these data on where differences lie.

One limitation of this study is its qualitative focus. The study was complex; participatory techniques allow direct debate and the exchange of ideas. Moreover, the mapping of gender and Spanish S&T experts included a small number of participants due to the equally small number of studies developed so far. From the initial 25 experts identified, 14 participated in the project meeting, which can be considered a good ratio bearing in mind their various agendas and workloads.

Nevertheless, there are some biases related, perhaps, to absences from the final participatory meeting. Firm associations were not represented and, therefore, there are few firm indicator proposals. Those that were proposed were prioritized by academic experts, who were likely conscious of the higher level of gender bias in the private sector compared to public institutions.

A second limitation is that LGTBIQ (lesbian, gay, transgender, bisexual, intersexual, queer) options did not appear as a gender aspect or indicator, although aspects related to these options are increasingly considered to be pure gender issues. This was an unexpected limitation of the results, not necessarily related to the expert recruitment, but one that should be noted.

Finally, although abandoning a research career is a frequent problem in the sector, this issue was not considered when setting the indicators. This might be because the experts have built their careers over several decades. It would be interesting to include in further debates early-career researchers and foreign researchers with different and more precarious career paths. Their expertise would rely on their sociological diversity and probably different visions and values, and less on their accumulated knowledge and experience. That focus could rely on the inclusive dimension that RRI perspectives suggest [41], which intend to open the debate to collectives that are not used to being consulted, as well as in intersectional approaches that are highly cited in gender studies [42,43], that remark the importance of how gender interact with other traits as age or origin.

These limitations do not undermine the results presented due to the exploratory nature of the study and the novelty of possible national adaptations of the RRI gender approach. They suggest directions for future research and policy-making initiatives based on the results presented here.

7. Contributions to Policy Making

The study contributes by providing 30 new indicators that can be used to measure gender issues in science and innovation. Most are associated to aspects related to measuring resources and the substantive presence of women, gender expertise enhancement, and visibility of women researchers, and highlight preferences related to work-life balance, time-use, and transparency in research and organizational culture and work relations. These aspects have been referred to in the gender literature. In the case of gender expertise enhancement, no other indicator sets include this aspect, and work-life balance, time-use, and transparency are missing from most previous indicator sets, apart from She Figures. The EU Expert Group on RRI panel also did not address this dimension. The focus on work-life balance and time-use contributed to filing the gap observed in the equality policies, which exists in the private sphere and women’s difficulties in masculinized work environments, as was previously introduced. The emergence of this issue might be due to the scarce public support for careers that increase the visibility of the problem in Spain.

Second, the study shows indicators being proposed in more than one indicators setting (Table 2, global list of indicators). The following were present in three of the four indicators settings analysed and are prioritized in the Spanish case, although the focus of the indicator settings differed: 3.4 (SH, CF, MO), which is related to evaluation panels; 4.6 (SH, CF, MO), which is related to research institutions leaders, and 7.4 (SH, GE, MO), which is related to projects with gender impact. Their importance should be highlighted in the gender measurement processes.

Third, the Spanish panel offers a different approximation of an indicators setting, based on relevant aspects (work relations, vertical segregation, research content, and gender expertise enhancement), and which differ in form from other RRI settings (MoRRI or EU RRI EG), centered on types of indicators (perception, benefit, outcome, etc.). Approximation based on relevant aspects highlights what should be measured. As already mentioned, sometimes the construction of indicators is symbolic in stakeholder debates on policy priorities and, especially, if the policy objectives are undefined or are general, such as “gender equality”, which carries a different meaning for different stakeholders. An approximation based on relevant aspects allows the problematic areas and key issues—and their possible transformative nature—to be at the center of the task.

Fourth, the Spanish results focused on monitoring equality policy developments in science, but overlook sex distribution, which is monitored mainly by other panels. This could steer the contribution of gender monitoring from an RRI perspective toward complementarity with other monitoring initiatives, although it should be noted that further research would be necessary to establish whether the results are similar in other European countries.

8. Conclusions and Further Research Questions

This study presents the results of a participatory methodology that was used to determine a set of RRI gender indicators generated by experts in a Spanish S&T policy-making context. The final list obtained could be used to develop further policy-making initiatives in Spain from an RRI perspective. It differs from existing sets of indicators at the EU level and for Spain.

The type of indicators chosen by the Spanish experts suggest a different use of the indicators from an RRI perspective, focusing on the measurement of policy making initiatives at the meso-level of application, meaning organizations and projects. The Spanish proposal differs in some key aspects that should be measured. The relevant aspects are: differential and asymmetric socialization and education, organizational culture, substantive representation, vertical segregation, work relations, visibility of women researchers in references and research content, and gender expertise enhancement and resources. Some of these aspects are new in comparison with previous indicators settings: substantive women’s presence and gender expertise enhancement. In addition, considerable importance is placed on work-life balance, time-use, and transparency issues. The study provides new indicators to measure these issues. These differences, and the fact that Spanish experts selected only two indicators from the EU Expert Group list, suggest the need for discussion about the gender vision supporting the EU RRI policy, and could trigger debate if aspects such as work-life balance, time use, and transparency can be considered new RRI contributions in gender monitoring rather than only country particularities. Further research is needed to confirm this hypothesis.

The need for a further debate on what an RRI vision adds to decades of work on gender in S&T is also motivated by other considerations. The focus on the institutional level does not explicitly address the cultural and institutional transformations that constitute an RRI perspective. However, this trend in observing institutional transformations is highlighted also by the introduction of relevant aspects specifically entitled organizational cultures or work relations. This difference in the Spanish setting might complement the country-level indicators and monitoring of women’s presence developed in She Figures and Científicas en Cifras.

The emphasis in the final Spanish set, on impact-related indicators of the effectiveness of equality measures, should be considered in the debate on the limits of monitoring using mainly quantitative approaches. The EU Expert Group also suggested that the collection of information on the gender dimension in R&I likely requires non-quantitative analysis techniques and other specific approaches, such as evaluation rather than monitoring. This does not contradict the debate about a new balance between monitoring and evaluation. Again, further research in other European countries would shed light on these issues.

New governance processes need to be explored. The EU launched an indicators report in the early stages of the RRI debate, before suggesting in-depth discussion of possible RRI areas, dimensions, and key traits. This study shows that only two indicators from the EU Expert Group were considered important by the Spanish experts, demonstrating that the aspects that need to be measured are quite diverse. This suggests the need for a bottom-up policy process. The early stages of RRI policy in European countries suggests caution in relation to the final implementation of indicators. As previously mentioned, measurement is not always accompanied by discourse on the necessity for or relevance of what is to be measured and can lead to resistance. It is recommended that RRI discourse should adopt a participatory approach involving multiple stakeholders in the policy-making process. We propose an approximation focused on relevant aspects beyond the types of indicators that might be useful, because differences in meanings and interpretation of equality problems are central. The differences identified in actors’ perceptions in the Spanish case suggest the need for nuanced debate that includes the objectives of the indicators: contributing to the management of gender issues in organizations or policy monitoring or both.

Finally, it would be interesting to observe complementary outputs from other RRI measurement practices, such as transparency, public engagement, and open science, among others, that might enhance gender equality.

Last but not least, concerning the development of the AHP participatory methodology used as a tool for prioritization, we want to state that there has been considerable agreement among the experts, and that the first meeting for defining and agreeing the indicators was fruitful. All the stakeholders felt that the AHP procedure allowed them to deal with prioritization in an organized and systematic way. They all agreed that the procedure enhanced participation and transparency and it was a necessary source of information and support for defining indicators based on consensus. Concerning the utility and applicability of this tool and findings in similar cases, the procedure is easily adaptable to other RRI areas, as is the case for Public Engagement or Open Science. For that, facilitators must bear in mind three key rules: first, to arrange a panel of stakeholders fully representative and motivated; second, to provide an appropriate means of communication among them; and third, to take AHP as a whole procedure and to devote the necessary time. As such, the AHP procedure becomes not only interesting in terms of reaching a final prioritization of indicators, but mainly in terms of enabling debates and reflections.