Research Ethics, Open Science and CRIS

Abstract

The purpose of this paper is to analyze how current research information systems (CRIS) take into account ethical issues, especially in the environment of open science. The analysis is based on a review of the literature on research information management, CRIS, open science and research ethics. The paper provides a framework for the assessment of CRIS on two levels: are CRIS (= their data model, format, functionalities, etc.) compliant with ethical requirements from the research community, funding bodies, government, etc., i.e., can they appropriately process data on research ethics (protocols, misconduct, etc.), and which are the ethical issues of the development, implementation and usage of CRIS? What is the impact of new ethical requirements from the open science movement, such as integrity or transparency? Can CRIS be considered as ethical infrastructures or “infraethics”? Concluding this analysis, the paper proposes an empirical approach for further investigation of this topic. The originality of the paper is that there are very few studies so far that assess the implications of research ethics and open science on the CRIS.

1. The Challenge

The evaluation and monitoring of research performance is one major challenge of research management. Increasing societal expectations, increasing demand for R&D and innovation and increasing funding of public and corporate research require accurate and reliable assessment of research activities, outcomes, resources, infrastructures, etc. Urgent issues like climate change, diversity loss or the COVID-19 pandemic highlight the need for high-quality and high-performance research.

Research information management systems are designed to assess this performance and to contribute to the steady improvement of research. These systems, also called current research information systems (CRIS)1, have been described as software for “the aggregation, curation, and utilization of metadata about research activities” [1], in order to produce useful and reliable knowledge about research and to support research institutions in the provision of funding information and reporting [2]. They aggregate and process information about projects, results, organizations, persons, infrastructures, equipment, facilities, etc., and they produce indicators and assessment for research management. Thus, they can also be described as specialized databases or specialized federated information systems in which distributed information about research produced from different sources (administration, science, etc.) is aggregated in order to provide a structured view of the equipment and services of an organization and its organizational units [3].

Research on CRIS most often puts the focus on technical aspects, on issues related to implementation, acceptance, functionalities, interoperability, data model and format, quality and so on. Yet, insofar as research ethics are highly relevant for the evaluation, the monitoring and the governance of research activities in universities and research organizations, our assumption is that CRIS will become increasingly impacted by ethical issues at least on two different levels:

• Their data models should be able to represent ethical aspects of research projects.
• Their development, implementation and functioning should be compliant with usual (actual) ethical standards of scientific research.

CRIS should comply with good scientific practice, with legal framework conditions and with the principles and values that guide research action for all types of research [4,5,6]. Despite the strategic importance of ethics, few CRIS project teams actually have ethics consultants; usually, research ethics seem to be considered by CRIS developers and operators as a kind of “low-risk, low-impact” issue without priority [7]. Now, especially because of the emerging ecosystem of open science, but also of data protection, privacy, research participation and so on, values and ethics are gaining importance; more than before, researchers’ legal and ethical awareness is required, and this awareness and related practice should be adequately represented by metadata systems such as CRIS [8].

Our paper provides an overview on relevant topics and papers in this field and contributes to the awareness of CRIS as a special kind of ethical infrastructure, also called (enabling) “infraethics” [9]. An “infraethics” is not necessarily morally good in itself; so, (how) can research information management infrastructure foster a kind of “CRIS morality”? This is all the more critical as the development of open science reinforces the role and importance of research ethics, such as integrity, openness and transparency. “It is quite likely that in the future research funding will depend on commitment to open science principles. Conforming to open science principles may be decisive for the success of grant applications” [10] (p. 1053). How do CRIS cope with scientific misconduct? To which extent are CRIS compliant with the ethical requirements of open science?

• First, we describe elements of research ethics that seem particularly relevant for research information management and CRIS.
• Second, we provide a review of former studies on CRIS-related ethical aspects.
• Third, we analyze the potential impact of the new open science movement on ethics, research assessment and CRIS.
• Fourth, we address the question of a particular “CRIS ethics”.

The paper is based on a selective review of communications, articles and reports retrieved from the euroCRIS repository and from Web of Science and on own research in the field of research information systems, research ethics and open science, and it concludes with perspectives for further research and investigation.

2. Research Ethics, Assessment and Monitoring

The concept of research ethics is generally used in a broad and inclusive way, similar to [11], who consider research ethics as “doing good science in a good manner” (p. 550). Good science means, research conducted according to common standards of excellence, while good manners include, among others, appropriate data storage, management of conflicts of interest, protection of human participants and animal subjects, honest reporting of findings and proper citation of sources.

Studies in the field of research ethics mention a large variety of ethical aspects and principles applied to scientific values and “rigor”, like honesty, objectivity, integrity, carefulness, openness, trust, accountability, respect for colleagues and for intellectual property, confidentiality, fairness, efficiency, human subject protection, animal care, etc. However, evaluation and monitoring require more clearly defined, operational elements (data). For example, [12] describe scientific integrity in terms of individual behavior, covering scientific misconduct, such as falsifying research data, ignoring or circumventing major aspects of human subject requirements, not properly disclosing conflicts of interest, changing the design, methodology or results of a study in response to pressure from a funding source, inappropriately assigning authorship credit and so on.

Regarding research management and based on published literature and our own experience with research projects, a number of relevant elements for the assessment and monitoring of research ethics can be identified, on different levels:

• Persons
  • Experts
  • Members of ethics committee
• Events
  • Training interventions
  • Ethics committee meetings
• Facility
  • Ethics committee (local, high level)
• Output (results)
  • Ethics committee reviews (audits)
  • Ethics committee deliberations
  • Reporting of individual scientific misconduct
  • Retractions
• Expertise and skills
  • Ethical expertise

Regarding the work of ethics committees, the German Data Forum proposes to develop indicators that can be used to assess which research projects require a more comprehensive audit or full review, those for which this may not be appropriate and in which cases a well-documented self-assessment or self-certification is sufficient and those for which no review is needed. The Forum also suggests the evaluation and follow-up of high-level ethical committees, like those of research organizations [13]. Similar approaches can be found in other fields, like health technology assessments [14]. The handling of research ethics varies—due to different risk potentials—very much between the disciplines. First and foremost are the health sciences and medicine, whose research and results have a major impact on the health and lives of people. In second place comes the social and economic sciences, where personal data and, consequently, data protection and privacy are often at stake. In the natural sciences, questions of dual use tend to play a role. While a CRIS should function independently of disciplines, it would perhaps be useful to take into account the disciplinary issues of ethics. On the other hand, it is perhaps enough if a CRIS contains information about an ethics committee’s (positive) vote, leaving the consideration of disciplinary issues to the committee itself.

Ensuring ethical conduct is an essential part of basic, applied and clinical research [15]. While research ethics is usually operationalized as an (absence of) individual misconduct, the role of the broader research environment in compromising individual scientific integrity should not be neglected [12]. They mention the significant associations between scientific misbehavior and perceptions of inequities in the resource distribution processes in science. Insofar as the policy in favor of open science pursues the goal of changing the rules and procedures of doing science, including criteria of career development and funding, there is an evident link between open science and integrity.

3. Research Ethics and CRIS: A Literature Review

Although there are numerous studies on research ethics and evaluation and a growing number of publications on research information systems, studies that establish a connection between ethical questions, evaluation and the research information management systems are rare; in fact, there are hardly any studies that address this connection explicitly [16,17], and a sufficiently systematic compilation of ethical requirements for these systems is missing.

A number of case studies suggest that and how CRIS should integrate information on protocols on ethical votes. This involves the exchange of data between CRIS and the databases of ethics committees on the one hand and the platforms for the administration of data management plans on the other hand [18]. Opinions of ethics committees are particularly necessary in projects in the fields of medicine and health care [19]. The question as to how the opinions of ethics committees should be represented in the systems in a uniform manner, both regionally and nationally, and in a comparable manner, independent of discipline, also requires corresponding competencies on the part of the CRIS operators [20,21].

Further studies show how CRIS operators cooperate with ethics advisors. This is seen as part of the necessary competencies for the development, implementation and application of CRIS [22]. In certain institutional contexts, as for instance in the German leading applied research organization Fraunhofer-Gesellschaft, the ethics committee emphasizes the responsibility of the institution and articulates it as a requirement towards the CRIS operator [23]. In the Swedish Science Council, a model has been implemented in which the ethics committee is an integral part of the federal leadership of the national CRIS for grant management [24].

A further series of studies, especially those published by CRIS providers, describe system functionalities and services related to ethical issues, in particular the central role for CRIS in the early research life cycle, including informed consent/ethical approval procedures and impacting privacy and security issues [25]. For example, the Norwegian national CRIStin research management system supplies the databases of the regional ethics committees with data on research projects via an open application interface (API) [26]. Other case studies show how institutional CRIS can produce relevant information for the ethics committees even without such a connection [27,28]. The recent Hércules initiative, promoted by the Conference of Rectors of Spanish Universities and led by the University of Murcia, develops a functional module for ethics management [29].

The same applies to the large commercial systems of Clarivate Analytics (Converis) and Elsevier (Pure), which are able to support ethical procedures with the necessary information and documents [30,31]. Here, interoperability of data formats and standards, such as the CASRAI profiles [32] or the Common European Research Information Format (CERIF)2, play a particularly important role.

Recently, a paper from the Online Computer Library Center (OCLC) has drawn attention to another ethical issue, i.e., the use of analytical data produced by CRIS, including the control of use, appropriate vs. inappropriate use and other ethical questions [33].

4. The Case of Data Collection with CRIS

Data collection with CRIS opens up new avenues for empirical research. However, it also comes with specific challenges. These include the data acquisition, the reliability of the feature acquisition and the validity of statements that are made on the basis of the research data. In addition, there are specific requirements as follows:

Research ethics and data protection: Data collection with CRIS often takes place in the everyday life of CRIS employees. The multitude of data that are collected using CRIS and their integration interface make it increasingly difficult to ensure their anonymity. This results in a number of research ethical and data protection issues [34], which can be listed as follows:

• How can the anonymity of persons (e.g., CRIS employees) be preserved?
• How should data be stored to rule out the risk of de-anonymization?
• How can persons make their own decisions about which data they provide and which they do not?
• What requirements result from the EU General Data Protection Regulation (GDPR), in particular Art. 35 on data protection impact assessment?
• How should informed consent be structured?
• What are the consequences of recording behavior that is relevant under criminal law?

Management of research data: CRIS provide a large amount of data that can make special demands on data storage and thus data management (e.g., data preparation and data provision for reuse, metadata). There is also a paradigm shift towards open science. As a result, new research approaches are developed that aim to collect data from research that are used in everyday life, e.g., are collected by CRIS employees to voluntarily make research accessible via internet platforms. In addition, data are increasingly being made available for reanalysis. This entails challenges to ensure the privacy of data when they are stored and made available, among other things, in repositories of research institutions and research data centers that increasingly are organized in a national (or even European) research data infrastructure.

5. The Impact of Open Science

There are various approaches to defining open science, some broader and more inclusive, others more selective, and more or less useful [35]. Some definitions are presented as taxonomies of open practices or principles (e.g., open data, open access and open peer review), others are rather simple yet not very helpful because they raise new questions about concepts, meanings, limits, etc. (“right to use, reuse, modify and redistribute scholarly knowledge” and so on). In its broadest sense, “open science (…) refers to efforts to make the scientific process more open and inclusive for all relevant actors, within and beyond the scientific community, as enabled by digitalisation” [36]. However, [10] cautions that some principles of open science may be “quite underdetermined” and that concepts like “social responsiveness” and the engagement with societal challenges and stakeholders need more specifications.

The European Commission defined open science as “the transformation, opening up and democratisation of science, research and innovation” [37], with three objectives:

• making science more efficient, transparent and interdisciplinary,
• changing the interaction between science and society,
• enabling broader societal impact and innovation.

Significant emphasis is placed on research data, especially through the European Open Science Cloud (EOSC) and the preparation of the FP9 funding program [38].

Transparency, removing barriers, is at the heart of Rentier’s concept of open science. Open science, with the words of the former Rector of the University of Liège, “aims to broaden the scope and freedom of use of research results, by facilitating the transfer of information, reducing its costs and, ultimately, preventing exclusion” [39] (p. 30). Good science becomes synonymous with open science, and the link with research ethics is obvious: “it has now become essential to recognize and value researchers’ full commitment to Open Science, and to make it clear how this goodwill will be rewarded in the various evaluations they will have to undergo more and more frequently” (p. 29). Personal commitment and institutional reward and funding systems, as part of the broader research environment, should foster and reinforce each other, in a dialectical and somehow virtuous circle.

This approach is in line with the “norms of openness” as defined 50 years ago by Merton in his Sociology of Science (1973) [40]. Expanding Merton’s principles of open science (communalism, universalism, disinterestedness, originality and skepticism) to the new environment of digital open science means to include, for instance, open access publishing, data sharing, open innovation and citizen (participatory) science in a new concept of academic “ethos of science” [41].

As mentioned above, research ethics is a multidimensional concept, with normative ethics (what is right and wrong, informed consent, harmfulness, etc.), but also other aspects like reproducibility, trust (accountability) and social value (importance and relevance for society). Regarding data acquisition and analysis, for instance, rigorous and careful research data management, based on good practices and standard procedures, is essential to ensure data integrity and transparency [15], while rich and standard data documentation and open data sharing contribute to the accountability and reproducibility of research. Normative ethics about the reuse of data have been summarized in [42], intending to improve the findability, accessibility, interoperability and reuse of digital assets.3

For the need for evaluation, the different aspects of emergent open science (or research) ethics can mostly be defined as elements of output and facilities, but also as a specific attribute of the core element project:

• Facility
  • Open (institutional) repository
  • Data repository
• Output (results)
  • Open access publishing (green, gold, etc.)
  • Data sharing
• Project
  • Participation (citizen science)

Depending on the scope of evaluation, other aspects could be added, such as data management plans (output), open access and open science training programs, for instance, for PhD students (event), or the nomination of an open science coordinator (person). Finally, how can reusability and reproducibility be evaluated: as attributes of publication and data; or as a methodology of doing science (project)?

Yet, the process of evaluation itself has become an object of the transformation of research practices towards open science. The San Francisco Declaration on Research Assessment (DORA)4, for instance, begins with a statement on “a pressing need to improve the ways in which the output of scientific research is evaluated by funding agencies, academic institutions, and other parties” and posits that, as “funding agencies, institutions that employ scientists, and scientists themselves, all have a desire, and need, to assess the quality and impact of scientific outputs (it is) imperative that scientific output is measured accurately and evaluated wisely”. The main goal of DORA is the elimination of the use of journal-based metrics, i.e., journal impact factors, in funding, appointment and promotion considerations. DORA’s major claims are for the need to assess research “on its own merits” and to capitalize on the opportunities provided by online publication, especially through new indicators of significance and impact.

More than 2000 organizations and many thousands of scientists from all around the world have already signed the declaration. Wellcome, the US National Institutes of Health and National Science Foundation, the European Commission, the Dutch Research Council, the Swiss National Science Foundation, EMBO and the French National Research Agency all signed the declaration, along with many research institutes, universities and academic societies. Moreover, they started to change their way of doing evaluation and they report on their examples of good practice in research assessment on the DORA site. This is a challenge for CRIS: how do they cope with these new good practices of research assessment? The relevance of DORA for research information management systems has been mentioned by different authors [43,44] but without any real implication for the data model, the metadata, the semantics or the reporting. An Italian case study on distributed research information management systems reports that the support of DORA by the biological research community may have “mitigated” the impact of a new performance-based funding system on the scientists’ publishing behavior but does not specify the impact on the CRIS itself [45]. The question remains open, to date, if and to what extent new and more qualitative assessment methods are compliant with systems that have been originally designed for traditional metrics.

6. CRIS as an Issue of Applied Ethics

Ethics is a branch of philosophy and is often referred to as practical philosophy. The core of ethical considerations is morality, which in turn is composed of values or goals, and norms or instructions for action. In every individual there are attitudes to how one should act. Ethics deals with the question of whether a person’s actions are right or not [46]. If this approach is applied to research information management, the morality of CRIS becomes an issue. In other terms, what is, really or potentially, right or wrong with a CRIS?

As a kind of sub-area of applied ethics, “CRIS ethics” deals above all with ethical questions raised by the development, implementation and usage of CRIS. Project management, system administration and the selection of data sources as well as the configuration of indicators, standard reports and so on: these CRIS-related dimensions are usually evaluated in terms of their usefulness, effectiveness and outcome, rarely regarding their compliance with research ethics.

Sometimes, the development and implementation of CRIS encounter questions, fears and concerns in the institutions, raised essentially by the different user groups. However, the need for a debate on values essential for the development of CRIS does not only exist on the side of the users. Such a debate on values is also required by and for people who work in the IT area, when faced with decisions in their everyday work and the negative consequences they could have for users or society at large [47]. An ethical discussion is necessary and desirable in the technical context of CRIS.

Ethical questions are negotiated depending on the context among the participants or within the framework of social conventions. In the context of the development of CRIS, they serve to promote the well-being of the institution and the individual and to initiate innovations that deal explicitly with these questions. For their design, this means that, as with business rules, there will be no generally applicable, concrete implementation patterns.

However, best practices, standards and aspects should be available at the level of the development phases and special requirements [48]. Some of the topics are data protection (privacy), system security, data integrity, transparency of data processing and system administration, system acceptance, acceptance of evaluation and quality of data, indicators, reports, etc. Some of these topics, in particular privacy, are well known and issues of concern. Others are (too) often addressed as a mainly technical challenge; yet, as the IFLA international code of ethics for information professionals reminds us, service quality, integrity and transparency are among the fundamental values of the profession’s deontology5. The exploration and integration of the ethical aspects into the digital solutions can only be done through ongoing discussion and continuous collaboration with the users and developers of CRIS. There are data models for research information management; what is needed, today, is an ethics framework for these systems.

In a broader way, [49] states that “ethical reasoning should be an integral part of data science, which helps researchers to critically valuate and discuss the allocation of responsibilities and accountabilities within highly distributed and globalized trajectories of data production, dissemination and re-use” (p. 3). For Leonelli, all people involved must take some responsibility for potential implications, in relation to their specific roles (p. 7):

• Computer engineers should be able to “reflect on the ethical dimensions of alternative ways of developing software, the variety of publics involved when engaging in such processes and the types of feedback and interdependencies that this involves”.
• Data providers should “evaluate the consequences of disseminating specific types of data, in terms of potential infringement of privacy laws, the replicability and reliability of the datasets at hand, and the wider implications of data sharing for local communities”.
• Data users should be trained “to consider the history and potential significance of the data before and during reanalysis, so as to spot potential bias or misalignment between the conditions under which data from different sources were originally collected (…) and the ways in which they were processed to enable comparison and integration”.

It is obvious that these general requirements of “data science ethics” also apply to the specific case of data-intensive research information systems, and to their development, management and usage.

7. Further Perspectives

A study should be conducted that has the purpose of exploring how the development, management and usage of CRIS in the new academic ecosystem of open science cope with research ethics in a general way and, more specifically, with the emerging “open science ethos”. Based on a review of available case studies and research, we can already establish a list of independent variables, i.e., different levels of ethical requirements, from at least five different sources:

• Research funding: funding agencies, foundations, research programs, etc.
• Science evaluation: authorities, ministries, research organizations, evaluation agencies, etc.
• Academic community: learned societies, institutions, library associations, etc.
• Legal framework: data protection, intellectual property rights, etc.
• Civil society: lobbies, non-governmental organizations, associations, citizens, etc.

Stakeholders on each level have their own concerns, demands and requirements regarding research ethics, often convergent (data protection, privacy, etc.), sometimes not (transparency, secrecy, etc.). Some requirements represent a strong and legal constraint, others do not. A model for CRIS-related ethics must be aware of the multidimensional nature of ethical requirements and address these issues. What is needed, moreover, is an evaluation of the way these requirements are perceived by the developers, managers and users of the research management systems, an investigation of their level of awareness, opinions, experiences, priorities and preparedness, and an analysis of to what extent recent open science strategies change the game. The variable of preparedness includes the need to assess to what degree existing data models and formats, in particular CERIF, are able to represent and communicate relevant data in an appropriate way.

In order to obtain empirical evidence on the ethical challenge for research information management, for a better understanding and an improved representation of the relevant data, attributes and relationships, we would suggest a two-level methodology:

• First, an exploratory survey with CRIS experts (project managers, editors, system administrators, scientists, librarians) from different European countries in order to investigate their opinions on ethical requirements from the academic communities, funders, research organizations, authorities and society as a whole, and to assess their attitudes towards ethical principles and scientific misconduct.
• Second, a small-scale auditing approach, above all with CRIS providers and administrators, in order to assess whether and to what extent ethical issues and aspects are considered in the design, implementation and application of CRIS.

While the expected outcome of the first part of the study is a detailed and realistic catalog of ethical requirements of research information management and its systems, the expected outcome of the second part is a landscape study of the compliance of existing CRIS with these requirements.

The underlying question is: can CRIS be considered as ethical infrastructures, as “infraethics”? Are they good infraethics, “oriented towards facilitating the occurrence of what is morally good” [50] (p. 393)? Key technology in the field of research, these systems represent a challenge for institutions: for instance, how much trust can be placed in the CRIS and who controls its quality? What methods can be used to ensure quality in CRIS? Where do the research data go and who has access to them? What technical potential should be promoted and where should boundaries be drawn? All of these challenges have been addressed by CRIS researchers. The new topic of research ethics—standards of conduct that distinguish between right and wrong, good and bad science [15]—opens up unprecedented opportunities in CRIS; but, at the same time, it brings with it new challenges. Ethical principles cannot be ignored.

Technology design is never ethically neutral but embeds some values, whether explicitly or implicitly [50]. The overall objective of our research is to contribute to a systematic and exhaustive overview of ethical issues related to CRIS, which should make it possible to describe the situation of the CRIS in the current context of open science and to make suggestions on how these systems can be further developed and thus contribute to the realization of the principles of open science from the point of view of research ethics.