Purposeful Evaluation of Scholarship in the Open Science Era

Abstract

In most of the world’s countries, scholarship evaluation for tenure and promotion continues to rely on conventional criteria of publications in journals of high impact factor and achievements in securing research funds. Continuing to hire and promote scholars based on these criteria exposes universities to risk because students, directly and indirectly through government funds, are the main source of revenues for academic institutions. At the same time, talented young researchers increasingly look for professors renowned for excellence in mentoring doctoral students and early career researchers. Purposeful scholarship evaluation in the open science era needs to include all three areas of scholarly activity: research, teaching and mentoring, and service to society.

1. Introduction

Plentiful research has been devoted in the last three decades (1990–2019) to scholarship evaluation for granting tenure and promotion to higher professor rankings [1,2,3]. Academic tenure (from Latin tenere “to hold”) is permanent employment at universities, safeguarding scholarly freedom to conduct research in any area without endangering the tenured scholar future position at the university [4].

Following the broadened scholarship concept proposed by Boyer [5], today’s scholars in academic evaluation generally agree that beyond achievements in research, evaluation should take into account teaching as well as scholarship of service and integration [3].

Suggesting his broadened concept for which, next to research (discovery) and education, scholarship includes “integration”, namely making connections across disciplines and shaping a more integrated use of knowledge, and application of disciplinary expertise, Boyer explicitly called for the use of these new criteria to evaluate scholars for professorship [6].

In practice, however, research on current practices of academic evaluation used by universities to recruit and promote professors in Canada [7], in the United States of America [8], or at the international level [9], invariably find that publicly orientated faculty work is undervalued [7] and that traditional criteria of peer reviewed publications, authorship order, journal impact factor, and grant funding are used in the recruitment, promotion and tenure guidelines [9].

Since the early “objectives-based” efforts of Dressel in the 1970s [10] to those of Boyer in the subsequent two decades [6], calls for changing the academic evaluation criteria are regularly published in the international literature. In 2018, for example, a team of scientists, academic administrators, and funders in the biomedical sciences introduced a few principles for assessing scientists: contributing to societal needs, new “responsible” indicators for assessing scientists, complete publication of results, openness and rewarding researchers for intellectual risk-taking [8]. The new criteria, in other words, include aspects such as rewarding researchers for open science practices, and the transparent and complete reporting of research.

In this study, after showing how current calls for professors in distant countries clearly reveal the interest of universities for highly cited scholars with a track of securing research funds, I suggest why and how universities should realign scholarship evaluation with the unchanged purpose of the university.

In this rapidly changing context, young scholars seeking tenure and promotion benefit from the practice of open science [11], because it provides better and more impactful results with respect to each of the three areas of scholarship.

2. Current Scholarship Evaluation Criteria

Reviewing the requirements for the recruitment of professors and senior researchers advertised by late 2020 in Nature, a reputed scientific journal published in Great Britain since 1869, reveals that universities across the world generally look for scholars author of “top-class publications” and “familiar with acquiring third-party funding”.

For example, in Germany the Technical University of Dresden, Faculty of Physics, the Institute of Solid-State and Materials Physics recently invited applications for the Chair of Nanoscale Quantum Materials. To be eligible for the position:

“You need to have a doctorate in physics or a related discipline as well as a habilitation or habilitation-equivalent achievements in research and teaching. We furthermore expect you to be familiar with acquiring third-party funding and to be experienced in project and group management. We place special emphasis on top-class publications, strong international contacts, as well as independently acquired and successfully conducted research projects”

[12].

In China, for the faculty positions available at the School of Environmental Science and Engineering, Southern University of Science and Technology based in Shenzhen, in exchange for “globally competitive (including US and Hong Kong) salaries and benefit packages”, applicants having a Ph.D. in environmental science and engineering, earth and atmospheric sciences or related disciplines were required to have:

“… a proven and consistent track record of high-quality scientific publications and good communication skills”

[13].

In Switzerland the University of Basel seeking a new professor of Biochemistry required candidates to show evidence of:

“Internationally reputed track record of independent research in the broad field of cancer biology and experience in leading a research group. Documented ability to acquire competitive third-party funding”

[14].

In Russia, St. Petersburg’s ITMO University’s Department of Physics and Engineering inviting candidates “for all ranks, with priority for the assistant professor rank”, required candidates to have a:

“Solid research record evidenced by high quality publications in high-impact journals» and a «demonstrated ability to develop and sustain internally and externally funded research”

[15].

In Israel, the University of Haifa, advertising a tenure-track faculty position in neuroscience, was seeking candidates with:

“Two or more years of postdoctoral experience and a strong publication record … expected to engage in independent research funded by external competitive funding”

[16].

In Canada, the University of British Columbia seeking an assistant professor in physical chemistry required successful applicants to have:

“an excellent research track record» being expected to «obtain external funding …”

[17].

The short list above could continue to include most of the world’s countries. Scholars seeking tenure invariably need to have published numerous research papers in “top journals” and a demonstrated ability to raise funds.

How these “top journals” are identified almost invariably translates into journals having a high journal impact factor (JIF), a poor statistical indicator largely determined by very few papers published by the journal [18], whereas the vast majority of papers will have far lesser citations than the JIF.

So, what can young scholars actually do using the approach of open science to research, teaching and mentoring, and service to society [11] to make their professional profile attractive to recruiting universities mostly interested in their ability to raise research funds and publish in “top class” journals?

3. Open Science for Impactful Research

Citations reflect the interest and partly the real use of published research findings from other researchers. Hence, the original idea of Garfield to measure the impact of a research paper through the number of citations [19] remains useful.

The most important factor driving citations, however, is not the JIF but rather the open access (OA) nature of published work, with papers self-archived and made freely accessible on the internet receiving from 2.5 to 5.7 times the number of citations, when compared to non-OA papers published in the same journal [20]. Similarly, by immediately publishing their work in preprint form, scholars benefit from higher citations and online mentions [21].

The final published articles in the basic sciences differ from the corresponding preprints only to a minor extent [22]. Hence, rather than pursuing publication of their research findings in high JIF journals, scholars would seek publication on the peer reviewed journal most suitable to their research findings (whether OA or pay-walled) taking care to make their research findings immediately available to the scholarly community first as a preprint, and then as self-archived published article in “green” open access after the embargo period (typically, 12 months) [20].

In almost the opposite fashion, the share of scholars “green” self-archiving their papers is currently slightly above 10% (only 12% of total annual articles actually self-archived [23]) making evident the urgent widespread need for the global scholarly community to receive updated education of practical value in the field of open science [24].

Besides citations, scholars seeking research funds, tenure and promotion are interested in visibility which, in the digital age, starts with online mentions of their research work. Making research findings openly accessible on the internet thanks to preprints and green self-archiving has substantial benefits on research visibility, and thus on securing research funds and on international collaborations.

For example, companies interested in financing applied research projects easily find practitioners of open science. Whatever the demand for applied research, the company’s research and development (R&D) managers will start looking for perspective collaborators through a search on Google, an online search engine which by October 2020 accounted for >90% of the global search market [25]. Having all their own work published online and easy to find by the aforementioned search engine directly enhances the chances to be identified as potential partners of the company seeking collaboration from a scholarly team.

The same is true for scholars seeking international collaboration. Accordingly, and perhaps surprisingly, low-income countries have the highest percentage of OA publications in biomedical field: 69% [26], and this even though papers in biomedicine resulting from international collaborations have a far higher proportion of OA papers with respect to other scientific fields: 57.2% vs. 41%.

The fraction of papers with international co-authorships in the last three decades has increased dramatically. For comparison, in 1983 the percentage of papers co-authored by scholars based in different countries was 5.8% in biology, 5.4% in chemistry and 9.4% in physics. In 2012, the shares had respectively grown to 56.1%, 48.5% and 64.2% [27]. From the impact viewpoint, furthermore, papers from international collaborations receive a higher number of citations, especially in the basic and applied sciences. For example, one additional country was found in 2013 to increase the mean citation count by 8.6% in chemistry and 5.5% in the life sciences [28].

4. Rankers Become Ranked

I agree with Edwards and Roy: academia and research funding agencies should support science “as a public good”, creating new research and science policies in place of the current ones based on quantity in place of quality which “selectively weeds out ethical and altruistic actors, while selecting for academics who are more comfortable and responsive to perverse incentives from the point of entry” [29].

Yet, as shown above, the recruitment practices of leading universities seek scholarly authors of “top-class publications” and “familiar with acquiring third-party funding”, and do not even mention achievements in two key areas of the academic activity: teaching and mentoring, and societal service.

“In my 20 years of mentoring at the Yale University School of Medicine I have helped numerous trainees cope with having to deal with toxic principal investigators … Mentorship evaluation should be taken into account for tenure decisions and annual salary decisions for all faculty members. On the other hand, exemplary mentors should be duly rewarded with a salary increase and promotion”

[30].

University managers should be aware that today’s universities competing for students and talented post-docs not only at the national level but also internationally [31], are increasingly selected by students based on the quality of teaching, whereas doctoral students no longer choose their professor/principal investigator based only on her/his research achievements, but also on achievements as a mentor of doctoral students.

Besides being intrinsically flawed as shown by plentiful research [1,2,3,5,8], scholarship evaluation based on publications in high JIF journals and research funds secured puts universities at risk of serious financial crisis because students and young researchers will simply opt for studying and carrying out research at universities whose professors excel in teaching and in doctoral student mentoring.

For example, the Times Higher Education (THE) World University Rankings, a ranking of more than 1900 global research universities published yearly with the aim of giving “students and their families the information they need to help them choose where to study”, evaluates universities based on five criteria, the most important of which are teaching (30%) and research (30%) [32]. Teaching metrics is measured by five performance indicators, the most important of which is a reputation survey among both academics and students (for example, in the USA this ranking is undertaken based on a survey of more than 170,000 current students).

Regardless of widespread academic criticism [33,34], similar rankings are published by other organizations such as the Shanghai Ranking Consultancy (producing the Academic Ranking of World Universities [35]), Quacquarelli Symonds (publishing the QS World University Ranking [36]), and by Leiden University (publishing the Centre for Science and Technology Studies Leiden Ranking [37]).

With ongoing globalization and easy online access to bibliometric data, these rankings are likely to become even more numerous. In brief, universities—the rankers—have become the ranked. The very same organizations that still in late 2020 required candidates applying for assistant and full professorship to provide evidence to be author of “top-class publications” and “familiar with acquiring third-party funding”, when ranked raise reasonable objections to ranking criteria and outputs.

Criticism spans from noting that “rankings apply a combination of indicators that might not represent universities’ particular missions, and often overlook societal impact or teaching quality” [34], through another finding that “QS and THE manipulate affect like fear, anxiety, mutuality, assurance, and so on to mobilize policy perpetuating the idea of rankers as authority” [38] in a global higher education market in which commercial rankers “by means of periodic yearly publication, construct the reputation for excellence as a scarce resource which universities are expected to compete for” [39].

In other words, universities are ranked based on criteria similar to those they actually use to recruit and promote their academic personnel, whereas thoughtful academic calls for global university ranking “boycotts and banning” [40] have mostly failed.

To resolve this paradox and prevent the aforementioned economic troubles which await universities continuing to rely on achievements in research and fund raising only to evaluate scholarship for tenure and promotion, universities in different nations must realign scholarship evaluation with the unchanged purpose of the modern university.

5. Alignment of Purpose and Scholarship Evaluation

The purpose of the university is the same conceived by Humboldt in 1810 when establishing Berlin’s University [41], namely to serve the supporting society through knowledge creation (by scientific research) and knowledge dissemination (via student education). The major difference with Humboldt’s times lies in the largely increased role of the university in contributing to societal needs, for example via policy advice to government or via consulting services to industry, taking into account that society benefits a lot from these activities whose impact cannot be measured by bibliometric indexes. For example, in geography there are numerous interesting research approaches which have a high impact on society but are not necessarily published in high-impact journals. For instance, the up-to-date topic of open geospatial data significantly contributes to the improvement of public administration, via methods developed by researchers using public (open) data to improve approaches to, for instance, the visualization of three-dimensional city models, maintaining cadastral systems, and urban planning.

Universities across nations should, therefore, comprehensively evaluate the achievements of scholars seeking tenure and promotion with respect to each of the three main scholarly activities: research, teaching and mentoring, and societal service. Accordingly, excellence in teaching and mentorship should always be explicitly mentioned in professor recruitment calls for applications, asking candidates to provide evidence of such excellence.

5.1. Research Scholarship Evaluation

Academics members of selection and promotions committees need to be aware that scientometric scholars have already introduced better indicators than the JIF or the h-index [42] in response to the demand of fairer and more comprehensive research scholarship evaluation. For example, Ioannidis and co-workers have lately introduced a composite indicator that combines six citation metrics [43].

By combining total citations, h-index, coauthorship-adjusted h_m-index, number of citations to papers as single author, number of citations to papers as single or first author, and number of citations to papers as single, first, or last author, the new metrics avoid using metrics that focus on single aspects of the scientific impact of a scholar. The team, for example, successfully applied the method to produce a publicly available database of 100,000 “top” scientists [44].

Another purposeful approach suggested by Bornmann and Marx makes use of the number of publications for a researcher belonging to the 10% of the most-cited publications in their field and publication year [45]. The resulting percentile indicator (P_{top 10%}) thus focuses onto successful publications normalised for time and field.

A practically useful innovation for preliminary and quick assessment research scholarship has been proposed by Shekman, a Nobel laureate in medicine, suggesting the inclusion in any academic job application of an “impact statement”, namely a succinct, statement summarizing the impact of the candidate’s main discoveries to be used as the basis for short-listing candidates:

“It should be written in a way a broader group of other scholars can read and say ‘Oh, wow, he discovered that’. ‘I didn’t know that, but that sounds really important’. Then the committee can create a short list of candidates and then look into the papers and letters of recommendation and refine judgments”

[46].

5.2. Teaching and Mentoring Scholarship Evaluation

There are many ways to provide evidence of successful scholarship in teaching. A professorship advertisement might for example ask candidates to provide evidence that they use a scholarly approach to teaching. One, suggested by Trigwell upon investigating the effectiveness of the scholarly approach to teaching at universities in Australia, adopts a student-centered approach to teaching and learning based on the use of pedagogical theory and faculty peer’s scrutiny [47].

Young scholars seeking tenure and promotion will emphasize in their applications how their scholarly approach to research, teaching and mentoring is capable to provide students and co-workers in the research team with plentiful uninterrupted time for studying, carrying out research, writing, and lecturing, namely for academic “deep work” [48] in “a distracted world” [48]. For example by teaching the applicant’s students how to effectively use the e-mail in the academic environment [49], or how to practically benefit from open science tools such as preprints, green self-archiving, and OA publishing [24].

Evidence of excellence in mentoring, in its turn, can be shown by including in the application recommendation letters from former students who successfully gained their M.Sci or Ph.D. degrees under the candidate’s guidance.

Quality in mentoring doctoral students starts from identifying what is currently lacking in current academic mentoring practices from the student’s viewpoint. A recent study of doctoral education carried out on 688 Ph.D. students at an American university, for example, suggests that the main gaps are in areas such as preparation for grant writing, teaching, and leading research teams [50], with significant differences in perceived preparation among disciplines. In closer detail, the proportion of Ph.D. students agreeing they were prepared professionally to write a grant proposal amounted to only 57.1% for students in engineering, physical sciences, and mathematics, whereas only 55.4% of the life sciences students felt comfortable with teaching. It is further instructive to learn that many students suggested that they would welcome professional education on teaching to develop their teaching skills [50].

Doctoral student mentors who are competent and able to deliver critical feedback within a climate of kindness and respect will shape successful Ph.D. students [51]. On the other hand, these mentoring skills are often absent amid principal investigators [30]. Hence, it is perhaps not surprising to learn that about half of all students who begin doctoral programs in the USA do not complete their degrees [52].

By the same token, calling for the urgent need for teaching doctoral students how to teach, Brightman reported in 2009 that few doctoral programs in management education in the USA offered systematic teacher training, ascribing the finding to the fact that “the typical academic reward system overvalues research and scholarship and undervalues teaching” [53].

That similar needs are common to many countries beyond North America is shown, for instance, by the widespread demand for the 2-day scientific writing course to improve the clarity and accuracy of scientific writing developed by Rothenberg and Lowe in the early 2000s at the University of Amsterdam [54]. Born as an “in house” course in the Faculty of Science, the workshop was subsequently required by organizations as diverse as research councils, research funding agencies and even banks.

However, when programs on how to effectively teach are deployed the outcomes are generally successful and publicly recognized. For example, University College Dublin in the academic year 2010–2011 required doctoral students and postdoctoral researchers in information and communication technology to develop and teach an undergraduate course “Web 2.0 and Social Media”. “Undergraduate students found varied instructor perspectives and teaching approaches stimulating” [55], and was awarded the university teaching award.

5.3. Societal Service Scholarship Evaluation

Service is generally offered by academic scholars either by consulting with industry or by engaging with society through policy advice, informing the media on research advances, and taking part in public debates.

Society at large benefits from these forms of service. Hence, university managers will wisely include societal service as one of the three key scholarship areas’ object of evaluation for tenure and promotion.

Besides being a source of revenues for the university, collaboration with industry is useful also to incorporate in teaching courses updated insight on industrial processes and technologies as well as to better “understand the nature of the firms’ activities and problems” [56]. For example, analyzing the outcomes of interaction with industry in three engineering-related disciplines (digital signal processing, geo-sciences, and transportation and logistics) at universities in Sweden, Holmén and Ljungberg found that professors use examples derived from the said interaction to illustrate and explain specific issues in class, which facilitates bridging the gap between theory and concepts [56], as well as provide the opportunity to conduct in collaboration with industry as part of the Ph.D. research work.

By engaging with society, humanities scholars, in turn, deliver multiple and highly valued services to the community [57]. It is sufficient to cite the case of the eminent archaeologist and professor Sebastiano Tusa who since 1972 participated or directed missions and archaeological researches, including excavations and underwater archaeology explorations in Italy, Malta, Tunisia, Iraq, Iran, Pakistan, Japan, Kenya and Turkey, eventually establishing the Sicily’s Superintendency of Sea. Through the latter institution, his team discovered archaeological finds of exceptional value such as the “Dancing Satyr” nowadays on display in a museum dedicated to the bronze statue in a small port city in Sicily, where it attracts thousands of tourists yearly from across the world [58].

6. Conclusions

In the digital era, universities are continuously scrutinized and evaluated by different stakeholders. Continuing to hire and promote scholars for their achievements in research and in securing research funds exposes them to serious financial risk because students, directly and indirectly (through government funds), are the main source of revenues for academic institutions, whereas talented young researchers are those who actually carry out most of the published research.

Students and young researchers look for universities whose professors excel, respectively, in teaching and mentoring, whereas society increasingly needs scholarly advice in a number of areas. Firms and local governments, for example, today need advice on how to tackle the challenges of the energy transition, namely the transition from consuming energy supplied in the form of natural gas or power supplied by gas or electric utilities to consuming energy self-generated from renewable energy sources via new energy technologies [59].

The focus on publishing and citations has a further negative effect which puts at risk also the quality of research because because scholars tend to chose the areas of study according to how easy they think it will be to publish a paper and gather citations, sacrificing those which could be useful investigations but less “profitable” in terms of citations.

In brief, purposeful scholarship evaluation for tenure and promotion in the open science era needs to include all three areas of scholarly activity: research, teaching and mentoring, and service to society.

Table 1. Framework for assessing candidates for tenure and promotion based on achievements in research, teaching and mentoring, and societal service. Adapted from reference [45].

Indicator	Candidate
Research
Original research article
Review
Letter
Conference Abstract
Editorial
Proceedings Paper
Patents
Total publications
Number of publications as first author
Number of single author publications
Number of publications as corresponding author
Year of first publication
Number of years between the first publication and date of evaluation
Average number of publications per year
Invited lectures at international meetings
Keynote lectures at international meetings
Number and overall value of research grants
International scientific conferences organized
National scientific meetings organized
Impact
Total citations (and self citations)
Proportion of self-citations in total citations
Average number of citations per publication
h-index
m-quotient (h-index divided by number of years since the first published study)
Ptop 10%
PPtop 10% (%)
Ptop 10% quotient
Teaching and mentoring
Number and nature of courses taught
International workshops and schools organized
Published studies in scientific education research
Number of PhD students mentored
Number of MSci students mentored
Impact
Student ratings of effectiveness in teaching
Recognition related to visiting professorships, teaching awards
Recommendation letters written by former M.Sci. students mentored
Recommendation letters written by former Ph.D. students mentored
Societal service
Advice to public authorities
Technologies transferred to marketplace
Talks at public conferences
Articles in newspapers and magazines
Interviews with the press
Divulgation of books
Consulting services to firms

summarizes a list of indicators for evaluating scholarship in this purposeful way. Selection criteria may vary depending on the specific needs of the university. One university might instruct the selection committee to grant, for instance, up to 40 points out of 100 for research, 40 for teaching and mentoring, and 20 for service to society.

Open science matters to scholarship evaluation because its practice, from publishing preprints through green self-archiving, provides more impactful results with respect to each of the three areas of scholarship. As put it by an early career researcher, “being an open researcher is not only the right thing to do, but is also the best thing to do” [60].

Publishing open access articles and books removes the economic barrier to accessing scientific knowledge. Of course, there are economic costs in OA publishing. However, in the digital era when all scientific articles are published only in digital format in the World Wide Web (and print articles made available on demand in exchange of a fee), these costs are a small fraction of the costs faced when scientific papers were printed and published in journal’s issues distributed via post to hundreds or thousands of libraries (and to industrial customers) across different countries. For instance, a highly profitable OA publishing company (50% profit on the 22,000 OA articles published in 2012) in early 2013 reported a production cost of $290 per article [61].

By reforming scholarship evaluation for tenure and promotion along the guidelines suggested in this and in several related studies [1,6,7,9], today’s universities competing on national and international levels will eventually realign the tenure and promotion processes with the unchanged purpose of the university [41], namely to serve the supporting society through knowledge creation, dissemination, and use for the betterment of society at large.