Performance of 4 Pre-Trained Sentence Transformer Models in the Semantic Query of a Systematic Review Dataset on Peri-Implantitis

Abstract

Systematic reviews are cumbersome yet essential to the epistemic process of medical science. Finding significant reports, however, is a daunting task because the sheer volume of published literature makes the manual screening of databases time-consuming. The use of Artificial Intelligence could make literature processing faster and more efficient. Sentence transformers are groundbreaking algorithms that can generate rich semantic representations of text documents and allow for semantic queries. In the present report, we compared four freely available sentence transformer pre-trained models (all-MiniLM-L6-v2, all-MiniLM-L12-v2, all-mpnet-base-v2, and All-distilroberta-v1) on a convenience sample of 6110 articles from a published systematic review. The authors of this review manually screened the dataset and identified 24 target articles that addressed the Focused Questions (FQ) of the review. We applied the four sentence transformers to the dataset and, using the FQ as a query, performed a semantic similarity search on the dataset. The models identified similarities between the FQ and the target articles to a varying degree, and, sorting the dataset by semantic similarities using the best-performing model (all-mpnet-base-v2), the target articles could be found in the top 700 papers out of the 6110 dataset. Our data indicate that the choice of an appropriate pre-trained model could remarkably reduce the number of articles to screen and the time to completion for systematic reviews.

1. Introduction

Systematic reviews and literature searches serve as the foundation for evidence-based medicine, facilitating the identification of relevant research findings [1]. Based on this approach, diagnostic and therapeutic interventions should be based on the existing evidence [2]. Given the increasing volume of available research in academic literature, however, searching for pertinent manuscripts has become an arduous endeavor [3]. Reviewing the literature has thus become increasingly valued in academia [4], with systematic reviews being recognized as a crucial tool to overcome the limitations of individual studies. By pooling data across multiple studies, systematic reviews help provide a comprehensive understanding of the effectiveness of various therapies, identify research gaps, inform clinical practice guidelines, and support evidence-based decision-making by healthcare professionals, policymakers, and researchers [5]. Generating systematic reviews, however, is challenging and time-consuming.

Systematic reviews follow a predefined protocol that outlines the search strategy, inclusion and exclusion criteria, data extraction methods, and statistical analysis procedures. This protocol ensures transparency and minimizes bias in the review process [6]. The systematic review process typically begins with the formulation of a specific and relevant research question [7]. Next, an extensive literature search is conducted using predetermined search terms across various databases like PubMed, Scopus, and Web of Science [8]. Therefore, after the initial search, studies are screened based on predefined inclusion and exclusion criteria to identify those that meet the criteria for inclusion in the review [9]. However, due to the vast amount of available literature, it can be challenging to manually assess all potential studies for relevance. The selected studies are then systematically analyzed and relevant data are extracted [10]. Statistical techniques such as meta-analysis may be employed to synthesize findings from individual studies when appropriate [11]. Additionally, each included study undergoes quality evaluation using established criteria to assess any bias present.

Automation has great potential to make systematic reviews quicker and cheaper, and recent advances in text mining, NLP, and machine learning have demonstrated that several tasks within the systematic review process can be automated or assisted by automation [12]. In recent years, there have been innovations proposed to accelerate the process of systematic reviews, and software is available to help scholars filter out the literature of interest, such as Abstractr, ASReviews, EPPI-reviewer, or RobotSearch, which relies on a convolutional neural network-based classifier to identify RCTs [13]. In particular, the development of large language models, such as BERT (Bidirectional Encoder Representations from Transformers), has shown promise in automating certain aspects of the systematic review process [14]. For example, BERT can be utilized to aid in the identification of relevant studies by improving literature retrieval capabilities through semantic understanding and contextual analysis of search terms, and software like ASReviews leverages these new architectures [15]. End-users, however, have little awareness of the underlying algorithms and models that underlie these tools. The foundation of semantic understanding of a text is the use of embeddings, i.e., numerical representations, through a vector of a given dimensionality of the meaning of a word or sentence [16]. Although vectors can be generically defined as geometric objects that represent a directed quantity, possessing both magnitude and direction, in the context of NLP, vectors can be conceived as sequences of numbers that encode the semantic information of words or phrases [17].

Although sparse vectors have been used in the semantic representation of words and texts for a long time [18], embeddings were first introduced by Mikolov and a team of researchers at Google in 2013 with an algorithm known as Word2Vec [19]. Word2vec consisted of a shallow artificial neural network that could be trained on a text or a series of texts and proceeded to model 300-dimensional embeddings for each word in such a way that embeddings of words that occurred together in the text within a pre-set context frame were more similar, while embeddings of words that never occurred together differed to a greater extent [20]. This idea, which may sound not only simple but even simplistic, is rooted in the distributional semantic theory of Harris and Firth [21], and, more importantly, has proven to be extremely effective in NLP tasks. By training such algorithms on large text datasets, it was possible to provide a way for machines to have an internal representation of the meaning of words, which would be distributed in a multidimensional semantic space according to their semantic features [22].

In 2017, a seminal paper by Vaswani et al. with the title “Attention is all you need” outlined the concept of attention mechanisms in NLP, revolutionizing the field by allowing models to capture more complex relationships and dependencies between words and sentences through specific architectures known as transformers [23]. Transformers can create richer embeddings using at least two key mechanisms: self-attention and positional encoding. Self-attention allows the model to weigh the importance of different words in a sentence and capture the relationships between them. Positional encoding is a way to provide information about the positions or order of words or tokens in a sequence by summing the embedding of each word into a positional encoding vector that, as its name suggests, encodes its position within the sequence, often relying on the sine and cosine functions. The embeddings obtained with transformer architectures in models such as BERT or GPT (for text generations) have exceeded benchmark performances in every aspect of NLP, including tasks like machine translation, sentiment analysis, named entity recognition, and question answering. Many pre-trained models are available in the literature, and models such as SBERT have been developed to generate embeddings for whole sentences or texts, relying on a special Siamese network architecture that has proven computationally faster and more agile than previous approaches [24].

These pre-trained models can be easily applied to datasets of abstracts from the literature, as a more accurate representation of the meaning of an abstract could potentially lead to better and more precise identification of similarities between published literature and a desired query, thus reducing the need for manual screening and making literature searches faster. The purpose of the current work is to retrospectively compare the performance of four freely available pre-trained models to identify a group of target papers in a dataset that was used for a previously published systematic review in implant dentistry.

2. Materials and Methods

2.1. Dataset

The generation of the dataset was conducted during a previously published systematic review by Donos et al. [25], and it has been described elsewhere. The purpose of the review was to identify Randomized Controlled Trials, Controlled Clinical Trials, and Prospective Case Series that addressed two Focused Questions (FQs):

FQ1. 

In patients with peri-implantitis, what is the efficacy of different bone reconstructive therapies compared to access flap surgery (AFS) in terms of pocket reduction, change in bleeding, and suppuration on probing (BOP and SOP), at a minimum of 12 months of follow-up?

FQ2. 

In patients with peri-implantitis, what is the long-term (≥12 months) performance of reconstructive therapies in terms of pocket reduction and change in BOP/SOP?

We chose this dataset because this corpus of articles had already been screened manually and 24 articles of interest (query target, QT) had already been identified for the systematic review, and we could use them as a measure of effectiveness for our investigation (

Table 1. List of the target articles that meet FQ1 requirements, identified by manual search.

ID	Authors	Title	Ref.
1	Andersen H., Aass AM. and Wohlfahrt, JC.	Porous titanium granules in the treatment of peri-implant osseous defects-a 7-year follow-up study	[26]
2	Jepsen K., Jepsen S., Laine, M. L., Anssari Moin D., Pilloni A., Zeza B., Sanz M., Ortiz-Vigon A., Roos-Jansaker AM., and Renvert S.	Reconstruction of Peri-implant Osseous Defects: A Multicenter Randomized Trial	[27]
3	Wohlfahrt JC., Lyngstadaas SP., Ronold HJ., Saxegaard EE., Jan Eirik KS., and Aass AM.	Porous titanium granules in the surgical treatment of peri-implant osseous defects: a randomized clinical trial	[28]
4	Emanuel N., Machtei EE., Reichart M., and Shapira, L.	D-PLEX500: a local biodegradable prolonged release doxycycline-formulated bone graft for the treatment for peri-implantitis. A randomized controlled clinical study	[29]
5	Renvert S., Giovannoli JL., Roos-Jansaker AM., and Rinke S.	Surgical treatment of peri-implantitis with or without a deproteinized bovine bone mineral and a native bilayer collagen membrane: A randomized clinical trial	[30]
6	Isehed C., Holmlun, A., Renvert S., Svenson B., Johansson I., and Lundberg P.	Effectiveness of enamel matrix derivative on the clinical and microbiological outcomes following surgical regenerative treatment of peri-implantitis. A randomized controlled trial	[31]
7	Isehed C., Svenson B., Lundberg P., and Holmlund A.	Surgical treatment of peri-implantitis using enamel matrix derivative, an RCT: 3- and 5-year follow-up	[32]
8	Renvert S., Roos-Jansaker AM., and Persson GR.	Surgical treatment of peri-implantitis lesions with or without the use of a bone substitute-a randomized clinical trial	[33]
9	Nct	Peri-implantitis-Reconstructive Surgical Therapy	[34]

and

Table 2. List of the target articles that meet FQ2 requirements, identified by manual search.

ID	Authors	Title	Ref.
1	Froum SJ., Froum SH., and Rosen PS.	A Regenerative Approach to the Successful Treatment of Peri-implantitis: A Consecutive Series of 170 Implants in 100 Patients with 2- to 10-Year Follow-up	[35]
2	Gonzalez Regueiro I., Martinez Rodriguez N., Barona Dorado C., Sanz-Sanchez I., Montero E., Ata-Ali J., Duarte F., and Martinez-Gonzalez JM.	Surgical approach combining implantoplasty and reconstructive therapy with locally delivered antibiotic in the treatment of peri-implantitis: A prospective clinical case series	[36]
3	Isler SC., Soysal F., Ceyhanli T., Bakirarar B., and Unsal B.	Regenerative surgical treatment of peri-implantitis using either a collagen membrane or concentrated growth factor: A 12-month randomized clinical trial	[37]
4	La Monaca G., Pranno N., Annibali S., Cristalli MP., and Polimeni A.	Clinical and radiographic outcomes of a surgical reconstructive approach in the treatment of peri-implantitis lesions: A 5-year prospective case series	[38]
5	Mercado F., Hamlet S., and Ivanovski S.	Regenerative surgical therapy for peri-implantitis using deproteinized bovine bone mineral with 10% collagen, enamel matrix derivative and Doxycycline-A prospective 3-year cohort study	[39]
6	Polymeri A., Anssari-Moin D., van der Horst J., Wismeijer D., Laine ML., and Loos BG.	Surgical treatment of peri-implantitis defects with two different xenograft granules: A randomized clinical pilot study	[40]
7	Roccuzzo M., Gaudioso L., Lungo M., and Dalmasso P.	Surgical therapy of single peri-implantitis intrabony defects, by means of deproteinized bovine bone mineral with 10% collagen	[41]
8	Roccuzzo M., Mirra D., Pittoni D., Ramieri G., and Roccuzzo A.	Reconstructive treatment of peri-implantitis infrabony defects of various configurations: 5-year survival and success	[42]
9	Isrctn	Reconstructive surgical therapy of peri-implantitis bone defects	[43]
10	Aghazadeh A., Persson RG., and Renvert S.	A single-centre randomized controlled clinical trial on the adjunct treatment of intra-bony defects with autogenous bone or a xenograft: results after 12 months	[44]
11	Aghazadeh A., Persson, RG., and Renvert S.	Impact of bone defect morphology on the outcome of reconstructive treatment of peri-implantitis	[45]
12	Nct	Evaluation of Photodynamic Therapy in Treatment of Peri-implantitis	[46]
13	Roos-Jansaker AM., Renvert H., Lindahl C., and Renvert S.	Submerged healing following surgical treatment of peri-implantitis: a case series	[47]
14	Roos-Jansaker AM., Lindahl C., Persson RG., and Renvert S.	Long-term stability of surgical bone regenerative procedures of peri-implantitis lesions in a prospective case-control study over 3 years	[48]
15	Roos-Jansaker AM., Persson RG., Lindahl C., and Renvert S.	Surgical treatment of peri-implantitis using a bone substitute with or without a resorbable membrane: a 5-year follow-up	[49]

). The QT corpus comprised two subsets, namely the articles that addressed FQ1 (Table 1) and the articles that addressed FQ2 (Table 2).

Briefly, to identify relevant articles, Donos et al. implemented a search strategy using terms related to peri-implantitis. Three major databases were included in the search: MEDLINE via OVID, EMBASE, and The Cochrane Database (including CENTRAL), and the search was updated until 10 April 2022. The result of the search consisted of 6277 records that were saved as a .csv file. The abstracts for these articles can be found as Appendix A.

2.2. Data Pre-Processing

Data were pre-processed and analyzed using both Jupyter notebooks version 6.5.2 [50] in the Anaconda distribution running Python 3.10.10 on a local laptop computer [51] and, in parallel, remotely using the Google Colab Pro cloud-based development environment, accelerated with the NVIDIA T4 Tensor Core GPU. The corpus was pre-processed by removing numbers and lowercasing all titles; stopwords, however, i.e., common grammatical words that are semantically empty [52], were not removed, to maximize the capability of transformers to create contextual embeddings. Records without an abstract (n = 167) were removed. The final corpus contained 6110 articles. Data plotting was conducted using the Matplotlib [53] and Seaborn libraries [54].

2.3. Models

Four pre-trained sentence-transformer models were used to generate sentence embeddings. All models were freely available on Huggingface.com.

(1)

sentence-transformers/all-MiniLM-L6-v2. This model is based on the nreimers/MiniLM-L6-H384-uncased model and was further fine-tuned using a dataset of 1 billion sentence pairs. The embeddings’ length is 384. (https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 (accessed on 15 January 2024))

(2)

sentence-transformers/all-MiniLM-L12-v2. This model is based on the microsoft/MiniLM-L12-H384-uncased model and was further fine-tuned using a dataset of 1 billion sentence pairs. The embeddings’ length is 384. (https://huggingface.co/sentence-transformers/all-MiniLM-L12-v2 (accessed on 15 January 2024))

(3)

sentence-transformers/all-mpnet-base-v2. The model underwent pretraining on the microsoft/mpnet-base model and was subsequently fine-tuned on a dataset consisting of 1 billion sentence pairs. The embeddings’ length is 768. (https://huggingface.co/sentence-transformers/all-mpnet-base-v2 (accessed on 15 January 2024))

(4)

sentence-transformers/all-distilroberta-v1. The model underwent pretraining on the distilroberta-base model and was subsequently fine-tuned on a dataset consisting of 1 billion sentence pairs. The embeddings’ length is 768. (https://huggingface.co/sentence-transformers/all-distilroberta-v1 (accessed on 15 January 2024))

2.4. Sentence Encoding

The usage of a pre-trained model (model_name in our example) for sentence transformers is straightforward, thanks to the SentenceTransformer library, and follows the general scheme:

                  from sentence_transformers import SentenceTransformer
      model = SentenceTransformer(model_name)
embeddings = model.encode(sentences)

where ‘sentences’ is the text to encode, the abstracts in our case. We imported the .csv corpus as a pandas Dataframe, with the following structure:

• Column name
• authors      6089 non-null object
• title             6109 non-null object
• journal       6106 non-null object
• abstract      6110 non-null object
• year            6110 non-null int64
• volume      5623 non-null object
• issue          4984 non-null float64
• pages         5426 non-null object

The ‘abstract’ column was pre-processed as described, and the encoding method was run iteratively on it to generate sentence embeddings for every abstract and each model.

2.5. Semantic Text Similarity and Semantic Search

To calculate the semantic similarity across embeddings generated by the Sentence transformer model, cosine similarity was employed. This similarity metric measures the cosine of the angle between two vectors (regardless of their dimensionality), indicating how similar or related they are in terms of their meaning. To calculate cosine similarity on our dataset, the ‘util’ tool was used. The general usage of this library is as follows:

from sentence_transformers import util

util.cos_sim(embedding1, embedding2)

Similarity was calculated both across the QT corpus (constituted by the 24 target articles only) and between each of the FQs and the whole corpus of 6110 abstracts.

3. Results

3.1. Sentence Encoding

Sentence encoding proceeded seamlessly, but the computing time varied according to the model used. The computer we employed for this experiment was a mid-tier commercial laptop, as could be used by a scholar for a systematic review, and it was equipped with Intel(R) Core (TM) i5-10210U CPU 1.60 GHz processor and 16 Gb RAM, without dedicated GPU and running Windows 10.

This configuration took approximately 3 to 6 s to calculate the embeddings for the abstracts of the corpus when using the all-MiniLM-L6-v2 model (depending on the abstract’s length), about 4 to 7 s when using the all-MiniLM-L12-v2 model, and up to about 30–50 s when using the all-mpnet-base-v2 model, whose embeddings are twice as long as the ones of the first two models. This makes the total computational time considerably longer for a corpus of 6110 references and scarcely manageable, with several hours to days of computer work necessary to encode it all. The same procedure was repeated using the free online tool Google Colab, a cloud-based Jupyter notebook environment, which offers the choice of running code on CPU or T4 GPU architectures, and this considerably reduced runtime (

Table 3. Processing time for sentence encoding using different models.

	all-MiniLM-L6-v2	all-MiniLM-L12-v2	all-Mpnet-Base-v2	all-Distilroberta-v1
Local computer (no GPU)	3–6	4–7	30–50	16–30
Cloud environment (CPU)	0.16	0.12	1.4	0.75
Cloud environment (GPU)	0.01	0.02	0.03	0.02

).

3.2. Semantic Text Similarity in the QT Corpus

Figure 1 shows the cosine similarity for the abstracts that belong to the QT corpus after encoding with the smaller all-MiniLM-L6-v2 pre-trained model. The similarity matrix ranges from 0.59 to 0.9, with a mean of 0.73 (and median of 0.72) for the FQ1 subset, and ranges from 0.55 to 0.99, with a mean of 0.73 (and median of 0.72) for the FQ2 subset. The distribution of the similarity values is close to normality for both subsets (Figure 1, bottom), with skewness of 0.8 and 0.45, respectively. Figure 2 reports the cosine similarity for the abstracts of the two subsets of the QT corpus after encoding with the all-MiniLM-L12-v2 pre-trained model. The similarity score ranges from 0.60 to 0.99, with a mean of 0.71 and median of 0.69 for the FQ1 subset, and ranges from 0.55 to 0.99, with a mean of 0.74 and median of 0.74 for the FQ2 subset. The distribution of the cosine similarity values for the embeddings obtained with the all-MiniLM-L12-v2 model approximates a normal distribution for the FQ2 subset (skewness = 0.17) and is slightly right-skewed for the FQ1 subset (skewness = 1.17). Figure 3 reports the cosine similarity for the abstracts of the two subsets of the QT corpus, after encoding with the larger all-mpnet-base-v2 pre-trained model. The similarity score ranges from 0.66 to 0.99, with a mean of 0.77 and median of 0.75 for the FQ1 subset, and ranges from 0.69 to 0.99, with a mean of 0.80 and median of 0.79 for the FQ2 subset. Overall, the cosine similarity of the embeddings obtained with the all-mpnet-base-v2 model approximates a normal distribution, though slightly right-skewed (skewness of 0.98 and 0.90, respectively), and is significantly higher than that obtained with the embeddings from the all-MiniLM-L6-v2 model.

Figure 4 shows the cosine similarity for the abstracts of the two subsets of the QT corpus after encoding with the all-distilroberta-v1 model. This similarity matrix ranges from 0.62 to 0.99, with a mean of 0.76 (and median 0.74) for the FQ1 subset and ranges from 0.55 to 0.99, with a mean of 0.78 (and median 0.80) for the FQ2 subset; the similarity values for the QT abstracts are visibly (and significantly) higher with the models “all-mpnet-base-v2” and “all-distilroberta-v1” than with the previous two models. The skewness of the distribution is 0.86 for the FQ1 subset and 0.44 for the FQ2 subset.

3.3. Semantic Text Similarity of the Whole Dataset to FQs

We then proceeded to encode FQ1 and FQ2 using the four models and computed the cosine similarity of these two queries with the whole corpus of 6110 abstracts. Figure 5 reports the distribution of the similarity score for (A) FQ1 and (B) FQ2 vs. the whole corpus, using the all-MiniLM-L6-v2 model. Overall, though both distributions are roughly normal, the abstracts of the corpus appear to have a higher similarity to FQ1 than to FQ2 (mean 0.41 vs. 0.37). Unsurprisingly, the range of similarities is wider than among the QT corpus, as the whole corpus is expected to include very diverse research papers, which may be only marginally related to the focused questions. Figure 6 reports the distribution of the similarity score for (A) FQ1 and (B) FQ2 vs. the whole corpus, using the all-MiniLM-L12-v2 model. Overall, though both distributions are roughly normal, the cosine similarity values of the abstracts of the corpus to FQ1 are higher than those to FQ2 (mean 0.41 vs. 0.37).

Figure 7 represents the distribution of the similarity score for (A) FQ1 and (B) FQ2 vs. the whole corpus, using the all-mpnet-base-v2 model. The distribution is left-skewed for both (A) FQ1 and (B) FQ2 vs. the total corpus. When compared to the first model, the similarity score is higher, with a mean of 0.43 for FQ1 (vs. 0.41 with all-MiniLM-L6-v2) and 0.44 for FQ2 (vs. 0.37), though the standard deviation is also slightly higher. This latter distribution appears to have lower kurtosis, which can be envisaged as a pointier peak and slimmer tails than the former model.

Figure 8 shows the distribution of the similarity score for (A) FQ1 and (B) FQ2 vs. the whole corpus, using the all-distilroberta-v1 model. The distribution is approximately normal and differs slightly from the preceding models because its mean (0.40 and 0.42) and its standard deviation are about the same (around 0.12 for both focused questions). To better understand how effective these models were in identifying our articles of interest, we ordered all the abstracts by cosine similarity, from those with the highest similarity to the lowest, and plotted the rank of the articles of interest. Ideally, the abstracts of the QT corpus should rank in the top positions.

3.4. Semantic Query

Figure 9 shows the rank of the target articles from the two subsets when ordered by semantic similarity to FQ1 (left column) or FQ2 (right column). When the all-MiniLM-L6-v2 model is used (Figure 9, first row) and the corpus is searched using FQ1, all target articles to FQ1 are contained in the top 844 papers, although the median rank is 128. Overall, FQ2 performs more poorly, as the median rank of the FQ2 target papers is 507, and they are all to be found in the top 1149 articles out of the total corpus. When the all-MiniLM-L12-v2 model is used (Figure 9, second row), the FQ1 target articles are contained in the first 292 papers, and the median rank is 56. Again, FQ2 tends to perform worse than FQ1, as the median rank of the QT papers is 140, though they are all to be found in the top 829 articles out of the overall corpus. When the all-mpnet-base-v2 model is used (Figure 9, third row), all the articles of the FQ1 corpus are contained in the first 257 papers, although the median rank is 79. FQ2 tends to perform slightly worse than FQ1, as the median rank of the QT papers is 73, though they are all to be found in the top 685 articles out of the overall corpus. When the All-distilroberta-v1 model is used (Figure 9, bottom row), the results are consistent but not better than with the previous models. All the articles of interest to FQ1 are contained in the top 781 papers, with a median of 130. FQ2 tended to perform similarly to FQ1, with a median rank for the FQ2 papers of 43, but they were all to be found in the top 2271 articles out of the overall corpus.

4. Discussion

Transformers have emerged as a groundbreaking tool in the field of NLP, revolutionizing the ability to develop algorithms that can comprehend and generate natural language, surpassing previous benchmark performances [55]. The considerable power of transformers enables them to build effective commercial products capable of sifting through scientific literature and autonomously extracting valuable knowledge and information [56,57]. This holds tremendous value, particularly in Life Sciences, where the vast volume of publications frequently exceeds an individual’s capacity or necessitates a team of experts investing significant time combing through databases for relevant insights [3]. Finding the papers of interest, discarding the background information noise, extracting useful information, and summarizing it are all steps that might soon be demanded by Artificial Intelligence [58,59]. To do that, however, it is necessary for users to improve their digital proficiency and know the strengths and limits of the models they are using.

Transformers can generate word and sentence embeddings in a highly efficient manner by incorporating contextual information, which enriches their content and subsequently improves their performance [55]. These embeddings capture the underlying semantic information of sentences, allowing for accurate comparison and similarity calculations. Additionally, transformers can be trained on extensive textual datasets, with pre-trained models readily available online to researchers and developers.

The objective of this report was to assess the effectiveness of four transformer models, particularly Sentence Transformers [24], in extracting and comparing semantic similarities between abstracts to assist scholars in selecting articles for systematic reviews. To do that, we retrospectively used data from a previously published systematic review, where clinical studies were identified that could help answer two focused questions, which were labeled as FQ1 and FQ2. The investigators who performed the systematic review had to manually scan a corpus of more than 6000 articles and extract 24 reports, mostly RCTs, that addressed the FQs; more specifically, 9 studies addressed FQ1 and 15 studies addressed FQ2. Therefore, besides possessing the whole dataset, we also had the desired corpus of documents that we wanted to identify, and this allowed us to easily evaluate the performance of our system.

Ideally, instead of going through all the 6110 articles one by one, it would have been preferable to possess an algorithm that could select those 24 articles automatically, or at least to identify a smaller subset of papers where these articles were more likely to be contained, which could relieve part of the scholars’ burden. This latter task can be defined as a semantic search task, i.e., given a document, in this case, the FQs, we wanted to find the closest articles in the whole corpus. To assess the similarity between the two documents, cosine similarity was employed as a metric [60]. Abstracts were chosen among other features, e.g., instead of titles, because they contained the highest amount of information, although we knew that their length could significantly slow down computing. We then decided to use four models, choosing them from those freely available in the Huggingface repository. Two models (all-MiniLM-L6-v2 and all-MiniLM-L12-v2) were smaller and encoded 384-long embeddings, which translates into a faster deployment. The other two models, by contrast (all-mpnet-base-v2 and all-distilroberta-v1) encoded longer, 768-dimensional embeddings. Longer embeddings have the potential to contain richer and more nuanced semantic representations and could therefore perform better, though their computation speed is obviously slower. Huggingface reports an encoding speed of 14,200 sentences/s for the all-MiniLM-L6-v2 model, 7500 sentences/s for the all-MiniLM-L12-v2 model, and 2800 sentences/s for the larger all-mpnet-base-v2. Noticeably, these speed measurements were recorded on computers equipped with NVIDIA Tesla V100 GPUs, which are among the best-performing GPUs on the current market and remarkably faster than all the systems we decided to employ to try and recreate standard working conditions of a scholar in the Life Science area. All these four models had been trained with very large corpora of various origins. The deployment of these models was made very simple by the SentenceTransformer library, which required a few command-line commands to start encoding and very little coding knowledge to be used. In addition, no specific software was required; besides, at the very minimum, the (freely available) Python interpreter of choice, or, alternatively, a cloud-based development environment [61], the abstract text was also minimally pre-processed. Numbers were removed, and the text was lower-cased, to make words more homogeneous, but no word was removed, including stopwords, i.e., very frequent words that have a purely grammatical or functional meaning (e.g., the conjunction ‘and’ or articles such as ‘the’) and are not usually considered to contain the ‘gist’ of the sentence [62]. Guidelines for older algorithms recommend removing stopwords to improve the creation of embeddings [63], but transformers do not need stopwords to be removed. Their inner working relies on all the lexical elements of the sentence to get context information, and more recent approaches leave stopwords untouched [64,65].

By using sentence transformers, we were able to encode all the abstracts in the dataset, calculate the cosine similarity of the embeddings of these abstracts to the embeddings of the focused questions, and sort the most similar documents. We first compared the similarity between the 24 abstracts that constituted our target, which we called the Query Target corpus, differentiating between those articles that addressed FQ1 and those that addressed FQ2. We expected that a well-performing model could be able to understand that the articles of these two QT subcorpora were somehow related because they all concerned common topics that corresponded to FQ1 and FQ2 and we, therefore, expected a good model to yield high similarity scores in the QT corpus. In this regard, the all-mpnet-base-v2 and all-distilroberta-v1models performed the best because their mean similarity was 0.77 for FQ1, 0.8 for FQ2, and 0.76 for FQ1 and 0.78 for FQ2, respectively (Figure 3 and Figure 4), higher than with any other model we tested.

If we review our data, we notice that we generated interesting results even just using the faster all-MiniLM-L6-v2 model to encode the corpus and the FQs. If we had sorted all the 6110 abstracts of the corpus based on their semantic similarity to FQ1 and FQ2 and had decided to limit our manual search to the top 1000 articles (e.g., to save time instead of manually examining over 6000 abstracts), by using FQ1 as a query, we would have identified all the 9 FQ1 target articles, and using FQ2 as a query, we would have missed only 2 papers out of the 15 FQ2 target articles. If we had decided to use the all-MiniLM-L12-v2, which was about as fast as the previous one, and had limited our manual search to the top 1000 papers, this time we would not have missed any article of interest, either with FQ1 or FQ2. The semantic search for this model was so good that we could have limited our manual search to the top 800 papers, further restricting the number of articles that we would have had to go through, and we would have still been able to find all the FQ1 target papers while missing only 1 FQ2 target paper, using FQ1 and FQ2, respectively. Furthermore, if we only focused on the FQ1 target papers, the top 300 papers of the corpus would have been enough to identify them, which would have meant a significant saving in hands-on time for the reviewers. Using the larger, more massive (and slower) all-mpnet-base-v2 model yielded similarly good results. To identify all the FQ1 target articles, it would be enough to limit our search to the top 300 articles, while the top 700 articles would be enough to locate the FQ2 target articles (using their respective FQs as a query). Interestingly, if we search by FQ1, all the 24 articles of interest are in the top 400 papers (see Figure A1 in Appendix B), which means that FQ1 is closer to all the 24 QT articles, not only its own 9 target papers but also the 15 FQ2 target articles, and this raises a significant issue, i.e., the importance of the formulation of the Focused Questions.

Focused Questions are commonly generated according to established criteria, such as PICO, to contain the relevant terms that will be used for database searches. Usually, these relevant terms, or keywords, are extracted from the PICO query, isolated, and combined with Boolean operators to create complex search strategies to feed into the search engine of the database [66]. This is not necessarily the case with semantic searches using sentence transformers, because they do not rely just on the presence of keywords in the sentences, and future research should address the need to implement a new set of recommendations for queries to optimize semantic searches and improve cosine similarities with abstracts.

Another important issue is the choice of threshold to use. We mentioned how different models would allow us to limit our search to the top n fraction of articles out of the total corpus. We were able to state that because we knew our QT articles in advance, as per the design of our study. Under normal conditions, however, we would have the whole corpus we retrieved from the usual databases, and we would have the appropriate FQs, but we would not know how many target articles are present in the corpus and where they are, and it would be difficult to decide where to stop our search. We could, e.g., arbitrarily decide to limit our manual search to the top 1000 papers, which could be a reasonable number of articles to screen manually, but that would really go against most guidelines for systematic reviews, which have always stressed, among others, the need to expand the search range, include grey literature, and perform hand-searching of relevant literature and conference abstracts [67], and we could hardly justify such an arbitrary choice, which could likely lead to the loss of otherwise easily identifiable articles, just for the sake of time-saving! A preferable approach would be to have Artificial Intelligence segment the corpus in some way and identify subsets of articles that better match our FQs and can therefore contain our target. This procedure is commonly known in machine learning as clustering, and several algorithms are available to tackle this problem, both following supervised and unsupervised approaches [68]. Thus, another equally important line of research in the near future will be comparing the performance of clustering protocols to get the system to automatically identify a sensible proposal to limit our manual search based on the cosine similarities of the dataset to the available FQs.

The data we have so far indicates that sentence transformers are a viable approach that meets the requirements of scholars who desire to streamline their processing of literature searches. Unsurprisingly, the all-mpnet-base-v2 model emerges as the best-performing pre-trained model among those tested, and if faster computer set-ups are available, our data indicate that this model is preferable over the alternatives we have compared. However, its performance is only marginally better than that of faster models, such as all-MiniLM-L12-v2, which can encode sentences in about 1/10th of the time necessary to all-mpnet-base-v2, and this kind of performance can be of great advantage with corpora that contain tens of thousands of articles. Adopting this kind of model in the routine workflow could lead to abandoning one of the mainstays of literature searches for systematic reviews, i.e., defining complex search strategies to filter out undesired papers and avoid overinflating the burden for the reviewers, as already proposed by Hamel et al. [69]. A somewhat comforting-for-reviewers scenario could be envisaged, where a generic, broad, and wide-encompassing literature search is performed on the appropriate databases, which could retrieve dozens, if not hundreds, of thousands of articles with no need to restrict the search to a few thousand items to keep it manageable, and the whole burden of scanning it would be left to a semantic search algorithm, possibly combining it with keyword search to enhance results. Using both search methods, i.e., semantic searches and keyword-based searches, could enhance the search accuracy and precision, and thus its reliability, and future research will have to explore the possible synergy between these two search approaches. An even better scenario could be to be able to perform the database search using semantic search in the first place, as it is better equipped than keyword searches to find articles of interest, but that would require possessing embeddings for millions of items, which is still a lengthy procedure and still unfeasible until a consensus is reached on which model to use (which might not even happen at all, considering that models are continuously evolving), or until our computing capabilities have progressed to such an extent to make the encoding of a whole database so quick that it can be reasonably performed at the click of a button by the final users when doing their search.