Performance Assessment of SARS-CoV-2 PCR Assays Developed by WHO Referral Laboratories

Abstract

A reliable diagnostic assay is crucial to early detect new COVID-19 cases and limit severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) transmission. Since the onset of the COVID-19 pandemic, the World Health Organization has published several diagnostic molecular approaches developed by referral laboratories, including Charité (Germany), HKU (Hong Kong), China CDC (China), US CDC (United States), and Institut Pasteur, Paris (France). We aimed to compare the sensitivity and specificity of these different RT-PCR assays using SARS-CoV-2 cell culture supernatants and clinical respiratory samples. Overall, the different RT-PCR assays performed well for SARS-CoV-2 detection and were all specific except the N Charité (Germany), and N2 US CDC (United States) assays. RdRp Institut Pasteur (IP2, IP4), N China CDC, and N1 US CDC were found to be the most sensitive assays. The data presented herein are of prime importance to facilitate the equipment choice of diagnostic laboratories, as well as for the development of marketed tests.

1. Introduction

A new human coronavirus called severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) emerged in China in December 2019 [1]. SARS-CoV-2 is responsible for coronavirus disease 2019 (COVID-19), which was declared a pandemic on 12 March 2020 by the World Health Organization (WHO) [2]. As of 26 May 2020, 5,404,512 cases have been reported, including 343,514 deaths [3]. A reliable diagnostic assay is crucial to limit the spread of SARS-CoV-2 as early detection of new cases leads to patient isolation and contact tracing. The first SARS-CoV-2 genome was published on 10 January 2020 [4], enabling the rapid design of a real-time reverse-transcriptase polymerase chain reaction (RT-PCR) assay by Charité (Germany) [5,6]. This test was the first to be dispatched by WHO [7] and was widely implemented in clinical virology laboratories worldwide [8]. Since then, WHO has published [9] other approaches developed by referral laboratories, including HKU (Hong Kong) [10,11], China CDC (China) [12], US CDC (United States) [13], and Institut Pasteur, Paris (France) [14]. These assays included several RT-PCRs targeting two or three different SARS-CoV-2 gene regions, including RdRp (RNA-dependent RNA polymerase), N (nucleocapsid protein), E (envelope protein), ORF1ab nsp10 (non-structural protein 10), and ORF1b nsp14 (non-structural protein 14). In the present study, we aimed to compare the sensitivity and specificity of these different RT-PCR assays. Overall, the different RT-PCR assays performed well for SARS-CoV-2 detection and were all specific except N Charité, (Germany) and N2 US CDC (United States) assays. RdRp Institut Pasteur (IP2, IP4), N China CDC, and N1 US CDC were found to be the most sensitive assays.

2. Materials and Methods

2.1. Study Design

To assess sensitivity, different RNA concentrations of SARS-CoV-2 cell culture supernatants, as well as clinical samples with different viral loads (n = 4), were tested using all RT-PCR assays. For the three most sensitive assays, limit of detection (LoD) was assessed, and additional clinical samples (n = 16) with low viral loads were also tested. To assess specificity, clinical samples negative for SARS-CoV-2 (n = 50) were tested using all RT-PCR assays. All clinical samples (nasopharyngeal aspirates) were provided by the Hospices Civils de Lyon—University Hospital, France, and frozen at 80 °C before extraction.

2.2. Sensitivity

The sensitivity for each RT-PCR assay was first assessed using ten-fold serial dilutions from 10⁻³ to 10⁻⁹ of SARS-CoV-2 cell culture supernatants (one replicate for 10⁻³ and 10⁻⁴, three replicates for 10⁻⁵ and 10⁻⁶, and five replicates for 10⁻⁷ to 10⁻⁹). Four positive clinical samples were then tested using all RT-PCR assays to confirm these results.

For the three most sensitive assays, we estimated the LoD using a probit analysis by including five additional replicates of each dilution of the cell culture supernatants. Probit analysis consists of describing the relationship between the probability of detection and concentration using a cumulative probability curve. For each dilution, the ratio (the hit rate) is computed as the number of replicates with a detected outcome per the total number of replicates tested. These hit rates are converted mathematically into cumulative normal probability units (probits) and fitted using a regression model vs. their respective concentrations. The LoD is defined as the lowest amount of viral genome that can be detected with a 95% hit rate. For these three most sensitive assays, the results obtained with probit analysis were confirmed by additional testing of sixteen clinical samples with low viral concentration.

2.3. Specificity

The specificity for each RT-PCR assay was assessed using clinical samples (n = 50) tested negative for SARS-CoV-2, including clinical samples (n = 30) tested positive for other respiratory viruses: human coronaviruses 229E, OC43, HKU1, and NL63, human influenza A and B viruses, rhinovirus, respiratory syncytial virus, parainfluenzavirus, adenovirus, metapneumovirus, and picornavirus.

Exploration of false-positive results was performed with additional negative clinical samples, water, and one additional clinical sample tested positive for each target. Amplicon size was analyzed using Agilent DNA 1000 kit (Agilent Technologies, Santa Clara, CA, USA).

2.4. SARS-CoV-2 Cell Culture Supernatants

Cell culture supernatants were obtained from a positive clinical sample cultivated in a biosafety level 3 laboratory on buffalo green monkey cells (cell line provided by the Université Louis Pasteur, Strasbourg, France) [15]. The SARS-CoV-2 culture had an infectious titer of 8.27 log₁₀TCID₅₀/mL as assessed by the Reed and Muench statistical method [16].

2.5. Extraction and RT-PCR

RNA extraction was performed using the EMAG^® platform (bioMérieux, Marcy-l’Étoile, France), according to manufacturer’s instructions. RT-PCR was performed following published instructions [5,6,10,11,12,13,14], which are summarized in

Table 1. Summary of the five RT-PCR assays targeting severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2).

Country (Institute)	Target	Oligonucleotide	Sequence	Amplicon Size a	Polymerase	Thermocycler Used in the Reference Publication	Volume of RNA Extract
Charité (Germany) [5,6]	RdRp b	Charité_RdRp_F	GTGARATGGTCATGTGTGGCGG	100 bp	SuperScript™ III Platinum® One-Step Quantitative RT-PCR System	Light Cycler ® 480II (Roche) or Applied Biosystems ViiA™7 (Therom Fisher)	5 µL
		Charité_S_RdRp_P c	FAM-CAGGTGGAACCTCATCAGGAGATGC-BBQ
		Charité_NS_RdRp_P d	FAM-CCAGGTGGWACRTCATCMGGTGATGC-BBQ
		Charité_RdRp_R	CARATGTTAAASACACTATTAGCATA
	E e	Charité_E_F	ACAGGTACGTTAATAGTTAATAGCGT	113 bp
		Charité_E_P	FAM-ACACTAGCCATCCTTACTGCGCTTCG-BBQ
		Charité_E_R	ATATTGCAGCAGTACGCACACA
	N	Charité_N_F	CACATTGGCACCCGCAATC	128 bp
		Charité_N_P	FAM-ACTTCCTCAAGGAACAACATTGCCA-BBQ
		Charité_N_R	GAGGAACGAGAAGAGGCTTG
HKU (Hong Kong) [10,11]	ORF1b-nsp14 f	HKU_ORF_F	TGGGGYTTTACRGGTAACCT	132 bp	TaqMan Fast Virus Master mix	Applied Biosystems ViiA™7 (Therom Fisher)	4 µL
		HKU_ORF_P	FAM-TAGTTGTGATGCWATCATGACTAG-TAMRA
		HKU_ORF_R	AACRCGCTTAACAAAGCACTC
	N e	HKU_N_F	TAATCAGACAAGGAACTGATTA	110 bp
		HKU_N_P	FAM-GCAAATTGTGCAATTTGCGG-TAMRA
		HKU_N_R	CGAAGGTGTGACTTCCATG
China CDC (China) [12]	N	ChinaCDC_N_F	GGGGAACTTCTCCTGCTAGAAT	99 bp	Unspecified	Unspecified	Unspecified
		ChinaCDC_N_P	FAM-TTGCTGCTGCTTGACAGATT-TAMRA
		ChinaCDC_N_R	CAGACATTTTGCTCTCAAGCTG
	ORF1ab-nsp10	ChinaCDC_ORF_F	CCCTGTGGGTTTTACACTTAA	119 bp
		ChinaCDC_ORF_P	FAM-CCGTCTGCGGTATGTGGAAAGGTTATGG-BHQ1
		ChinaCDC_ORF_R	ACGATTGTGCATCAGCTGA
US CDC (United States) [13]	N1 c	USCDC_N1_F	GACCCCAAAATCAGCGAAAT	72 bp	TaqPath™ 1-Step RT-qPCR Master Mix, CG (Thermo Fisher)	Applied Biosystems™ 7500 Fast (Thermo Fisher)	5 µL
		USCDC_N1_P	FAM-ACCCCGCATTACGTTTGGTGGACC-BHQ1
		USCDC_N1_R	TCTGGTTACTGCCAGTTGAATCTG
	N2 c	USCDC_N2_F	TTACAAACATTGGCCGCAAA	67 bp
		USCDC_N2_P	FAM-ACAATTTGCCCCCAGCGCTTCAG-BHQ1
		USCDC_N2_R	GCGCGACATTCCGAAGAA
	N3 d	USCDC_N3_F	GGGAGCCTTGAATACACCAAAA	72 bp
		USCDC_N3_P	FAM-AYCACATTGGCACCCGCAATCCTG-BHQ1
		USCDC_N3_R	TGTAGCACGATTGCAGCATTG
Institut Pasteur, Paris (France) [14]	RdRp IP2 (Flo2)	Pasteur_IP2_F	ATGAGCTTAGTCCTGTTG	108 bp	SuperScript™ III Platinum® One-Step Quantitative RT-PCR System	LightCycler ® 480 (Roche)	5 µL
		Pasteur_IP2_P	HEX-AGATGTCTTGTGCTGCCGGTA-BHQ1
		Pasteur_IP2_R	CTCCCTTTGTTGTGTTGT
	RdRp IP4 (Flo4)	Pasteur_IP4_F	GGTAACTGGTATGATTTCG	107 bp
		Pasteur_IP4_P	FAM-TCATACAAACCACGCCAGG-BHQ1
		Pasteur_IP4_R	CTGGTCAAGGTTAATATAGG

^a Amplicon size in base-pairs (bp) deduced from BetaCoV/Wuhan-Hu-1/2019 sequence (GISAID |EPI ISL 402125). ^b Target used for confirmation and discrimination of SARS-CoV-2 and SARS-CoV. ^c Probe specific for SARS-CoV-2. ^d Probe detecting SARS-CoV-2, SARS-CoV, and bat-SARS-related CoVs. ^e Target used for screening. ^f Target used for confirmation.

;

Table 2. Amplification cycles of the five RT-PCR assays targeting SARS-CoV-2.

Institute (Country)	Charité (Germany) [5,6]			HKU (Hong Kong) [10,11]			China CDC (China) [12]			US CDC (United States) [13]			Institut Pasteur, Paris (France) [14]
Amplification Cycles	T °C	Time (min)	Number of Cycles	T °C	Time (min)	Number of Cycles	T°C	Time (min)	Number of Cycles	T °C	Time (min)	Number of Cycles	T °C	Time (min)	Number of Cycles
Uracil-N-glycosylase activation							Unspecified			25	02:00	1
Reverse transcription	55	10:00	1	50	05:00	1				50	15:00		55	20:00	1
RT inactivation/Enzyme activation	95	03:00		95	00:20					95	02:00		95	03:00
Denaturation	95	00:15	45	95	00:05	40				95	00:03	45	95	00:15	50
Annealing/Extending	58	00:30		60	00:30					55	00:30		58	00:30
Cooling													40	00:30	1

. Since the China CDC protocol does not specify polymerase, thermocycler, volume of nucleic acid extract, and amplification cycles, the same instructions as those for the HKU assay were applied. RdRp IP2 and IP4 assays from Institut Pasteur, Paris (France) can be multiplexed or used in simplex [14]. Preliminary comparison on SARS-CoV-2 cell culture supernatants found that RdRp IP4 performed better when used in multiplex, whereas IP2 was not significantly impacted (Supplementary Table S1). The CFX 96 Touch™ Real-Time PCR (Bio-Rad, Hercules, CA, USA) was used for all RT-PCR assays.

3. Results

3.1. Sensitivity Comparison of the Five RT-PCR Assays

The E Charité and N2 US CDC assays were positive for all specimens, including negative samples and negative controls (water). These false-positive results were explored (details below), but the sensitivity of these assays was not further assessed.

Sensitivity was first assessed using SARS-CoV-2 cell culture supernatants. Using both specific SARS-CoV-2 (S) and non-specific (NS; detecting SARS-CoV-2, SARS-CoV, and bat-SARS-related CoVs) RdRp Charité assays, all replicates of the 10⁻⁵ dilution (and inferior); 1/3 (S RdRp) and 3/3 (NS RdRp) of the 10⁻⁶ dilution replicates; and none of the 10⁻⁷, 10⁻⁸, and 10⁻⁹ dilutions (0/5) were detected. ORF1b and N HKU, and ORF1ab China CDC assays detected all replicates of dilutions inferior or equal to 10⁻⁶ and detected 4/5, 3/5, and 2/5 for 10⁻⁷ dilutions, respectively. None of these assays detected replicates of 10⁻⁸ (0/5) and 10⁻⁹ (0/5) dilutions. In contrast, N Charité, N China CDC, N1 and N3 US CDC, and duplex RdRp IP2/IP4 were positive for most replicates of the 10⁻⁷ (5/5, 5/5, 5/5, 4/5, 5/5 and 5/5, respectively) and 10⁻⁸ dilutions (3/5, 2/5, 4/5, 5/5, 3/5, 3/5, respectively; Figure 1). At 10⁻⁹ dilution, N China, N1 and N3 US CDC, and duplex RdRp IP2/IP4 assays were able to detect replicates (1/5, 1/5, 2/5, 3/5, 1/5, respectively).

The mean cycle threshold (Ct) values obtained for each assay were then compared for dilutions 10⁻⁵ to 10⁻⁸ (Figure 1, Supplementary Table S2). Since the accepted technical variability of RT-PCR is below 0.5 log₁₀, we considered a difference of 2 Ct as significant [17]. At 10⁻⁵ dilution, the lowest Ct value was 27.7 for RdRp IP4. No significant difference in Ct values (Ct ranged from 28.0 to 29.1) was reported with N1 and N3 US CDC, and RdRp IP2. A similar Ct profile was observed for these assays at 10⁻⁶ and 10⁻⁷ dilutions. At 10⁻⁸ dilution, only N Charité had significantly higher Ct values (41.0 vs. 36.7 to 39.0 for N China CDC, N1 and N3 US CDC, and duplex RdRp IP2/IP4).

Clinical samples (n = 4) were then tested using all RT-PCR assays to confirm the results obtained on SARS-CoV-2 cell culture supernatants (Supplementary Table S3). ORF1b and N HKU, ORF1ab and N China CDC, N1 and N3 US CDC, and RdRp IP2 and RdRp IP4 assays detected all four positive samples. S and NS RdRp, and N Charité assays did not detect the positive sample with the lowest viral concentration.

Taken together, N China CDC, N1 and N3 US CDC, as well as RdRp IP2 and IP4 were the most sensitive assays.

3.2. Limit of Detection for the Most Sensitive Assays

Due to a limited quantity of clinical sample available, only one target from each referral laboratory providing the most sensitive assays was tested: N China CDC, N1 US CDC, and RdRp IP2. The LoD of N3 US CDC was not determined as this assay is not specific for SARS-CoV-2 detection and was removed from the new version of the US CDC assay [13]. We chose to determine the LoD for RdRp IP2 and not IP4 as IP2 detected more replicates at the 10⁻⁹ dilution on cell culture supernatants.

The 95% hit rate obtained was 1.36 log₁₀TCID₅₀/mL [0.8; 3.09] for N China CDC, 0.44 log₁₀TCID₅₀/mL [0.05; 1.83] for N1 US CDC, and 0.63 log₁₀TCID₅₀/mL [0.25; 1.9] for RdRp IP2. The differences observed were not statistically significant. For these three assays, the results were confirmed by additional testing of clinical samples (Figure 2, Supplementary Table S4). N1 US CDC and RdRp IP2 had lower Ct values than N China CDC (Figure 2), but no significant differences (Ct difference < 2) were observed.

3.3. Specificity

No false-positive results were obtained on clinical samples that tested negative for SARS-CoV-2 and/or positive for other viruses than SARS-CoV-2, except for E Charité and N2 US CDC, which were positive for all specimens (Supplementary Table S5).

3.4. Exploration of E Charité and N2 US CDC False-Positive Results

Since E Charité and N2 US CDC assays were positive for all specimens and replicates, including negative samples and controls, false-positive results were further explored (Supplementary Figure S1).

For E Charité, negative samples showed two amplicons, one at 84 base pairs (bp) and one at 121 bp, whereas the positive sample only had one amplicon at 121 bp, which is close to the expected size of a specific amplification (Table 1). Thus, the false-positive amplification obtained using E Charité might be derived from a contamination (amplicon size at 121 bp) but could also be associated with an aspecific amplification (amplicon size at 84 bp). Using the N2 US CDC assay, negative samples showed one amplicon at 73 bp, which is close to the expected size of a specific amplification (Table 1). Thus, the false-positive amplification obtained using N2 US CDC might be due to a contamination. Sequencing of these amplicon products should be performed for further investigation.

4. Discussion

The present study compared the performance of five RT-PCR-based methods developed by referral laboratories. N China CDC, N1 US CDC, and RdRp IP2 and IP4 were found to be the most sensitive assays on SARS-CoV-2 cell culture supernatants and clinical respiratory samples. Vogels et al. compared the performance of SARS-CoV-2 PCR assays developed by the same referral laboratories, except those from Institut Pasteur. Using RNA-spiked mock samples, they found that ORF HKU was one of the most sensitive assays [18]. Herein, ORF HKU was more sensitive than RdRp Charité but slightly less sensitive than other assays, such as N1 US CDC or N China. Although RdRp Charité performed well for the lowest dilutions, it was nevertheless found to be less sensitive than others, a result in line with those of Vogels et al. [18]. It is worth noting that the Charité assay was the first to be published at the early stage of the pandemic [9] and has been widely used worldwide [8]. This assay was initially designed for the diagnosis of SARS-related CoVs and then optimized for SARS-CoV-2 detection [5]. Thanks to this assay, an important number of COVID-19 diagnoses were made, which contributed to limiting the spread of the outbreak. In line with the present results, it was reported that RdRp IP2 and RdRp IP4 sensitivity was similar when used in multiplex [14], suggesting that the Institut Pasteur assay should preferentially be used in multiplex. Of note, we did not apply the Ct cut-off values above, in which a sample would be considered negative, since such values were not provided in the protocols made available by the referral laboratories.

As previously reported [19], we identified probable primer contamination using N2 US CDC and E Charité, which prevented us from further evaluating their sensitivity and specificity. Except for these two assays, no false-positive results were observed when testing a wide range of respiratory viruses, a result consistent with previous studies [5,6,10,11,13,14]. Although not observed herein, the amplification of non-specific products for ORF1 and N China CDC, and N2 and N3 US CDC has also been reported [18].

The performance of other RT-PCR tests recently developed [7] should be explored in further studies. In addition, the quantification of SARS-CoV-2 could be performed to assess the effectiveness of potential treatments. The data presented herein are of prime importance to facilitate the equipment choice of all diagnostic laboratories, as well as for the development of marketed tests. Sensitive tests should be widely implemented to limit the spread of the current outbreak and prepare for the post-epidemic phase and future seasonal epidemics.