Next Article in Journal
Chronotropic Incompetence in Non-Hospitalized Patients with Post-COVID-19 Syndrome
Next Article in Special Issue
The Role of Heparin in COVID-19: An Update after Two Years of Pandemics
Previous Article in Journal
Predictive Modeling of Poor Outcome in Severe COVID-19: A Single-Center Observational Study Based on Clinical, Cytokine and Laboratory Profiles
Previous Article in Special Issue
Venous Thromboembolism in COVID-19 Compared to Non-COVID-19 Cohorts: A Systematic Review with Meta-Analysis
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
JCM
Volume 10
Issue 22
10.3390/jcm10225433
jcm-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Elderly Population with COVID-19 and the Accuracy of Clinical Scales and D-Dimer for Pulmonary Embolism: The OCTA-COVID Study

by
Maribel Quezada-Feijoo
1,2,*,†,
Mónica Ramos
1,2,†,
Isabel Lozano-Montoya
2,3,
Mónica Sarró
4,
Verónica Cabo Muiños
5,
Rocío Ayala
1,2,
Francisco J. Gómez-Pavón
2,3 and
Rocío Toro
6,7,†
1
Cardiology Departament, Hospital Central de la Cruz Roja, C/Reina Victoria, 24, 28003 Madrid, Spain
2
Facultad de Medicina, Universidad Alfonso X El Sabio, Avda. De la Universidad, 1, Villanueva de la Cañada, 28691 Madrid, Spain
3
Geriatric Departament, Hospital Central de la Cruz Roja, C/Reina Victoria, 24, 28003 Madrid, Spain
4
Radiology Departament, Hospital Central de La Cruz Roja, C/Reina Victoria, 24, 28003 Madrid, Spain
5
Biochemistry Laboratory, Hospital Central de la Cruz Roja, C/Reina Victoria, 24, 28003 Madrid, Spain
6
Biomedical Research and Innovation Institute of Cádiz (INiBICA), Research Unit, Puerta del Mar University Hospital, Av/Ana de Viya 21, 11009 Cádiz, Spain
7
Medicine Department, School of Medicine, Cádiz University, Edificio Andrés Segovia 30 Floor, C/Dr Marañón S/N, 21001 Cádiz, Spain
*
Author to whom correspondence should be addressed.
†
These authors contributed equally to this work.
J. Clin. Med. 2021, 10(22), 5433; https://doi.org/10.3390/jcm10225433
Submission received: 11 October 2021 / Revised: 16 November 2021 / Accepted: 17 November 2021 / Published: 20 November 2021
(This article belongs to the Special Issue COVID-19 and Venous Thromboembolism)
Browse Figures
Review Reports Versions Notes

Abstract

:
Background: Elderly COVID-19 patients have a high risk of pulmonary embolism (PE), but factors that predict PE are unknown in this population. This study assessed the Wells and revised Geneva scoring systems as predictors of PE and their relationships with D-dimer (DD) in this population. Methods: This was a longitudinal, observational study that included patients ≥75 years old with COVID-19 and suspected PE. The performances of the Wells score, revised Geneva score and DD levels were assessed. The combinations of the DD level and the clinical scales were evaluated using positive rules for higher specificity. Results: Among 305 patients included in the OCTA-COVID study cohort, 50 had suspected PE based on computed tomography pulmonary arteriography (CTPA), and the prevalence was 5.6%. The frequencies of PE in the low-, intermediate- and high-probability categories were 5.9%, 88.2% and 5.9% for the Geneva model and 35.3%, 58.8% and 5.9% for the Wells model, respectively. The DD median was higher in the PE group (4.33 mg/L; interquartile range (IQR) 2.40–7.17) than in the no PE group (1.39 mg/L; IQR 1.01–2.75) (p < 0.001). The area under the curve (AUC) for DD was 0.789 (0.652–0.927). After changing the cutoff point for DD to 4.33 mg/L, the specificity increased from 42.5% to 93.9%. Conclusions: The cutoff point DD > 4.33 mg/L has an increased specificity, which can discriminate false positives. The addition of the DD and the clinical probability scales increases the specificity and negative predictive value, which helps to avoid unnecessary invasive tests in this population.

Graphical Abstract

1. Introduction

The diagnosis of pulmonary embolism (PE) in older patients is challenging due to the associated comorbidity [1] and masked symptoms [2]. An increased risk of PE, which is a medical emergency with a high mortality rate, and the development of a hypercoagulable state have been widely known to be associated with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) disease 2019 (COVID-19) [3,4,5].
The preliminary diagnosis of PE begins with clinical suspicion established by a pretest probability using diagnostic scoring systems such as the Wells criteria or the Geneva scoring criteria [6,7]. These scales have been previously validated and may be useful to identify patients who have a low and intermediate probability of developing a PE.
To date, scales have been validated with a normal D-dimer (DD) level; however, this assessment is limited within the elderly population because as the age of the patient increases, the DD specificity decreases. Additionally, the plasma levels of DD increase as part of COVID-19 infection, making the use of these scales difficult. DD has been proven to be a significant predictor of increased mortality in patients infected by COVID-19 [8,9] but is not specific for the diagnosis of PE. There are also other markers, such as troponin and N-terminal pro-brain natriuretic peptide (NT-pro-BNP), that are increased in COVID-19 patients [10] and are also related to a worse prognosis [11]. For this reason, it is crucial to determine the cutoff level for DD in this population to improve the diagnosis of PE in frail and elderly populations, and this can add diagnostic value to the clinical scales for the requirements of tailoring imaging tests in highly suspicious PE cases [12].
We sought to explore the diagnostic accuracy and reproducibility of the Wells and Geneva clinical probability scales and their association with DD in the diagnosis of PE in elderly patients with COVID-19.

2. Materials and Methods

2.1. Study Population

A longitudinal, observational study was designed. Patients over 75 years of age hospitalized with COVID-19 with a clinical suspicion of PE were recruited from the Acute Geriatrics Unit between March and May 2020. This study belongs to the OCTA-COVID-19 cohort. Patients under 75 years of age, those with palliative needs, those diagnosed by the attending team and those who did not meet the diagnostic criteria for COVID-19 were excluded. Patients with a high suspicion of PE who could not undergo a computed tomography (CT) scan and those who declined to participate were also excluded.

2.2. Ethics Approvals

The study protocol was approved by the ethics committee (PI-4134) and was conducted in full compliance with the Declaration of Helsinki. All participants provided written informed consent. The collected data were appropriately made anonymous, and each patient was identified by a unique alphanumeric identification code.

2.3. Assessment of Clinical Probability

The Wells and revised Geneva scores were calculated to evaluate the probability of PE. Recruited patients were classified based on both scores into one of three categories. Based on the Wells scale [13,14], we considered low risk to be less than 2 points, moderate risk from 2 to 6 points and high risk over 6 points. Meanwhile, the Geneva scale [15] was scored as low probability for PE under 3 points, intermediate from 4 to 10 points and high probability over 11 points. A positive computed tomography pulmonary arteriography (CTPA) confirmed the presence of PE. Patients with a decreased renal filtration <30 mL/mL/min/1.73 m2 did not undergo the CT scan. In patients with renal filtration between 30 and 45 mL/min, a hydration protocol was carried out prior to the scan (Figure 1).
We considered terminal criteria as advanced organ disease, such as severe lung failure with irreversible damage, end-stage heart disease, advanced kidney disease without dialysis criteria or advanced treatment subsidiary, among others, according to The National Hospice Organization.

2.4. Data Collection

The biodemographic data and clinical characteristics were collected prospectively from patients with a clinical suspicion of PE. The clinical signs that were assessed included heart rate, breathing rate, oxygen saturation, pain in the deep vein of the lower limb during palpation and unilateral edema. The risk factors that were considered included atrial fibrillation, deep vein thrombosis (DVT) or PE, cancer, bed rest for more than 3 days, newly confirmed DVT events and the presence of associated arterial ischemia. Data regarding clinical complications were collected during hospitalization. The CURB-65 scale, as recommended by the British Thoracic Society [16], was applied to stratify the severity of pneumonia. The degree of frailty was assessed by the Clinical Frailty Scale (CFS) two weeks prior to hospitalization [17]. The degree of dependence was calculated using the Barthel Scale [18], and the presence of dementia was assessed by the Global Deterioration Scale (GDS) [19].

2.5. Laboratory Procedures

SARS-CoV-2 detection was performed using real-time reverse transcriptase-polymerase chain reaction on nasal swabs. The routine blood examinations that were performed included a complete blood count and serum biochemical tests. The determination of plasmatic DD was carried out using the VIDAS® DD technique (bioMérieux, Lyon, France), a sandwich-type immuno-enzymatic method in 2 stages, with a final detection by the enzyme-linked fluorescence assay (ELFA). The concentration was expressed in micrograms per milliliter of fibrinogen equivalent units. We included the peak DD value, either from the beginning of admission or during the course of hospitalization. Cardiac biomarkers were measured in patients who were suspected of having cardiac involvement. NT-ProBNP was analyzed using a VIDAS analyzer (bioMérieux, Lyon, France). Troponin T (TnT) levels were measured using a (DADE Stratus®, Roche Diagnostics, Switzerland), fluorometric enzyme immunoassay analyzer with specific reagents and calibrators.

2.6. Definitions

The DD value was adjusted based on the patient’s age following the current PE guidelines [20] and was considered elevated when it was above 1 mg/L. NT-proBNP was considered elevated when it was above 450 pg/mL in patients with a normal sinus rhythm and when it was above 1100 pg/mL in patients with atrial fibrillation [21]. Acute myocardial damage was considered in patients with TnT levels above the 99th percentile of the upper limit of normal (50 ng/L) [22].

2.7. Statistical Analysis

Continuous variables are summarized as the median and interquartile range (IQR), and categorical data are summarized as frequencies and percentages. For comparisons, the Mann–Whitney U test was used because of the non-normal distribution of the continuous data. Categorical data were compared using the chi-square test or Fisher’s test, based on the expected counts. Ordinal data were compared with the Cochran-Armitage trend test.
The performances of the Wells score, the revised Geneva score and DD as diagnostic predictors of PE were assessed. The area under the receiver operating characteristic (ROC) curve was estimated, and a cutoff value was calculated using Youden’s index. The sensitivity, specificity and predictive values and their 95% confidence limits were calculated. The combinations of DD and the clinical scales were performed using both positive rules for higher specificity. All analyses were performed with SAS® 9.4 (SAS Institute Inc., Cary, NC, USA). A p-value ≤ 0.05 was considered statistically significant.

3. Results

3.1. Participant Characteristics

The OCTA-COVID study included 305 patients admitted for COVID-19 pneumonia, and 50 patients who had suspected PE based on CTPA were included in our study. Seventeen patients were confirmed to have a diagnosis of PE, and the mean hospitalization stay was 14.5 days (IQR 11–21). Thus, the prevalence of PE in the global population was 5.6%. In our cohort, 70% of the patients were previously treated with heparin prophylaxis, and 24% were treated with full doses of anticoagulation. Regarding the distribution of PE, 11 patients (64.7%) had peripheral PE, 5 patients (29.4%) had central and peripheral PE and only 1 patient (5.9%) had central PE. Segmental artery involvement was seen in 8 patients (50%), subsegmental arteries were seen in 2 patients (12%) and 37.5% of the patients had both segmental and subsegmental involvement.
The anthropometric, clinical and geriatric characteristics of all patients with suspected PE related to COVID-19 infection are shown in Table 1.
While the patients with PE were more obese (p = 0.046), no difference was noted as related to clinical antecedents, such as previous PE, DVT, trauma or palliative cancer treatment. Previous oncological diseases were noted in 35.3% of the patients with PE, with a trend toward statistical significance between the groups (p = 0.07).
Based on their clinical situation, patients diagnosed with PE presented with a significantly higher heart rate of 96 beats per minute than non-PE patients (p = 0.015). Asthenia was more prevalent as a clinical symptom in patients without PE, with a statistically significant difference (p = 0.052).
Regarding the biomarkers, the DD median was higher in the PE group, with a statistically significant difference (4.3 mg/L; IQR 2.40–7.17 vs. 1.3 mg/L; IQR 1.01–2.75; p < 0.001) Table 2.
Considering the biomarker evolution during hospitalization in our elderly cohort, the DD level remained elevated until the fifth day of infection in patients who presented with PE, while the non-PE cohort showed a gradually decreased DD level. A high value remained in the PE group on the third day, with a statistically significant difference from the group without PE (3.5 vs. 1.2 mg/L; p < 0.006). Likewise, the C-reactive protein (CRP) level was increased in the PE group on the third day but decreased gradually starting on the first day in the non-PE group (Figure 2a,b).

3.2. Predictive Values of the Wells Score and the Revised Geneva Score in the Elderly Group with COVID-19 and Pulmonary Embolism

The frequencies of PE in the low-, intermediate- and high-probability categories were 5.9%, 88.2% and 5.9% for the Geneva model and 35.3%, 58.8% and 5.9% for the Wells scale, respectively. The average score of the Wells scale in the PE cohort was higher than that in the group without PE (3 (IQR 1.5–3) vs. 1.5 (1.5–2.5); p = 0.06), but the difference was not statistically significant. In contrast, the median Geneva scale score was significantly higher in the PE group than in the non-PE group (6 (6–8) vs. 4 (2.5–6); p = 0.005).
To identify a good discrimination of PE in our elderly population, the cutoff point that maximized sensitivity and specificity on the Wells scale was 2.5 and that on the Geneva scale was 5. After taking these values into account, in older people infected with COVID-19, the positive predictive value (PPV) of the Wells scale was similar to that of the Geneva scale (55% vs. 51%). Regarding diagnostic accuracy, the sensitivity of the Geneva scale performed better than the Wells score in the probability of detecting PE, and the area under the curve (AUC) for the Geneva scale was better (Figure 3a) than that for the Wells score (Figure 3b).

3.3. DD and Clinical Score for the Geriatric Population with PE and COVID-19

A cutoff value of 4.3 mg/L in patients over 75 years of age was calculated to categorize the patients with and without PE, and this optimal value showed an intermediate sensitivity of 52.9% and a high specificity of 93.9% (80.4–98.3%) (Figure 4). The cutoff point adjustment for the DD level from 1.0 to 4.3 mg/L resulted in a significant rise in the specificity (30.3% to 93.9%) but reduced the sensitivity (Table 3). Moreover, the combination of a 4.3 mg/L DD cutoff point with several variables to determine the pretest probability of PE, such as the value of 2.5 on the Wells scale and 5 as intermediate risk on the Geneva scale indicating intermediate risk (both obtained by ROC curves), led to improvements in the PPV and the negative predictive value (NPV) (Table 4).

4. Discussion

We demonstrated a novel DD cutoff value for the oldest COVID-19 patients to rule out PE. To date, no study has been carried out in extremely geriatric patients to improve the diagnosis of PE in this population. For the first time, we showed that increased DD values and clinical rules can help improve the diagnosis of PE in this extremely old population.
The diagnosis of PE in elderly patients with COVID-19 is challenging, and several situations complicate the diagnosis: (i) age, (ii) the initiation of anticoagulant treatment, (iii) infection by SARS-CoV-2 that causes a prothrombotic effect with a high incidence of embolic phenomena and excessive DD levels and (iv) the lack of symptoms. Therefore, the usefulness of this isolated biomarker with a new cutoff point can be very helpful in clinical practice for the initial diagnosis of PE [23].
The combination of clinical criteria and DD to determine the probability of PE in a general population using several predictive rules and a scheme of three categories based on the clinical probability has been previously studied, but this kind of study has not been performed in a COVID-19 cohort [12,13]. In COVID-19, a hypercoagulable status with impaired DD levels affects the pretest probability diagnosis, making this algorithm difficult to use for follow-up. CTPA is the gold standard [24] technique for diagnosing PE, but during the first stage of the COVID-19 pandemic, its use was limited by several factors, such as the isolation of patients, the risk of viral spread and the significant number of hospitalized patients [25].
Thus, elevated DD levels are a sign of excessive activation of coagulation and hyperfibrinolysis [26]. In our cohort, DD levels over 4.33 mg/L to detect PE in elderly COVID-19 patients showed a sensitivity of 52.9%. Compared with other series of non-COVID-19 patients, this sensitivity is low, and when the DD value is negative, the sensitivity of this biomarker is over 95%, especially in low- and intermediate-risk patients [27,28,29]. However, it must be emphasized that 96% of our population was elderly, had COVID-19 and had previous treatment with heparin, and there is a decrease in the DD value of approximately 25% within the first 24 h after starting heparin. All these factors may have influenced the lower sensitivity (from 95.5% to 89.4%), as has been shown [30]. However, the specificity of DD in our cohort was high (93.9%), with a PPV and NPV that were not negligible (81.8% and 79.5%, respectively). It is known that the specificity of DD decreases as the age of the patient increases; however, when the cutoff point is adjusted, this specificity increases, as has been shown in other studies in non-COVID patients [31,32]. An increased DD cutoff value will increase the specificity, which will reduce the number of false positives. This may be crucial in clinical practice in two important aspects: (i) when referring a frail, elderly patient with multiple comorbidities to receive CTPA, which would reduce the risk of contrast nephropathy and contagion as well as the medical costs during the pandemic, and (ii) these preliminary data may suggest the necessity for early anticoagulant treatment in the appropriate patients.
Similar studies in non-COVID-19 populations have shown the importance of increasing the DD value to minimize unnecessary examinations without influencing the efficacy of DD to rule out PE [33]. Likewise, according to our results, a low sensitivity of 65%, specificity of 96.7%, PPV of 86% and NPV of 89.4% have been reported in COVID-19 patients with DDs above 3.5 mg/L [34], but when the cutoff point is increased to >5 mg/L, the predictive value is 75.5% [35].
There is much controversy regarding standard prophylaxis or anticoagulation treatment in hospitalized patients with COVID-19. Today, anticoagulants probably reduce thromboembolic events compared to prophylactic doses, but at the expense of a higher risk of bleeding. An isolated DD would not be an indicator for anticoagulation if the patient is at low or moderate risk. However, if this value added to a high preclinical risk of PE and if imaging techniques were not available, the patient would benefit from anticoagulant treatment. It is important to take into account the embolic risk vs. the hemorrhagic risk, since we are facing a very old, frail population with many comorbidities. To date, there are no validated venous and hemorrhagic embolic risk scales in COVID-19.
Some studies have demonstrated the usefulness of both scales and have confirmed the diagnostic performance of the Wells and Geneva scales [14,15] to estimate the pretest probability [36]. There was no significant difference between the Wells and Geneva rules when different pretest probability scales for PE in elderly COVID-19 patients were compared, although the Geneva scale had a greater sensitivity (82.4% vs. 64.7%) with an NPV of 87%.
Chagnon et al. [37] described a similar proportion of patients with low, intermediate and high risks based on the pretest probability for PE and by using the same two scales in patients from the emergency room. They concluded that the Geneva rule was more reliable when the clinical data were added. In contrast, our results showed a higher proportion of patients with moderate risk on the Geneva scale than on the Wells scale (82.2% vs. 58.8%). In the revised Geneva score, the high prevalence of patients in the intermediate probability range may be related to novel items, including age (all of our patients were older than 75 years) and heart rate of 75–94 beats per minute, which represents 3 points. Our cutoff was an average HR of 88 bpm, unlike the Wells scale, which values HR > 110 bpm as an item.
In our study cohort, the superiority of the modified Geneva scale in elderly patients with COVID-19 was proven, which is in disagreement with studies that evaluated elderly non-COVID-19 patients. Furthermore, Di Marca et al. reported that the Wells scale demonstrated a better discrimination index than the Geneva scale [7]. However, their study is not comparable to ours due to the features of the population recruited and the study methodology, where only patients in the high-probability risk group were referred for a CTPA. If the group had a moderate risk, the DD value was assessed and was only considered elevated when it was greater than 0.5 mg/L, as recommended by the European Society of Cardiology guidelines.
A meta-analysis [38] concluded that when the cutoff point of the Wells scale is less than 2, its sensitivity is greater, and when the cutoff point is 4, the sensitivity is reduced to 60% with a high specificity. Thus, the lower the clinical probability, the higher the Wells score sensitivity. These data are in line with our findings: the cutoff point of the Wells rule was over 2.5, and the specificity increased to exclude low- and moderate-risk cases, which influenced the sensitivity. Thus, intermediate case percentages are a flaw for these scales and need to be supported by an increased DD level. Clinical rules provide a tough basis that, in addition to DD, turn out to be a multiparametric tool that tailors the decision to perform a CTPA. According to a recently published study that evaluated COVID-19 patients [39], the different DD cutoff points associated with the Wells scale were valued over 2 points, demonstrating that the specificity of the Wells scale is higher when a DD cutoff point over 3000 ng/mL is added, at the cost of a low sensitivity of 57.1%.
Current guidelines show that a high pretest probability leads to an excessive use of CTPA, and using a more specific Wells score could safely reduce the number of unnecessary invasive tests. Herein, we proposed a combination of the Wells and Geneva rules that are associated with increased DD levels to accurately diagnose PE in an elderly COVID-19 population, which shows an improvement in the specificity of both rules to 96.8% and 93.9%, respectively. This clinical strategy was based on the association of a DD cutoff point with a Wells scale score of less than 2.5 and a Geneva scale score of less than 4. For the first time, we have presented a novel and accessible strategy to avoid unnecessary invasive tests in elderly COVID-19 patients to improve the diagnosis of PE.
Our results raised several questions about the diagnosis of PE in older patients with COVID-19, such as which is the best DD value for a clinical suspicion of PE, and what is the best pretest patient assessment that may avoid unnecessary tests. Therefore, in light of the results obtained, new algorithms and multiparametric clinical tools should be proposed for this fragile population.

5. Limitations

Our work has some limitations. First, there is a lack of scientific literature on COVID-19 in the elderly population and the associated biomarkers. In patients infected by COVID-19, most studies carried out are retrospective; therefore, it is important to validate a displaced cutoff point combined with age and COVID-19 infection. Second, confounding biases, including the clinical diagnosis, and limited knowledge of the pathophysiology and biomarkers in COVID-19 patients, need to be supported by future multicenter studies. Third, the incidence of PE could have been underestimated at the first stage of the pandemic due to the lower number of patients referred for CTPA, mainly because of the critical situation of elderly patients. Finally, the dynamic changes in the DD levels from admission to discharge and the low experience with the use of this biomarker in COVID-19 patients could have been influenced by the age of our cohort.

6. Conclusions

We proposed a novel, noninvasive approach to the elderly COVID-19 population to rule out PE. We demonstrated that the modified Geneva scale was more accurate than the Wells scale for classifying patients with suspected PE. A DD cutoff point of >4.33 mg/L has a high specificity and a high NPV for discriminating false positives. The combination of the DD level over 4.33 mg/L and the clinical probability scales increases the specificity and NPV, leading to reduced exposure to unnecessary tests in this fragile population that commonly has comorbidities.

Author Contributions

The study design, database recording and manuscript writing were by M.Q.-F. and M.R.; The study design and epidemiological evaluation were by R.T. and F.J.G.-P.; The epidemiological and clinical data were recorded by I.L.-M. and R.A.; The CTPA and parameter quantification were performed by M.S.; The laboratory quantification analysis was by V.C.M.; The English language and manuscript were reviewed by R.T. and M.R. All authors have read and agreed to the published version of the manuscript.

Funding

This work was partially supported by grants (to M.Q.-F., M.R. and R.T.) from the “New announcement for extraordinary initiative fund UAX-Santander COVID-19”, under ID 1.011.103, Universidad Alfonso X el Sabio. This study was also supported by the Fundación Pública Andaluza Progreso y Salud para la Financiación, co-financed by the European Regional Development Fund (ERDF) (PI-0048-2017 and PI0033_ 2019), and by a grant from the Spanish Society of Cardiology (SEC) for Basic Research (0011-2019).

Institutional Review Board Statement

The study was conducted according to the principles of the Declaration of Helsinki and approved by the clinical research ethics committee of Hospital La Paz Institute for Health Research (IdiPaz) 04-05-2020 under ID: I-4131.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The data that support the findings of this study are available upon request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.

Acknowledgments

This research was carried out using the extraordinary fund of initiatives “UAX-Santander COVID-19”, a resource of the Alfonso X el Sabio Universidad, as part of the registered project 1,011,103. We would like to thank Xavier Vidal for statistical analysis support.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

References

  1. Lozano-Montoya, I.; Quezada-Feijoo, M.; Jaramillo-Hidalgo, J.; Gómez-Pavón, F.J. Atypical symptoms of COVID-19 in hospitalised oldest old adults. Rev. Esp. Geriatr. Gerontol. 2021, 56, 120–121. [Google Scholar] [CrossRef] [PubMed]
  2. Gómez, L.C.G.; Guerra, L.C.; Moro, M.G.; Alba, R.C.; García, M.P.; Camacho, M.Á.C. Pulmonary embolism in very elderly patients. A diagnostic challenge. Rev. Clín. Esp. 2019, 219, 310–314. [Google Scholar]
  3. Kichloo, A.; Dettloff, K.; Aljadah, M.; Albosta, M.; Jamal, S.; Singh, J.; Wani, F.; Kumar, A.; Vallabhaneni, S.; Khan, M.Z. COVID-19 and hypercoagulability: A review. Clin. Appl. Thromb. Hemost. 2020, 26, 1076029620962853. [Google Scholar] [CrossRef]
  4. Connors, J.M.; Levy, J.H. Thromboinflammation and the hypercoagulability of COVID-19. J. Thromb. Haemost. 2020, 18, 1559–1561. [Google Scholar] [CrossRef]
  5. Gupta, R.; Ammari, Z.; Dasa, O.; Ruzieh, M.; Burlen, J.J.; Shunnar, K.M.; Nguyen, H.T.; Xie, Y.; Brewster, P.; Chen, T.; et al. Long-term mortality after massive, submassive, and low-risk pulmonary embolism. Vasc. Med. 2020, 25, 141–149. [Google Scholar] [CrossRef] [PubMed]
  6. Shen, J.H.; Chen, H.L.; Chen, J.R.; Xing, J.L.; Gu, P.; Zhu, B.F. Comparison of the Wells score with the revised Geneva score for assessing suspected pulmonary embolism: A systematic review and meta-analysis. J. Thromb. Thrombolysis 2016, 41, 482–492. [Google Scholar] [CrossRef]
  7. Di Marca, S.; Cilia, C.; Campagna, A.; D’Arrigo, G.; Abd ElHafeez, S.; Tripepi, G.; Puccia, G.; Pisano, M.; Mastrosimone, G.; Terranova, V.; et al. Comparison of wells and revised geneva rule to assess pretest probability of pulmonary embolism in high-risk hospitalized elderly adults. J. Am. Geriatr. Soc. 2015, 63, 1091–1097. [Google Scholar] [CrossRef]
  8. Griffin, D.O.; Jensen, A.; Khan, M.; Chin, J.; Chin, K.; Saad, J.; Parnell, R.; Awwad, C.; Patel, D. Pulmonary embolism and increased levels of d-dimer in patients with coronavirus disease. Emerg. Infect. Dis. 2020, 26, 1941–1943. [Google Scholar] [CrossRef]
  9. Zhang, L.; Yan, X.; Fan, Q.; Liu, H.; Liu, X.; Liu, Z.; Zhang, Z. D-dimer levels on admission to predict in-hospital mortality in patients with Covid-19. J. Thromb. Haemost. 2020, 18, 1324–1329. [Google Scholar] [CrossRef]
  10. Cannata, F.; Bombace, S.; Stefanini, G.G. Cardiac biomarkers in patients with COVID-19: Pragmatic tools in hard times. Rev. Esp. Cardiol. 2021, 74, 566–568. [Google Scholar] [CrossRef]
  11. Guo, T.; Fan, Y.; Chen, M.; Wu, X.; Zhang, L.; He, T.; Wang, H.; Wan, J.; Wang, X.; Lu, Z. Cardiovascular implications of fatal outcomes of patients with coronavirus disease 2019 (COVID-19). JAMA Cardiol. 2020, 5, 811–818. [Google Scholar] [CrossRef] [Green Version]
  12. Ortel, T.L.; Neumann, I.; Ageno, W.; Beyth, R.; Clark, N.P.; Cuker, A.; Hutten, B.A.; Jaff, M.R.; Manja, V.; Schulman, S.; et al. American Society of Hematology 2020 guidelines for management of venous thromboembolism: Treatment of deep vein thrombosis and pulmonary embolism. Blood Adv. 2020, 4, 4693–4738. [Google Scholar] [CrossRef] [PubMed]
  13. Ceriani, E.; Combescure, C.; Le Gal, G.; Nendaz, M.; Perneger, T.; Bounameaux, H.; Perrier, A.; Righini, M. Clinical prediction rules for pulmonary embolism: A systematic review and meta-analysis. J. Thromb. Haemost. 2010, 8, 957–970. [Google Scholar] [CrossRef]
  14. Wells, P.S.; Anderson, D.R.; Rodger, M.; Ginsberg, J.S.; Kearon, C.; Gent, M.; Turpie, A.G.; Bormanis, J.; Weitz, J.; Chamberlain, M.; et al. Derivation of a simple clinical model to categorize patients probability of pulmonary embolism: Increasing the models utility with the SimpliRED D-dimer. Thromb. Haemost. 2000, 83, 416–420. [Google Scholar]
  15. Klok, F.A.; Mos, I.C.; Nijkeuter, M.; Righini, M.; Perrier, A.; Le Gal, G.; Huisman, M.V. Simplification of the revised Geneva score for assessing clinical probability of pulmonary embolism. Arch. Intern. Med. 2008, 168, 2131–2136. [Google Scholar] [CrossRef]
  16. McNally, M.; Curtain, J.; O’Brien, K.K.; Dimitrov, B.D.; Fahey, T. Validity of British thoracic society guidance (the CRB-65 rule) for predicting the severity of pneumonia in general practice: Systematic review and meta-analysis. Br. J. Gen. Pract. 2010, 60, e423–e433. [Google Scholar] [CrossRef] [Green Version]
  17. Rockwood, K.; Song, X.; MacKnight, C.; Bergman, H.; Hogan, D.B.; McDowell, I.; Mitnitski, A. A global clinical measure of fitness and frailty in elderly people. CMAJ 2005, 173, 489–495. [Google Scholar] [CrossRef] [Green Version]
  18. Mahoney, F.I.; Barthel, D.W. Functional evaluation: The barthel index. Md State Med. J. 1965, 14, 61–65. [Google Scholar]
  19. Reisberg, B.; Ferris, S.H.; de Leon, M.J.; Crook, T. The global deterioration scale for assessment of primary degenerative dementia. Am. J. Psychiatry 1982, 139, 1136–1139. [Google Scholar]
  20. Konstantinides, S.V.; Meyer, G.; Becattini, C.; Bueno, H.; Geersing, G.J.; Harjola, V.P.; Huisman, M.V.; Humbert, M.; Jennings, C.S.; Jiménez, D.; et al. 2019 ESC Guidelines for the diagnosis and management of acute pulmonary embolism developed in collaboration with the European Respiratory Society (ERS). Eur. Heart J. 2020, 41, 543–603. [Google Scholar] [CrossRef] [PubMed]
  21. Pieske, B.; Tschöpe, C.; de Boer, R.A.; Fraser, A.G.; Anker, S.D.; Donal, E.; Edelmann, F.; Fu, M.; Guazzi, M.; Lam, C.S.; et al. How to diagnose heart failure with preserved ejection fraction: The HFA-PEFF diagnostic algorithm: A consensus recommendation from the Heart Failure Association (HFA) of the European Society of Cardiology (ESC). Eur. Heart J. 2019, 40, 3297–3317. [Google Scholar] [CrossRef] [Green Version]
  22. Apple, F.S.; Ler, R.; Murakami, M.M. Determination of 19 cardiac troponin I and T assay 99th percentile values from a common presumably healthy population. Clin. Chem. 2012, 58, 1574–1581. [Google Scholar] [CrossRef] [Green Version]
  23. Quezada-Feijoo, M.; Ramos, M.; Lozano-Montoya, I.; Toro, R.; Jaramillo-Hídalgo, J.; Fernández de la Puente, E.; Garmendia, B.; Carrillo, P.; Cristofori, G.; Goñi Rosón, S.; et al. Predictive factors of pulmonary embolism in older patients with SARS-CoV-2: The OCTA-COVID-19 study. J. Clin. Med. 2021, 10, 2998. [Google Scholar] [CrossRef] [PubMed]
  24. Estrada-Y-Martin, R.M.; Oldham, S.A. CTPA as the gold standard for the diagnosis of pulmonary embolism. Int. J. Comput. Assist. Radiol. Surg. 2011, 6, 557–563. [Google Scholar] [CrossRef]
  25. Kampouri, E.; Filippidis, P.; Viala, B.; Méan, M.; Pantet, O.; Desgranges, F.; Tschopp, J.; Regina, J.; Karachalias, E.; Bianchi, C.; et al. Predicting venous thromboembolic events in patients with coronavirus disease 2019 requiring hospitalization: An observational retrospective study by the COVIDIC initiative in a Swiss University Hospital. BioMed Res. Int. 2020, 2020, 9126148. [Google Scholar] [CrossRef]
  26. Thachil, J.; Tang, N.; Gando, S.; Falanga, A.; Cattaneo, M.; Levi, M.T.; Clark, C.; Iba, T. ISTH interim guidance on recognition and management of coagulopathy in COVID-19. J. Thromb. Haemost. 2020, 18, 1023–1026. [Google Scholar] [CrossRef]
  27. Wells, P.S.; Anderson, D.R.; Rodger, M.; Stiell, I.; Dreyer, J.F.; Barnes, D.; Forgie, M.; Kovacs, G.; Ward, J.; Kovacs, M.J.; et al. Excluding pulmonary embolism at the bedside without diagnostic imaging: Management of patients with suspected pulmonary embolism presenting to the emergency department by using a simple clinical model and d-dimer. Ann. Intern. Med. 2001, 135, 98–107. [Google Scholar] [CrossRef]
  28. Righini, M.; Perrier, A.; De Moerloose, P.; Bounameaux, H. D-Dimer for venous thromboembolism diagnosis: 20 years later. J. Thromb. Haemost. 2008, 6, 1059–1071. [Google Scholar] [CrossRef]
  29. Wang, L.; Baser, O.; Wells, P.; Peacock, W.F.; Coleman, C.I.; Fermann, G.J.; Schein, J.; Crivera, C. Benefit of early discharge among patients with low-risk pulmonary embolism: Focus on net clinical impact. Circulation 2016, 134, A13112. [Google Scholar]
  30. Couturaud, F.; Kearon, C.; Bates, S.M.; Ginsberg, J.S. Decrease in sensitivity of D-dimer for acute venous thromboembolism after starting anticoagulant therapy. Blood Coagul. Fibrinolysis 2002, 13, 241–246. [Google Scholar] [CrossRef]
  31. Douma, R.A.; Le Gal, G.; Söhne, M.; Righini, M.; Kamphuisen, P.W.; Perrier, A.; Kruip, M.J.; Bounameaux, H.; Büller, H.R.; Roy, P.M. Potential of an age adjusted D-dimer cut-off value to improve the exclusion of pulmonary embolism in older patients: A retrospective analysis of three large cohorts. BMJ 2010, 340, c1475. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  32. Farm, M.; Siddiqui, A.; Onelöv, L.; Chaireti, R.; Holmström, M.; Antovic, J.P. Age-adjusted D-dimer cutoffs for DVT and pulmonary embolism: A comparison of five assays. Blood 2016, 128, 1419. [Google Scholar] [CrossRef]
  33. Alhassan, S.; Bihler, E.; Patel, K.; Lavudi, S.; Young, M.; Balaan, M. Assessment of the current D-dimer cutoff point in pulmonary embolism workup at a single institution: Retrospective study. J. Postgrad. Med. 2018, 64, 150–154. [Google Scholar] [CrossRef]
  34. Cui, S.; Chen, S.; Li, X.; Liu, S.; Wang, F. Prevalence of venous thromboembolism in patients with severe novel coronavirus pneumonia. J. Thromb Haemost. 2020, 18, 1421–1424. [Google Scholar] [CrossRef]
  35. Mestre-Gómez, B.; Lorente-Ramos, R.M.; Rogado, J.; Franco-Moreno, A.; Obispo, B.; Salazar-Chiriboga, D.; Saez-Vaquero, T.; Torres-Macho, J.; Abad-Motos, A.; Cortina-Camarero, C.; et al. Incidence of pulmonary embolism in non-critically ill COVID-19 patients. Predicting factors for a challenging diagnosis. J. Thromb. Thrombolysis 2021, 51, 40–46. [Google Scholar] [CrossRef]
  36. Miniati, M.; Bottai, M.; Monti, S. Comparison of 3 clinical models for predicting the probability of pulmonary embolism. Medicine 2005, 84, 107–114. [Google Scholar] [CrossRef]
  37. Chagnon, I.; Bounameaux, H.; Aujesky, D.; Roy, P.M.; Gourdier, A.L.; Cornuz, J.; Perneger, T.; Perrier, A. Comparison of two clinical prediction rules and implicit assessment among patients with suspected pulmonary embolism. Am. J. Med. 2002, 113, 269–275. [Google Scholar] [CrossRef]
  38. Lucassen, W.; Geersing, G.J.; Erkens, P.M.; Reitsma, J.B.; Moons, K.G.; Büller, H.; van Weert, H.C. Clinical decision rules for excluding pulmonary embolism: A meta-analysis. Ann. Intern. Med. 2011, 155, 448–460. [Google Scholar] [CrossRef]
  39. Kirsch, B.; Aziz, M.; Kumar, S.; Burke, M.; Webster, T.; Immadi, A.; Sam, M.; Lal, A.; Estrada-Y-Martin, R.M.; Cherian, S.; et al. Wells score to predict pulmonary embolism in patients with coronavirus disease 2019. Am. J. Med. 2021, 134, 688–690. [Google Scholar] [CrossRef] [PubMed]
Jcm 10 05433 g001 550
Figure 1. Algorithm for pulmonary embolism in elderly patients with COVID-19. High-risk patients were considered those who scored >6 on the Wells scale or >11 on the revised Geneva scale, and those with D-dimer ≥ 1 mg/L with a torpid evolution; RV—Right ventricle; IVS—Interventricular septum; GDS—Global deterioration scale; CFS—Clinical frailty scale.
Figure 1. Algorithm for pulmonary embolism in elderly patients with COVID-19. High-risk patients were considered those who scored >6 on the Wells scale or >11 on the revised Geneva scale, and those with D-dimer ≥ 1 mg/L with a torpid evolution; RV—Right ventricle; IVS—Interventricular septum; GDS—Global deterioration scale; CFS—Clinical frailty scale.
Jcm 10 05433 g001
Jcm 10 05433 g002 550
Figure 2. Schemes following biomarker evolution during hospitalization: (a) D-dimer evolution curve according to groups, and (b) C-reactive protein evolution curve according to groups.
Figure 2. Schemes following biomarker evolution during hospitalization: (a) D-dimer evolution curve according to groups, and (b) C-reactive protein evolution curve according to groups.
Jcm 10 05433 g002
Jcm 10 05433 g003 550
Figure 3. Predictive values of the Wells score and the revised Geneva score: (a) ROC curve scale modified Geneva and (b) ROC curve scale Wells.
Figure 3. Predictive values of the Wells score and the revised Geneva score: (a) ROC curve scale modified Geneva and (b) ROC curve scale Wells.
Jcm 10 05433 g003
Jcm 10 05433 g004 550
Figure 4. Receiver operating characteristic curve for D-dimer levels as a parameter for predicting pulmonary embolism in the COVID-19 cohort. The AUC for the model was 0.7897 (0.652–0.927). The threshold for the D-dimer level was 4.3 mg/L, which showed a sensitivity of 52.9%, specificity of 93.9%, positive predictive value of 81.8% and negative predictive value of 79.5%.
Figure 4. Receiver operating characteristic curve for D-dimer levels as a parameter for predicting pulmonary embolism in the COVID-19 cohort. The AUC for the model was 0.7897 (0.652–0.927). The threshold for the D-dimer level was 4.3 mg/L, which showed a sensitivity of 52.9%, specificity of 93.9%, positive predictive value of 81.8% and negative predictive value of 79.5%.
Jcm 10 05433 g004
Table
Table 1. Characteristics of the population.
Table 1. Characteristics of the population.
FeaturesGlobal PopulationPE (n = 17)Non-PE (n = 33)p
Age, years85.5 (80–90)83 (80–86)88 (81–91)0.264
Sex (% male)26 (52.0)9 (52.9)17 (51.5)0.924
Place of origin (%) 0.475
Home11 (22.0)5 (29.4)6 (18.2)
Nursing home39 (78.0)12 (70.6)27 (81.8)
BMI (n = 49) 0.046
Low weight5 (10.2)1 (5.9)4 (12.5)
Normal weight25 (51.0)7 (41.2)18 (56.3)
Overweight8 (16.3)2 (11.8)6 (18.8)
Obesity11 (22.4)7 (41.2)4 (12.5)
Time from clinical symptoms to admission, days8 (5–10)7 (4.5–9)8 (5–10)0.403
Time from COVID diagnosis to CT scan, days14 (8–23)15 (10–23)12 (8–22)0.362
Days of hospitalization14.5 (11–21)15 (13–28)14 (10–20)0.246
Comorbidities, n (%)
Oncological history10 (20.0)6 (35.3)4 (12.1)0.070
DVT1 (2.0)1 (5.9)0 (0)0.340
PE3 (6.0)0 (0)3 (9.1)0.542
Trauma 1 (2.0)0 (0)1 (3.0)1.000
Neoplasia in palliative treatment2 (4.0)1 (5.9)1 (3.0)1.000
Lower limbs pain2 (4.0)1 (5.9)1 (3.0)1.000
PE symptoms
Heart rate, beats/min (range)88 (80–100)96 (86–109)86 (76–96)0.015
Tachycardia classification 0.013
75–9422 (44.0)5 (29.4)17 (51.5)
>9420 (40.0)11 (64.7)9 (27.3)
DVT signs3 (6.0)2 (11.8)1 (3.0)0.264
New DVT3 (6.0)2 (11.8)1 (3.0)0.264
Pain/edema lower limbs4 (8.0)3 (17.6)1 (3.0)0.108
Arterial embolic event 0.108
Lower limb ischemic events2 (4.0)1 (5.9)1 (3.0)
Embolic stroke2 (4.0)2 (11.8)0 (0)
Severity of the disease: CURB653 (2-3)2 (2-3)3 (2-3)0.431
Geriatric assessment
Dependency35 (70.0)10 (58.8)25 (75.8)0.216
Frailty32 (64.0)10 (58.8)22 (66.7)0.584
Polypharmacy34 (68.0)11 (64.7)23 (69.7)0.720
Dementia20 (40.0)6 (35.3)14 (42.4)0.626
Symptoms at hospitalization
Fever22 (44.0)6 (35.3)16 (48.5)0.373
Falls9 (18.0)5 (29.4)4 (12.1)0.242
Dyspnea41 (82.0)13 (76.5)28 (84.8)0.468
Loss of appetite12 (24.0)2 (11.8)10 (30.3)0.181
Asthenia18 (36.0)3 (17.6)15 (45.5)0.052
Delirium 13 (26.0)3 (17.6)10 (30.3)0.499
Cough11 (22.0)3 (17.6)8 (24.2)0.728
Pneumonia 1.000
Unilateral12 (26.7)4 (25.0)8 (27.6)
Bilateral33 (73.3)12 (75.0)21 (72.4)
Medication
Hydroxychloroquine37 (75.5)12 (75.0)25 (75.8)1.000
Azithromycin27 (55.1)8 (50)19 (57.6)0.617
Steroids 24 (49)8 (50)16 (48.5)0.921
PE prophylaxis47 (94.0)16 (94.1)31 (93.9)1.000
Type of anticoagulation 0.725
Prophylactic dose35 (70.0)11 (64.7)24 (72.7)
Full anticoagulation12 (24.0)5 (29.4)7 (21.2)
Time of prophylaxis10 (8-14)10 (9-13)12 (6-15)0.623
Mortality10 (20.0)3 (17.6)7 (21.2)1.000
BMI—body mass index; CURB-65—severity score for predicting mortality from community-acquired pneumonia; DVT—deep vein thrombosis; NOACS—non-vitamin K antagonist oral anticoagulants; PE—pulmonary embolism. Data are presented as the medians and interquartile ranges or numbers (%). Bold indicates statistically significant variables (p < 0.05).
Table
Table 2. Laboratory characteristics between the PE and non-PE groups.
Table 2. Laboratory characteristics between the PE and non-PE groups.
CharacteristicsPE (n = 17)Non-PE (n = 33)p-Value
D-Dimer mg/L4.33 (2.40–7.17)1.39 (1.01–2.75)<0.001
NT-Pro-BNP pg/mL1273 (444–1908)1003 (501–2240)0.946
Troponin ng/L40 (40–53)40 (40–55)ND
CRP mg/L39.4 (21.0–248.0)62.5 (31.6–170.9)0.802
Ferritin225 (159–463)243 (185–737)0.316
Lymphocytes0.72 (0.55–1.20)0.75 (0.40–1.06)0.630
PE—pulmonary embolism; NT-Pro-BNP—N-terminal pro-brain natriuretic peptide; CRP—C-reactive protein. Bold indicates statistically significant variables (p < 0.05).
Table
Table 3. D-Dimer values in the geriatric population with PE and COVID-19.
Table 3. D-Dimer values in the geriatric population with PE and COVID-19.
DD mg/LSensitivitySpecificityPPVNPVFalse Positives
1.0100%30.3%42.5%100%23%
1.582.4%54.5%48.3%85.7%15%
2.076.5%60.6%50%83.3%13%
2.570.6%69.7%54.5%82.1%10%
3.064.7%78.8%61.1%81.3%7%
3.558.8%81.8%62.5%79.4%6%
4.3352.9%93.9%81.8%79.5%2%
DD—D-dimer; PPV—positive predictive value; NPV—negative predictive value.
Table
Table 4. Predictive values of the DD levels, Wells score, revised Geneva score and their combination to detect pulmonary embolism in elderly patients with COVID-19.
Table 4. Predictive values of the DD levels, Wells score, revised Geneva score and their combination to detect pulmonary embolism in elderly patients with COVID-19.
ItemsSEPPVNPV
Wells score64.572.755.080.0
Revised Geneva score82.460.651.087.0
D-dimer52.993.981.879.5
Wells score with D-dimer35.396.885.774.4
Geneva score with D-dimer47.193.98077.5
S—sensitivity; E—specificity; PPV—positive predictive value; NPV—negative predictive value.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Quezada-Feijoo, M.; Ramos, M.; Lozano-Montoya, I.; Sarró, M.; Cabo Muiños, V.; Ayala, R.; Gómez-Pavón, F.J.; Toro, R. Elderly Population with COVID-19 and the Accuracy of Clinical Scales and D-Dimer for Pulmonary Embolism: The OCTA-COVID Study. J. Clin. Med. 2021, 10, 5433. https://doi.org/10.3390/jcm10225433

AMA Style

Quezada-Feijoo M, Ramos M, Lozano-Montoya I, Sarró M, Cabo Muiños V, Ayala R, Gómez-Pavón FJ, Toro R. Elderly Population with COVID-19 and the Accuracy of Clinical Scales and D-Dimer for Pulmonary Embolism: The OCTA-COVID Study. Journal of Clinical Medicine. 2021; 10(22):5433. https://doi.org/10.3390/jcm10225433

Chicago/Turabian Style

Quezada-Feijoo, Maribel, Mónica Ramos, Isabel Lozano-Montoya, Mónica Sarró, Verónica Cabo Muiños, Rocío Ayala, Francisco J. Gómez-Pavón, and Rocío Toro. 2021. "Elderly Population with COVID-19 and the Accuracy of Clinical Scales and D-Dimer for Pulmonary Embolism: The OCTA-COVID Study" Journal of Clinical Medicine 10, no. 22: 5433. https://doi.org/10.3390/jcm10225433

APA Style

Quezada-Feijoo, M., Ramos, M., Lozano-Montoya, I., Sarró, M., Cabo Muiños, V., Ayala, R., Gómez-Pavón, F. J., & Toro, R. (2021). Elderly Population with COVID-19 and the Accuracy of Clinical Scales and D-Dimer for Pulmonary Embolism: The OCTA-COVID Study. Journal of Clinical Medicine, 10(22), 5433. https://doi.org/10.3390/jcm10225433

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop