Next Article in Journal
A Multimarker Model for Aberrant Cardiac Geometry after Preeclampsia
Next Article in Special Issue
Preventive Health Screening during the COVID-19 Pandemic: A Cross-Sectional Survey among 102,928 Internet Users in Poland
Previous Article in Journal
Temporal Trends and Geographic Variability in the Prescription of Antiretroviral Treatments in People Living with HIV in Spain, 2004–2020
Previous Article in Special Issue
Exercise Ventilatory Inefficiency in Post-COVID-19 Syndrome: Insights from a Prospective Evaluation
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Multimorbidity Profile of COVID-19 Deaths in Portugal during 2020

by
Paulo Jorge Nogueira
1,2,3,4,*,
Miguel de Araújo Nobre
4,5,
Cecília Elias
4,6,
Rodrigo Feteira-Santos
1,2,3,4,
António C.-V. Martinho
4,7,
Catarina Camarinha
4,8,
Leonor Bacelar-Nicolau
1,2,3,4,
Andreia Silva Costa
3,4,9,10,
Cristina Furtado
1,3,4,11,
Liliane Morais
3,
Juan Rachadell
12,
Mário Pereira Pinto
13,
Fausto Pinto
14,15 and
Antó Vaz Carneiro
1,3,16,17
1
Instituto de Medicina Preventiva e Saúde Pública, Faculdade de Medicina, Universidade de Lisboa, Avenida Professor Egas Moniz, 1649-028 Lisbon, Portugal
2
Área Disciplinar Autónoma de Bioestatística (Laboratório de Biomatemática), Faculdade de Medicina, Universidade de Lisboa, Avenida Professor Egas Moniz, 1649-028 Lisbon, Portugal
3
Instituto de Saúde Ambiental, Faculdade de Medicina, Universidade de Lisboa, Avenida Professor Egas Moniz, 1649-028 Lisbon, Portugal
4
EPI Task-Force FMUL, Faculdade de Medicina, Universidade de Lisboa, Avenida Professor Egas Moniz, 1649-028 Lisbon, Portugal
5
Clínica Universitária de Estomatologia, Faculdade de Medicina, Universidade de Lisboa, Avenida Professor Egas Moniz, 1649-028 Lisbon, Portugal
6
Unidade de Saúde Pública Francisco George, Agrupamento de Centros de Saúde Lisboa Norte, Administração Regional de Saúde de Lisboa e Vale do Tejo, 1500-559 Lisbon, Portugal
7
AIBILI—Associação para Investigação Biomédica em Luz e Imagem, Azinhaga Sta, Comba, Celas, 3000-548 Coimbra, Portugal
8
Unidade de Epidemiologia, Instituto de Medicina Preventiva e Saúde Pública, Faculdade de Medicina, Universidade de Lisboa, Avenida Professor Egas Moniz, 1649-028 Lisbon, Portugal
9
CIDNUR—Centro de Investigação, Inovação e Desenvolvimento em Enfermagem de Lisboa Escola Superior de Enfermagem de Lisboa, Avenida Professor Egas Moniz, 1600-190 Lisbon, Portugal
10
CRC-W—Católica Research Centre for Psychological, Family and Social Wellbeing, Universidade Católica Portuguesa, Palma de Cima, 1649-023 Lisbon, Portugal
11
National Institute of Health Dr. Ricardo Jorge, Av. Padre Cruz, 1600-560 Lisbon, Portugal
12
Unidade de Saúde Pública, Agrupamento de Centros de Saúde Lisboa Ocidental e Oeiras, Administração Regional de Saúde de Lisboa e Vale do Tejo, 2770-219 Lisbon, Portugal
13
Casa Civil da Presidência da República, Presidência da República de Portugal, 1349-022 Lisbon, Portugal
14
Centro Cardiovascular da Universidade de Lisboa, Centro Académico de Medicina de Lisboa, Faculdade de Medicina, Universidade de Lisboa, Avenida Professor Egas Moniz, 1649-028 Lisbon, Portugal
15
Departamento do Coração e Vasos, Centro Hospitalar Universitário Lisboa Norte (CHULN), EPE, Avenida Professor Egas Moniz, 1649-028 Lisbon, Portugal
16
Instituto de Saúde Baseada na Evidência, Faculdade de Medicina, Universidade de Lisboa, Avenida Professor Egas Moniz, 1649-028 Lisbon, Portugal
17
Cochrane Portugal, Faculdade de Medicina, Universidade de Lisboa, Avenida Professor Egas Moniz, 1649-028 Lisbon, Portugal
*
Author to whom correspondence should be addressed.
J. Clin. Med. 2022, 11(7), 1898; https://doi.org/10.3390/jcm11071898
Submission received: 22 December 2021 / Revised: 24 March 2022 / Accepted: 25 March 2022 / Published: 29 March 2022

Abstract

:
Background: COVID-19 is caused by SARS-CoV-2 infection and has reached pandemic proportions. Since then, several clinical characteristics have been associated with poor outcomes. This study aimed to describe the morbidity profile of COVID-19 deaths in Portugal. Methods: A study was performed including deaths certificated in Portugal with “COVID-19” (ICD-10: U07.1 or U07.2) coded as the underlying cause of death from the National e-Death Certificates Information System between 16 March and 31 December 2020. Comorbidities were derived from ICD-10 codes using the Charlson and Elixhauser indexes. The resident Portuguese population estimates for 2020 were used. Results: The study included 6701 deaths (death rate: 65.1 deaths/100,000 inhabitants), predominantly males (72.1). The male-to-female mortality ratio was 1.1. The male-to-female mortality rate ratio was 1.2; however, within age groups, it varied 5.0–11.4-fold. COVID-19 deaths in Portugal during 2020 occurred mainly in individuals aged 80 years or older, predominantly in public healthcare institutions. Uncomplicated hypertension, uncomplicated diabetes mellitus, congestive heart failure, renal failure, cardiac arrhythmias, dementia, and cerebrovascular disease were observed among COVID-19 deceased patients, with prevalences higher than 10%. A high prevalence of zero morbidities was registered using both the Elixhauser and Charlson comorbidities lists (above 40.2%). Nevertheless, high multimorbidity was also identified at the time of COVID-19 death (about 36.5%). Higher multimorbidity levels were observed in men, increasing with age up to 80 years old. Zero-morbidity prevalence and high multimorbidity prevalences varied throughout the year 2020, seemingly more elevated in the mortality waves’ peaks, suggesting variation according to the degree of disease incidence at a given period. Conclusions: This study provides detailed sociodemographic and clinical information on all certificated deaths from COVID-19 in Portugal during 2020, showing complex and extreme levels of morbidity (zero-morbidity vs. high multimorbidity) dynamics during the first year of the pandemic in Portugal.

1. Introduction

The COVID-19 pandemic has plunged the world into an unprecedented health, economic, and social crisis [1]. As of 8 September 2021, the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), which emerged in China in December 2019, has infected more than 222 million individuals worldwide and generated 4.5 million COVID-19-related deaths [1,2]. Complications such as acute respiratory distress syndrome (ARDS), acute kidney injury (AKI), myocardial injury, and death may occur [3,4].
The first COVID-19 patients were diagnosed in Portugal on 2 March 2020. Since then, more than one million cases and more than 17,000 deaths have been reported in Portugal. Indeed, the introduction of this new disease has been associated with an increase in the number of all-cause deaths compared with the average number of all-cause deaths in the previous five years. A total of 99,356 deaths were reported in Portugal between 2 March and 27 December, an increase of 12,852 deaths compared to the mean of the previous five years in the same period [5]. From these excess deaths, Statistics Portugal, I.P. (INE), reported that 6683 (52%) were directly due to COVID-19 [5].
Several individual [6] and clinical characteristics [7,8] have been associated with poor COVID-19 outcomes. Age has been the most consistently associated risk factor with death [6]. The male sex [9], race/ethnicity [10], and pre-existing comorbidities, such as cardiovascular disease, hypertension, diabetes, chronic respiratory diseases, cerebrovascular disease, malignancy, kidney disease, and liver disease [11], can also exert a significant impact on the prognosis of the disease [12].
Assessing the distribution and coexistence of morbidities and estimating their impact on death prognosis allow for the identification of high-risk COVID-19 patients. The Charlson et al. [12] and the Elixhauser et al. [13] indexes are commonly used to create comorbidity risk-adjustment models for individuals’ risk and outcome predictions. These indexes are available for coding 17 and 31 different medical conditions, respectively, according to the International Statistical Classification of Diseases and Related Health Problems, 10th Revision (ICD-10) [14]. Previous studies on severe and critical COVID-19 patients using these indexes have shown them to be able to stratify these patients’ risk of death adequately [15,16].
Our study aimed to characterize the morbidity profiles of fatalities due to COVID-19 infection in Portugal, intending to contribute evidence for potentially improving the evaluation of COVID-19 patients and optimizing medical interventions and approaches.

2. Materials and Methods

2.1. Study Population and Data Sources

This study used retrospective data of death certificates during 2020 in Portugal to characterize the distribution of comorbidities and characteristics of COVID-19-related deaths. The dataset used is a subset of the National e-Death Certificates Information System (Sistema de Informação do Certificados de Óbito) (SICO). SICO is the Portuguese national mortality information system. It is a web-based system used by Portuguese medical doctors to certify deaths and has been fully electronic since 1 January 2014. The Portuguese Directorate-General of Health (DGS)—the technical and normative arm of the Portuguese Ministry of Health—is responsible for managing the SICO database and provided the dataset used here. Medical doctors write causes of death and comorbidities in the database in an open text form. Subsequently, the DGS’s team of specialized coders for mortality coded the causes of death and related comorbidities.
For this study, all deaths certificated in Portugal (mainland and the Autonomous Regions of Azores and Madeira) in which the underlying cause of death was “COVID-19” (ICD–10 codes: U07.1 or U07.2) and registered between 16 March and 31 December 2020 were included in this study. The codes used to identify COVID-19-related deaths followed the coding guidelines of the WHO for the COVID-19 pandemic [17].
Resident population estimates for Portugal are available at the Statistics Portugal (INE) website [18] for 2020 to calculate global crude mortality rates and age-specific rates per 100,000 individuals. Population estimates were segregated according to territorial units for statistics classification, level 2 (NUTS II; approximately Health Administrative Regions (ARS)), districts, municipalities, and age groups.
Additional data characterizing the number of COVID-19 daily infections were obtained from the https://github.com/CSSEGISandData repository (accessed on 21 December 2021) [2].

2.2. Population Characteristics and Morbidity

Population characteristics available in the SICO dataset were described, such as age (recoded into age groups), sex, region, health region, district, municipality, and death place.
Comorbidities were derived from the ICD10 codes associated with each death certificate, which include the codification of the underlying cause of death, diseases, and other conditions. Comorbidities relating to each COVID-19 death were then used to calculate the Elixhauser index [13] as a primary approach and the Charlson index [12] as a supplementary approach. These indexes are methods of categorizing patient comorbidities according to respective ICD-10-adapted algorithms (Appendix A). Elixhauser-associated indexes were adopted for statistical analysis: the Elixhauser index, the weighted Elixhauser with Agency for Healthcare Research and Quality (AHRQ) modification [19], and the Elixhauser index with the van Walraven modification [20]. These weighted indexes stratify the respective weighted scores into four groups: <0, 0, 1–4, and ≥5 (i.e., from less probability of death, or potentially low morbidity, to a greater likelihood of death, or potentially high multimorbidity). The place-of-death variable was constructed from the available text descriptions in the database.
A variable dividing the pandemic into three periods during 2020 was also created and used to compare COVID-19-related deaths in each stage. Each period had distinct characteristics in the evolution of COVID-19 infections, according to the observed 7 days incidence and assuming that two infectious waves occurred, with period 1 as the first infection wave (between 2 March and 2 June); period 2 as a calmer infectious stage (between 3 June and 11 August); and period 3 as the initial increasing period of the extensive second wave of infections, starting on 12 August and lasting until 31 December (the second wave of infections went on at least until March 2021; however, it was mostly outside of the defined period for this study).

2.3. Statistical Analysis

Descriptive statistics of single variables were performed using absolute and relative frequencies, means, medians, standard deviations (SDs), interquartile ranges (IQRs), and minimum as well as maximum statistics.
Global crude mortality rates, age-specific rates, and age-standardized rates per 100,000 individuals were calculated using the absolute number of deaths divided by the estimated resident population’s respective number (obtained at the Statistics Portugal (INE) website). The WHO 2000–2025 standard population [21] was used for the calculation of age-standardized rates. Additionally, the absolute number of deaths and rate ratios per sex were calculated for the main variables and morbidities.
Comparisons of quantitative variables were done descriptively and using nonparametric tests (Mann–Whitney U tests for two groups and Kruskal–Wallis tests for three or more groups). When necessary, associations between categorical variables were performed using Fisher’s exact test resorting to Monte Carlo (simulated p-value). Distributions’ homogeneity by category variables within each comorbidity was tested using Pearson’s goodness-of-fit test, considering each categorical variable’s total/marginal distribution.
For all the tests performed, the level of statistical significance was set at 0.05. All the analyses were conducted using the R software version 3.6.3 (R Foundation for Statistical Computing, Vienna, Austria). The determination of comorbidities was performed using the R “Comorbidity” package, version 0.5.3 [22].

3. Results

3.1. Study Population Characteristics

During 2020, a total of 6701 COVID-19-related deaths were recorded in the e-death certificate database in Portugal: 3502 males and 3199 females, with an average age (SD) and median of 82.0 (10.6) and 84 years, respectively. Two COVID-19-related deaths peaks were observed in Portugal, the first during the 13th week and the second in the 50th week of 2020 (Supplementary Material, Figure S1A). It is noteworthy that the highest number of deaths per week was similar between the sexes during the first peak and co-occurring. During the second peak, the highest number of deaths per week was lowest for men and occurred about two weeks earlier (Supplementary Materials, Figure S1B). Both peaks of deaths per week reflect different evolutions in the incidence of COVID-19 across the three observed pandemic stages: two waves of infections separated by a period with a consistent and low number of COVID-19 cases per week (Supplementary Materials, Figure S1C).
Regarding the distribution of deaths by age group, more than two-thirds of COVID-19-related deaths were in individuals aged 80 years and older. The distribution of deaths by sociodemographic variables and death rates are detailed in Table 1. The global death rate was estimated at 65.5 deaths per 100,000 inhabitants, higher in males (72.1) than females (58.8). Additionally, the death rate per 100,000 inhabitants was differently distributed by region. In more densely populated regions with over 1,000,000 inhabitants (North, Center, and Lisbon Metropolitan Area), higher crude mortality rates (CMRs) were registered in men; however, in less densely populated regions with fewer than 1,000,000 inhabitants (Alentejo, Algarve, and both Autonomous Regions of the Azores and Madeira), higher CMRs were registered in women.
It is noteworthy that although, globally, the ratio between male and female COVID-19 deaths was estimated as 1.1 (approximately 10% more men than women), within age groups, the panorama was substantially different. From 20 to 70 years, absolute deaths were consistently higher in males within each age decade. Most total death number ratios by age decade were more than 1.5-fold higher in males (namely, in individuals in their 20s, 30s, and 70s) and about 2-fold in those in their 40s and 50s, while a 2.7-fold-higher difference was verified in those aged in their 60s. The mean age of death in men and women was statistically different (p < 0.001), with 84.3 years in females and 79.9 years in males (Supplementary Materials, Table S1 and Figure S2). In relative terms, using CMR ratios, this discrepancy was even more remarkable, with 5- to almost 12-fold male-to-female mortality difference within age groups (Table 1). The ratio between both sexes’ age-standardized mortality rates consistently indicated a global 6.9 for males compared to females. In terms of geographical distribution, the highest absolute number of deaths was observed in Porto at the district level, and the highest mortality rate was in Bragança. At the municipal level, the highest absolute number of fatalities and the highest mortality rate were observed in Lisbon and the municipality of Vimioso, respectively (Supplementary Materials, Figures S3 and S4). The Northern ARS registered the most elevated mortality at the regional level in the absolute number of COVID-19 deaths and mortality rates (Supplementary Materials, Table S1, Figures S3 and S4). The most COVID-19-related deaths occurred mainly in a “public health institution” (90.8%), with a higher proportion in males (93.3%) than in females (88.0%). Considerably fewer, 4.3%, deaths occurred “at home” (slightly more in females than males). Moreover, the third-highest place of death was identified as at a “nursing home” with 4.2%, this proportion being 2.2-fold higher in females than in males (Table 1). A very residual number of deaths was identified as occurring in a “private health institution” (just 0.2%).
Considering the age distribution by death location, a higher mean age at death was observed in the “unknown” and “nursing homes” categories (84.4 and 87.1 years, respectively). In comparison, lower mean ages were observed for deaths occurring in “public health institutions” and “at home” (81.8 and 82.6 years, respectively) (Supplementary Material, Table S1).

3.2. Comorbidities

Among the deceased, a total of 29,775 ICD-10 codes were registered, with an average of 4.44 codes per death (SD: 3.04; range: 1 to 15). Evaluating this information according to the presence of comorbidities on Elixhauser’s list, a total of 9234 occurrences were observed. The number of comorbidities per death ranged from 0 to 8, with an average (SD) of 1.38 (1.44) and a median of 1. Thus, about 37.9% of deaths had zero Elixhauser’s comorbidities associated with them, while 40.9% had at least two Elixhauser’s comorbidities identified with this approach. The top ten Elixhauser’s comorbidities associated with COVID-19 fatalities that occurred in Portugal during 2020 included (in order of importance): uncomplicated hypertension (30.0%), uncomplicated diabetes mellitus (16.8%), congestive heart failure (14.5%), renal failure (12.8%), cardiac arrhythmias (10.6%), pulmonary circulation disorders (7.1%), solid tumors (7.1%), obesity (5.2%), other neurological disorders (4.8%), and fluid as well as electrolyte disorders (3.8%) (Table 2 or Table S2 for a complete table).
The presence of comorbidities differed between the sexes. For example, in Elixhauser’s comorbidities, associated morbidity rankings differed slightly, as renal failure was third in women and fourth in men. In addition, several morbidities were shown to differ by sex, such as chronic heart failure, solid tumors, obesity, hypothyroidism, depression, and rheumatoid arthritis/collagen vascular disease. At the same time, age groups were significantly associated with almost all morbidities.
Additionally, there were no substantial differences by health region of residence, where only two morbidities showed some heterogeneity: depression and blood loss anemia. Additionally, comparing the three defined periods of the pandemic in Portugal, it is noteworthy that the morbidity rankings vary from period to period. However, no substantial heterogeneities were identified within morbidity (exceptions were fluid and electrolyte disorders; weight loss; and AIDS or HIV) (Supplementary Materials, Table S3).
The construction of Charlson’s comorbidities generated 6605 occurrences that are displayed in Table S5 (Supplementary Materials). The number of comorbidities per death ranged from 0 to 6, with an average (SD) of 0.99 (1.03) and a median of 1. Thus, approximately 40.3% of deaths had zero Charlson comorbidities associated with them. On the other hand, 28.2% of deaths had at least two Charlson comorbidities identified. It is noteworthy that two comorbidities are identical in the Charlson and Elixhauser indexes: congestive heart failure (CHF) and peripheral vascular disorders (PVD), with CHF ranking in the top three in both approaches. The top five Charlson comorbidities included uncomplicated dementia, diabetes, CHF, renal disease, and cerebrovascular disease. The inclusion of dementia in the Charlson top five is an important departure from the Elixhauser approach, but overall, the top morbidities in both approaches show a similar message.
Additionally, no substantial differences by sex were observed (with the exceptions of dementia, chronic heart failure, malignant cancer, and rheumatoid disease). Here, all morbidities showed heterogeneity by age groups.
In COVID-19 deaths observed in Portugal during 2020, correlations between comorbidities observed together and identified using Elixhauser were few. Although no remarkable negative correlations were found, some positive correlations were observed. The correlation between pairs of comorbidities seems to be even less relevant in what concerns Charlson’s estimated comorbidities. However, a slight negative correlation was observed between dementia and several remaining comorbidities (Figure 1B).
Elixhauser’s list: chf—congestive heart failure; carit—cardiac arrhythmias; valv—valvular disease; pcd—pulmonary circulation disorders; pvd—peripheral vascular disorders; hypunc—hypertension, uncomplicated; hypc—hypertension, complicated; para—paralysis; ond—other neurological disorders; cpd—chronic pulmonary disease; diabunc—diabetes, uncomplicated; diabc—diabetes, complicated; hypothy—hypothyroidism; rf—renal failure; ld—liver disease; pud—peptic ulcer disease, excluding bleeding; aids—AIDS/HIV; lymph—lymphoma; metacanc—metastatic cancer; solidtum—solid tumor, without metastasis; rheumd—rheumatoid arthritis/collaged vascular disease; coag—coagulopathy; obes—obesity; wloss—weight loss; fed—fluid and electrolyte disorders; blane—blood loss anemia; dane—deficiency anemia; alcohol—alcohol abuse; drug—drug abuse; psycho—psychoses; and depre—depression.
Charlsons’ list: ami—acute myocardial infarction; chf—congestive heart failure; pvd—peripheral vascular disease; cevd—cerebrovascular disease; dementia—dementia; copd—chronic obstructive pulmonary disease; rheumd—rheumatoid disease; pud—peptic ulcer disease; mld—mild liver disease; diabetes—diabetes without complications; diabetes w/com—diabetes with complications; hp—hemiplegia or paraplegia; rend—renal disease; cancer—cancer (any malignancy); msld—moderate or severe liver disease; metacanc—metastatic solid tumor; and aids—AIDS/HIV.

3.3. Multimorbidity

The distribution of COVID-19 deaths according to Elixhauser morbidity composite scores revealed high percentages of deceased individuals associated with low morbidity (low expected probability of death) and with high multimorbidity (high expected probability of death) (Table 3). Higher levels of multimorbidity (present in COVID-19 fatalities) were more frequently estimated in men than in women. By age group, the morbidity indexes’ analyses are more challenging. Overall, morbidity increases with age. However, in the 80 years old and older group, a slight reduction in multimorbidity and a concomitant increase in the percentual occurrence of death were observed to be associated with nonidentified morbidity. The distribution patterns of fatalities along the various comorbidity categories of both the Elixhauser and Charlson indexes varied between residence districts, suggesting differences between districts with higher (Bragança and Porto) and lower (Beja and Faro) mortality rates. The same is observed at the level of health regions. More than 90% of deaths occurred in a “public health institution”, influencing the overall observed morbidity profile. Deaths observed “at home” seem to have happened in individuals with lower levels of morbidity.
In contrast, deaths registered as occurring in “nursing homes” revealed a morbidity profile similar to those found globally and in health institutions, with slightly lower multimorbidity. The number of deaths reported in “private health institutions” was very low, accounting only for 0.2% of deaths. Considering the Elixhauser composite scores as well as the three periods of 2020 defining the evolution of the pandemic, evidence emerged that the morbidity profile of observed deaths differed in each period. The differences suggest higher multimorbidity levels during lower infection incidence phases and an association between higher infection incidence stages and somewhat lower multimorbidity profiles.
To further investigate the notion that the consecutive periods had different morbidity profiles, the Elixhauser indexes were separately analyzed for each period. A comparison between levels of multiple Elixhauser morbidities observed per death in the three pandemic periods is detailed in Supplementary Materials Table S4. Considering the occurrence of two or more morbidities (Elixhauser score > 1), the percentage of individuals with this level of multimorbidity was 38.1% in the first period (first infection wave), 42% between the first and second infection waves, and 26.6% in the third period (ascending phase of the second infection wave). When considering the simultaneous presence of three or more morbidities (Elixhauser score > 2 and Elixhauser score > 3), the pattern was similar, although the difference between periods one and two was minimal. Still, in Supplementary Materials Table S4, the extreme level of Elixhauser morbidity conveys a similar message. However, the score’s weighted versions that account more closely for the probability of death at hospital admission, considering the respective morbidities, show a more evident discrepancy among the three periods. In fact, extreme multimorbidity is higher during period two (lower incidence of the disease), while period one seems to have about 5% less excessive multimorbidity in its observed COVID-19 deaths. Last, the morbidity observed in period three has nearly half of the previous period’s extreme level of morbidity.

4. Discussion

This study assessed a Portuguese national death certificate database and retrospectively included all 6701 individuals deceased from COVID-19 between 16 March and 31 December 2020. The authors used comorbidities registered as directly contributing to death. The current methodology represents a strength of the present study, enabling the characterization of underlying comorbidities of COVID-19-related deaths.
Registered increased mortality for men was observed, characterized by an overall modest absolute increase but a dramatic increase within the age groups when applying subgroup analysis. Besides the sex differences in biological terms [23], the possible explanations for this result include the following: (a) the fact that women are more likely to search for healthcare services than men [24], accessing medical care at an earlier stage of COVID-19 disease; (b) the generally stronger response to infectious pathogens in women [25]; (c) the increased likelihood of the adoption of nonpharmaceutical measures to avoid infection, particularly hand hygiene [26] and preventive behaviors [27]; or (d) sex-based differences in multimorbidity registered in this study that may be associated with severe states of COVID-19 [19,20,28,29,30,31,32,33].
The Elixhauser comorbidity index was not used to predict mortality in this study, given that we did not have enough data to do so. However, given the statistical superiority of the Elixhauser comorbidity system in predicting mortality outcomes [32] and the present study’s large dataset (with a reduced risk of overfitting), this index was used as a primary multimorbidity analysis tool, to the detriment of the Charlson comorbidity index, and revealed to be advantageous. The authors used two composite score modifications [19,33] to discriminate associations between multimorbidity and mortality by COVID-19. This allowed for managing the computational challenge in the current dataset using the Elixhauser comorbidity index. This strategy was previously reported with high accuracy in the literature for studying in-hospital mortality [20].
The data show that 37.9% (n = 2541) of COVID-19 deaths did not have Elixhauser morbidities, potentially indicating a high proportion of fatalities without comorbidities. About 11.2% (n = 284) and 3.7% (n = 95) of the deceased patients without death codes related to the Elixhauser list of comorbidities were under 70 and 60 years of age, respectively. The demographic analysis of this group of patients revealed a higher number of men (n = 1338) compared to women (n = 1203) and a mean age of 82.5 years, with 69.9% (n = 1771) of the patients aged 80 years or more. This result hints at a high global absence of morbidities, which is highly unlikely considering the Portuguese senior population’s characteristics. This finding may be related to an issue with death records, since these patients died in periods of an increased number of deaths, which could have led to less time to certify each death and thus less time to register comorbidities. The absence of comorbidities assumed by the Elixhauser index can also explain this issue. Of the 2541 patients, 306 (12.0%) had dementia, considering the Charlson comorbidity list. According to a systematic review, this condition is one of the identified prognostic factor pathologies for mortality by COVID-19, with a 54% increase in the odds [34].
The data suggested that the presence of multimorbidity was strongly concomitant with fatalities by COVID-19, as multimorbidity was registered in 40.9% of deaths. Previous studies reported the association of several comorbidities with an increased mortality rate. Using surveillance data of the first 20,293 cases during the initial stage of the pandemic in Portugal, a previous study investigated the role of preconditions in COVID-19 deaths. In that study, cardiac diseases, kidney disorders, and neuromuscular disorders were registered as potential risk indicators for COVID-19 deaths [35]. The same study also registered a detrimental effect on COVID-19 deaths for several comorbidities. For example, cardiac diseases and renal failure—included in the top 10 diseases present at the time of death—cancer, hypertension (uncomplicated), diabetes (uncomplicated), congestive heart failure, obesity, and chronic pulmonary disease were also indicated as prognostic factors for severity or mortality in patients infected with COVID-19. These associations were also recognized in systematic reviews and meta-analyses on the subject [34,36,37,38,39,40,41,42].
A COVID-19 pandemic early-stage study reported estimating excess mortality due to COVID-19, nonidentified COVID-19, and a decrease in healthcare access of approximately 2400 to 4000 deaths [43]. This three-partied mechanism could partly explain the high lethality in multimorbidity patients registered in this analysis. Furthermore, a systematic review reported an association between multimorbidity and increased loneliness levels [44], implying decreased healthcare access for individuals highly susceptible to severe COVID-19 outcomes. However, the number of studies investigating the effect of multimorbidity on the lethality of COVID-19 is limited and in great need of expansion. A similar association between multimorbidity and COVID-19 lethality was registered in a study investigating populations from the South American continent. That study reported different diseases, including high blood pressure, diabetes mellitus, obesity, and cardiovascular, respiratory, and chronic kidney disease [45]. Therefore, the authors consider it very important that the results of the present study using the current methodology confirmed the association of multimorbidity with COVID-19 death suggested by previous studies in the literature [34,35,36,37,38], providing a point of reference for future research.
Considering the geographical location, the epidemiological assessment of COVID-19 mortality is complex, given the distinct perspectives of mortality absolute figures and rates, especially in less densely populated areas. Moreover, considering the region or district of residence as a factor for COVID-19 mortality lacks direct epidemiological plausibility. It implies a biased causal path given its association with latent unobserved factors, such as socioeconomic factors, the timing of the introduction and propagation of the disease, public health metrics (e.g., case detection and testing capacity, healthcare service practices and proficiency, and the implementation of mitigation strategies), and its condition as the ancestor of exposure and outcomes with both age and preconditions [46,47,48]. The authors discuss the region/district of residence only in the COVID-19 geographical fatality distribution context. When evaluating the health regions and districts of residence with at least 20 deaths, the cutoff for a higher fatality ratio in men was areas with over 1,000,000 inhabitants (districts located in the North, LVT, and Center administrative health regions). In comparison, women’s fatality ratio was higher in less densely populated areas in southern Portugal (below 1,000,000 inhabitants). In these less densely populated regions and districts, the average age of death was above 80 years. This result was expectable in terms of observing a higher death ratio for women given the significant difference between the sexes in the average age of death (79.8 years for men; 84.3 years for women). The exception was the Guarda district, where the CMR ratio was higher for women. Still, this district’s average age of death was also the highest in Portugal, above two standard deviations from the national average.
The present study’s mortality data suggested a difference in the characteristics of the affected population between the three time periods. A decrease in mortality was observed for individuals with higher multimorbidity levels, while an increase in mortality was observed for individuals with lower multimorbidity levels. Several potential reasons arise for the explanation of this result. First, the completeness of the records decreased in period three (average of 1.39 morbidities per individual (in 5046 deaths) in period three compared to an average of 1.46 morbidities per individual (in 264 deaths) in period two and an average of 1.29 morbidities per individual (in 1332 deaths) in period one). Second, the significant increase in mortality that occurred in time period three might have caused the collapse of health services. Last, the Alpha variant of the SARS-CoV-2 virus, which may be responsible for higher mortality [49], became predominant in Portugal in period three. This result warrants further research to clarify the health system or etiopathogenesis mechanisms involved in these results. Administrative datasets help study and estimate health outcomes [28]. Still, the data are more vulnerable to potential sources of bias and limitations, including inaccuracy, missing data, or a lack of updating. Using administrative datasets, measuring multimorbidity (as the existence of multiple conditions ≥ 2 in the same person) [29] can be complex. To overcome administrative data limitations, the use of proper comorbidity adjustment is essential [30,31].
Despite being positively impressed with the comprehensiveness and detail of causes of death codes, the authors acknowledge the effect of death certificate errors as a limitation to the present study. They may also lead to a potential underestimation of some causes, especially in the elderly population [50]. In health, information and communication technologies have manifested themselves as essential tools for providing health services. The e-death certification system (SICO platform) intended, from its conception, to optimize mortality data quality and epidemiological surveillance. It proposed to do so through the electronic certification of deaths, improving both the access to and quality of mortality data [51,52]. Moreover, the electronic certification of deaths allows electronic communication between information systems with relevant information associated with cause of death and worldwide comparability (ICD-10 coding) [53,54]. Standardized death certification and coding practices also ensure quality mortality data and accurate tracking of multimorbidity related to COVID-19 mortality, which are crucial in public health/epidemiological matters. However, data completeness may be somewhat compromised for several reasons, including their use by different physicians, the little clinical information available, the absence of close contacts, and difficulty prioritizing the conditions leading to death in elderly patients due to multimorbidity [50,55]. Additionally, the highly stressful work environment due to the current pandemic crisis may decrease the records’ quality. There is also the possibility that the high level of detail observed may stem from the particular focus on COVID-19, potentiated by multiple sources of information circulating at health authorities’ institutions, which may not be the case in normal conditions for all kinds of mortality. In fact, in regular conditions, year N codes of mortality are only available after October of year N + 1; hence, authors were working on these codes for COVID-19 deaths more than six months before that schedule.
The Elixhauser comorbidity index was chosen as the primary multimorbidity analysis tool, to the detriment of Charlson’s. The use of 30 disease indicators, statistical superiority in predicting mortality outcomes [32], and the present study’s large dataset (with reduced risk of overfitting) revealed advantages. To manage the computational challenge in the current dataset using the Elixhauser comorbidity index, the authors used two composite scores modifications [19,33] to discriminate associations between multimorbidity and mortality by COVID-19. This strategy was previously reported with high accuracy in the literature for studying in-hospital mortality [20]. Both the Elixhauser and Charlson indexes were essentially used here to identify diseases/comorbidities associated with COVID-19 deaths. As they are usually seen as in-hospital mortality predictive tools, and as about 90% of COVID-19 deaths were observed as occurring in a health institution, this might be considered a limitation in our study. Nevertheless, this study was not focused on in-hospital mortality predictions.
The present study detailed the characteristics of COVID-19 deaths in Portugal during the year 2020. Social demographic characteristics and underlying health conditions (morbidities) provide important clues that may be used to identify noninfected individuals with high risk (ages and multimorbidity patterns). Furthermore, targeting specific preventive measures/management and hospital triage can help maximize the probability of COVID-19 survival. Future research should include a qualitative assessment of the procedure used herein to obtain the comorbidities associated with COVID-19 deaths.

5. Conclusions

In the first year of the COVID-19 pandemic, considering the 6701 COVID-19 deaths in Portugal during 2020, the respective global mortality rate was estimated as being 65.5 per 100,000 inhabitants, with significantly different and higher mortality rates in men than in women. Overall, the mean age at death was 81.9 years old (higher within females), with the majority of these deaths occurring in individuals over 80 years old. However, differences between the male-to-female number of death ratios were observed within age groups, as this indicator ranged between 5- and 11-fold. Consistently, the ratio of age-standardized rates between men and women was 6.9-fold. In more densely populated regions, increased mortality was observed, and men predominated; in contrast, less-populated regions registered less mortality and a higher female proportion.
A high prevalence of zero morbidities was registered using both the Elixhauser and Charlson comorbidities lists, namely 53.8% in the VW Elixhauser index and 40.2% in Charlson’s. Nevertheless, the opposite is also true. In fact, a wide range of diseases/morbidities were identified at the time of COVID-19 death. The most prevalent were uncomplicated hypertension, dementia, uncomplicated diabetes mellitus, congestive heart failure, renal failure, and cardiac arrhythmias. Higher multimorbidity levels were observed in men, increasing with age up to 80 years old. The zero-morbidity prevalence and multimorbidity prevalence varied throughout the year 2020, seemingly more elevated in the mortality wave’s peaks, which could suggest a variation according to the degree of incidence at a given period.
This study provides detailed sociodemographic and clinical information on all certificated deaths with COVID-19 in Portugal during 2020, showing complex and extreme levels of morbidity (no morbidity vs. high levels of comorbidity) dynamics during the first year of the pandemic in Portugal.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/jcm11071898/s1, Figure S1: Weekly evolution of the number of COVID-19 deaths, total (A) and by sex (B), and of new infected cases (C) in Portugal during 2020, Figure S2: Age distribution of COVID-19 deaths in Portugal during 2020, Figure S3: Distribution of COVID-19 number of deaths and COVID-19 death rates by health administrative region (A,B), district (C,D), and municipality of residence (E,F), Figure S4: Geographical distribution of COVID-19 deaths by municipalities in absolute number and crude mortality rates for total population (A,B), men (C,D), and women (E,F), Portugal, 2020, Table S1: Age distribution and comparison by characteristics, Table S2: Percentual presence of comorbidities (according to the Elixhauser comorbidity index) by health region of residence and by time periods, Table S3: Percentage of Elixhauser multimorbidity levels and Elixhauser weighted multimorbidity indexes’ highest levels by time periods.

Author Contributions

Conceptualization, P.J.N.; methodology, P.J.N.; software, P.J.N. and R.F.-S.; formal analysis, P.J.N. and R.F.-S.; investigation, P.J.N., M.d.A.N., C.E., R.F.-S., A.C.-V.M., C.C., L.B.-N., A.S.C., C.F., L.M., J.R., M.P.P., F.P. and A.V.C.; resources, P.J.N., A.S.C., M.P.P., F.P. and A.V.C.; data curation, P.J.N.; writing—original draft preparation, P.J.N., M.d.A.N., C.E., R.F.-S., A.C.-V.M., C.C., L.B.-N., A.S.C., C.F., L.M., J.R. and A.V.C.; writing—review and editing, P.J.N., M.d.A.N., C.E., R.F.-S., A.C.-V.M., C.C., L.B.-N., A.S.C., C.F., L.M., J.R., M.P.P., F.P. and A.V.C.; visualization, P.J.N., M.d.A.N., C.E., R.F.-S., A.C.-V.M., C.C., L.B.-N., A.S.C., C.F., L.M., J.R., M.P.P., F.P. and A.V.C.; supervision, M.P.P., F.P. and A.V.C.; project administration, P.J.N. All authors have read and agreed to the published version of the manuscript.

Funding

This publication was supported by Fundação para a Ciência e Tecnologia (FCT) under the references UIDB/04295/2020 and UIDP/04295/2020.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki. Furthermore, this noninterventional study used a dataset from an administrative database. Thus, according to the Centro Académico de Medicina de Lisboa Ethics Committee (Refª. Nº365/21), ethical approval was not required.

Informed Consent Statement

Not applicable.

Data Availability Statement

E-death certificates’ Data analyzed is owned by Direção-Geral da Saúde (MoH, Portugal) it is not publicly available but can be requested to [email protected], the institution has a specific procedure to provide data that needs to be followed and approved by the Director-General of Health. All additional used data are available at the provided sites and links.

Acknowledgments

The authors would like to thank the Presidency of the Portuguese Republic for providing the data to perform the present study. Words of appreciation also go to the Direcção-Geral da Saúde and its professionals’ work in data management and preparation. This work was produced with the computational support of INCD—Infraestrutura Nacional de Computação Distribuída—which is funded by Fundação para a Ciência e a Tecnologia and FEDER under project 01/SAICT/2016, no. 022153.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

ICD-10 codes for the derivation of the Charlson and Elixhauser comorbidity indexes.

Appendix A.1. CHARLSON Morbidity Index ICD-10 Codes

  • Myocardial infarction: I21.x, I22.x, and I25.2.
  • Congestive heart failure: I09.9, I11.0, I13.0, I13.2, I25.5, I42.0, I42.5–I42.9, I43.x, I50.x, and P29.0.
  • Peripheral vascular disease: I70.x, I71.x, I73.1, I73.8, I73.9, I77.1, I79.0, I79.2, K55.1, K55.8, K55.9, Z95.8, and Z95.9.
  • Cerebrovascular disease: G45.x, G46.x, H34.0, and I60.x–I69.x.
  • Dementia: F00.x–F03.x, F05.1, G30.x, and G31.1.
  • Chronic pulmonary disease: I27.8, I27.9, J40.x–J47.x, J60.x–J67.x, J68.4, J70.1, and J70.3.
  • Rheumatic disease: M05.x, M06.x, M31.5, M32.x–M34.x, M35.1, M35.3, and M36.0.
  • Peptic ulcer disease: K25.x–K28.x.
  • Mild liver disease: B18.x, K70.0–K70.3, K70.9, K71.3–K71.5, K71.7, K73.x, K74.x, K76.0, K76.2–K76.4, K76.8, K76.9, and Z94.4.
  • Diabetes without chronic complication: E10.0, E10.1, E10.6, E10.8, E10.9, E11.0, E11.1, E11.6, E11.8, E11.9, E12.0, E12.1, E12.6, E12.8, E12.9, E13.0, E13.1, E13.6, E13.8, E13.9, E14.0, E14.1, E14.6, E14.8, and E14.9.
  • Diabetes with chronic complication: E10.2–E10.5, E10.7, E11.2–E11.5, E11.7, E12.2–E12.5, E12.7, E13.2–E13.5, E13.7, E14.2–E14.5, and E14.7.
  • Hemiplegia or paraplegia: G04.1, G11.4, G80.1, G80.2, G81.x, G82.x, G83.0–G83.4, and G83.9.
  • Renal disease: I12.0, I13.1, N03.2–N03.7, N05.2–N05.7, N18.x, N19.x, N25.0, Z49.0–Z49.2, Z94.0, and Z99.2.
  • Any malignancy, including lymphoma and leukemia, except malignant neoplasm of skin: C00.x–C26.x, C30.x–C34.x, C37.x–C41.x, C43.x, C45.x–C58.x, C60.x–C76.x, C81.x–C85.x, C88.x, and C90.x–C97.x.
  • Moderate or severe liver disease: I85.0, I85.9, I86.4, I98.2, K70.4, K71.1, K72.1, K72.9, K76.5, K76.6, and K76.7.
  • Metastatic solid tumor: C77.x–C80.x.
  • AIDS/HIV: B20.x–B22.x and B24.x.

Appendix A.2. ELIXHAUSER Morbidity List ICD-10 Codes

  • Congestive heart failure: I09.9, I11.0, I13.0, I13.2, I25.5, I42.0, I42.5–I42.9, I43.x, I50.x, and P29.0.
  • Cardiac arrhythmias: I44.1–I44.3, I45.6, I45.9, I47.x–I49.x, R00.0, R00.1, R00.8, T82.1, Z45.0, and Z95.0.
  • Valvular disease: A52.0, I05.x–I08.x, I09.1, I09.8, I34.x–I39.x, Q23.0–Q23.3, and Z95.2–Z95.4.
  • Pulmonary circulation disorders: I26.x, I27.x, I28.0, I28.8, and I28.9.
  • Peripheral vascular disorders: I70.x, I71.x, I73.1, I73.8, I73.9, I77.1, I79.0, I79.2, K55.1, K55.8, K55.9, Z95.8, and Z95.9.
  • Hypertension, uncomplicated: I10.x.
  • Hypertension, complicated: I11.x–I13.x and I15.x.
  • Paralysis: G04.1, G11.4, G80.1, G80.2, G81.x, G82.x, G83.0–G83.4, and G83.9.
  • Other neurological disorders: G10.x–G13.x, G20.x–G22.x, G25.4, G25.5, G31.2, G31.8, G31.9, G32.x, G35.x–G37.x, G40.x, G41.x, G93.1, G93.4, R47.0, and R56.x.
  • Chronic pulmonary disease: I27.8, I27.9, J40.x–J47.x, J60.x–J67.x, J68.4, J70.1, and J70.3.
  • Diabetes, uncomplicated: E10.0, E10.1, E10.9, E11.0, E11.1, E11.9, E12.0, E12.1, E12.9, E13.0, E13.1, E13.9, E14.0, E14.1, and E14.9.
  • Diabetes, complicated: E10.2–E10.8, E11.2–E11.8, E12.2–E12.8, E13.2–E13.8, and E14.2–E14.8.
  • Hypothyroidism: E00.x–E03.x and E89.0.
  • Renal failure: I12.0, I13.1, N18.x, N19.x, N25.0, Z49.0–Z49.2, Z94.0, and Z99.2.
  • Liver disease: B18.x, I85.x, I86.4, I98.2, K70.x, K71.1, K71.3–K71.5, K71.7, K72.x–K74.x, K76.0, K76.2–K76.9, and Z94.4.
  • Peptic ulcer disease, excluding bleeding: K25.7, K25.9, K26.7, K26.9, K27.7, K27.9, K28.7, and K28.9.
  • AIDS/HIV: B20.x–B22.x and B24.x.
  • Lymphoma: C81.x–C85.x, C88.x, C96.x, C90.0, and C90.2.
  • Metastatic cancer: C77.x–C80.x.
  • Solid tumor without metastasis: C00.x–C26.x, C30.x–C34.x, C37.x–C41.x, C43.x, C45.x–C58.x, C60.x–C76.x, and C97.x.
  • Rheumatoid arthritis/collagen vascular diseases: L94.0, L94.1, L94.3, M05.x, M06.x, M08.x, M12.0, M12.3, M30.x, M31.0–M31.3, M32.x–M35.x, M45.x, M46.1, M46.8, and M46.9.
  • Coagulopathy: D65–D68.x, D69.1, and D69.3–D69.6.
  • Obesity: E66.x.
  • Weight loss: E40.x–E46.x, R63.4, and R64.
  • Fluid and electrolyte disorders: E22.2, E86.x, and E87.x.
  • Blood loss anemia: D50.0.
  • Deficiency anemia: D50.8, D50.9, and D51.x–D53.x.
  • Alcohol abuse: F10, E52, G62.1, I42.6, K29.2, K70.0, K70.3, K70.9, T51.x, Z50.2, Z71.4, and Z72.1.
  • Drug abuse: F11.x–F16.x, F18.x, F19.x, Z71.5, and Z72.2.
  • Psychoses: F20.x, F22.x–F25.x, F28.x, F29.x, F30.2, F31.2, and F31.5.
  • Depression: F20.4, F31.3–F31.5, F32.x, F33.x, F34.1, F41.2, and F43.2.

References

  1. Zhou, P.; Yang, X.-L.; Wang, X.-G.; Hu, B.; Zhang, L.; Zhang, W.; Si, H.-R.; Zhu, Y.; Li, B.; Huang, C.-L.; et al. A pneumonia outbreak associated with a new coronavirus of probable bat origin. Nature 2020, 579, 270–273. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  2. Dong, E.; Du, H.; Gardner, L. An interactive web-based dashboard to track COVID-19 in real time. Lancet Infect. Dis. 2020, 20, 533–534. [Google Scholar] [CrossRef]
  3. Li, Q.; Guan, X.; Wu, P.; Wang, X.; Zhou, L.; Tong, Y.; Ren, R.; Leung, K.S.M.; Lau, E.H.Y.; Wong, J.Y.; et al. Early Transmission Dynamics in Wuhan, China, of Novel Coronavirus–Infected Pneumonia. N. Engl. J. Med. 2020, 382, 1199–1207. [Google Scholar] [CrossRef] [PubMed]
  4. Huang, C.; Wang, Y.; Li, X.; Ren, L.; Zhao, J.; Hu, Y.; Zhang, L.; Fan, G.; Xu, J.; Gu, X.; et al. Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. Lancet 2020, 395, 497–506. [Google Scholar] [CrossRef] [Green Version]
  5. Instituto Nacional de Estatística. A Mortalidade em Portugal n Contexto da Pandemia COVID-19. Available online: https://www.ine.pt/ngt_server/attachfileu.jsp?look_parentBoui=472955720&att_display=n&att_download=y (accessed on 20 December 2021).
  6. Harrison, S.L.; Fazio-Eynullayeva, E.; Lane, D.A.; Underhill, P.; Lip, G.Y.H. Comorbidities associated with mortality in 31,461 adults with COVID-19 in the United States: A federated electronic medical record analysis. PLOS Med. 2020, 17, e1003321. [Google Scholar] [CrossRef] [PubMed]
  7. Deng, Y.; Liu, W.; Liu, K.; Fang, Y.-Y.; Shang, J.; Zhou, L.; Wang, K.; Leng, F.; Wei, P.-F.; Chen, L.; et al. Clinical characteristics of fatal and recovered cases of coronavirus disease 2019 in Wuhan, China: A retrospective study. Chin. Med. J. 2020, 133, 1261–1267. [Google Scholar] [CrossRef]
  8. Zhang, B.; Zhou, X.; Qiu, Y.; Song, Y.; Feng, F.; Feng, J.Y.; Song, Q.; Jia, Q.; Wang, J. Clinical characteristics of 82 cases of death from COVID-19. PLoS ONE 2020, 15, e0235458. [Google Scholar] [CrossRef]
  9. Peckham, H.; de Gruijter, N.M.; Raine, C.; Radziszewska, A.; Ciurtin, C.; Wedderburn, L.R.; Rosser, E.C.; Webb, K.; Deakin, C.T. Male sex identified by global COVID-19 meta-analysis as a risk factor for death and ITU admission. Nat. Commun. 2020, 11, 6317. [Google Scholar] [CrossRef] [PubMed]
  10. Sze, S.; Pan, D.; Nevill, C.R.; Gray, L.J.; Martin, C.A.; Nazareth, J.; Minhas, J.S.; Divall, P.; Khunti, K.; Abrams, K.R.; et al. Ethnicity and clinical outcomes in COVID-19: A systematic review and meta-analysis. EClinicalMedicine 2020, 29, 100630. [Google Scholar] [CrossRef] [PubMed]
  11. Zhou, Y.; Yang, Q.; Chi, J.; Dong, B.; Lv, W.; Shen, L.; Wang, Y. Comorbidities and the risk of severe or fatal outcomes associated with coronavirus disease 2019: A systematic review and meta-analysis. Int. J. Infect. Dis. 2020, 99, 47–56. [Google Scholar] [CrossRef]
  12. Charlson, M.E.; Pompei, P.; Ales, K.L.; MacKenzie, C.R. A new method of classifying prognostic comorbidity in longitudinal studies: Development and validation. J. Chronic Dis. 1987, 40, 373–383. [Google Scholar] [CrossRef]
  13. Elixhauser, A.; Steiner, C.; Harris, D.R.; Coffey, R.M. Comorbidity Measures for Use with Administrative Data. Med. Care 1998, 36, 8–27. [Google Scholar] [CrossRef] [PubMed]
  14. WHO. International Statistical Classification of Diseases and Related Health Problems, 10th ed.; WHO: Geneva, Switzerland, 2011.
  15. Zhou, W.; Qin, X.; Hu, X.; Lu, Y.; Pan, J. Prognosis models for severe and critical COVID-19 based on the Charlson and Elixhauser comorbidity indices. Int. J. Med. Sci. 2020, 17, 2257–2263. [Google Scholar] [CrossRef] [PubMed]
  16. Kuswardhani, R.T.; Henrina, J.; Pranata, R.; Lim, M.A.; Lawrensia, S.; Suastika, K. Charlson comorbidity index and a composite of poor outcomes in COVID-19 patients: A systematic review and meta-analysis. Diabetes Metab. Syndr. Clin. Res. Rev. 2020, 14, 2103–2109. [Google Scholar] [CrossRef] [PubMed]
  17. World Health Organization. International Guidelines for Certification and Classification (Coding) of COVID-19 as Cause of Death; WHO: Geneva, Switzerland, 2020.
  18. Portal do Instituto Nacional de Estatística. População residente (No) por Local de residência (NUTS-2013), Sexo e Grupo etário; Anual. 2020. Available online: https://www.ine.pt/xportal/xmain?xpid=INE&xpgid=ine_indicadores&indOcorrCod=0008273&xlang=en (accessed on 20 December 2021).
  19. Moore, B.J.; White, S.; Washington, R.; Coenen, N.; Elixhauser, A. Identifying Increased Risk of Readmission and In-hospital Mortality Using Hospital Administrative Data: The AHRQ Elixhauser Comorbidity Index. Med. Care 2017, 55, 698–705. [Google Scholar] [CrossRef] [PubMed]
  20. van Walraven, C.; Austin, P.C.; Jennings, A.; Quan, H.; Forster, A.J. A Modification of the Elixhauser Comorbidity Measures Into a Point System for Hospital Death Using Administrative Data. Med. Care 2009, 47, 626–633. [Google Scholar] [CrossRef]
  21. Ahmad, O.B.; Boschi-Pinto, C.; Lopez, A.D.; Murray, C.J.L.; Lozano, R.; Inoue, M. Age Standardization of Rates: A New WHO Standard; Citeseer: Geneva, Switzerland, 2001; Volume 9. [Google Scholar]
  22. Gasparini, A. comorbidity: An R package for computing comorbidity scores. J. Open Source Softw. 2018, 3, 648. [Google Scholar] [CrossRef] [Green Version]
  23. Scully, E.P.; Haverfield, J.; Ursin, R.L.; Tannenbaum, C.; Klein, S.L. Considering how biological sex impacts immune responses and COVID-19 outcomes. Nat. Rev. Immunol. 2020, 20, 442–447. [Google Scholar] [CrossRef] [PubMed]
  24. Bertakis, K.D.; Azari, R.; Helms, L.J.; Callahan, E.J.; Robbins, J.A. Gender differences in the utilization of health care services. J. Fam. Pract. 2000, 49, 147–152. [Google Scholar]
  25. Klein, S.L.; Flanagan, K.L. Sex differences in immune responses. Nat. Rev. Immunol. 2016, 16, 626–638. [Google Scholar] [CrossRef]
  26. Machida, M.; Nakamura, I.; Saito, R.; Nakaya, T.; Hanibuchi, T.; Takamiya, T.; Odagiri, Y.; Fukushima, N.; Kikuchi, H.; Kojima, T.; et al. Adoption of personal protective measures by ordinary citizens during the COVID-19 outbreak in Japan. Int. J. Infect. Dis. 2020, 94, 139–144. [Google Scholar] [CrossRef]
  27. Suen, L.K.P.; Siu, G.K.H.; Guo, Y.P.; Yeung, S.K.W.; Lo, K.Y.K.; O’Donoghue, M. The public washroom-friend or foe? An observational study of washroom cleanliness combined with microbiological investigation of hand hygiene facilities. Antimicrob. Resist. Infect. Control 2019, 8, 47. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  28. Aylin, P.; Bottle, A.; Majeed, A. Use of administrative data or clinical databases as predictors of risk of death in hospital: Comparison of models. BMJ 2007, 334, 1044. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  29. Johnston, M.C.; Crilly, M.; Black, C.; Prescott, G.J.; Mercer, S.W. Defining and measuring multimorbidity: A systematic review of systematic reviews. Eur. J. Public Health 2019, 29, 182–189. [Google Scholar] [CrossRef]
  30. Kaplan, M.H.; Feinstein, A.R. The importance of classifying initial co-morbidity in evaluating the outcome of diabetes mellitus. J. Chronic Dis. 1974, 27, 387–404. [Google Scholar] [CrossRef]
  31. Sharabiani, M.T.A.; Aylin, P.; Bottle, A. Systematic Review of Comorbidity Indices for Administrative Data. Med. Care 2012, 50, 1109–1118. [Google Scholar] [CrossRef] [PubMed]
  32. Southern, D.; Quan, H.; Ghali, W.A. Comparison of the Elixhauser and Charlson/Deyo Methods of Comorbidity Measurement in Administrative Data. Med. Care 2004, 42, 355–360. [Google Scholar] [CrossRef] [PubMed]
  33. Thompson, N.R.; Fan, Y.; Dalton, J.E.; Jehi, L.; Rosenbaum, B.P.; Vadera, S.; Griffith, S.D. A New Elixhauser-based Comorbidity Summary Measure to Predict In-Hospital Mortality. Med. Care 2015, 53, 374–379. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  34. Izcovich, A.; Ragusa, M.A.; Tortosa, F.; Marzio, M.A.L.; Agnoletti, C.; Bengolea, A.; Ceirano, A.; Espinosa, F.; Saavedra, E.; Sanguine, V.; et al. Prognostic factors for severity and mortality in patients infected with COVID-19: A systematic review. PLoS ONE 2020, 15, e0241955. [Google Scholar] [CrossRef] [PubMed]
  35. Nogueira, P.J.; Nobre, M.D.A.; Costa, A.; Ribeiro, R.M.; Furtado, C.; Nicolau, L.B.; Camarinha, C.; Luís, M.; Abrantes, R.; Carneiro, A.V. The Role of Health Preconditions on COVID-19 Deaths in Portugal: Evidence from Surveillance Data of the First 20293 Infection Cases. J. Clin. Med. 2020, 9, 2368. [Google Scholar] [CrossRef]
  36. Tian, W.; Jiang, W.; Yao, J.; Nicholson, C.J.; Li, R.; Sigurslid, H.; Wooster, L.; Rotter, J.I.; Guo, X.; Malhotra, R. Predictors of mortality in hospitalized COVID-19 patients: A systematic review and meta-analysis. J. Med. Virol. 2020, 92, 1875–1883. [Google Scholar] [CrossRef]
  37. Dorjee, K.; Kim, H.; Bonomo, E.; Dolma, R. Prevalence and predictors of death and severe disease in patients hospitalized due to COVID-19: A comprehensive systematic review and meta-analysis of 77 studies and 38,000 patients. PLoS ONE 2020, 15, e0243191. [Google Scholar] [CrossRef]
  38. Wang, Z.; Deng, H.; Ou, C.; Liang, J.; Wang, Y.; Jiang, M.; Li, S. Clinical symptoms, comorbidities and complications in severe and non-severe patients with COVID-19: A systematic review and meta-analysis without cases duplication. Medicine 2020, 99, e23327. [Google Scholar] [CrossRef]
  39. Miller, L.E.; Bhattacharyya, R.; Miller, A.L. Diabetes mellitus increases the risk of hospital mortality in patients with COVID-19: Systematic review with meta-analysis. Medicine 2020, 99, e22439. [Google Scholar] [CrossRef]
  40. Alguwaihes, A.M.; Al-Sofiani, M.E.; Megdad, M.; Albader, S.S.; Alsari, M.H.; Alelayan, A.; Alzahrani, S.H.; Sabico, S.; Al-Daghri, N.M.; Jammah, A.A. Diabetes and COVID-19 among hospitalized patients in Saudi Arabia: A single-centre retrospective study. Cardiovasc. Diabetol. 2020, 19, 205. [Google Scholar] [CrossRef]
  41. Yang, J.; Tian, C.; Chen, Y.; Zhu, C.; Chi, H.; Li, J. Obesity aggravates COVID-19: An updated systematic review and meta-analysis. J. Med. Virol. 2021, 93, 2662–2674. [Google Scholar] [CrossRef]
  42. Huang, Y.; Lu, Y.; Huang, Y.-M.; Wang, M.; Ling, W.; Sui, Y.; Zhao, H.-L. Obesity in patients with COVID-19: A systematic review and meta-analysis. Metabolism 2020, 113, 154378. [Google Scholar] [CrossRef]
  43. Nogueira, P.J.; Nobre, M.D.A.; Nicola, P.J.; Furtado, C.; Carneiro, A.V. Excess Mortality Estimation During the COVID-19 Pandemic: Preliminary Data from Portugal. Acta. Med. Port. 2020, 33, 376–383. [Google Scholar] [CrossRef]
  44. Hajek, A.; Kretzler, B.; König, H.-H. Multimorbidity, Loneliness, and Social Isolation. A Systematic Review. Int. J. Environ. Res. Public Health 2020, 17, 8688. [Google Scholar] [CrossRef]
  45. Fernández-Niño, J.A.; Guerra-Gómez, J.A.; Idrovo, A.J. Multimorbidity patterns among COVID-19 deaths: Proposal for the construction of etiological models. Rev. Panam. Salud Pública 2020, 44, e166. [Google Scholar] [CrossRef]
  46. Zhang, C.H.; Schwartz, G.G. Spatial Disparities in Coronavirus Incidence and Mortality in the United States: An Ecological Analysis as of May 2020. J. Rural Health 2020, 36, 433–445. [Google Scholar] [CrossRef]
  47. Peixoto, V.R.; Vieira, A.; Aguiar, P.; Sousa, P.; Carvalho, C.; Thomas, D.; Abrantes, A.; Nunes, C. COVID-19: Determinants of Hospitalization, ICU and Death among 20,293 reported cases in Portugal. medRxiv 2020. [Google Scholar] [CrossRef]
  48. Bialek, S.; Bowen, V.; Chow, N.; Curns, A.; Gierke, R.; Hall, A.; Hughes, M.; Pilishvili, T.; Ritchey, M.; Roguski, K.; et al. Geographic Differences in COVID-19 Cases, Deaths, and Incidence—United States, February 12–April 7, 2020. MMWR. Morb. Mortal. Wkly. Rep. 2020, 69, 465–471. [Google Scholar] [CrossRef] [Green Version]
  49. Davies, N.G.; Jarvis, C.I.; Edmunds, W.J.; Jewell, N.P.; Diaz-Ordaz, K.; Keogh, R.H. Increased mortality in community-tested cases of SARS-CoV-2 lineage B.1.1.7. Nature 2021, 593, 270–274. [Google Scholar] [CrossRef]
  50. Mieno, M.N.; Tanaka, N.; Arai, T.; Kawahara, T.; Kuchiba, A.; Ishikawa, S.; Sawabe, M. Accuracy of Death Certificates and Assessment of Factors for Misclassification of Underlying Cause of Death. J. Epidemiol. 2016, 26, 191–198. [Google Scholar] [CrossRef] [Green Version]
  51. Nunes, A.M. Tecnologias de informação e comunicação no registro de óbitos em Portugal. J. Health Inform. 2017, 9, 137–141. [Google Scholar]
  52. Sousa Pinto, C.; Anderson, R.N.; Marques, C.; Maia, C.; Martins, H.; do Carmo Borralho, M. Improving the mortality information system in Portugal. Eurohealth 2016, 22, 48–51. [Google Scholar]
  53. Eisfeld, J. International Statistical Classification of Diseases and Related Health Problems. TSQ Transgender Stud. Q. 2014, 1, 107–110. [Google Scholar] [CrossRef] [Green Version]
  54. Pinto, C.S.; Anderson, R.N.; Martins, H.; Marques, C.; Maia, C.; Borralho, M.D.C. Mortality Information System in Portugal: Transition to e-death certification. Eurohealth 2016, 22, 1–53. [Google Scholar] [PubMed]
  55. Brooks, E.G.; Reed, K.D. Principles and Pitfalls: A Guide to Death Certification. Clin. Med. Res. 2015, 13, 74–82. [Google Scholar] [CrossRef] [Green Version]
Figure 1. Pairwise correlations between morbidities of the Elixhauser comorbidity index (A) and the Charlson comorbidity index (B).
Figure 1. Pairwise correlations between morbidities of the Elixhauser comorbidity index (A) and the Charlson comorbidity index (B).
Jcm 11 01898 g001
Table 1. COVID-19 death distributions and mortality rates (/100,000 individuals) by age group, district of residence, health region of residence, and location of death (total and according to sex).
Table 1. COVID-19 death distributions and mortality rates (/100,000 individuals) by age group, district of residence, health region of residence, and location of death (total and according to sex).
TotalSexRatio Rates M/FRatio Deaths M/Fp 1
FemaleMale
n%Rate /105n%Rate /105n%Rate /105
Total6701100.0 319947.5 350252.5 1.10.001
Age Group <0.001
0 to 9 years10.00.110.00.00-0.00.0-
10 to 19 years20.00.210.00.010.00.25.01.0
20 to 29 years50.10.420.10.130.10.56.61.5
30 to 39 years130.21.150.20.280.21.37.91.6
40 to 49 years600.93.8210.70.5391.15.211.31.9
50 to 59 years1652.511.1531.71.41123.216.011.42.1
60 to 69 years5378.040.91464.65.839111.264.611.22.7
70 to 79 years137220.5137.251516.124.285724.5197.38.11.7
>80 years453267.6664.9244876.5152.4208459.5849.25.60.9
Unknown10.00.110.00.00-0.00.0-
Crude Mortality Rate 65.5 58.8 72.11.2
Age-Standardized Mortality Rate 20.2 4.0 27.76.9
District of Residence 0.094
Aveiro4396.664.61936.054.02467.068.81.31.3
Beja600.942.9341.147.2260.736.10.80.8
Braga6529.778.932910.376.03239.274.61.01.0
Bragança1532.3124.4702.2107.9832.4127.91.21.2
Castelo Branco1221.868.7632.067.4591.763.10.90.9
Coimbra2123.252.11003.146.41123.252.01.11.1
Évora871.357.6501.663.2371.146.80.70.7
Faro590.913.5300.913.0290.812.61.01.0
Guarda1231.886.1702.292.3531.569.90.80.8
Leiria2223.348.41093.445.31133.247.01.01.0
Lisboa145121.763.167421.155.177722.263.61.21.2
Portalegre661.064.0351.164.4310.957.00.90.9
Porto165924.893.176824.081.489125.494.51.21.2
Santarém2503.758.21203.853.01303.757.51.11.1
Setúbal4807.256.32307.251.22507.155.61.11.1
Viana do Castelo1301.956.8652.052.8651.952.81.01.0
Vila Real1562.382.0722.371.4842.483.31.21.2
Viseu2053.157.9973.052.01083.157.81.11.1
Madeira Island130.25.290.36.840.13.00.40.4
Santa Maria Island170.312.4140.419.830.1112.75.70.2
Terceira Island20.03.610.03.510.033.79.61.0
Graciosa Island10.023.90-0.010.01.4--
Unknown1422.1-662.1-762.2--1.2
Health Region of Residence 0.013
ARS Alentejo2283.432.61304.135.7982.829.20.80.8
ARS Algarve590.924.4300.924.0290.824.81.01.0
ARS Centro96414.443.246814.639.949614.247.01.21.1
ARS LVT223633.477.9104932.868.7118733.988.51.31.1
ARS Norte303945.485.2143244.876.0160745.995.51.31.1
AR Açores200.37.9150.511.150.14.20.40.3
AR Madeira130.23.090.33.940.11.90.50.4
Unknown1422.1-662.1-762.2--1.2
Location of Death <0.001
Unknown410.6-270.8-140.4--0.5
In a public health institution608390.8-281588.0-326893.3--1.2
At home2864.3-1595.0-1273.6--0.8
Nursing home2804.2 1936.0 872.5 0.5
Private health institution110.2-50.2-60.2--1.2
Notes: 1 Homogeneity test. Abbreviations: ARS, administrative region of health; AR, autonomous region.
Table 2. Comorbidities present (identified according to the Elixhauser comorbidity index), total, by sex and age group.
Table 2. Comorbidities present (identified according to the Elixhauser comorbidity index), total, by sex and age group.
ConditionsTotalMaleFemalep 1<40 Years40–49 Years50–59 Years60–69 Years70–79 Years80+ Yearsp 1
Hypertension, uncomplicated30.0031.0129.070.1474.7613.3318.7928.8631.8530.32<0.001
Diabetes, uncomplicated16.8317.3516.360.3254.7615.0019.3920.1122.2314.81<0.001
Congestive heart failure14.4816.8512.31<0.0019.523.339.709.6812.8315.91<0.001
Renal failure12.8312.2513.360.20514.295.0012.7310.6113.1913.11<0.001
Cardiac arrhythmias10.6310.5710.680.8864.763.337.887.459.6911.52<0.001
Chronic pulmonary disease7.136.757.480.2640.005.0010.918.947.656.69<0.001
Solid tumor, without metastasis7.065.978.050.0014.7613.3313.338.578.096.27<0.001
Obesity5.245.854.680.0384.768.337.8810.616.204.19<0.001
Other neurological disorders4.864.415.280.1050.0011.675.457.084.664.55<0.001
Fluid and electrolyte disorders3.844.003.680.5070.005.002.423.543.284.10<0.001
Blood loss anemia0.030.030.030.9490.000.000.000.000.000.040.323
Notes: 1 Homogeneity test; significant results are presented in bold font. Only the 10 most frequent conditions are presented; the full table is presented in the Supplementary Materials (Table S2).
Table 3. COVID-19 deaths distributions (as a percentage of total deaths) and the Charlson and Elixhauser comorbidity (AHRQ and VW) weighted indexes by sex, age group, health region of residence, district of residence, location of death, and pandemic period.
Table 3. COVID-19 deaths distributions (as a percentage of total deaths) and the Charlson and Elixhauser comorbidity (AHRQ and VW) weighted indexes by sex, age group, health region of residence, district of residence, location of death, and pandemic period.
Charlson Weighted IndexAHRQ Elixhauser Comorbidity IndexVW Elixhauser Comorbidity Index
01–23–4≥5p<001–4≥5p<001–4≥5p
Total40.242.013.64.1 15.442.97.434.2 4.149.79.536.7
Sex
Female40.142.313.93.80.59915.942.76.934.50.2774.849.58.8370.011
Male40.541.813.44.414.943.17.9343.549.910.236.5
Age Group
0 to 18 years100000<0.0010100000.103010000<0.001
19 to 49 years5031.2108.810505356.25011.232.5
50 to 59 years39.44013.96.713.338.89.738.24.84313.338.8
60 to 69 years4040.613.26.116.439.79.534.57.144.913.234.8
70 to 79 years37.743.113.55.816.841.58.533.24.249.311.734.8
80 to 106 years40.942.213.83.21543.86.834.43.750.58.237.6
Health Region of Residence
ARS Alentejo3943.4143.50.02819.738.67.534.2<0.001747.4738.6<0.001
ARS Algarve30.544.116.98.520.335.63.440.73.447.510.239
ARS Centro39.743.413.23.714.643.47.534.52.250.48.439
ARS LVT37.242.415.15.315.838.27.738.23.945.810.439.9
ARS Norte42.641.412.73.314.946.47.431.34.752.29.433.7
AR Açores3035251025305400501040
AR Madeira38.546.2015.47.753.8038.5038.530.830.8
District of Residence
Aveiro45.839.210.94.1<0.00112.851.56.629.2<0.0012.557.49.630.5<0.001
Beja4048.311.7026.73011.731.711.74011.736.7
Braga39.340.815.64.31642.28.633.3 4.848.610.136.5
Bragança3445.8155.212.437.96.543.14.641.26.547.7
Castelo Branco4147.59.81.61845.16.630.34.954.95.734.4
Coimbra38.744.812.34.29.942.5839.61.945.310.442.5
Évora43.741.412.62.317.243.75.733.34.655.24.635.6
Faro30.544.116.98.520.335.63.440.73.447.510.239
Guarda40.739.817.12.415.441.56.536.61.646.35.746.3
Leiria38.741.913.55.917.1414.537.41.8508.140.1
Lisboa3642.515.46.115.437.66.9403.945.310.340.5
Portalegre39.434.816.79.116.743.94.534.87.6476.139.4
Porto44.141.711.82.515.746.87.929.55.25310.431.3
Santarém39.239.6165.214.840.46.8383.246.88.841.2
Setúbal38.145.214.22.517.737.311.533.54.245.811.938.1
Viana do Castelo43.841.510.83.812.354.64.628.53.160.85.430.8
Vila Real37.842.9163.213.547.43.835.34.550.63.241.7
Viseu40.543.912.72.914.143.99.832.21.550.77.340.5
Madeira Island38.546.2015.47.753.8038.5038.530.830.8
Santa Maria Island29.429.429.411.823.523.55.947.1041.211.847.1
Terceira Island505000505000010000
Graciosa Island010000010000010000
Place of Death
Public heath institution39.84213.94.3<0.00115.5427.535<0.0014.348.79.537.5<0.001
At home5730.89.82.412.257.74.225.92.163.38.725.9
Nursing home34.651.811.42.116.146.81027.12.55510.731.8
Private health institution18.263.618.2027.327.39.136.49.145.59.136.4
Unknown46.346.37.3017.153.74.924.44.956.12.436.6
Time Periods
T142.74013.83.50.01714476.432.60.0043.552.59.234.80.326
T234.146.611.77.618.635.25.340.94.946.29.839
T34042.313.7415.642.47.834.24.249.29.537.1
Notes: Significant results are presented in bold font. Abbreviations: ARS, administrative region of health; AR, autonomous region. Periods: T1—2 March–2 June; T2—3 June–11 August; and T3—12 August–31 December.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Nogueira, P.J.; de Araújo Nobre, M.; Elias, C.; Feteira-Santos, R.; Martinho, A.C.-V.; Camarinha, C.; Bacelar-Nicolau, L.; Costa, A.S.; Furtado, C.; Morais, L.; et al. Multimorbidity Profile of COVID-19 Deaths in Portugal during 2020. J. Clin. Med. 2022, 11, 1898. https://doi.org/10.3390/jcm11071898

AMA Style

Nogueira PJ, de Araújo Nobre M, Elias C, Feteira-Santos R, Martinho AC-V, Camarinha C, Bacelar-Nicolau L, Costa AS, Furtado C, Morais L, et al. Multimorbidity Profile of COVID-19 Deaths in Portugal during 2020. Journal of Clinical Medicine. 2022; 11(7):1898. https://doi.org/10.3390/jcm11071898

Chicago/Turabian Style

Nogueira, Paulo Jorge, Miguel de Araújo Nobre, Cecília Elias, Rodrigo Feteira-Santos, António C.-V. Martinho, Catarina Camarinha, Leonor Bacelar-Nicolau, Andreia Silva Costa, Cristina Furtado, Liliane Morais, and et al. 2022. "Multimorbidity Profile of COVID-19 Deaths in Portugal during 2020" Journal of Clinical Medicine 11, no. 7: 1898. https://doi.org/10.3390/jcm11071898

APA Style

Nogueira, P. J., de Araújo Nobre, M., Elias, C., Feteira-Santos, R., Martinho, A. C. -V., Camarinha, C., Bacelar-Nicolau, L., Costa, A. S., Furtado, C., Morais, L., Rachadell, J., Pinto, M. P., Pinto, F., & Vaz Carneiro, A. (2022). Multimorbidity Profile of COVID-19 Deaths in Portugal during 2020. Journal of Clinical Medicine, 11(7), 1898. https://doi.org/10.3390/jcm11071898

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop