Next Article in Journal
Development of a Method for Detecting and Estimating Moniliophthora roreri Spore Loads Based on Spore Traps and qPCR
Next Article in Special Issue
Neuroimaging of Cryptococcal Meningitis in Patients without Human Immunodeficiency Virus: Data from a Multi-Center Cohort Study
Previous Article in Journal
Impact of Hyperglycemia and Diabetes Mellitus on Breakthrough Mucormycosis Outcomes in Patients with Hematologic Malignancies—Complex and Intriguing Associations
Previous Article in Special Issue
Prevalence and Associated Factors of Cryptococcal Antigenemia in HIV-Infected Patients with CD4 < 200 Cells/µL in São Paulo, Brazil: A Bayesian Analysis
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Sterile Cerebrospinal Fluid Culture at Cryptococcal Meningitis Diagnosis Is Associated with High Mortality

1
Department of Medicine, University of Minnesota, Minneapolis, MN 55455, USA
2
Infectious Diseases Institute, Makerere University, Kampala 7062, Uganda
3
School of Public Health, University of Minnesota, Minneapolis, MN 55455, USA
4
Department of Medicine, Mbarara University of Science and Technology, Mbarara 1410, Uganda
*
Author to whom correspondence should be addressed.
J. Fungi 2023, 9(1), 46; https://doi.org/10.3390/jof9010046
Submission received: 20 October 2022 / Revised: 21 December 2022 / Accepted: 22 December 2022 / Published: 28 December 2022
(This article belongs to the Special Issue Cryptococcus and Cryptococcosis 2.0)

Abstract

:
Cryptococcus is the leading cause of AIDS-related meningitis in sub-Saharan Africa. The clinical implications of a sterile cerebrospinal fluid (CSF) culture among individuals diagnosed with cryptococcal meningitis using CSF cryptococcal antigen (CrAg) are unclear. We prospectively enrolled 765 HIV-positive Ugandans with first-episode cryptococcal meningitis from November 2010 to May 2017. All persons were treated with amphotericin-based induction therapy. We grouped participants by tertile of baseline CSF quantitative Cryptococcus culture burden and compared clinical characteristics, CSF immune profiles, and 18-week mortality. We found 55 (7%) CSF CrAg-positive participants with sterile CSF cultures. Compared to the non-sterile groups, participants with sterile CSF cultures had higher CD4 counts, lower CSF opening pressures, and were more frequently receiving ART. By 18 weeks, 47% [26/55] died in the sterile culture group versus 35% [83/235] in the low culture tertile, 46% [107/234] in the middle tertile, and 56% [135/241] in the high tertile (p < 0.001). The sterile group had higher levels of CSF interferon-gamma (IFN-γ), IFN-α, interleukin (IL)-6, IL-17, G-CSF, GM-CSF, and chemokine CXCL2 compared with non-sterile groups. Despite persons with sterile CSF cultures having higher CD4 counts, lower CSF opening pressures, and CSF cytokine profiles associated with better Cryptococcus control (e.g., IFN-γ predominant), mortality was similar to those with higher fungal burdens. This unexpected finding challenges the traditional paradigm that increasing CSF fungal burdens are associated with increased mortality but is consistent with a damage-response framework model.

1. Introduction

Cryptococcus neoformans is the leading cause of AIDS-related meningitis in sub-Saharan Africa [1,2,3]. Cryptococcal meningitis is often diagnosed by detecting cryptococcal antigen (CrAg) in cerebrospinal fluid (CSF), although CSF fungal culture is the traditional gold standard. CrAg testing has proven more sensitive than fungal culture in the CSF [4]; 10–15% of persons with symptomatic cryptococcal meningitis may be CrAg positive yet without any Cryptococcus yeast isolated on culture. In ART-naïve populations, higher CSF fungal burdens generally correlate with increasing mortality in first-episode cryptococcal meningitis [5]. However, it is unknown whether persons with no culture growth (i.e., sterile CSF) have lower mortality per the traditional paradigm.
Despite increasing access to ART throughout Africa in the last decade, many persons still present with advanced HIV disease [6]. In general, routine HIV care has moved away from CD4 testing at diagnosis and rather focused on early ART initiation and viral load monitoring. Test-and-treat strategies may carry a disadvantage in persons with early cryptococcal disease, as initiating ART without antifungal therapy could unmask cryptococcal meningitis. Rhein et al. demonstrated that persons who developed cryptococcal meningitis after initiating ART within the prior 14 days of diagnosis had 2-fold worse mortality [7]. It is becoming increasingly common to encounter ART-experienced persons with cryptococcal meningitis, but it remains unknown what impact ART might have on those with cryptococcal meningitis but sterile CSF cultures.
During a large randomized trial [8], we found a notable proportion of persons with cryptococcal meningitis who presented with positive CSF CrAg LFA but with sterile CSF fungal cultures [9]. In order to study this unique patient subset, we subsequently identified all persons in our database with sterile CSF cultures over a 7-year period of prospective enrollment. We compared the baseline characteristics, CSF cytokine profiles, and 18-week survival outcomes of persons with sterile CSF against those in tertile groupings of low, middle, or high CSF quantitative Cryptococcus growth, and analyzed the impact that recently initiated ART might play in this unique subset of persons with cryptococcal meningitis.

2. Materials and Methods

We prospectively enrolled HIV-positive persons with cryptococcal meningitis into clinical cohorts from November 2010 to May 2017 in Kampala and Mbarara, Uganda. We screened hospitalized persons with clinical evidence of meningitis by lumber puncture for cryptococcal meningitis using the CrAg lateral flow assay (LFA) (IMMY, Norman, Oklahoma, USA). Participants would then typically enter either of the two main trials occurring over that period. The Cryptococcal Optimal ART Timing (COAT) trial and the Adjunctive Sertraline for the Treatment of HIV-Associated Cryptococcal Meningitis (ASTRO-CM) trials have been previously described [8,10,11]. Full inclusion and exclusion criteria can be found at ClinicalTrials.gov (NCT01075152; NCT01802385). All participants received antifungal induction therapy at meningitis diagnosis with amphotericin B deoxycholate (0.7–1.0 mg/kg/day) and fluconazole (800 mg/day) for up to 2 weeks, followed by fluconazole consolidation and maintenance therapy. All participants also received key supportive therapies including therapeutic lumbar pressures for elevated intracranial pressure, intravenous fluids, and electrolyte repletion. The COAT trial followed participants for 46 weeks, while the primary endpoint for the ASTRO-CM trial was 18-week survival. This analysis includes participants enrolled in any of the dose-finding pilot studies, the main interventional trials, or an open label compassionate care arm over that time period.
Quantitative CSF cultures were obtained on all participants at baseline, day 3, 7, and 14, as described [12]. Cryptococcal meningitis diagnosis was based on a positive CrAg LFA in the CSF regardless of CSF culture positivity. Quantitative CSF cultures were performed with five serial 1:10 dilutions of 100 μL of CSF. Sterile cultures were defined as no growth of Cryptococcus after 10 days of incubation on Sabouraud Dextrose agar (1 colony forming unit (CFU)/mL limit-of-detection) [12]. Clinical characteristics, laboratory results, and outcomes were collected via study-specific case report forms through the 18 weeks of study follow up.
To assess whether persons with sterile CSF cultures had an immune response differing from those with fungal growth, we measured biomarkers from cryopreserved baseline CSF specimens. We analyzed 44 different cytokines using the Human XL Cytokine Magnetic Luminex assay (RD Biosystems, Minneapolis, MN, USA) according to manufacturer’s protocols.
The primary analysis was mortality through 18 weeks, which accounts for 95% of 5-year mortality in cryptococcal meningitis [13]. We categorized participants by tertile of CSF culture quantification, resulting in four groups: (1) persons with sterile cultures (Sterile), (2) persons with 1–14,700 CFU/mL (Low Tertile), persons with 14,701–206,000 CFU/mL (Middle Tertile), and persons with >206,000 CFU/mL (High Tertile). We compared baseline characteristics across groups by Chi-square for categorical variables or Kruskall-Wallis for continuous variables. Cytokines were analyzed by comparing the log2 mean of any given biomarker across all groups using unadjusted linear regression. We prioritized identifying biomarkers that met criteria for both p-value <0.05 and that were at least 2-fold different in the sterile group versus any of the fungal growth tertiles. Both 30-day mortality and 18-week mortality were compared by Chi-square testing of group proportions. We used a Kaplan–Meier time-to-event model with log-rank testing to visualize 18-week mortality by each group, and a Cox proportional hazards model to determine hazard ratios by tertile grouping. Survival analyses presented in the main manuscript are unadjusted. As a sensitivity analysis, we included two models adjusting for the impact of CD4 count and receiving ART. The first model adds baseline CD4 count as a co-variate. The second model adds both baseline CD4 count and baseline ART receipt as co-variates. Both models account for missing values using a multivariate imputation by chained equations (MICE) methodology. Analyses were completed using both SAS (SAS Institute; Cary, NC, USA) and R (R Foundation; Vienna, Austria).
All participants provided written informed consent as part of the parent trials, including consent for the future testing of clinical samples for research purposes. IRB approval was obtained from the appropriate Ugandan and US institutions.

3. Results

We enrolled a total of 765 HIV-positive participants with cryptococcal meningitis for this analysis. Of those, 55 (7%) participants had sterile CSF cultures. By CSF quantitative tertile, 235 participants were grouped in the low 1–14,700 CFU/mL category, 234 participants in the middle 14,701–206,000 category, and 241 participants in the high >206,000 CFU/mL category. Compared across the tertile groupings, those with sterile cultures had significantly higher CD4 counts (median 70 cells/µL [IQR, 25–96]) and were more likely to be receiving ART at time of diagnosis (71% [39/55]) (Table 1). The sterile culture group also had significantly fewer participants with lumbar puncture opening pressures >250 mm H2O (21% [10/55]) and lower overall mean opening pressure (170 mm H2O [11,12,13,14,15,16,17,18,19,20,21,22,23,24]). Finally, the proportion with CSF pleocytosis (>5 white cells/µL) was significantly higher in both the sterile group (46% [23/55]) and low tertile group (54% [124/235]) compared with the high tertile group (26% [61/241]).
We analyzed mortality by CSF quantitative culture tertile. By 30 days, 40% [22/55] died in the sterile culture group versus 26% [62/235] in low tertile, 34% [79/234] in the middle tertile, and 41% [98/241] in the high tertile groups (p < 0.01). Similarly through 18 weeks, 47% [26/55] died in the sterile culture group versus 35% [83/235] in low tertile, 46% [107/234] in the middle tertile, and 56% [135/241] in the high tertile groups (p < 0.001) (Figure 1). When analyzing survival through 18 weeks in a time-to-event unadjusted Cox regression model (using the low tertile group as reference), the sterile group had a mortality hazard ratio of 1.57 [95%CI, 1.01–2.44], falling between the middle tertile group (hazard ratio = 1.42 [95%CI, 1.07–1.89]) and the high tertile group (hazard ratio = 1.88 [95%CI, 1.43–2.48]) in terms of risk. As a sensitivity analysis, we created two multivariate models to adjust for CD4 count and ART receipt (Table 2). The hazard ratio for the sterile group was similar (hazard ratio = 1.55 [95%CI, 0.99–2.45]) after adjustment for CD4 count and ART status.
Persons with sterile cultures were more likely to be receiving ART than persons in the other tertiles. Given the high mortality found in the sterile culture group, we analyzed the timing of ART initiation in this group. We found that persons with sterile cultures initiated ART a median of 2.3 months (IQR, 1.1–12.8) prior to diagnosis, a shorter time frame as compared to the low tertile group at 4.1 months (0.8–19.7), although this difference was not statistically significant (Table 1).
A subset of 334 participants had baseline CSF cytokine measurements, including 24 persons in the sterile group. Of the 44 cytokines quantified from CSF in our participants, a total of 7 were found in statistically higher concentrations in the sterile group compared to those with any tertile of fungal growth (Table 3). The cytokines found to be at least 2-fold higher in the sterile group include: interleukin (IL)-6, IL-17, interferon (IFN)-γ, IFN-α, granulocyte colony-stimulating factor (G-CSF), granulocyte-macrophage colony-stimulating factor (GM-CSF), and chemokine CXCL2 (macrophage inflammatory protein 2-alpha). No cytokines measured were two-fold lower in the sterile culture group. A full table of all 44 cytokines categorized by functional group can be found in Supplemental Table S1.

4. Discussion

Overall, our results demonstrate that persons with HIV-associated cryptococcal meningitis but without fungal growth in CSF have higher early 30-day mortality, comparable to those with the highest fungal burden. By 18 weeks, mortality falls between the middle and high tertile groups, but remains significantly higher than the low tertile group. The traditional paradigm considered increasing CSF fungal burden as having increased risk of mortality. However, with the introduction of highly sensitive, non-culture based diagnostics [4], cases of cryptococcal meningitis without fungal growth are increasingly recognized [9]. Until now, it was not known how cryptococcal meningitis outcomes might differ between those with sterile cultures versus those with varying degrees of fungal burden. We have observed that those with sterile cultures have unexpected relatively high mortality, although the cause of that higher mortality is not entirely clear.
We note some surprising features about this cohort. The first being that the sterile culture group paradoxically features several characteristics classically associated with lower mortality in cryptococcal meningitis. Only 21% [10/55] of the sterile culture group had substantially elevated CSF opening pressures (>250 mm H2O), compared to all other tertiles having at least 45% of persons with high opening pressures. Prior studies have demonstrated that high baseline opening pressures may be associated with increased mortality [14,15], and control of intracranial pressure is associated with improved survival in cryptococcal meningitis [16]. Thus, one could expect that the sterile group might have better outcomes given that significantly fewer participants in this group have elevated opening pressures. Second, persons with sterile cultures had higher levels of CSF white cells than those with the highest fungal burden, suggesting a more robust immune response in the central nervous system. Lack of white blood cells in the CSF has previously been linked to poorer outcomes [5], and we otherwise find a fairly linear relationship of decreasing CSF white cells with increasing fungal burden in those with CSF growth. Thus, it was surprising to find increased CSF cellular infiltrate in our high mortality sterile group, which lead us to additional inquiries about the immune response in persons with sterile CSF.
In order to better understand the immune response in the sterile culture group, we compared quantitative CSF cytokine measurements across the groups. We found seven cytokines to be significantly higher in the sterile culture group as compared to any fungal growth tertile: IL-6, IL-17, IFN-γ, IFN-α, G-CSF, GM-CSF, and CXCL2. To date, IFN-γ, IL-6, G-CSF, and GM-CSF have been the best described in the literature. A principal component analysis by Jarvis et al. found that IL-6 and IFN-γ were key factors associated with rapid clearance of Cryptococcus yeast from the CSF [17]. Interestingly, Jarvis et al. found that IL-17 positively correlated with IFN-γ concentrations and may also play a role in fungal clearance. Herein, we found decreasing IL-17 and decreasing IFN-γ associated with higher fungal burdens. Elevated IFN-γ and G-CSF concentrations have previously been reported in persons who develop paradoxical immune reconstitution inflammatory syndrome (IRIS) [18]. Elevated GM-CSF at diagnosis has been associated with an increased likelihood of later developing paradoxical IRIS [17]. IFN-α is a Type 1 interferon classically associated with upregulation as a viral response, while CXCL2 (MIP2-α) is secreted by macrophages as a chemotactic for polymorphonuclear leukocytes. Neither has been particularly implicated with the cryptococcal disease before. In summary, our participants with sterile CSF cultures have a cytokine profile that is consistent with fungal clearance, but which may also predispose to an IRIS-like phenomena. However, the elevated concentration of cytokines such as IFN-α and CXCL2 and the lack of other cytokines classically associated with fungal clearance (i.e., TNF-α) [19], in the setting of high mortality, may suggest this is a unique immunologic entity that is not appropriately considered by current treatment guidelines which draw on historically ART-naïve populations.
One hypothesis for the increased cellular infiltrate and upregulated cytokine production seen in the sterile culture group relates to that of immune reconstitution. Persons with sterile cultures were more likely to be receiving ART at the time of their meningitis diagnosis, were more likely to have recently initiated ART, and had significantly higher median CD4 counts. Presumably, more functioning CD4 cells allows for a more robust immune response against the opportunistic infection. However, this may be a proverbial “double-edged sword” in the population. Scriven et al. demonstrated in a randomized trial that persons who started ART early after cryptococcal meningitis diagnosis (and whom subsequently had worse outcome) had increased cellular infiltrate and type 2 T helper responses in the CSF [20]. Additionally, Rhein et al. found that persons with advanced HIV who recently initiated ART and were subsequently diagnosed with meningitis within 2 weeks had poorer outcomes than those who were ART-naïve [7]. Termed “unmasking” disease, the hypothesis is that ART-triggered recovery of CD4 cells and reactivation of the immune system may be detrimental in persons with cryptococcal infection without first treating with antifungal therapy [21,22].
Pirofski and Casadevall proposed the damage-response framework model whereby clinical disease can occur by either uncontrolled pathogen dissemination or a detrimental host immune response, but with the common denominator being host damage (Figure 2) [23,24]. Immune reconstitution inflammatory syndrome (IRIS) is a dysregulated immune response to an opportunistic pathogen upon restoration of a depleted component of the immune system [22]. Excess inflammation in the brain in cryptococcal meningitis IRIS can be fatal [25,26]. In contrast to unmasking disease, paradoxical cryptococcal IRIS occurs subsequent to diagnosis and treatment of infection and after initiating ART. Paradoxical IRIS may present anywhere from a few days to years after ART initiation in cryptococcal meningitis, while the incidence and mortality are variable, with incidence ranging between 8–49% and mortality between 0–36% [22,27,28]. While unmasking cryptococcal disease and paradoxical cryptococcal IRIS occur at different time points in relation to diagnosis, the pathogenesis of the two may be similar.
Alternatively, the unique pathogenesis of this sterile cohort may not be due to ART-associated unmasking IRIS, but rather some other immune related entity. In contrast to the Rhein et al. findings, the larger ACTA trial group observed only weak evidence of a trend toward increased mortality in those recently initiating ART, reporting a 2-week mortality of 23% for those initiating ART within 2 weeks prior to diagnosis compared to 15% initiating ART >2 weeks (Hazard Ratio = 1.70; 95%CI, 0.85–3.39; p = 0.13) [30]. One possibility is that the specific strain of Cryptococcus may be driving the unique immune response. Nielsen and colleagues have demonstrated in mouse models that different clinically isolated human Cryptococcus strains yield different survival outcomes in the mice, and have identified numerous genes that may influence this differential pathogenicity [31,32]. Unfortunately, identifying strain specific disease in real time is not currently possible, but would certainly be a future area of research interest. Lastly, we feel misidentified disease due to false positive antigen testing is highly unlikely given the clinical presentation is consistent with cryptococcal meningitis, the fact that the majority do still respond to treatment, and the well-established excellent specificity of the CrAg LFA [4].
A limitation in studying the global cytokine profile of a severely immunocompromised person is the potential for other pathogens to influence the findings. All of our patients were admitted for presentations consistent with meningitis but may have had other opportunistic infections. In particular, the presence of IFN-α may suggest the presence of viral co-infections in higher proportions in the sterile culture group. Unfortunately, we do not consistently collect HIV viral loads in this cohort, which could also influence IFN-α. However, our large sample size of similarly immunocompromised persons should reflect a relatively even distribution of concomitant opportunistic infections. We cannot determine how cytokine profiles might have changed over time as our measurements are at baseline. We do not have paired blood cytokine measurements to determine if the CSF findings are a compartmentalized phenomenon versus a whole body response. It is possible that in rare situations cryptococcal yeast might not grow within 10 days of culture, and it is therefore possible persons in our cohort considered sterile might have had growth after 10 days. Lastly, we cannot generalize these findings to the subset of patients who have HIV-negative cryptococcal meningitis; however these findings may provide insight into the diversity of immune responses present in HIV-negative meningitis.
The other major limitation of our study was our survival analysis. We recognize that there are limitations with either an unadjusted or partially adjusted survival analysis, but we felt it valuable to draw early attention to the surprising features of this group to facilitate future study. We performed a sensitivity analysis to determine if the hazard ratios were similar when adjusting for baseline CD4 count and ART receipt. In general, the hazard ratio for the sterile culture group is similar (Table 2). We do think there are several features that are reassuring that an association exists. One, the hazard ratio measures the effect size, which is maintained after adjustment; if sample size was increased, we would anticipate the effect size would be similar but the power to detect a difference would increase. Two, neither CD4 count or ART receipt were independently associated with mortality. Additionally, three, we would argue that receiving ART at baseline (and subsequently higher CD4 counts) are in the causal pathway for CSF culture sterility in persons with cryptococcal meningitis. Variables in the causal pathway are not considered confounders and not typically adjusted for. The scope of this manuscript was to highlight the high level differences in this sterile culture group. Ultimately, a robust study would need to explore the complex interactions of ART status, adherence, timing of initiation, and immune reconstitution timing and quality to better understand these potentially complex interactions.
In conclusion, we find that persons with sterile CSF cultures at cryptococcal meningitis diagnosis have high mortality, similar to those with higher fungal burdens. This finding is particularly unexpected given the higher CD4 cell count, lower CSF opening pressures, and IFN-γ predominant CSF pro-inflammatory cytokine profile of the group. Our observations challenge the traditional paradigm that increasing CSF fungal burdens are associated with increased mortality. However, sterile culture cryptococcal meningitis may rather represent a unique disease entity consistent with a parabolic damage-response framework model. Further research is needed from a prospective, controlled cohort.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/jof9010046/s1, Supplemental Table S1. Baseline Mean CSF Biomarkers by CSF Quantitative Culture Tertile.

Author Contributions

Conceptualization: C.P.S., J.R., D.R.B. and D.B.M.; methodology: C.P.S., K.H.H., E.O., J.R., D.R.B. and D.B.M.; formal analysis: C.P.S., K.H.H., A.S. and D.R.B.; investigation: K.S., L.T., E.N., A.A., A.K.M. and M.A.; data curation: K.H.H. and A.S.; writing—original draft preparation: C.P.S.; writing—review and editing: K.H.H., A.S., D.A.W., K.S., E.O., L.T., E.N., A.A., A.K.M., M.A., C.M., J.R., D.R.B. and D.B.M.; supervision: J.R., D.R.B. and D.B.M.; project administration: D.A.W. and C.M.; funding acquisition: J.R., D.R.B. and D.B.M. All authors have read and agreed to the published version of the manuscript.

Funding

This research was made possible through support from the National Institutes of Health’s National Center for Advancing Translational Sciences (KL2TR002492 and UL1TR002494); National Institute of Allergy and Infectious Diseases (T32AI055433, U01AI089244, K23AI138851); National Institute of Neurologic Disorders and Stroke (R01NS086312); and the Fogarty International Center (K01TW010268, K43TW010718).

Institutional Review Board Statement

The study was conducted in accordance with the Declaration of Helsinki, and approved by the Institutional Review Board (or Ethics Committee) of the University of Minnesota (COAT: IRB code number 0810M49622; ASTRO: IRB Code Number 1304M3136), and The Uganda National Council of Science and Technology (COAT: REF 2009-022; ASTRO: HS1406).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The data presented in this study are available on request from the corresponding author in a de-identified dataset.

Acknowledgments

We are appreciative of the study team and hospital staff working continuously to provide compassionate care to the participants and staff at Mulago National Referral Hospital and Mbarara Regional Referral Hospital. We thank Kirsten Nielsen, for conceptual input regarding Figure 2.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Durski, K.N.; Kuntz, K.M.; Yasukawa, K.; Virnig, B.A.; Meya, D.B.; Boulware, D.R. Cost-effective diagnostic checklists for meningitis in resource-limited settings. J. Acquir. Immune Defic. Syndr. 2013, 63, e101–e108. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  2. Ellis, J.; Bangdiwala, A.S.; Cresswell, F.V.; Rhein, J.; Nuwagira, E.; Ssebambulidde, K.; Tugume, L.; Rajasingham, R.; Bridge, S.C.; Muzoora, C.; et al. The Changing Epidemiology of HIV-Associated Adult Meningitis, Uganda 2015–2017. Open Forum Infect. Dis. 2019, 6, ofz419. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  3. Rajasingham, R.; Govender, N.P.; Jordan, A.; Loyse, A.; Shroufi, A.; Denning, D.W.; Meya, D.B.; Chiller, T.M.; Boulware, D.R. The global burden of HIV-associated cryptococcal infection in adults in 2020: A modelling analysis. Lancet Infect. Dis. 2022, 22, 1748–1755. [Google Scholar] [CrossRef]
  4. Boulware, D.R.; Rolfes, M.A.; Rajasingham, R.; von Hohenberg, M.; Qin, Z.; Taseera, K.; Schutz, C.; Kwizera, R.; Butler, E.K.; Meintjes, G.; et al. Multisite validation of cryptococcal antigen lateral flow assay and quantification by laser thermal contrast. Emerg. Infect. Dis. 2014, 20, 45–53. [Google Scholar] [CrossRef] [PubMed]
  5. Jarvis, J.N.; Bicanic, T.; Loyse, A.; Namarika, D.; Jackson, A.; Nussbaum, J.C.; Longley, N.; Muzoora, C.; Phulusa, J.; Taseera, K.; et al. Determinants of mortality in a combined cohort of 501 patients with HIV-associated Cryptococcal meningitis: Implications for improving outcomes. Clin. Infect. Dis. 2014, 58, 736–745. [Google Scholar] [CrossRef]
  6. Carmona, S.; Bor, J.; Nattey, C.; Maughan-Brown, B.; Maskew, M.; Fox, M.P.; Glencross, D.K.; Ford, N.; MacLeod, W.B. Persistent High Burden of Advanced HIV Disease Among Patients Seeking Care in South Africa’s National HIV Program: Data From a Nationwide Laboratory Cohort. Clin. Infect. Dis. 2018, 66, S111–S117. [Google Scholar] [CrossRef] [Green Version]
  7. Rhein, J.; Hullsiek, K.H.; Evans, E.E.; Tugume, L.; Nuwagira, E.; Ssebambulidde, K.; Kiggundu, R.; Mpoza, E.; Musubire, A.K.; Bangdiwala, A.S.; et al. Detrimental Outcomes of Unmasking Cryptococcal Meningitis With Recent ART Initiation. Open Forum Infect. Dis. 2018, 5, ofy122. [Google Scholar] [CrossRef] [Green Version]
  8. Rhein, J.; Hullsiek, K.H.; Tugume, L.; Nuwagira, E.; Mpoza, E.; E Evans, E.; Kiggundu, R.; A Pastick, K.; Ssebambulidde, K.; Akampurira, A.; et al. Adjunctive sertraline for HIV-associated cryptococcal meningitis: A randomised, placebo-controlled, double-blind phase 3 trial. Lancet Infect. Dis. 2019, 19, 843–851. [Google Scholar] [CrossRef]
  9. Williams, D.A.; Kiiza, T.; Kwizera, R.; Kiggundu, R.; Velamakanni, S.; Meya, D.B.; Rhein, J.; Boulware, D.R. Evaluation of fingerstick cryptococcal antigen lateral flow assay in HIV-infected persons: A diagnostic accuracy study. Clin. Infect. Dis. 2015, 61, 464–467. [Google Scholar] [CrossRef] [Green Version]
  10. Boulware, D.R.; Meya, D.B.; Muzoora, C.; Rolfes, M.A.; Hullsiek, K.H.; Musubire, A.; Taseera, K.; Nabeta, H.W.; Schutz, C.; Williams, D.A.; et al. Timing of antiretroviral therapy after diagnosis of cryptococcal meningitis. N. Engl. J. Med. 2014, 370, 2487–2498. [Google Scholar] [CrossRef]
  11. Rhein, J.; Morawski, B.M.; Hullsiek, K.H.; Nabeta, H.W.; Kiggundu, R.; Tugume, L.; Musubire, A.; Akampurira, A.; Smith, K.D.; Alhadab, A.; et al. Efficacy of adjunctive sertraline for the treatment of HIV-associated cryptococcal meningitis: An open-label dose-ranging study. Lancet Infect. Dis. 2016, 16, 809–818. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  12. Dyal, J.; Akampurira, A.; Rhein, J.; Morawski, B.M.; Kiggundu, R.; Nabeta, H.W.; Musubire, A.K.; Bahr, N.C.; A Williams, D.; Bicanic, T.; et al. Reproducibility of CSF quantitative culture methods for estimating rate of clearance in cryptococcal meningitis. Med. Mycol. 2016, 54, 361–369. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  13. Butler, E.K.; Boulware, D.R.; Bohjanen, P.R.; Meya, D.B. Long term 5-year survival of persons with cryptococcal meningitis or asymptomatic subclinical antigenemia in Uganda. PLoS ONE 2012, 7, e51291. [Google Scholar] [CrossRef] [Green Version]
  14. Graybill, J.R.; Sobel, J.; Saag, M.; Van Der Horst, C.; Powderly, W.; Cloud, G.; Riser, L.; Hamill, R.; Dismukes, R.; NIAID Mycoses Study Group; et al. Diagnosis and management of increased intracranial pressure in patients with AIDS and cryptococcal meningitis. The NIAID Mycoses Study Group and AIDS Cooperative Treatment Groups. Clin. Infect. Dis. 2000, 30, 47–54. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  15. Van der Horst, C.M.; Saag, M.S.; Cloud, G.A.; Hamill, R.J.; Graybill, J.R.; Sobel, J.D.; Johnson, P.C.; Tuazon, C.U.; Kerkering, T.; Moskovitz, B.L.; et al. Treatment of cryptococcal meningitis associated with the acquired immunodeficiency syndrome. National Institute of Allergy and Infectious Diseases Mycoses Study Group and AIDS Clinical Trials Group. N. Engl. J. Med. 1997, 337, 15–21. [Google Scholar] [CrossRef] [PubMed]
  16. Rolfes, M.A.; Hullsiek, K.H.; Rhein, J.; Nabeta, H.W.; Taseera, K.; Schutz, C.; Musubire, A.; Rajasingham, R.; Williams, D.A.; Thienemann, F.; et al. The effect of therapeutic lumbar punctures on acute mortality from cryptococcal meningitis. Clin. Infect. Dis. 2014, 59, 1607–1614. [Google Scholar] [CrossRef] [Green Version]
  17. Jarvis, J.N.; Meintjes, G.; Bicanic, T.; Buffa, V.; Hogan, L.; Mo, S.; Tomlinson, G.; Kropf, P.; Noursadeghi, M.; Harrison, T.S. Cerebrospinal fluid cytokine profiles predict risk of early mortality and immune reconstitution inflammatory syndrome in HIV-associated cryptococcal meningitis. PLoS Pathog. 2015, 11, e1004754. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  18. Boulware, D.R.; Bonham, S.C.; Meya, D.; Wiesner, D.L.; Park, G.S.; Kambugu, A.; Janoff, E.N.; Bohjanen, P. Paucity of initial cerebrospinal fluid inflammation in cryptococcal meningitis is associated with subsequent immune reconstitution inflammatory syndrome. J. Infect. Dis. 2010, 202, 962–970. [Google Scholar] [CrossRef] [Green Version]
  19. Bicanic, T.; Muzoora, C.; Brouwer, A.E.; Meintjes, G.; Longley, N.; Taseera, K.; Rebe, K.; Loyse, A.; Jarvis, J.; Bekker, L.-G.; et al. Independent association between rate of clearance of infection and clinical outcome of HIV associated cryptococcal meningitis: Analysis of a combined cohort of 262 patients. Clin. Infect. Dis. 2009, 49, 702–709. [Google Scholar] [CrossRef]
  20. Scriven, J.E.; Rhein, J.; Hullsiek, K.H.; Von Hohenberg, M.; Linder, G.; Rolfes, M.A.; Williams, D.A.; Taseera, K.; Meya, D.B.; Meintjes, G.; et al. Early ART after cryptococcal meningitis is associated with cerebrospinal fluid pleocytosis and macrophage activation in a multisite randomized trial. J. Infect. Dis. 2015, 212, 769–778. [Google Scholar] [CrossRef]
  21. Rajasingham, R.; Meya, D.B.; Boulware, D.R. Integrating cryptococcal antigen screening and pre-emptive treatment into routine HIV care. J. Acquir. Immune Defic. Syndr. 2012, 59, e85–e91. [Google Scholar] [CrossRef] [Green Version]
  22. Haddow, L.; Colebunders, R.; Meintjes, G.; Lawn, S.D.; Elliott, J.H.; Manabe, Y.C.; Bohjanen, P.; Sungkanuparph, S.; Easterbrook, P.J.; French, M.; et al. Cryptococcal immune reconstitution inflammatory syndrome in HIV-1-infected individuals: Proposed clinical case definitions. Lancet Infect. Dis. 2010, 10, 791–802. [Google Scholar] [CrossRef] [Green Version]
  23. Pirofski, L.A.; Casadevall, A. Immune-Mediated Damage Completes the Parabola: Cryptococcus neoformans Pathogenesis Can Reflect the Outcome of a Weak or Strong Immune Response. mBio 2017, 8, e02063-17. [Google Scholar] [CrossRef] [Green Version]
  24. Pirofski, L.A.; Casadevall, A. The damage-response framework of microbial pathogenesis and infectious diseases. Adv. Exp. Med. Biol. 2008, 635, 135–146. [Google Scholar]
  25. Wiesner, D.L.; Boulware, D.R. Cryptococcus-Related Immune Reconstitution Inflammatory Syndrome(IRIS): Pathogenesis and Its Clinical Implications. Curr. Fungal Infect. Rep. 2011, 5, 252–261. [Google Scholar] [CrossRef] [Green Version]
  26. Boulware, D.R.; Meya, D.B.; Bergemann, T.L.; Wiesner, D.L.; Rhein, J.; Musubire, A.; Lee, S.J.; Kambugu, A.; Janoff, E.N.; Bohjanen, P.R. Clinical features and serum biomarkers in HIV immune reconstitution inflammatory syndrome after cryptococcal meningitis: A prospective cohort study. PLoS Med. 2010, 7, e1000384. [Google Scholar] [CrossRef] [Green Version]
  27. Longley, N.; Harrison, T.S.; Jarvis, J.N. Cryptococcal immune reconstitution inflammatory syndrome. Curr. Opin. Infect. Dis. 2013, 26, 26–34. [Google Scholar] [CrossRef]
  28. Katchanov, J.; Blechschmidt, C.; Nielsen, K.; Branding, G.; Arastéh, K.; Tintelnot, K.; Meintjes, G.; Boulware, D.R.; Stocker, H. Cryptococcal meningoencephalitis relapse after an eight-year delay: An interplay of infection and immune reconstitution. Int. J. STD AIDS 2015, 26, 912–914. [Google Scholar] [CrossRef] [Green Version]
  29. Skipper, C.; Abassi, M.; Boulware, D.R. Diagnosis and Management of Central Nervous System Cryptococcal Infections in HIV-Infected Adults. J. Fungi 2019, 5, 65. [Google Scholar] [CrossRef] [Green Version]
  30. Kalata, N.; Ellis, J.; Kanyama, C.; Kuoanfank, C.; Temfack, E.; Mfinanga, S.; Lesikari, S.; Chanda, D.; Lakhi, S.; Nyazika, T.; et al. Short-term Mortality Outcomes of HIV-Associated Cryptococcal Meningitis in Antiretroviral Therapy-Naïve and -Experienced Patients in Sub-Saharan Africa. Open Forum Infect. Dis. 2021, 8, ofab397. [Google Scholar] [CrossRef]
  31. Altamirano, S.; Jackson, K.M.; Nielsen, K. The interplay of phenotype and genotype in Cryptococcus neoformans disease. Biosci. Rep. 2020, 40, BSR20190337. [Google Scholar] [CrossRef] [PubMed]
  32. Gerstein, A.C.; Jackson, K.M.; McDonald, T.R.; Wang, Y.; Lueck, B.D.; Bohjanen, S.; Smith, K.D.; Akampurira, A.; Meya, D.B.; Xue, C.; et al. Identification of Pathogen Genomic Differences That Impact Human Immune Response and Disease during Cryptococcus neoformans Infection. mBio 2019, 10, e01440-19. [Google Scholar] [CrossRef] [PubMed]
Figure 1. Kaplan–Meier Curve of 18-week Survival by CSF Fungal Burden Tertile in Cryptococcal Meningitis. Figure 1 demonstrates cumulative survival probability by Kaplan–Meier curve with Log Rank testing. At 18 weeks, the high tertile group has the poorest survival with 56% [135/241] dead, followed by the sterile culture group with 47% [26/55] dead, the middle tertile group with 46% [107/234] dead, and the low tertile group with 35% [83/235] dead. Log Rank p-value is calculated across all groups. CSF quantitative culture tertiles: Low Tertile = 1–14,700 CFU/mL; Middle Tertile = 14,701–206,000 CFU/mL; High Tertile > 206,000 CFU/mL.
Figure 1. Kaplan–Meier Curve of 18-week Survival by CSF Fungal Burden Tertile in Cryptococcal Meningitis. Figure 1 demonstrates cumulative survival probability by Kaplan–Meier curve with Log Rank testing. At 18 weeks, the high tertile group has the poorest survival with 56% [135/241] dead, followed by the sterile culture group with 47% [26/55] dead, the middle tertile group with 46% [107/234] dead, and the low tertile group with 35% [83/235] dead. Log Rank p-value is calculated across all groups. CSF quantitative culture tertiles: Low Tertile = 1–14,700 CFU/mL; Middle Tertile = 14,701–206,000 CFU/mL; High Tertile > 206,000 CFU/mL.
Jof 09 00046 g001
Figure 2. Simplified Damage-Response Parabolic Framework Model of Cryptococcal Meningitis Disease. Figure 2 demonstrates a hypothesized damage-response model for HIV-associated cryptococcal disease. The model proposes that clinical disease can occur by either of two mechanisms: (1) Uncontrolled fungal dissemination characterized by suboptimal IFN-γ predominated Type 1 helper (Th1) response with increasing CSF quantitative culture burden, or (2) Excessive damaging host immune response characterized by exaggerated IFN-γ predominated Th1 host response with persons presenting with sterile CSF cultures. The host is most likely protected by a moderate, balanced immune response. (Figure is originally adapted from Pirofski and Casadevall concept, and further modified from our prior publication [29]).
Figure 2. Simplified Damage-Response Parabolic Framework Model of Cryptococcal Meningitis Disease. Figure 2 demonstrates a hypothesized damage-response model for HIV-associated cryptococcal disease. The model proposes that clinical disease can occur by either of two mechanisms: (1) Uncontrolled fungal dissemination characterized by suboptimal IFN-γ predominated Type 1 helper (Th1) response with increasing CSF quantitative culture burden, or (2) Excessive damaging host immune response characterized by exaggerated IFN-γ predominated Th1 host response with persons presenting with sterile CSF cultures. The host is most likely protected by a moderate, balanced immune response. (Figure is originally adapted from Pirofski and Casadevall concept, and further modified from our prior publication [29]).
Jof 09 00046 g002
Table 1. Baseline characteristics by CSF quantitative culture burden in first-episode HIV-related cryptococcal meningitis.
Table 1. Baseline characteristics by CSF quantitative culture burden in first-episode HIV-related cryptococcal meningitis.
CSF Fungal BurdenSterileLow CFU
Tertile *
Middle CFU Tertile *High CFU
Tertile *
p-Value
(N = 55)(N = 235)(N = 234)(N = 241)
Demographics
Age, years37 [30, 42]34 [29, 40]36 [30, 40]35 [29, 40]0.09
Women27 (49.1%)99 (42.1%)84 (35.9%)100 (41.5%)0.25
Weight, kg55 [50, 60]53 [47, 60]52 [47, 60]52 [48, 59]0.52
Glasgow Coma Score < 1525 (45.5%)98 (41.9%)81 (34.8%)110 (45.6%)0.09
CD4 Count, cells/μL70 [25, 96]25 [9, 62]15 [6, 39]10 [6, 28]<0.001
Receiving HIV Therapy39 (70.9%)108 (46.2%)75 (32.1%)63 (26.1%)<0.001
Months of HIV Therapy †2.3 [1.1, 12.8]4.1 [0.8, 19.7]3.0 [0.7, 24.4]15.0 [1.1, 38.8]0.10
Baseline CSF Parameters
Opening Pressure > 250 mm H2O10 (20.8%)89 (45.2%)125 (60.7%)145 (67.1%)<0.001
Opening Pressure, mm H2O168
[113, 240]
230
[160, 340]
288
[200, 401]
348
[220, 473]
<0.001
CSF > 5 White Cells/μL23 (46.0%)124 (54.1%)72 (32.3%)61 (26.2%)<0.001
CSF Protein, mg/dL55 [20, 124]76 [28, 140]57 [23, 97]56 [24, 117]0.13
Data are N (%) or medians with [P25, 75]. p-value from Kruskal–Wallis test for medians, Chi-squared test for proportions. * Fungal burden tertiles: Low tertile = 1–14,700 CFU/mL; Middle tertile = 14,701–206,000 CFU/mL; High tertile > 206,000 CFU/mL. † Among those on ART at diagnosis. Abbreviations: CFU = colony forming units; CSF = cerebrospinal fluid. Data missingness as follows: Age (N = 0), Sex (N = 0), Weight (N = 165), Glasgow Coma Score (N = 2), CD4 (N = 40), Receiving HIV Therapy (N = 0), Months of HIV Therapy (N = 195), Opening pressure (N = 98), CSF White Cells (N = 30), CSF Protein (N = 96).
Table 2. Proportional Hazards Regression Models for 18 Week Mortality by CSF Quantitative Culture Burden using Low Tertile Grouping (1–14,700 CFU/mL) as the Reference Group.
Table 2. Proportional Hazards Regression Models for 18 Week Mortality by CSF Quantitative Culture Burden using Low Tertile Grouping (1–14,700 CFU/mL) as the Reference Group.
VariableHazard RatioLower Bound
95%CI
Upper Bound
95%CI
Univariate Unadjusted [N = 765]
Sterile Group1.571.012.44
Middle Tertile, 14,701–206,000 CFU/mL1.421.071.89
High Tertile, >206,000 CFU/mL1.881.432.48
Multivariate Adjusted Model #1 [N = 765]
Sterile Group1.631.042.56
Middle Tertile, 14,701–206,000 CFU/mL1.401.051.87
High Tertile, >206,000 CFU/mL1.841.402.45
CD4+ T cell count, per 10 cells/μL1.000.991.01
Multivariate Adjusted Model #2 [N = 765]
Sterile Group1.550.992.45
Middle Tertile, 14,701–206,000 CFU/mL1.431.071.91
High Tertile, >206,000 CFU/mL1.901.442.52
CD4+ T cell count, per 10 cells/μL1.000.991.01
Receiving ART at baseline1.170.941.47
ART = antiretroviral therapy. Missing values for CD4 (N = 40) were imputed using the multivariate imputation by chained equations (MICE) method (in R). Interpretation: We performed a sensitivity analysis to assure the hazard ratios were similar when adjusting for baseline CD4 count and ART receipt. In the first model, CD4 is added as a continuous co-variate. In the second model, CD4 is added as a continuous co-variate and receiving ART at baseline is added as a categorical co-variate. The hazard ratio for the sterile culture group is very similar. We do note that after adjustment in model #2, the 95%CI for the sterile culture group “crosses one” by less than one-hundreth of a point. Despite the fact that would yield a p-value slightly greater than 0.05, we feel strongly that an association likely exists for the following reasons. One, the hazard ratio measures the effect size, which is maintained after adjustment; if sample size was increased, we would anticipate the effect size would be similar but the power to detect a difference would increase. Two, neither CD4 count (hazard ratio = 1.00 [95%CI, 0.99–1.01]) or ART receipt (hazard ratio = 1.17 [95%CI, 0.94–1.47]) was independently associated with mortality. Additionally, three, we would argue that receiving ART at baseline (and subsequently higher CD4 counts) are in the causal pathway for CSF culture sterility in persons with cryptococcal meningitis. Variables in the causal pathway are not considered confounders and are typically not adjusted for. We further examine this concept in our damage-response framework model in the discussion.
Table 3. CSF biomarkers with more than two-fold difference between those with sterile cultures and those with fungal growth as quantified on baseline CSF.
Table 3. CSF biomarkers with more than two-fold difference between those with sterile cultures and those with fungal growth as quantified on baseline CSF.
Sterile CSF
(N = 24)
Low CFU Tertile
(N = 98)
Middle CFU Tertile
(N = 99)
High CFU Tertile
(N = 113)
p-Value
LABSMedian [IQR]
CD4 cells/μL64 [26, 121]20 [7, 49]15 [6, 32]10 [6, 30]<0.001
CSF white cells/μL25 [4, 155]20 [<5, 95]<5 [<5, <5]<5 [<5, <5]<0.001
CYTOKINESMean (95% CI) on log2 pg/mL scale
IL-69.2 (7.8, 10.6)8.3 (7.6, 8.9)7.4 (6.7, 8.1)7.4 (6.9, 8.0)0.02
IL-172.0 (0.8, 3.1)0.6 (0.0, 1.2)−0.2 (−0.7, 0.4)−0.3 (−0.8, 0.2)<0.001
IFN-γ3.0 (2.0, 3.9)1.7 (1.2, 2.2)1.4 (0.9, 1.8)1.3 (0.9, 1.7)0.02
G-CSF7.0 (5.9, 8.1)5.1 (4.6, 5.6)4.7 (4.3, 5.0)5.0 (4.6, 5.3)<0.001
GM-CSF6.2 (5.4, 7.0)5.2 (4.9, 5.5)4.9 (4.6, 5.2)5.0 (4.7, 5.2)<0.001
CXCL2 (GRO-b)4.8 (3.9, 5.6)3.5 (3.0, 3.9)3.3 (2.8, 3.7)3.7 (3.3, 4.0)0.02
IFN-α3.1 (2.2, 3.9)1.6 (1.0, 2.2)1.7 (1.3, 2.1)2.1 (1.7, 2.4)0.02
CSF quantitative culture tertiles: Low Tertile = 1–14,700 CFU/mL; Middle Tertile = 14,701–206,000 CFU/mL; High Tertile > 206,000 CFU/mL. p-value from unadjusted linear regression. Abbreviations: IQR, interquartile range; 95% CI, 95% confidence interval; CSF, cerebrospinal fluid; CFU, colony forming units; IL-6, interleukin 6; IL-17, interleukin 17; IFN-γ, interferon gamma; G-CSF, granulocyte colony stimulating factor; GM-CSF, granulocyte macrophage colony stimulating factor; CXCL2 (GRO-b), growth regulated protein beta; IFN-α, interferon alpha. Full set of CSF biomarkers provided as Supplemental Table S1.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Skipper, C.P.; Hullsiek, K.H.; Stadelman, A.; Williams, D.A.; Ssebambulidde, K.; Okafor, E.; Tugume, L.; Nuwagira, E.; Akampurira, A.; Musubire, A.K.; et al. Sterile Cerebrospinal Fluid Culture at Cryptococcal Meningitis Diagnosis Is Associated with High Mortality. J. Fungi 2023, 9, 46. https://doi.org/10.3390/jof9010046

AMA Style

Skipper CP, Hullsiek KH, Stadelman A, Williams DA, Ssebambulidde K, Okafor E, Tugume L, Nuwagira E, Akampurira A, Musubire AK, et al. Sterile Cerebrospinal Fluid Culture at Cryptococcal Meningitis Diagnosis Is Associated with High Mortality. Journal of Fungi. 2023; 9(1):46. https://doi.org/10.3390/jof9010046

Chicago/Turabian Style

Skipper, Caleb P, Katherine Huppler Hullsiek, Anna Stadelman, Darlisha A Williams, Kenneth Ssebambulidde, Elizabeth Okafor, Lillian Tugume, Edwin Nuwagira, Andrew Akampurira, Abdu K Musubire, and et al. 2023. "Sterile Cerebrospinal Fluid Culture at Cryptococcal Meningitis Diagnosis Is Associated with High Mortality" Journal of Fungi 9, no. 1: 46. https://doi.org/10.3390/jof9010046

APA Style

Skipper, C. P., Hullsiek, K. H., Stadelman, A., Williams, D. A., Ssebambulidde, K., Okafor, E., Tugume, L., Nuwagira, E., Akampurira, A., Musubire, A. K., Abassi, M., Muzoora, C., Rhein, J., Boulware, D. R., & Meya, D. B. (2023). Sterile Cerebrospinal Fluid Culture at Cryptococcal Meningitis Diagnosis Is Associated with High Mortality. Journal of Fungi, 9(1), 46. https://doi.org/10.3390/jof9010046

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop