Next Article in Journal
Spanish Paediatricians’ Knowledge, Attitudes and Practices Regarding Oral Health of Children under 6 Years of Age: A Cross-Sectional Study
Next Article in Special Issue
Prevalence, Predictors, and Experience of Moral Suffering in Nursing and Care Home Staff during the COVID-19 Pandemic: A Mixed-Methods Systematic Review
Previous Article in Journal
What Does It Take to Get Somebody Back to Work after Severe Acquired Brain Injury? Service Actions within the Vocational Intervention Program (VIP 2.0)
Previous Article in Special Issue
The North Italian Longitudinal Study Assessing the Mental Health Effects of SARS-CoV-2 Pandemic on Health Care Workers—Part I: Study Design and Psychometric Structural Validity of the HSE Indicator Tool and Work Satisfaction Scale
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
IJERPH
Volume 19
Issue 15
10.3390/ijerph19159541
ijerph-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

The North Italian Longitudinal Study Assessing the Mental Health Effects of SARS-CoV-2 Pandemic Health Care Workers—Part II: Structural Validity of Scales Assessing Mental Health

by
Emanuele Maria Giusti
1,2,
Giovanni Veronesi
3,
Camilla Callegari
4,
Gianluca Castelnuovo
2,5,
Licia Iacoviello
3,6 and
Marco Mario Ferrario
3,*
1
Psychology Research Laboratory, Istituto Auxologico Italiano IRCCS, 20149 Milan, Italy
2
Department of Psychology, Catholic University of the Sacred Heart, 20123 Milan, Italy
3
EPIMED Research Center, Department of Medicine and Surgery, University of Insubria, 21100 Varese, Italy
4
Division of Psychiatry, Department of Medicine and Surgery, University of Insubria, 21100 Varese, Italy
5
Psychology Research Laboratory, Istituto Auxologico Italiano IRCCS, 28824 Verbania, Italy
6
Department of Epidemiology and Prevention, IRCCS Neuromed, 86077 Pozzilli, Italy
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2022, 19(15), 9541; https://doi.org/10.3390/ijerph19159541
Submission received: 21 June 2022 / Revised: 29 July 2022 / Accepted: 29 July 2022 / Published: 3 August 2022
(This article belongs to the Special Issue Mental Health at the Workplace)
Browse Figures
Review Reports Versions Notes

Abstract

:
It is unclear if the factor structure of the questionnaires that were employed by studies addressing the impact of COVID-19 on the mental health of Healthcare Workers (HCW) did not change due to the pandemic. The aim of this study is to assess the factor structure and longitudinal measurement invariance of the Maslach Burnout Inventory (MBI) and the factor structure of the General Health Questionnare-12 (GHQ-12), PTSD Checklist for DSM-5-Short Form (PCL-5-SF), Connor-Davidson Resilience Scale-10 (CD-RISC-10) and Post-Traumatic Growth Inventory-Short Form (PTGI-SF). Out of n = 805 HCWs from a University hospital who responded to a pre-COVID-19 survey, n = 431 were re-assessed after the COVID-19 outbreak. A Confirmatory Factor Analysis (CFA) on the MBI showed adequate fit and good internal consistency only after removal of items 2, 6, 12 and 16. The assumptions of configural and metric longitudinal invariance were met, whereas scalar longitudinal invariance did not hold. CFAs and exploratory bifactor analyses performed using data from the second wave confirmed that the GHQ-12, the PCL-5-SF, the PTGI-SF and the CD-RISC-10 were unidimensional. In conclusion, we found support for a refined version of the MBI. The comparison of mean MBI values in HCWs before and after the pandemic should be interpreted with caution.

1. Introduction

The mental health of Healthcare Workers (HCW) is of paramount importance since its reduction is strictly linked to high direct and indirect individual and social costs [1]. Declining mental health in the workplace in general is known to be associated with loss of productivity, higher spending on social security programs and to cause direct expenditure on healthcare [2]. HCWs are at particular risk since they are exposed to a multitude of stressors potentially affecting their physical and mental health [1]. As a consequence, they experience high levels of anxiety, depression, burnout and, due to repeated exposure to patients who have experienced traumas, forms of vicarious traumatization that might give rise to symptoms of Post-Traumatic Stress Disorder (PTSD) [3,4]. The impact of these conditions is particularly of concern, since it might result in shortage of crucial professionals, reduced quality of patient care and medical errors [5].
Before the pandemic, it was estimated that about 11% of nurses and from 20% to 40% of physicians suffered from burnout [6,7]. The COVID-19 pandemic seems to have exacerbated this issue due to repeated exposure to life-threatening situations, fear of contagion, shift overload and changes in work organization [8,9]. The results of longitudinal studies and the comparison between the estimates of the studies performed before and during the pandemic show a high prevalence and a marked increase of the levels of anxiety, with estimates from studies performed during the pandemic ranging from 22% to 31%, depression (from 17% to 36%), PTSD symptoms (from 13% to 37%) and burnout (from 36% to 52%) [10,11,12,13,14,15,16,17]. Among the protective factors, studies have shown that resilience and post-traumatic growth helped to buffer the impact of the pandemic, suggesting that these factors should be taken into account in order to describe the mental health of HCWs [18,19].
However, a caveat applies to this evidence. Both the direct and indirect comparisons across time of the mental health levels are valid if the assumption of the longitudinal measurement invariance of the questionnaires that are used to assess them holds, that is to say that the questionnaires have the same meaning for people over time. If this assumption does not hold, differences in scores across time do not reflect differences in the latent construct of interest but to changes in how people respond to the questionnaire [20]. As a consequence, conclusions about changes in the construct of interest are not valid [21]. Given the enormous psychological, social and cultural impact of the pandemic on the general population, and the specific challenges that HCWs had to face during the pandemic [22], it is uncertain whether the assumption of the measurement invariance and the validity of factor structures analyzed before the pandemic holds. This has important consequences for both the assessment of changes in constructs in longitudinal studies, and for the interpretation of results in cross-sectional studies performed after the pandemic. Therefore, studies on the structural validity of questionnaires employed to assess the mental health of HCWs are needed.
This study is part of a broader research project which aimed to assess the mental health status of HCWs during the pandemic, the changes in burnout from the pre-pandemic to the pandemic period, and the impact of pre-pandemic work-related stressors on mental health during the pandemic. Based on previous literature, the mental health variables that were studied were general mental health, burnout, PTSD symptoms, post traumatic growth and resilience. In a parallel paper, the issues of the representativeness of the sample and of the structural validity of the questionnaires assessing pre-pandemic work-related stressors and work satisfaction are analyzed. In this paper, we aimed to investigate the factor structure of widely used questionnaires assessing the mental health constructs named above. In particular, we assessed the factor structure and the longitudinal measurement invariance of the Maslach Burnout Inventory (MBI) using both pre-pandemic data and data collected during the pandemic, as well as the factor structure of the validated Italian versions of the General Health Questionnaire-12 (GHQ-12), the PTSD Checklist for DSM-5-Short Form (PCL-5-SF), the Connor-Davidson Resilience Scale-10 (CD-RISC-10) and the Post-Traumatic Growth Inventory-Short Form (PTGI-SF), using data collected during the pandemic.

2. Materials and Methods

From August to September 2019 all the doctors, nurses, assistant nurses and office clerks working in 69 selected wards and offices of a large University hospital in the city of Varese participated in a first online survey investigating work-related stress and mental health variables (from now on: pre-COVID-19 wave). The wards and offices were selected to include those at expected high risk of work stress according to the representatives of the unions and of the hospital administration, and a random sample of the remaining wards. Then, from December 2020 to January 2021, the respondents to the first survey were asked to participate in a second online survey investigating the impact of the COVID-19 pandemic on their mental health (from now on: COVID-19 wave). Additional details regarding the study population, design and procedures are provided in the companion paper [23]. The study received approval from the Institutional Ethics Committee (approval ID 69/2020).

2.1. Measurement Instruments

The following measurement instrument was administered both during the first and second survey:
  • Maslach Burnout Inventory (MBI) [24]. The MBI is a self-report questionnaire which assesses the three theoretical components of burnout, i.e., emotional exhaustion, depersonalization and personal accomplishment. It is composed of 22 Likert-type items assessing the frequency of feelings or attitudes reflecting the three components of burnout on a scale ranging from 0 (“never”) to 6 (“every day”). Higher scores in the emotional exhaustion and depersonalization scales reflect higher levels of burnout, whereas higher scores in the personal accomplishment subscale reflect lower levels of burnout. Reviews on the psychometric properties of the MBI showed that this measurement instruments has adequate construct validity and internal consistency [25]. The Italian translation of the MBI was found to reflect the same three-factor structure as the original version and showed adequate internal consistency [26].
The following measurement instruments were administered only in the second survey:
  • General Health Questionnaire-12 (GHQ-12) [27]. The GHQ-12 is a self-report questionnaire originally developed for use in consulting settings to detect individuals with psychiatric disorders, whose 12-items version is an extensively used general measure of mental well-being. The GHQ-12 items assess the severity of a mental problem over the past two weeks using a 4-point Likert scale ranging from 0 (“less than usual”) to 3 (“much more than usual”). Higher scores indicate worse mental well-being. The factor structure of the GHQ-12 is debated, since unidimensional and multidimensional structures have been proposed [28,29,30]. However, it has been recognized that the retrieval of multiple dimensions is related to a wording effect rather than the presence of true distinct latent factors, since half of the GHQ-12 items have a positive wording and half a negative wording [31]. The Italian version of the GHQ-12 showed acceptable test–retest reliability [32].
  • The PTSD Checklist for DSM-5-Short Form (PCL-5-SF) [33]. The PCL-5-SF is a 5-item version of the original 20-item PCL-5 questionnaire, which is a self-report measure assessing the frequency of the DSM-5 symptoms of PTSD in the past month using a Likert scale ranging from 1 (“Not at all”) to 5 (“extremely”). Higher scores indicate higher frequency of PTSD symptoms. The Italian version of this questionnaire has shown adequate construct validity, criterion validity and internal consistency [34].
  • The Connor-Davidson Resilience Scale-10 item version (CD-RISC-10) [35]. The CD-RISC-10 is a self-report questionnaire assessing resilience based on the Connor and Davidson definition, i.e., the ability to thrive in the face of adversity [35]. In this study, we employed the 10-item version. The items are scored using Likert scale ranging from 0 (“not true at all”) to 4 (“true nearly all the time”). Higher scores indicate higher levels of resilience. The Italian version of the CD-RISC-10 has shown adequate internal consistency and concurrent validity [36].
  • The Post-Traumatic Growth Inventory-Short Form (PTGI-SF) [37]. The PTGI-SF is a self-rated questionnaire assessing positive outcomes reported by people who have experienced traumatic events. In this study we employed the 10-item version of this scale. The items are scored on a 6-point Likert scale ranging from 0 (“I did not experience this change as a result of my crisis”) to 5 (“I experienced this change to a very great degree as a result of my crisis”). Higher scores indicate higher levels of post-traumatic growth. The Italian version of the PTGI-SF has shown adequate internal consistency [38].

2.2. Statistical Analyses

2.2.1. Preliminary Analyses

Demographic and work-related variables were described using frequencies and percentages if categorical, and using means and standard deviations if continuous. Then, a missing data analysis was performed. Firstly, the absolute and relative amount of missing data for each measurement instrument was inspected. Then, patterns of missing data were inspected by plotting the frequency of each missing data pattern. The Little’s test was finally performed to assess if the missing data mechanism was Missing Completely At Random (MCAR) [39]. This test was run twice, once using the variables collected in the first survey, namely the sociodemographic variables and the MBI data, and once using the sociodemographic variables and the variables collected in the second survey. Then, the normality of each questionnaire was checked, performing both a Henze-Zirkler test for multivariate normality and, for each item, inspecting normal Q-Q plots and the results of Shapiro-Wilk tests.

2.2.2. Confirmatory Factor Analyses

Measurement instruments whose items are scored using Likert scales with less than 5 points were analyzed using a Diagonally Weighted Least Squares (DWLS) estimator, and those with more than 5 points using a Maximum Likelihood estimator with robust standard error and a Satorra-Bentler scaled test statistic (MLR). The use of the robust variant of the maximum likelihood estimator was chosen because the normality assumption of all the questionnaires was violated.
All the CFAs were performed by assessing the fit of the factor structures already analyzed by previous studies. In case of poor fit of all these factor structures, items were deleted one by one by combining theoretical considerations (e.g., presence of items that were flagged as problematic in other studies either for content issues or for displaying bad fit with the proposed structures) and the assessment of misfit through modification indices, which indicate presence of cross-loadings or of correlated errors between items. This procedure was performed iteratively until a structure with good fit was reached. Cut-offs for good fit were CFI ≥ 0.95 and RMSEA ≤ 0.06 for CFAs that were estimated using the MLR estimator and AGFI ≥ 0.90 and SRMR ≤ 0.08 for CFAs that were estimated using the DWLS estimator [40]. Structures were retained also when the fit indices were CFI ≥ 0.90 and RMSEA ≤ 0.08 [41] and further deletion of items had no theoretical significance.

2.2.3. Confirmatory Factor Analyses and Measurement Invariance of the Maslach Burnout Inventory

Firstly, the factor structure of the MBI was analyzed using the data from the whole sample during the pre-COVID-19 wave. The first structure that was evaluated was the three correlated factors structure employed by Sirigatti & Stefanile [26] for the Italian translation of the MBI, which is identical with the structure proposed by Maslach et al. [24]. Then, items were deleted following the procedure described above.
Then, the longitudinal invariance of the MBI across the waves was assessed using the data of HCWs who responded to both the pre-COVID-19 and the COVID-19 surveys. This was done by comparing the fit of models, progressively posing stricter equality constraints between the responses of each HCW at the two waves. Firstly, configural invariance was tested by constraining the items to load on the same latent factors in both waves. Secondly, metric invariance was tested by constraining item loadings to be equal across the waves. Finally, scalar invariance was tested by constraining item intercepts to be equal. The presence of each of these components of measurement invariance was established if the difference in CFI (ΔCFI) and RMSEA (ΔRMSEA) between the more constrained structure and the preceding structure was lower than 0.01 and 0.015, respectively [42]. The errors of each item across time were allowed to correlate.

2.2.4. Factor Analyses of the GHQ-12, of the PCL-5-SF, of the PTGI-SF and of the CD-RISC-10 Scale

The factor structure of the GHQ-12, of the PCL-5-SF, of the PTGI-SF and of the CD-RISC-10 scale was assessed using the data from the second wave. A unidimensional structure was firstly assessed using a confirmatory approach. If available, other factor structures based on previous studies were assessed. If no adequate factor structures were found, we performed a bifactor analysis to assess if the measurement instrument under consideration was “unidimensional enough” to allow for the calculation of a total score [43]. Briefly, in a bifactor structure, items are assumed to reflect both a general factor that accounts for covariation among items, and a specific factor that accounts for covariation not measured by general factor. Based on the resulting bifactor structure, it is possible to calculate the Explained Common Variance (ECV), which is an estimate of the amount of variance that is shared among a set of items, and the coefficient omega-hierarchical (ωH), which is an estimate of the general factor saturation of a test. An ECV > 0.70 and a ωH > 0.80 support the presence of a strong general factor [44]. The bifactor structures were assessed using a confirmatory approach. In case of non-convergence of the confirmatory bifactor structure, an exploratory bifactor analysis was performed using the Schmid-Leiman procedure [45].
Finally, the Cronbach’s alpha of the final versions of the questionnaires was computed. All the analyses were performed using the significance value of 0.05. The CFAs were performed using SAS OnDemand for Academics software (release 9.04) (SAS Institute Inc., Cary, NC, USA). The exploratory bifactor analysis was performed using the R (version 4.0.1) package psych [46].

3. Results

Of the 1286 HCWs who were invited, 805 participated in the pre-COVID-19 wave. The socio-demographic and work-related characteristics of the sample are reported in Table 1. Four hundred and thirty-one of them also responded to the COVID-19 wave. The flowchart of the participants is reported in Appendix A (Figure A1).

3.1. Missing Data Analysis

The percentage of cases with at least one missing value in the MBI items was 8.07%. The inspection of missing data patterns did not reveal any potential recurring pattern. The Little’s MCAR test performed using the data from the first survey was not significant (χ2(916) = 947.83, p = 0.22). Since missing data were less than 10% and the hypothesis of presence of MAR mechanisms was not corroborated, observations with missing data were excluded from the analysis of the factor structure of the MBI. Missing data of the questionnaires collected during the second survey were less than 1% for the GHQ-12, 1.6% for the PCL-5-SF, 2.7% for the PTGI-SF and 2.6% for the CD-RISC-10. The Little’s MCAR test performed using the data from the second survey was significant (χ2(1075) = 1186.32, p = 0.01). Nonetheless, the graphical analysis of the missing data revealed no pattern of missing data. Based on this finding and on the low amount of missing data, the CFAs of each of the questionnaires administered during the second wave were performed after excluding cases with missing observations.

3.2. Confirmatory Factor Analysis and Internal Consistency of the Maslach Burnout Inventory

The results of the CFA performed to assess the factor structure of the MBI are summarized in Table 2. The fit of the original three-correlated factors structure was not satisfactory. Based on presence of cross-loadings and the fact that these items were flagged as misfitting by previous studies, items 12 and 16 were excluded from the structure and the CFA was re-run. The fit of this solution was not satisfactory. Based on previous literature and presence of correlated errors between items 2 and 6, items 2 and 1 and items 2 and 3, we decided to remove item 2. The fit of the resulting structure did not reach the criteria. Therefore, we decided to remove item 6 due to presence of cross-loadings, correlated errors with item 5 and presence of misfit in previous studies. The resulting refined version of the MBI reached the cut-off criteria. This version was therefore employed in the analyses of the measurement invariance. The loadings of this solution are represented in the Appendix A (Figure A2). The Cronbach’s alphas of the subscales of the refined version of the MBI were above the cutoff (Emotional Exhaustion: alpha = 0.89, Depersonalization: alpha = 0.72, Personal Accomplishment: alpha = 0.74).

3.3. Longitudinal Measurement Invariance of the Refined Version of the Maslach Burnout Inventory

The results of the longitudinal measurement invariance are reported in Table 3. The configural and metric invariance of the MBI over time was corroborated. However, the scalar invariance assumption did not hold. After relaxing the equality constraints of the intercepts of five items based on their modification indices, the fit indices did not reach the pre-defined criteria. Therefore, the scalar invariance was not corroborated. The analysis of the sign of the mean differences in MBI scores, given the same amount of the latent trait, suggests that post-pandemic scores are higher (data not presented).

3.4. Factor Analysis and Internal Consistency of the General Health Questionnaire-12, of the PCL-5-SF, of the Post-Traumatic Growth Inventory and of the Connor-Davidson Resilience Scale

Regarding the GHQ-12, the fit of the unidimensional structure was satisfactory (AGFI = 0.97, SRMR = 0.08). We also assessed the fit of a bidimensional structure based on the wording of the items, in which items with positive wording loaded on a positive factor and items with negative wording loaded on a negative factor. This factor structure showed slightly better indices (AGFI = 0.98, SRMR = 0.07) with respect to the unidimensional structure. Since the fit of the unidimensional structure was satisfactory and theoretically sounder than the bidimensional structure, we decided to retain the unidimensional structure. The loadings of this structure are reported in the Appendix A (Figure A3). The Cronbach’s alpha of the scale was 0.88.
The fit of the unidimensional structure applied to the PCL-5-SF did not meet the pre-specified cutoffs (CFI = 0.98, RMSEA = 0.12). The Schmid-Leiman procedure did not converge. Since no other plausible factor structures were found in the literature and there was no theoretical background for the deletion of items, we did not attempt to perform additional analyses. The Cronbach’s alpha of the scale was 0.84.
The unidimensional CFA performed on the PTGI-SF did not reach the cutoff criteria (CFI = 0.92, RMSEA = 0.10). We tested a 5-factor structure based on the solution of the original version of the scale [49], but the covariance matrix based on this structure was not positively definite. We found no theoretical background for removal of individual items. Therefore, we performed an exploratory bifactor analysis using the Schmid Leiman procedure. The values of ωH and ECV calculated using this procedure were above the cutoffs (ωH = 0.80, ECV = 0.74), providing support for the calculation of a total score. The loadings of the exploratory bifactor analyses are reported in the Appendix A (Figure A4). The Cronbach’s alpha of the scale was 0.92.
Finally, the CFA performed on the CD-RISC-10 showed that the unidimensional structure did not reach the cutoff criteria (CFI = 0.94, RMSEA = 0.07). We found no alternative factor structures in the literature, nor any theoretical background for the removal of any item. Therefore, we performed an exploratory bifactor analysis of the scale. The bifactor indices were both above the cutoffs (ωH = 0.80, ECV = 0.74). The loadings of this solution are reported in Appendix A (Figure A5). The Cronbach’s alpha of the scale was 0.89.

4. Discussion

The aim of this study was to assess the factor structure and measurement invariance of measurement instruments of interest for the assessment of mental health in HCWs. The results show that a refined version of the MBI has adequate structural validity and longitudinal measurement invariance over the pandemic waves, that the GHQ-12 is a unidimensional scale, and that factor structure of the PCL-5-SF, the PTGI-SF and the CD-RISC-10 is uncertain but that these questionnaires can be scored as unidimensional instruments.
The factor structure of the MBI has been widely studied in different countries and work contexts [50,51,52]. The results of these studies are often conflicting, and this has led to a debate about the internal validity of the scale, with various authors noting that the MBI might have different characteristics in different populations, and various studies have suggested some modifications to the scale [51,53,54]. In most of these studies, the presence of three correlated factors reflecting the components of burnout is corroborated, but there are differences in the retained items. Loera et al. [50] performed multiple CFAs using data from Italian nurses evaluating the fit of ten factor structures analyzed by previous studies, finding support for the deletion of items 2, 12 and 16. The results of this study provide additional support for the removal of these items, with the addition of the removal of item 6. In our sample, removing these items helped to find a factor structure with good fit indices, and the resulting questionnaire showed adequate measurement invariance over time.
The misfit of items 12 (“I feel very energetic”) and item 16 (“Working with people directly puts too much stress on me”) has been previously recognized by Maslach herself [55] and is probably due to their content. The content of item 12 could refer both to absence of emotional exhaustion and presence of personal accomplishment, and the content of item 16 might be both a symptom of emotional exhaustion and, since it refers to difficulties in being in contact with people, might overlap with depersonalization. Item 2 has been recognized as misfitting by a previous study [47] and its deletion improved the fit on the study by Loera et al. [50]. In our study, we found that this item has cross-loadings with the components of emotional exhaustion and depersonalization and that its errors are correlated with those of items 1 (“I feel emotionally drained from my work”) and 3 (“I feel fatigued when I get up in the morning and have to face another day on the job”). The presence of cross-loadings could not be explained by content issues, since it refers to a symptom of emotional exhaustion (“I feel used up at the end of the workday”) that should not be related to depersonalization. However, the fact that its errors are correlated with those of three items of the same scale referring to fatigue seems to suggest that this item causes redundancy. Finally, item 6 (“Working with people all day is really a strain for me”) seems to suffer from the same issue as item 16, since it refers both to emotional exhaustion and difficulties in relationships with people, an aspect that might reflect depersonalization. The deletion of this item, which similarly to items 2 and 16 should reflect emotional exhaustion, improved the fit while still leaving a sufficient number of items in that factor, which preserves its content validity. The 18-items refined version of the MBI showed adequate structural validity and adequate internal consistency.
The refined version of the MBI meets the requirement for configural and metric longitudinal invariance, but not for scalar invariance. Presence of configural invariance suggests that item responses reflect the same latent factors across waves, meaning that responders at different waves have equal interpretations of the latent factors. Presence of metric invariance suggests that the relationships between the responses to the measurement instrument and the latent variable are similar over time. Violation of scalar invariance suggests that, given the same level of the latent trait, scores obtained during the pandemic are higher than scores obtained before the pandemic. Overall, these results provide support for the use of this refined version of the MBI for the assessment of burnout both in the pre-pandemic and in the post-pandemic period. However, the assessment of changes in burnout levels between the pre-pandemic and the post-pandemic period should be made with caution since differences can be attributable to different ways of scoring the questionnaire.
The absence of scalar invariance of the refined version MBI over the pandemic waves has some consequences. Cross-sectional studies have shown that burnout after the pandemic has a very high prevalence, and longitudinal studies that burnout levels have increased as a result of the pandemic [8,10,11,12,13,14,15,16,17]. If the lack of longitudinal scalar invariance that was found in this study is generalized to other countries and populations, it might be hard to disentangle the presence of true differences in the levels of burnout from differences in how HCWs respond to the items of the MBI from the pandemic on. This calls for further research to confirm the longitudinal measurement invariance of the MBI, possibly using advanced psychometric methods that might also help to provide information about the magnitude of its impact [56]. In addition, the stability of the cut-offs employed to detect presence of clinically significant levels of burnout should be specifically assessed. Given the absence of studies supporting the longitudinal invariance of the MBI, we suggest explicitly taking this issue into account. Methods for taking the absence of invariance into account have been proposed, and vary according to the type of study [57,58,59,60]. In any case, the comparison of burnout over time and of change scores should be interpreted with caution.
The analysis of the GHQ-12 showed that a unidimensional structure had a good fit to the data. The factor structure of the GHQ-12 has been debated but it has been argued that the retrieval of distinct latent factors is an artifact due to the positive and negative wording of the GHQ-12 items [31]. In this study, we found support for a slightly better fit of the bidimensional structure accounting for the wording effect. Nonetheless, since the unidimensional structure is in line with the theoretical structure underpinning the measurement instrument [30], we suggest consideration of the GHQ-12 as a unidimensional scale.
When we analyzed the factor structure of the PCL-5-SF, we found no support for the structural validity of this measurement instrument. Some caveats must be made. Firstly, from a statistical point of view, analyzing a scale composed of four items has the consequence that the fit indices, in particular the RMSEA, tend to detect misfit even in its absence [61]. Secondly, the PCL-5-SF was developed from the original measurement instrument using the maintenance of diagnostic ability as a criterion rather than focusing on its psychometric properties, and in such a way as to be consistent with DSM-V criteria [62]. Therefore, modifications of the scale are not warranted, and the absence of a good fit of a one-dimensional structure does not preclude its use.
Finally, both the CD-RISC-10 and, in particular, the PTGI-SF showed some departure from unidimensional factor structures. The PTGI-SF has been studied either as a unidimensional instrument or as a 5-factor instrument [49,63,64]. In this study, we found that both solutions did not reach a good fit to the data. However, the exploratory bifactor analysis provide some ground for the calculation of a total score. Similarly, the CD-RISC-10 has been described as a unidimensional scale [65,66], but the CFA showed that the fit of the unidimensional structure was not satisfactory. In this case also, the exploratory bifactor analysis showed the presence of a strong general factor, supporting the calculation of a total score. It is important to note that the results of the exploratory bifactor analysis should be analyzed with caution and should not be employed as a strong corroboration for the unidimensionality of these scales [43]. Analyses using advanced psychometric methods are warranted to gather insights about the factor structure of these scales [44].
There are various aspects that might impact the generalization of the results of this study. Firstly, the second survey was performed at a specific time within a period characterized by frequent and sudden changes in the characteristics of the pandemic and the burden on the healthcare system. In particular, the survey was performed in a period when the pandemic was at the beginning of its second wave and vaccines were not already available. It is therefore possible that the results of this study, in particular the results of the longitudinal measurement invariance, could have been different if the survey had been carried out at a different time. In addition, given the territorial variability in the effect if the pandemic burden on hospitals, the results of our study are not generalizable to contexts in which the impact of the pandemic was less severe.
Our study has some limitations. Firstly, we did not perform a formal analysis to determine the initial sample size. Therefore, we may lack sufficient precision for subgroup analysis by job title, especially in the COVID-19 wave. In this paper, for instance, we were not able to assess the measurement invariance of the questionnaires across HCW groups. Secondly, except for the MBI, the questionnaires analyzed in this study were administered only in the COVID-19 wave and we could not perform the assessment of longitudinal invariance of these instruments. Finally, the presence of selection bias due to minor rates of responses of workers with high levels of burnout in both waves cannot be excluded. Nonetheless, at least for work-related characteristics, the representativeness of the sample has been confirmed [23] and it is unlikely that this issue might have impacted our structural validity assessments.

5. Conclusions

In conclusion, we found support for the use of an 18-item refined version of the MBI for the assessment of burnout in Italian HCWs and we suggest that the GHQ-12, the PCL-5-SF, the PTGI-SF and the CD-RISC-10 should be scored as unidimensional scales. The comparison between pre-pandemic and post-pandemic levels of burnout should be made and interpreted with caution.

Author Contributions

Conceptualization, G.V., L.I. and M.M.F.; Data curation, G.V.; Formal analysis, E.M.G.; Funding acquisition, L.I. and M.M.F.; Methodology, E.M.G., G.V. and M.M.F.; Project administration, G.V.; Supervision, C.C. and G.C.; Writing—original draft, E.M.G.; Writing—review & editing, G.V., C.C., G.C., L.I. and M.M.F. All authors have read and agreed to the published version of the manuscript.

Funding

This study was sponsored by a grant provided by Regione Lombardia (DG-Welfare n. 7082/2020); a grant by Fondazione Veronesi (COVID-19 Insieme per la Ricerca 2020); and by a grant provided by Istituto Nazionale per l’Assicurazione Contro gli Infortuni sul Lavoro (Grant ID: BRIC-2019 ID 29A).

Institutional Review Board Statement

The study received ethical approval by the ASST Sette Laghi ethical committee (approval ID 69/2020).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

Anonymized study data are available upon motivated and reasonable request to the corresponding author.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study, in the collection, analyses, or interpretation of data, in the writing of the manuscript, or in the decision to publish the results.

Appendix A

  • Flowchart of the study participants
Ijerph 19 09541 g0a1 550
Figure A1. Flowchart of the study participants.
Figure A1. Flowchart of the study participants.
Ijerph 19 09541 g0a1
2.
Representations of the factor structures of the questionnaires assessed in the study
Ijerph 19 09541 g0a2 550
Figure A2. Factor structure of the refined version of the Maslach Burnout Inventory.
Figure A2. Factor structure of the refined version of the Maslach Burnout Inventory.
Ijerph 19 09541 g0a2
Ijerph 19 09541 g0a3 550
Figure A3. Factor structure of the General Health Questionnaire-12.
Figure A3. Factor structure of the General Health Questionnaire-12.
Ijerph 19 09541 g0a3
Ijerph 19 09541 g0a4 550
Figure A4. Factor structure resulting from the exploratory bifactor analysis of the Post-Traumatic Growth Inventory-Short Form.
Figure A4. Factor structure resulting from the exploratory bifactor analysis of the Post-Traumatic Growth Inventory-Short Form.
Ijerph 19 09541 g0a4
Ijerph 19 09541 g0a5 550
Figure A5. Factor structure resulting from the exploratory bifactor analysis of the Connor-Davidson Resilience Scale-10.
Figure A5. Factor structure resulting from the exploratory bifactor analysis of the Connor-Davidson Resilience Scale-10.
Ijerph 19 09541 g0a5

References

  1. Johnson, J.; Hall, L.H.; Berzins, K.; Baker, J.; Melling, K.; Thompson, C. Mental Healthcare Staff Well-Being and Burnout: A Narrative Review of Trends, Causes, Implications, and Recommendations for Future Interventions. Int. J. Mental. Health Nurs. 2018, 27, 20–32. [Google Scholar] [CrossRef]
  2. Santini, Z.I.; Becher, H.; Jørgensen, M.B.; Davidsen, M.; Nielsen, L.; Hinrichsen, C.; Madsen, K.R.; Meilstrup, C.; Koyanagi, A.; Stewart-Brown, S.; et al. Economics of Mental Well-Being: A Prospective Study Estimating Associated Health Care Costs and Sickness Benefit Transfers in Denmark. Eur. J. Health Econ. 2021, 22, 1053–1065. [Google Scholar] [CrossRef] [PubMed]
  3. Kim, J.; Chesworth, B.; Franchino-Olsen, H.; Macy, R.J. A Scoping Review of Vicarious Trauma Interventions for Service Providers Working With People Who Have Experienced Traumatic Events. Trauma Violence Abus. 2021. [Google Scholar] [CrossRef]
  4. Weaver, M.D.; Vetter, C.; Rajaratnam, S.M.W.; O’Brien, C.S.; Qadri, S.; Benca, R.M.; Rogers, A.E.; Leary, E.B.; Walsh, J.K.; Czeisler, C.A.; et al. Sleep Disorders, Depression and Anxiety Are Associated with Adverse Safety Outcomes in Healthcare Workers: A Prospective Cohort Study. J. Sleep Res. 2018, 27, e12722. [Google Scholar] [CrossRef] [Green Version]
  5. Melnyk, B.M.; Orsolini, L.; Tan, A.; Arslanian-Engoren, C.; Melkus, G.D.; Dunbar-Jacob, J.; Rice, V.H.; Millan, A.; Dunbar, S.B.; Braun, L.T. A National Study Links Nurses’ Physical and Mental Health to Medical Errors and Perceived Worksite Wellness. J. Occup. Environ. Med. 2018, 60, 126–131. [Google Scholar] [CrossRef] [PubMed]
  6. Woo, T.; Ho, R.; Tang, A.; Tam, W. Global Prevalence of Burnout Symptoms among Nurses: A Systematic Review and Meta-Analysis. J. Psychiatr. Res. 2020, 123, 9–20. [Google Scholar] [CrossRef] [PubMed]
  7. O’Connor, K.; Neff, D.M.; Pitman, S. Burnout in Mental Health Professionals: A Systematic Review and Meta-Analysis of Prevalence and Determinants. Eur. Psych. 2018, 53, 74–99. [Google Scholar] [CrossRef] [PubMed]
  8. Denning, M.; Goh, E.T.; Tan, B.; Kanneganti, A.; Almonte, M.; Scott, A.; Martin, G.; Clarke, J.; Sounderajah, V.; Markar, S.; et al. Determinants of Burnout and Other Aspects of Psychological Well-Being in Healthcare Workers during the Covid-19 Pandemic: A Multinational Cross-Sectional Study. PLoS ONE 2021, 16, e0238666. [Google Scholar] [CrossRef]
  9. Bolcato, M.; Aurilio, M.T.; Aprile, A.; Di Mizio, G.; Della Pietra, B.; Feola, A. Take-Home Messages from the COVID-19 Pandemic: Strengths and Pitfalls of the Italian National Health Service from a Medico-Legal Point of View. Healthcare 2021, 9, 17. [Google Scholar] [CrossRef] [PubMed]
  10. Giusti, E.M.; Pedroli, E.; D’Aniello, G.E.; Stramba Badiale, C.; Pietrabissa, G.; Manna, C.; Stramba Badiale, M.; Riva, G.; Castelnuovo, G.; Molinari, E. The Psychological Impact of the COVID-19 Outbreak on Health Professionals: A Cross-Sectional Study. Front. Psychol. 2020, 11, 1684. [Google Scholar] [CrossRef] [PubMed]
  11. Kok, N.; van Gurp, J.; Teerenstra, S.; van der Hoeven, H.; Fuchs, M.; Hoedemaekers, C.; Zegers, M. Coronavirus Disease 2019 Immediately Increases Burnout Symptoms in ICU Professionals: A Longitudinal Cohort Study. Crit. Care Med. 2021, 49, 419–427. [Google Scholar] [CrossRef]
  12. Gomez, S.; Anderson, B.J.; Yu, H.; Gutsche, J.; Jablonski, J.; Martin, N.; Kerlin, M.P.; Mikkelsen, M.E. Benchmarking Critical Care Well-Being: Before and After the Coronavirus Disease 2019 Pandemic. Crit. Care Explor. 2020, 2, e0233. [Google Scholar] [CrossRef]
  13. Lluch, C.; Galiana, L.; Doménech, P.; Sansó, N. The Impact of the COVID-19 Pandemic on Burnout, Compassion Fatigue, and Compassion Satisfaction in Healthcare Personnel: A Systematic Review of the Literature Published during the First Year of the Pandemic. Healthcare 2022, 10, 364. [Google Scholar] [CrossRef]
  14. Ghahramani, S.; Lankarani, K.B.; Yousefi, M.; Heydari, K.; Shahabi, S.; Azmand, S. A Systematic Review and Meta-Analysis of Burnout Among Healthcare Workers During COVID-19. Front. Psychiatry 2021, 12, 758849. [Google Scholar] [CrossRef] [PubMed]
  15. Pan, L.; Xu, Q.; Kuang, X.; Zhang, X.; Fang, F.; Gui, L.; Li, M.; Tefsen, B.; Zha, L.; Liu, H. Prevalence and Factors Associated with Post-Traumatic Stress Disorder in Healthcare Workers Exposed to COVID-19 in Wuhan, China: A Cross-Sectional Survey. BMC Psychiatry 2021, 21, 572. [Google Scholar] [CrossRef] [PubMed]
  16. Fernandez, R.; Sikhosana, N.; Green, H.; Halcomb, E.J.; Middleton, R.; Alananzeh, I.; Trakis, S.; Moxham, L. Anxiety and Depression among Healthcare Workers during the COVID-19 Pandemic: A Systematic Umbrella Review of the Global Evidence. BMJ Open 2021, 11, e054528. [Google Scholar] [CrossRef] [PubMed]
  17. Hill, J.E.; Harris, C.; Danielle, L.C.; Boland, P.; Doherty, A.J.; Benedetto, V.; Gita, B.E.; Clegg, A.J. The Prevalence of Mental Health Conditions in Healthcare Workers during and after a Pandemic: Systematic Review and Meta-Analysis. J. Adv. Nurs. 2022, 78, 1551–1573. [Google Scholar] [CrossRef]
  18. Feingold, J.H.; Hurtado, A.; Feder, A.; Peccoralo, L.; Southwick, S.M.; Ripp, J.; Pietrzak, R.H. Posttraumatic Growth among Health Care Workers on the Frontlines of the COVID-19 Pandemic. J. Affect. Disord. 2022, 296, 35–40. [Google Scholar] [CrossRef] [PubMed]
  19. Li, W.-Q.; Yuan, P.; Sun, J.; Xu, M.-L.; Wang, Q.-X.; Ge, D.-D.; Jiang, M.-M.; Xing, L.-Q.; Du, W.-J.; Li, Q. Resilience, Coping Style, and COVID-19 Stress: Effects on the Quality of Life in Frontline Health Care Workers. Psychol. Health Med. 2022, 27, 312–324. [Google Scholar] [CrossRef]
  20. Putnick, D.L.; Bornstein, M.H. Measurement Invariance Conventions and Reporting: The State of the Art and Future Directions for Psychological Research. Dev. Rev. 2016, 41, 71–90. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  21. Widaman, K.F.; Ferrer, E.; Conger, R.D. Factorial Invariance within Longitudinal Structural Equation Models: Measuring the Same Construct across Time. Child Dev. Perspect. 2010, 4, 10–18. [Google Scholar] [CrossRef] [PubMed]
  22. Osofsky, J.D.; Osofsky, H.J.; Mamon, L.Y. Psychological and Social Impact of COVID-19. Psychol. Trauma 2020, 12, 468–469. [Google Scholar] [CrossRef] [PubMed]
  23. Veronesi, G.; Giusti, E.M.; D’Amato, A.; Gianfagna, F.; Borchini, R.; Castelnuovo, G.; Iacoviello, L.; Ferrario, M.M. The North Italian Longitudinal Study Assessing the Mental Health Effects of SARS-CoV-2 Pandemic on Health Care Workers—Part I: Study Design and Psychometric Structural Validity of the HSE Indicator Tool and Work Satisfaction Scale. Int. J. Environ. Res. Public Health 2022, 19, 9514. [Google Scholar]
  24. Maslach, C.; Jackson, S.E. The Measurement of Experienced Burnout. J. Organ. Behav. 1981, 2, 99–113. [Google Scholar] [CrossRef]
  25. Shoman, Y.; Marca, S.C.; Bianchi, R.; Godderis, L.; van der Molen, H.F.; Guseva Canu, I. Psychometric Properties of Burnout Measures: A Systematic Review. Epidemiol. Psychiatr. Sci. 2021, 30, e8. [Google Scholar] [CrossRef]
  26. Sirigatti, S.; Stefanile, C. The Maslach Burnout Inventory: Adattamento e Taratura per l’Italia; Organizzazioni Speciali: Firenze, Italy, 1993. [Google Scholar]
  27. Goldberg, D.P.; Hillier, V.F. A Scaled Version of the General Health Questionnaire. Psychol. Med. 1979, 9, 139–145. [Google Scholar] [CrossRef]
  28. Graetz, B. Multidimensional Properties of the General Health Questionnaire. Soc. Psychiatry Psychiatr. Epidemiol. 1991, 26, 132–138. [Google Scholar] [CrossRef] [PubMed]
  29. Werneke, U.; Goldberg, D.P.; Yalcin, I.; Ustün, B.T. The Stability of the Factor Structure of the General Health Questionnaire. Psychol. Med. 2000, 30, 823–829. [Google Scholar] [CrossRef]
  30. Goldberg, D.P. User’s Guide to the General Health Questionnaire; NFER-NELSON: Windsor, UK, 1988. [Google Scholar]
  31. Hankins, M. The Factor Structure of the Twelve Item General Health Questionnaire (GHQ-12): The Result of Negative Phrasing? Clin. Pract. Epidemiol. Mental Health 2008, 4, 10. [Google Scholar] [CrossRef] [Green Version]
  32. Piccinelli, M.; Bisoffi, G.; Bon, M.G.; Cunico, L.; Tansella, M. Validity and Test-Retest Reliability of the Italian Version of the 12-Item General Health Questionnaire in General Practice: A Comparison between Three Scoring Methods. Compr. Psychiatry 1993, 34, 198–205. [Google Scholar] [CrossRef]
  33. Blevins, C.A.; Weathers, F.W.; Davis, M.T.; Witte, T.K.; Domino, J.L. The Posttraumatic Stress Disorder Checklist for DSM-5 (PCL-5): Development and Initial Psychometric Evaluation. J. Trauma Stress 2015, 28, 489–498. [Google Scholar] [CrossRef] [PubMed]
  34. Di Tella, M.; Romeo, A.; Zara, G.; Castelli, L.; Settanni, M. The Post-Traumatic Stress Disorder Checklist for DSM-5: Psychometric Properties of the Italian Version. Int. J. Environ. Res. Public Health 2022, 19, 5282. [Google Scholar] [CrossRef] [PubMed]
  35. Connor, K.M.; Davidson, J.R.T. Development of a New Resilience Scale: The Connor-Davidson Resilience Scale (CD-RISC). Depress. Anxiety 2003, 18, 76–82. [Google Scholar] [CrossRef]
  36. Di Fabio, A.; Palazzeschi, L. Connor-Davidson Resilience Scale: Proprietà Psicometriche Della Versione Italiana [Connor-Davidson Resilience Scale: Psychometric Properties of the Italian Version]. Counsel. Giornale Ital. Ric. Appl. 2012, 5, 101–110. [Google Scholar]
  37. Tedeschi, R.G.; Calhoun, L.G. The Posttraumatic Growth Inventory: Measuring the Positive Legacy of Trauma. J. Trauma Stress 1996, 9, 455–471. [Google Scholar] [CrossRef] [PubMed]
  38. Prati, G.; Pietrantoni, L. Italian Adaptation and Confirmatory Factor Analysis of the Full and the Short Form of the Posttraumatic Growth Inventory. J. Loss Trauma 2014, 19, 12–22. [Google Scholar] [CrossRef]
  39. Little, R.J.A. A Test of Missing Completely at Random for Multivariate Data with Missing Values. J. Am. Stat. Assoc. 1988, 83, 1198–1202. [Google Scholar] [CrossRef]
  40. Hu, L.; Bentler, P.M. Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conventional Criteria versus New Alternatives. Struct. Equ. Model. 1999, 6, 1–55. [Google Scholar] [CrossRef]
  41. Brown, T.A. Confirmatory Factor Analysis for Applied Research; Guilford Publications: New York, NY, USA, 2015; ISBN 1-4625-1779-X. [Google Scholar]
  42. Chen, F.F. Sensitivity of Goodness of Fit Indexes to Lack of Measurement Invariance. Struct. Equ. Model. Multidiscip. J. 2007, 14, 464–504. [Google Scholar] [CrossRef]
  43. Reise, S.P.; Moore, T.M.; Haviland, M.G. Bifactor Models and Rotations: Exploring the Extent to Which Multidimensional Data Yield Univocal Scale Scores. J. Pers. Assess. 2010, 92, 544–559. [Google Scholar] [CrossRef]
  44. Reise, S.P.; Scheines, R.; Widaman, K.F.; Haviland, M.G. Multidimensionality and Structural Coefficient Bias in Structural Equation Modeling: A Bifactor Perspective. Educ. Psychol. Meas. 2013, 73, 5–26. [Google Scholar] [CrossRef] [Green Version]
  45. Schmid, J.; Leiman, J.M. The Development of Hierarchical Factor Solutions. Psychometrika 1957, 22, 53–61. [Google Scholar] [CrossRef]
  46. Revelle, W. Psych: Procedures for Psychological, Psychometric, and Personality Research; Evanston, IL, USA, 2020. [Google Scholar]
  47. Kim, H.; Ji, J. Factor Structure and Longitudinal Invariance of the Maslach Burnout Inventory. Res. Soc. Work Pract. 2009, 19, 325–339. [Google Scholar] [CrossRef]
  48. Kanste, O.; Miettunen, J.; Kyngäs, H. Factor Structure of the Maslach Burnout Inventory among Finnish Nursing Staff. Nurs. Health Sci. 2006, 8, 201–207. [Google Scholar] [CrossRef] [PubMed]
  49. Taku, K.; Cann, A.; Calhoun, L.G.; Tedeschi, R.G. The Factor Structure of the Posttraumatic Growth Inventory: A Comparison of Five Models Using Confirmatory Factor Analysis. J. Trauma Stress 2008, 21, 158–164. [Google Scholar] [CrossRef] [PubMed]
  50. Loera, B.; Converso, D.; Viotti, S. Evaluating the Psychometric Properties of the Maslach Burnout Inventory-Human Services Survey (MBI-HSS) among Italian Nurses: How Many Factors Must a Researcher Consider? PLoS ONE 2014, 9, e114987. [Google Scholar] [CrossRef] [Green Version]
  51. Worley, J.A.; Vassar, M.; Wheeler, D.L.; Barnes, L.L.B. Factor Structure of Scores From the Maslach Burnout Inventory: A Review and Meta-Analysis of 45 Exploratory and Confirmatory Factor-Analytic Studies. Educ. Psychol. Meas. 2008, 68, 797–823. [Google Scholar] [CrossRef]
  52. Poghosyan, L.; Aiken, L.H.; Sloane, D.M. Factor Structure of the Maslach Burnout Inventory: An Analysis of Data from Large Scale Cross-Sectional Surveys of Nurses from Eight Countries. Int. J. Nurs. Stud. 2009, 46, 894–902. [Google Scholar] [CrossRef] [Green Version]
  53. De Beer, L.T.; Bianchi, R. Confirmatory Factor Analysis of the Maslach Burnout Inventory: A Bayesian Structural Equation Modeling Approach. Eur. J. Psychol. Assess. 2019, 35, 217–224. [Google Scholar] [CrossRef]
  54. Stalker, C.A.; Harvey, C.; Frensch, K.; Mandell, D.; Adams, G.R. Confirmatory Factor Analysis of the Maslach Burnout Inventory. Null 2008, 1, 77–94. [Google Scholar] [CrossRef]
  55. Maslach, C.; Jackson, S.E.; Leiter, M.P. Maslach Burnout Inventory, 3rd ed.; Scarecrow Education: Lanham, MD, USA, 1996. [Google Scholar]
  56. Meade, A.W.; Lautenschlager, G.J.; Hecht, J.E. Establishing Measurement Equivalence and Invariance in Longitudinal Data With Item Response Theory. Int. J. Test. 2005, 5, 279–300. [Google Scholar] [CrossRef]
  57. Murray, A.L.; Obsuth, I.; Eisner, M.; Ribeaud, D. Evaluating Longitudinal Invariance in Dimensions of Mental Health Across Adolescence: An Analysis of the Social Behavior Questionnaire. Assessment 2019, 26, 1234–1245. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  58. Brincks, A.; Montag, S.; Howe, G.W.; Huang, S.; Siddique, J.; Ahn, S.; Sandler, I.N.; Pantin, H.; Brown, C.H. Addressing Methodologic Challenges and Minimizing Threats to Validity in Synthesizing Findings from Individual-Level Data Across Longitudinal Randomized Trials. Prev. Sci. 2018, 19, 60–73. [Google Scholar] [CrossRef] [PubMed]
  59. Gross, R.A. Examining the Robustness of the Latent Growth Curve Model to Violations of Longitudinal Measurement Equivalence: A Methodological Study with Practical Applications in Child Development. Ph.D. Thesis, Indiana University, Bloomington, IN, USA, 2020. [Google Scholar]
  60. Marsh, H.; Guo, J.; Parker, P.; Nagengast, B.; Asparouhov, T.; Muthén, B.; Dicke, T. What to Do When Scalar Invariance Fails: The Extended Alignment Method for Multi-Group Factor Analysis Comparison of Latent Means Across Many Groups. Psychol. Methods 2017, 23, 524. [Google Scholar] [CrossRef]
  61. Kenny, D.A.; McCoach, D.B. Effect of the Number of Variables on Measures of Fit in Structural Equation Modeling. Struct. Equ. Model. 2003, 10, 333–351. [Google Scholar] [CrossRef]
  62. Zuromski, K.L.; Ustun, B.; Hwang, I.; Keane, T.M.; Marx, B.P.; Stein, M.B.; Ursano, R.J.; Kessler, R.C. Developing an Optimal Short-Form of the PTSD Checklist for DSM-5 (PCL-5). Depress. Anxiety 2019, 36, 790–800. [Google Scholar] [CrossRef] [PubMed]
  63. Kaur, N.; Porter, B.; LeardMann, C.A.; Tobin, L.E.; Lemus, H.; Luxton, D.D.; Armenta, R.; Bauer, L.; Bookwalter, D.; Bukowinski, A.; et al. Evaluation of a Modified Version of the Posttraumatic Growth Inventory-Short Form. BMC Med. Res. Methodol. 2017, 17, 69. [Google Scholar] [CrossRef] [PubMed]
  64. Cann, A.; Calhoun, L.G.; Tedeschi, R.G.; Taku, K.; Vishnevsky, T.; Triplett, K.N.; Danhauer, S.C. A Short Form of the Posttraumatic Growth Inventory. Anxiety Stress Coping 2010, 23, 127–137. [Google Scholar] [CrossRef] [PubMed]
  65. Nartova-Bochaver, S.; Korneev, A.; Bochaver, K. Validation of the 10-Item Connor–Davidson Resilience Scale: The Case of Russian Youth. Front. Psychiatry 2021, 12, 611026. [Google Scholar] [CrossRef] [PubMed]
  66. Notario, B.; Solera-Martínez, M.; Serrano-Parra, M.; Bartolomé, R.; Garcia-Campayo, J.; Martinez Vizcaino, V. Reliability and Validity of the Spanish Version of the 10-Item Connor-Davidson Resilience Scale (10-Item CD-RISC) in Young Adults. Health Qual. Life Outcomes 2011, 9, 63. [Google Scholar] [CrossRef] [Green Version]
Table
Table 1. Description of the participants in the pre-COVID-19 wave (n = 805).
Table 1. Description of the participants in the pre-COVID-19 wave (n = 805).
VariableLeveln (%)Missing Data n (%)
SexFemale622 (77.6)3 (0.3)
Male180 (22.4)
Age25–34 years126 (15.7)1 (0.1)
35–44 years178 (22.1)
45–54 years306 (38.1)
55 years or more194 (24.1)
Job profileClerk102 (12.7)
Doctor104 (12.9)
Nurse429 (53.3)
Nurse assistant170 (21.1)
Education attainmentPrimary school115 (14.3)1 (0.1)
Secondary school295 (36.7)
Attended university20 (2.5)
Bachelor’s degree192 (23.9)
Master’s degree61 (7.6)
Postgraduate programs or specialization121 (15.0)
Job seniorityLess than 1 year57 (7.1)3 (0.3)
From 1 to 5 years167 (20.8)
From 6 to 15 years190 (23.7)
From 16 to 30 years255 (31.8)
More than 30 years133 (16.6)
Work seniority in current roleLess than 1 year62 (7.7)2 (0.2)
From 1 to 5 years232 (28.9)
From 6 to 15 years223 (27.8)
From 16 to 30 years230 (28.6)
More than 30 years56 (7.0)
Type of contractFixed-term14 (1.7)2 (0.2)
Permanent789 (98.3)
Type of employmentFull-time693 (86.3)2 (0.2)
Part-time110 (13.7)
Work scheduling Non-shift work151 (18.8)2 (0.2)
Shift work (with night shifts)502 (62.5)
Shift work (without night shifts)150 (18.7)
Table
Table 2. Summary of the results of the Confirmatory Factor Analysis of the Maslach Burnout Inventory performed using the sample from the pre-COVID-19 wave (n = 740).
Table 2. Summary of the results of the Confirmatory Factor Analysis of the Maslach Burnout Inventory performed using the sample from the pre-COVID-19 wave (n = 740).
StructureModifications with Respect to the Original StructureCFIRMSEAIssues Affecting the Fit of the Structure *
Original three-correlated factors structure [24,26] 0.80840.0829Presence of cross-loadings of items 6 (EE and DEP), 12 (EE, PA) and 16 (EE, DEP). Presence of correlated errors between items 6 and 16, 1 and 2, 2 and 3.
Three-correlated factors structure, with deletion of items 12 and 16Removed items 12 and 160.86390.0730Presence of cross-loadings of items 6 (EE and DEP) and 2 (EE, DEP). Presence of local dependencies between items 1 and 2, 2 and 3.
Previous structure with further deletion of item 2, similarly to Kim and Ji, 2009 [47]Removed items 2, 12, 160.89020.0639Presence of cross-loadings of items 6 (EE and DEP) and 2 (EE, DEP). Presence of local dependencies between items 6 and 5.
Previous structure with further deletion of item 6, similarly to Kanste et al., 2006 [48]Removed items 2, 6, 12, 160.92600.0532
Note. * Issues affecting the fit of the examined structures were detected based on modification indices. Abbreviations: EE = Emotional Exhaustion; DEP = Depersonalization; PA = Personal Accomplishment.
Table
Table 3. Summary of the results of the Multigroup CFA performed to assess the measurement invariance of the Maslach Burnout inventory across study waves (n = 369).
Table 3. Summary of the results of the Multigroup CFA performed to assess the measurement invariance of the Maslach Burnout inventory across study waves (n = 369).
Type of InvarianceCFIRMSEAΔCFIΔRMSEAComment
Configural invariance0.91400.0457 Configural invariance was met
Metric invariance0.90750.04670.00650.001Metric invariance was met
Scalar invariance0.84400.05970.06350.013Complete scalar invariance was not met
Scalar invariance (freed the constraints to the intercepts of items 4, 8, 13, 14 and 20)0.88360.05180.0239 *0.005 *After freeing the loadings of five items, partial invariance was not met
Note. * Calculated using the fit indices of the structure employed to assess metric invariance as reference.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Giusti, E.M.; Veronesi, G.; Callegari, C.; Castelnuovo, G.; Iacoviello, L.; Ferrario, M.M. The North Italian Longitudinal Study Assessing the Mental Health Effects of SARS-CoV-2 Pandemic Health Care Workers—Part II: Structural Validity of Scales Assessing Mental Health. Int. J. Environ. Res. Public Health 2022, 19, 9541. https://doi.org/10.3390/ijerph19159541

AMA Style

Giusti EM, Veronesi G, Callegari C, Castelnuovo G, Iacoviello L, Ferrario MM. The North Italian Longitudinal Study Assessing the Mental Health Effects of SARS-CoV-2 Pandemic Health Care Workers—Part II: Structural Validity of Scales Assessing Mental Health. International Journal of Environmental Research and Public Health. 2022; 19(15):9541. https://doi.org/10.3390/ijerph19159541

Chicago/Turabian Style

Giusti, Emanuele Maria, Giovanni Veronesi, Camilla Callegari, Gianluca Castelnuovo, Licia Iacoviello, and Marco Mario Ferrario. 2022. "The North Italian Longitudinal Study Assessing the Mental Health Effects of SARS-CoV-2 Pandemic Health Care Workers—Part II: Structural Validity of Scales Assessing Mental Health" International Journal of Environmental Research and Public Health 19, no. 15: 9541. https://doi.org/10.3390/ijerph19159541

APA Style

Giusti, E. M., Veronesi, G., Callegari, C., Castelnuovo, G., Iacoviello, L., & Ferrario, M. M. (2022). The North Italian Longitudinal Study Assessing the Mental Health Effects of SARS-CoV-2 Pandemic Health Care Workers—Part II: Structural Validity of Scales Assessing Mental Health. International Journal of Environmental Research and Public Health, 19(15), 9541. https://doi.org/10.3390/ijerph19159541

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop