Learning Patterns at the Time of COVID-19-Induced School Closures
Round 1
Reviewer 1 Report
This paper draws on a very ambitous research focusing on the relevant topic of our recent times, that is, what is the impact of the school closures on students' learning losses and is based on data gathered among students themselves. Although the quantitative approach follows a classic pattern searching for correlations, the search of factors that may help understand differences in the students' perceptions through their responses is remarkable. However, the paper could be improved by acknowledging the limitations of the approach itself, that only relies on one source of data, and it may neglect the diverse types of learning environments that schools provide -differences are blurred there- enhancing the resulting inequalities identified beyond students' families and characteristics. Moreover, when linking the impact of school closures to the higher risks of droppouts and subsequent early school leaving the article ignores some literature right prior to the pandemic comparing European countries and the key concept of school engagement and its dimensions. Some deeper analysis in this sense would improve the article, which is already good, to make a more significant contribution considering all the effort made to produce the data.
Author Response
Dear Sir/Madam, Thank you for your review. We tried to answer all your questions and suggestions. Please see the attachment.
Author Response File: 
Author Response.docx
Reviewer 2 Report
The article is clear and detailed structured and relevant for social and educational science. The cited references are mostly recent publications within the last 2 years during the pandemic; relevant publications of EU and meta-analysis are mentioned. There are few numbers of self-citations. The methodology is detailed and clear explained. The experimental design is appropriate to test the hypothesis and to answer the research questions. The tables are appropriate, they show the quantitative data of different analysis. Interpretation and main results of each analysis are summarized. This could be high lightened in the data base of the table . Even though there are other studies on the impact of the Covid Pandemic, the design of the presented research and the results provide an advancement of the current knowledge about inequalities in school learning and the influence of individual and family factors.
Author Response
Dear Sir/Madam, Thank you for your review and appreciation. Please see the attachment.
Author Response File: Author Response.docx
Reviewer 3 Report
This is an interesting study and a contribution to extended knowledge about 7th graders at risk of dropouts and how they adapted to the Covid-19 situation and factors in interplay to their adaptation strategy.
There are however several concerns that need to be addressed before publication.
A major concern is recruitment, data collection, questionaries used in the survey and discussion. The authors also include a substantial number of variables. A suggestion is to divide this into at least two publications.
Line 204. RQ2 is unclear. Do not use i.e. in a RQ. Further on, the last part, from i.e. is not in accordance with the first part of RQ2.
Material and Methods.
The method is not clear and concisely described.
It seems like the sample was selected based on the early warning system for students at risk of dropping out of school. It is however unclear if the results used in this study were based on already collected data or if the questionaries used in the survey were developed especially for this study.
Line 210, the bracket is empty.
Line 215-216. The study is 7th graders while it is referred to as an index that includes 5th-8th grade students. The context in which the students were selected is unclear.
Are the questions in the survey validated or tested in some way?
The students in this study are at risk of dropping out of school, a total of 3,251 participated in the study, a response rate of 54.4%. Is the included student representative of the total population of students at risk of dropping out?
What is the total population of students in 7th grade in the 19 counties?
Line 255, the p-value is included. What was the level of the statical significant set to?
Line 261, «Physical health was measured by self-rated health status (poor, fair, good, excellent)(Cavallo et al. 2015) and the prevalence (none/existing) of a chronic non-communicable disease (e.g. diabetes, arthritis, allergies, birth injury) diagnosed by a doctor. Mental health was analysed using overall satisfaction with life as measured by the Cantril ladder». This describes how data on physical health was reported, information on how data on mental health was reported is missing. Is this self-reported or results from previously collected data?
Regarding questions described in 2.4, Family Background Characteristics, especially questions about parents smoking, alcohol consumption, family member in prison, and tranquillisers/sleeping pills use. Is information about the families from previously collected data and were the questions answered by the 7th graders or parents themselves?
Line 255, the level of significance was set at a 5% significant level. Was this for only the Bonferroni post-hoc test or for other analysis?
Results
The authors have included a substantial number of variables; it is easy to get lost in the information. What is essential and what is less important?
In Results, lines 356-369 results are presented. The same information is in Fig 1. Not needed to present the result twice.
Line 350, no significant result is reported. Not necessary if the level of significance is clear. However, it is of interest to report on what the significance is.
Line 387-389, stated that ‘Easy adapters’ are overrepresented in cities and larger towns and underrepresented in villages, while the opposite is true for the ‘Resigned’ group. Please explain how this conclusion was made. Based on the results, the majority of the easy adapters are living in cities and larger towns, and only 22% are living in villages. In the resigned cluster, also here the majority are living in cities and large towns, however, there is a slightly larger proportion living in the villages compared with those in the easy adapter cluster, 32.9% and 22.0% respectively.
Line 425-429. Unclear what the authors mean.
Discussion
Discussion is basically reporting on the results. There is a lack of reference to previous studies.
Table 6, if included should be in the result section or as additional material. It simply once again describes the results.
Strength and limitations are not included in the manuscript. It is not discussed potential risks i.e. self-reported data in regards to bias. Other potential biases are not discussed.
Tables and figures.
In Figure 1 a pie chart is chosen to illustrate the proportion in the respective cluster. A staple chart could also be chosen, it is often easier to illustrate the difference in the different clusters.
Table 2. 4 and 5. Mean and SD is reported. N is missing.
Table 6 should be included in the result section or as additional information.
Author Response
Dear Sir/Madam, Thank you for your review. We tried to answer all your questions and suggestions.
Response to Reviewer 3 Comments
Point 1: This is an interesting study and a contribution to extended knowledge about 7th graders at risk of dropouts and how they adapted to the Covid-19 situation and factors in interplay to their adaptation strategy. There are however several concerns that need to be addressed before publication. A major concern is recruitment, data collection, questionaries used in the survey and discussion.
Response 1: Thank you for your valuable and detailed review. In the following, we will try to answer all your questions and suggestions.
Point 2: The authors also include a substantial number of variables. A suggestion is to divide this into at least two publications.
Response 2: The aim of our analysis was to explore our research question from as many perspectives as possible, which is why we studied individual, family and school life-related variables together. It is precisely the originality of the article that would be lost if we were to break down the groups of variables and analyse them in separate papers.
Point 3: Line 204. RQ2 is unclear. Do not use i.e. in a RQ. Further on, the last part, from i.e. is not in accordance with the first part of RQ2.
Response 3: Thank you for your suggestion; we have adapted the research question.
«RQ2. What are the individual, family or school-related characteristics that differentiate students' opinions?»
Point 4: The method is not clear and concisely described. It seems like the sample was selected based on the early warning system for students at risk of dropping out of school. It is however unclear if the results used in this study were based on already collected data or if the questionaries used in the survey were developed especially for this study.
Response 4: Thank you for your comment. The data collection for the survey is described in lines 208-231. In lines 208-210 we have indicated that our analyses are based on the results of an online survey of students in the framework of a complex research project on early school leaving in Hungary. Several analyses of the survey have been produced and are in the process of being published. The results used in our study were therefore based on data already collected, adding that we also included questions on the epidemic situation in the online questionnaire, which we have detailed in this analysis. This part of the survey was specifically designed for this study.
Point 5: Line 210, the bracket is empty.
Response 5: Bracket is empty due to self-reference.
Point 6: Line 215-216. The study is 7th graders while it is referred to as an index that includes 5th-8th grade students. The context in which the students were selected is unclear. The students in this study are at risk of dropping out of school, a total of 3,251 participated in the study, a response rate of 54.4%. Is the included student representative of the total population of students at risk of dropping out? What is the total population of students in 7th grade in the 19 counties?
Response 6: The schools participating in the project were selected on the basis of the proportion of pupils at risk of dropping out, collected by the early warning system. The target group of the research project are the 7th-grade students of the selected schools (a total of 5,975 pupils from 159 schools). The early warning system does indeed contain data for pupils in grades 5-8, but data is also available by grade, i.e. data is available for grade 7. We have chosen grade 7 because the questionnaire items are best suited to this cohort and grade 7 can be linked to the previous academic year’s, i.e. grade 6, data in the Hungarian National Assessment of Basic Competences which we have used in our other analyses.
Accordingly, the text of the Survey Method and Respondents section (2.1.) of the article is amended as follows:
«The research project was carried out in five of Hungary's 19 counties (Vas, Zala, Győr-Moson-Sopron, Borsod-Abaúj-Zemplén and Veszprém) and Budapest. The schools participating in the project were selected on the basis of the proportion of students at risk of dropping out from the early warning system that was introduced nationwide in 2016. The early warning system aims to collect data on students at risk of dropping out that will provide a basis for schools for intervention and support. The early warning system includes an index for primary school students in grades 5–8, based on a number of elementary variables, both by grade and collectively, which shows the proportion of students at risk of dropping out at school level. The target group of the research project are the seventh-grade students of the selected schools, i.e. 5,975 pupils from 159 schools. A total of 3,251 seventh-graders from 117 schools participated in the student survey (return rate: 54.4%). The sample was reduced by the lack of response to the questions in the focus of the analysis. Thus, the sample analysed in our study consisted of responses from 3,222 pupils. The sample cannot be considered representative due to the purpose of the broader context of the complex research project (to study schools most affected by dropout in certain areas), however, the gender distribution of the sample matches the gender distribution of students in schools in the study areas (binomial test: p = 0.12).
The data used as a basis for the analyses were collected using the self-administered CAWI surveying technique. The data collection took place in the first semester of the 2020/2021 school year, i.e. in the autumn of 2020. The questionnaires were completed on a voluntary and anonymous basis, subject to the ethical approval of the research site.»
Furthermore, in describing the sample, it was explained that the sample cannot be considered representative (see lines 227-229) because the research project was carried out in five of Hungary's 19 counties and Budapest. Nevertheless, we believe that our sample is suitable for exploring differences between students at risk of dropping out, and fits our research goals. At the same time, this limitation has been added to the Limitations subsection (4.1) of the Discussion section (4) (see Response 18).
Point 7: Are the questions in the survey validated or tested in some way?
Response 7: Sections 2.2.-2.6. contain a description of the variables and groups of variables and how they are created. Validated measurement tools were also used in our analysis (e.g. Rosenberg Self-Esteem Scale). Where necessary, Cronbach's alpha values have been added to the description.
Point 8: Line 255, the p-value is included. What was the level of the statical significant set to?
Response 8: For all statistical analyses, the significance level was set at p < 0.05. This sentence has been added to the Analysis Procedure section (2.6).
Point 9: Line 261, «Physical health was measured by self-rated health status (poor, fair, good, excellent)(Cavallo et al. 2015) and the prevalence (none/existing) of a chronic non-communicable disease (e.g. diabetes, arthritis, allergies, birth injury) diagnosed by a doctor. Mental health was analysed using overall satisfaction with life as measured by the Cantril ladder». This describes how data on physical health was reported, information on how data on mental health was reported is missing. Is this self-reported or results from previously collected data? Regarding questions described in 2.4, Family Background Characteristics, especially questions about parents smoking, alcohol consumption, family member in prison, and tranquillisers/sleeping pills use. Is information about the families from previously collected data and were the questions answered by the 7th graders or parents themselves?
Response 9: In all cases, the data are from the online student questionnaire, i.e. the students' self-report (see Survey Method and Respondents section of the article).
Point 10: Line 255, the level of significance was set at a 5% significant level. Was this for only the Bonferroni post-hoc test or for other analysis?
Response 10: For all statistical analyses, the significance level was set at p < 0.05. This sentence has been added to the Analysis Procedure section (2.6).
Point 11: The authors have included a substantial number of variables; it is easy to get lost in the information. What is essential and what is less important?
Response 11: The aim of the analysis was to distinguish between groups of pupils on the basis of their individual, family and school-related. The aim was to illustrate the complexity of the phenomenon that individuals within disadvantaged groups are diverse and can be further subdivided into groups. In the Discussion section of this article, we have tried to highlight and compare our essential findings with the results of previous research.
Point 12: Line 350, no significant result is reported. Not necessary if the level of significance is clear. However, it is of interest to report on what the significance is.
Response 12: For all statistical analyses, the significance level was set at p < 0.05. This sentence has been added to the Analysis Procedure section (2.6). However, it is also important to highlight non-significant variables for several reasons. On the one hand, this can help the reader not to get lost in the information, and on the other hand, we believe that this information is also professionally relevant.
Point 13: In Results, lines 356-369 results are presented. The same information is in Fig 1. Not needed to present the result twice.
Response 13: We accept the reviewer's suggestion and deleted Figure 1 from the article.
Point 14: Line 387-389, stated that ‘Easy adapters’ are overrepresented in cities and larger towns and underrepresented in villages, while the opposite is true for the ‘Resigned’ group. Please explain how this conclusion was made. Based on the results, the majority of the easy adapters are living in cities and larger towns, and only 22% are living in villages. In the resigned cluster, also here the majority are living in cities and large towns, however, there is a slightly larger proportion living in the villages compared with those in the easy adapter cluster, 32.9% and 22.0% respectively.
Response 14: This is indicated by the absolute values greater than 2.0 for the adjusted standardised residuals in brackets in Table 3. An adjusted standardised residual that is more than 2.0 indicates that the number of cases in that cell is significantly larger, less than –2.0 indicates that the number of cases in that cell is significantly smaller than would be expected if the null hypothesis (2 variables are independent) were true, with a significance level of 0.05. So, there are fewer (underrepresented) or more (overrepresented) cases than would be expected if the 2 variables were independent.
Point 15: Line 425-429. Unclear what the authors mean.
Response 15: Thank you for your helpful comment. The sentence has been clarified as follows.
«There are no significant differences between ‘Struggling but coping’ and ‘Compensators with more time input’ as well as between ‘Struggling but coping’ and ‘Resigned’ groups in all indicators of the characteristics of school life.»
Point 16: Discussion is basically reporting on the results. There is a lack of reference to previous studies.
Response 16: There are eight references in the discussion, several of which are systematic literature reviews. However, additional references have been added to this section:
«Whether changes in family background increase or decrease student achievement gaps depends to a large extent on which family background variable is considered the most critical. Such a critical family background variable can be family income and wealth, family structure, parents’ education, health, or well-being. The precise impact of the changes in family factors is unclear (Hanuschek et al. 2022). In our study, we highlight the role of financial background and family structure.»
[Hanushek, Eric A., Jacob D. Light, Paul E. Peterson, Laura M. Talpey, and Ludger Woessmann. 2022. Long-run Trends in the U.S. SES-Achievement Gap. Available online: http://hanushek.stanford.edu/sites/default/files/publications/SES%20paper%20final%20version.pdf (accessed on 4 August 2022).]
Point 17: Table 6, if included should be in the result section or as additional material. It simply once again describes the results.
Response 17: Based on reviewer’s suggestion, Table 6 has been moved to the Main differences between the student groups subsection (3.5) of the Results section (3).
Point 18: Strength and limitations are not included in the manuscript. It is not discussed potential risks i.e. self-reported data in regards to bias. Other potential biases are not discussed.
Response 18: Thank you for reviewers’ suggestion. Based on your suggestion, Limitations subsection (4.1) has been added to the Discussion section (4).
«4.1. Limitations
Sampling was based on availability, and we analysed schools in particular regions where students at risk of dropping out are overrepresented. Consequently, the sample cannot be considered representative and the findings and conclusions derived from our analyses only refer to the students involved. Nevertheless, in our professional opinion, the sample fits our research goals, i.e. it is suitable for exploring differences between student responses and learning patterns to online/digital distance learning introduced due to the COVID-19 pandemic among students at risk of dropping out.
Another limitation is the survey was self-reported, i.e. our findings are based solely on student perceptions and opinions, which may also bias our results. Thus, our analysis does not include school background variables and the different types of actual learning environments provided by schools that may enhance identified inequalities beyond the characteristics of students and their families.
Taking all of this into account, we have sought to capture the phenomenon in a complex way, by exploring the internal structure of disadvantaged groups of pupils through a multi-perspective study.»
Point 19: In Figure 1 a pie chart is chosen to illustrate the proportion in the respective cluster. A staple chart could also be chosen, it is often easier to illustrate the difference in the different clusters. Table 2. 4 and 5. Mean and SD is reported. N is missing. Table 6 should be included in the result section or as additional information.
Response 19: We have deleted Figure 1 from the article. N has been added to Tables 2, 4 and 5. Table 6 has been moved to the Main differences between the student groups subsection (3.5) of the Results section (3).
