Hybrid Approach to Predicting Learning Success Based on Digital Educational History for Timely Identification of At-Risk Students

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The article, "Hybrid Approach to Predicting Learning Success Based on Digital Educational History for Timely Identifying At-Risk Students," discusses the implementation of a system designed to predict learning success using data on student interactions with a learning platform and their current and past academic performance at a specific university. The authors introduce two predictive models: one to forecast success in mastering a course and another to forecast successful completion of the semester. This review examines the application of these models at Siberian Federal University (SibFU).

General comments

The article's main contribution lies in documenting the implementation of a forecasting system that has been successfully utilized by a higher education institution (HEI). However, there are several areas where the article could be improved to enhance clarity and coherence:

1. The aims outlined at the beginning do not align clearly with the methods implemented, and the conclusions do not adequately reflect the success of the proposed aims.

2. The paragraph in lines 90-99 should include more detailed descriptions of how the forecasting methods were applied.

3. The relationship and differences between the two aims need to be clarified, as they currently appear to overlap significantly

4. This section requires expansion, for example, the paragraph in lines 271 to 277 should include more explanations of the methodologies employed, the rationale behind them, and the parameters used.

5. Relocate detailed methodological descriptions from the results section to the methods section for better organization.

6. Given the nature of this journal, a discussion on the tangible impacts of the models on educational quality at SibFU would enrich the article.

Specific recommendations:

1. Revise titles 1.3 and 2.3

2. Section 1.3 lacks detail and could benefit from more thorough editing to refine the presentation of information.

3. The paragraph in lines 64 to 73 needs clarification on the specific types of Neural Networks used.

4.  Line 74 contains a sentence that does not add significant value to the discussion.

5. Statements such as those on lines 304 to 306 and 707 to 709 need proper citations to support claims. 

6. Several sections, such as between lines 307 and318, would benefit from editing for clarity and readability. Simplifying complex sentences and ensuring logical flow can make the content more accessible.

Some typos:

• Line 74 *algorithms*. This sentence does seem to contribute much to the manuscript.

• Line 83 *shows*

• Line 439 *debts*

Comments on the Quality of English Language

English needs to be polished there are some sentences and paragraphs that are difficult to follow in the current form.

Author Response

Dear colleague,

On behalf of our team, I would like to thank you for your valuable feedback. I hope that in addressing your comments, we were able to create a more comprehensive and reader-friendly version of the article.

I will go now through your points explaining where they were addressed in the paper.

1) The aims outlined at the beginning do not align clearly with the methods implemented, and the conclusions do not adequately reflect the success of the proposed aims.

> 

We have edited the conclusion of the article to more clearly reflect the success in achieving the research goals.

There are two research aimes outlined in Section 1.4.

The first aim is to develop a hybrid approach to forecasting learning success. The development of the approach was based on the system approach (as mentioned in Section 2.4.) and on the representation of a student’s digital profile as a set components (Section 2.3). Each component can be separately considered as a source of academic success predictors. Thus, the idea is to make forecast based on different components of the digital profile and then combine then to make the final prediction. In Conclusion we briefly describe the resulting approach in paragraphs 1-3.

The second aim is to develop forecasting models based on the approach. We implemented several machine learning methods and data preprocessing techniques (Section 2.4) to create base models and a meta-model. The resulting forecasting system and the quality of forecasts are briefly described in paragraphs 4-5 of the conclusion.

2) The paragraph in lines 90-99 should include more detailed descriptions of how the forecasting methods were applied.

>

We have expanded the description of the forecasting models used in the research studies mentioned in Section 1.3. The additional text is highlighted in yellow.

3) The relationship and differences between the two aims need to be clarified, as they currently appear to overlap significantly

>

We agree that the research aims overlap. However, we consider them as two separate aims, since the results of their achieving are scalable to varying degrees. We added the explanation of our viewpoint at the end of Section 1.4 (highlighted in yellow).

4) This section requires expansion, for example, the paragraph in lines 271 to 277 should include more explanations of the methodologies employed, the rationale behind them, and the parameters used.

> 

We have expanded the description of the methodologies employed. We added information about the model based on Markov processes and its parameters. Additionally, we provided a rationale for using bagging and boosting algorithms, as well as for employing model stacking. The text highlighted in yellow is partially new and partially consists of text relocated from the Results section.

5) Relocate detailed methodological descriptions from the results section to the methods section for better organization.

>

The methodological descriptions from Sections 3.2.2 and 3.3 were partially relocated to Methods (Section 2.4). However, as we consider the presented hybrid approach a result of our research, we have retained the detailed description of the approach in the Results section.

6) Given the nature of this journal, a discussion on the tangible impacts of the models on educational quality at SibFU would enrich the article.

>

We have added Section 4.2 which includes the description of the pilot study on the development and application of support measures for students from at-risk groups. After implementing a set of pedagogical support measures based on learning success forecasts, we noticed an increase in student retention rates. However, additional research is necessary to analyze any possible causal relationships.

• Revise titles 1.3 and 2.3

>

We renamed titles 1.3 and 2.3

8) Section 1.3 lacks detail and could benefit from more thorough editing to refine the presentation of information.

>

We have expanded the description of the forecasting models in Section 1.3.

• The paragraph in lines 64 to 73 needs clarification on the specific types of Neural Networks used.

>

In the mentioned research authors use three and four-layer Artificial Neural Networks. This information was added and highlighted in yellow.

• Line 74 contains a sentence that does not add significant value to the discussion.

>

The sentence was excluded.

• Statements such as those on lines 304 to 306 and 707 to 709 need proper citations to support claims. 

>

We provided citations to the statement on lines 304-306. The statement on lines 707-709 was removed during the revision of the Conclusions

• Several sections, such as between lines 307 and318, would benefit from editing for clarity and readability. Simplifying complex sentences and ensuring logical flow can make the content more accessible.

>

Several sentences, including those between lines 307 and 318, were simplified for increasing readability. All changes are highlighted.

• Correct typos in Line 74, Line 83,Line 439

>

The typos were corrected.

We hope this version addresses most of your comments.

Reviewer 2 Report

Comments and Suggestions for Authors

This is a very interesting article that will contribute to the field of knowledge. The authors make a proposal for an automatic system to predict learning success based on multiple and variable-nature students data. They obtain promising results. Although the proposal is based on a very specific learning context of a specific university, it can be used by other researchers as a framework to create prediction models based on different types of educational data.
The article is very well written and comprehensive. Although some interesting implementation details are missing, they can be consulted in the references which are perfectly cited. 

I only have some minor corrections and suggestions:

- Line 280 of page 7: “assessing feature importance” instead of “accessing feature importance”?
- Line 306 of page 7: SFU stands for Siberian Federal University? You have used another acronym in the rest of the article (SibFU).
- To complete the study, have you considered collecting other information from students, related to study habits or student motivation? It seems that they may be of importance especially in the dropout of first-year students, as some studies suggests (DOI: 10.9781/ijimai.2023.06.002). You might consider or discuss this as future work in the article.

Comments on the Quality of English Language

Although the English Language is of sufficient quality, it would be desirable that it be proofread again (preferably by a native speaker) to correct small errors and improve the final result.

Author Response

Dear colleague,

On behalf of our team, I would like to thank you for your valuable feedback.

The misprints in lines 280 and 306 were corrected.

We share your opinion that motivation plays a crucial role in learning success. In the future, we plan to expand the list of personality traits used in forecasting based on research into their contribution to learning success. We have added motivation and the citation of the corresponding work to the list of characteristics with strong predictive power in the Literature Review (Section 1.3).