Usage of Gamification Techniques in Software Engineering Education and Training: A Systematic Review

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The paper deals with the use of gamification in Software Engineering Education and Training. Considering the number of relevant publications identified by the authors, this seems to be a vivid front of research and thus deserving analysis. Nonetheless, the paper has some flaws and needs to be refined before I could recommend accepting it for publication.

Detailed remarks:

There is a single block of text starting at page 2 and taking most of page 3. For readability, please divide it into shorter paragraphs.

Figure 1 shows a bit trivial vision of the architecture for gamified educational platforms (and in no way it is specialized for Software Engineering as the caption implies). There are papers describing far more sophisticated architectures for gamified educational platforms/learning management systems - I recommend reading them and possibly updating the figure and/or the text around it (which is very generic and lacks references).

The discussion of prior systematic reviews (the Related work section) is too superficial. It does not report the bibliographic databases used by those studies and their key findings.

Also, was not SEET included in any general surveys on gamificatiion in education (there are some of them)?

Please precisely specify the databases you have used: "from databases including" means the list is not complete.

Please explain what do you mean by "Studies for which the full text is not available."?  Not available to you or not available in their respective repository indicated by DOI?

There is a huge number of excluded papers. Please assign numbers of excluded papers to the respective reasons of exclusion.

It's COVID-19 not Covid-19.

I do not think RQ1: "How has Gamification in SEET evolved over time" is properly answered. For a question stated in such a form, I'd expect a topic evolution analysis, and we only get two charts: one showing the change in quantity of papers per year, the other the shares of conference and journal publications. Please change RQ1 or change the content that is supposed to answer it.

The number of papers in Table 2 does not seem to be 68.

As Table 2 spans over several pages, its header should be repeated on each page.

Figure 8 does not conform to RQ5 as continents are not countries (and "hybrid" is neither). Also, it is not clear how these number were obtained: for instance, how a paper having two authors affiliated to institutions in different countries was classified? Or a paper with an author having two affiliations in different countries?

Why RQ6 is answered based on just 3 papers? What about the remaining 65?

Author Response

Thank you very much for taking the time to review this manuscript. Please find the detailed responses below and the corresponding corrections highlighted in the re-submitted files. 

Comment 1: There is a single block of text starting at page 2 and taking most of page 3. For readability, please divide it into shorter paragraphs.
Response 1: Thank you for the comment. We divided the text into shorter paragraphs.
 
Comment 2: Figure 1 shows a bit trivial vision of the architecture for gamified educational platforms (and in no way it is specialized for Software Engineering as the caption implies). There are papers describing far more sophisticated architectures for gamified educational platforms/learning management systems - I recommend reading them and possibly updating the figure and/or the text around it (which is very generic and lacks references).
Response 2: Thank you for the comment. We corrected the Figure caption and, by moving it at the top of the subsection, we want to use it as a general starting point for gamified platforms, and not for specialized ones. We also inserted some references to the text.
 
Comment 3: The discussion of prior systematic reviews (the Related work section) is too superficial. It does not report the bibliographic databases used by those studies and their key findings.
Response 3: We updated the section with additional details on related reviews, such as the used databases and the main findings.

Comment 4: Also, was not SEET included in any general surveys on gamificatiion in education (there are some of them)?
Response 4: We updated the section by adding new related reviews on gamification and SEET.
 
Comment 5: Please precisely specify the databases you have used: "from databases including" means the list is not complete.
Response 5: We updated the sentence to include the used databases. We also updated the PRISMA flow diagram with included databases and number of gathered papers.

Comment 6: Please explain what do you mean by "Studies for which the full text is not available."?  Not available to you or not available in their respective repository indicated by DOI?
Response 6: We modified the methodology section to include more details.
 
Comment 7: There is a huge number of excluded papers. Please assign numbers of excluded papers to the respective reasons of exclusion.
Response 7: We modified the methodology section to include the number of excluded papers per criteria.
 
Comment 8: It's COVID-19 not Covid-19.
Response 8: We corrected it.
 
Comment 9: I do not think RQ1: "How has Gamification in SEET evolved over time" is properly answered. For a question stated in such a form, I'd expect a topic evolution analysis, and we only get two charts: one showing the change in quantity of papers per year, the other the shares of conference and journal publications. Please change RQ1 or change the content that is supposed to answer it.
Response 9: Thank you for the insight. We changed RQ1 to better suit the given answer.
 
Comment 10: The number of papers in Table 2 does not seem to be 68.
Response 10: Thank you for the comment. We updated Table 2 and included all the 68 papers.
 
Comment 11: As Table 2 spans over several pages, its header should be repeated on each page.
Response 11: We corrected it.
 
Comment 12: Figure 8 does not conform to RQ5 as continents are not countries (and "hybrid" is neither). Also, it is not clear how these number were obtained: for instance, how a paper having two authors affiliated to institutions in different countries was classified? Or a paper with an author having two affiliations in different countries?
Response 12: Thank you for the comment. We changed "countries" into "continents" and we added an explanation on "hybrid": this field includes papers where authors have affiliations from two or more continents, to avoid inconsistencies in the data extraction.
 
Comment 13: Why RQ6 is answered based on just 3 papers? What about the remaining 65?
Response 13: We updated RQ6 results by including the analyzed papers that explicitly evaluate advantages or disadvantages of gamification in SEET.

Reviewer 2 Report

Comments and Suggestions for Authors

The paper is well written and organized, following changes can be made to further improve the quality of the paper. 

1. Abbreviation SEET can be used more often for example Page 2, line 40 use SEET instead of Software Engineering Education

2. Page 2, line 43, give reference to PRISMA methodology

3. Use Figure instead of Fig (for example in Line number 119, page 3, Figure instead of Fig)

4. Figure 1 and the explanation written in paragraph can be moved in the beginning of 2.1 to have a better flow

5. Section 5.1, RQ1, it will be good to see the trends of Journal and Conference papers over the period under considered (2015-23)

6. Since universities offer more courses related to software engineering etc as compared to schools, there is more scope for gamification in such courses. Since, it is very difficult to find the percentages, this can be added as one of the explanations in discussion section etc

7. More description can be provided for “serious games” and how this categorization is done. Some examples can be explained (line 312)

8. Context of statement “difficulty in providing immediate feedback” needs to be provided, whether this feedback to the student/user, if yes, what are the challenges (line 357)

9. Similarly, “limited time and interest” (who is referred) (line 359)

10. There are 65 References, whereas authors have considered 68 papers, may include the remaining one

Author Response

Thank you very much for taking the time to review this manuscript. Please find the detailed responses below and the corresponding corrections highlighted in the re-submitted files. 

Comment 1: Abbreviation SEET can be used more often for example Page 2, line 40 use SEET instead of Software Engineering Education
Response 1: We defined acronyms at the end of the paper and used SEET instead of Software Engineering Education and Training.

Comment 2: Page 2, line 43, give reference to PRISMA methodology
Response 2: We inserted the citation for PRISMA methodology.

Comment 3: Use Figure instead of Fig (for example in Line number 119, page 3, Figure instead of Fig)
Response 3: We corrected it.

Comment 4: Figure 1 and the explanation written in paragraph can be moved in the beginning of 2.1 to have a better flow
Response 4: Thank you for the comment. We moved Figure 1 and the explanation to the top of Section 2.1.

Comment 5: Section 5.1, RQ1, it will be good to see the trends of Journal and Conference papers over the period under considered (2015-23)
Response 5: We modified RQ2 figure to distinguish between conferences and journals over the years, and we added some discussion on it.

Comment 6: Since universities offer more courses related to software engineering etc as compared to schools, there is more scope for gamification in such courses. Since, it is very difficult to find the percentages, this can be added as one of the explanations in discussion section etc
Response 6: Thank you for the comment. We illustrate the number of papers related to universities, schools, and industries in RQ2. In the discussion of that RQ, we inserted the percentages to highlight the universities role in the topic.

Comment 7: More description can be provided for “serious games” and how this categorization is done. Some examples can be explained (line 312)
Response 7: We updated the serious games paragraph by adding some description and examples.

Comment 8: Context of statement “difficulty in providing immediate feedback” needs to be provided, whether this feedback to the student/user, if yes, what are the challenges (line 357)
Response 8: We modified the sentence for improved clarity.

Comment 9: Similarly, “limited time and interest” (who is referred) (line 359)
Response 9: We modified the sentence for improved clarity.

Comment 10: There are 65 References, whereas authors have considered 68 papers, may include the remaining one
Response 10: Thank you for the comment. We updated Table 2 and included all the 68 papers in the references, plus the references of initial sections.

Reviewer 3 Report

Comments and Suggestions for Authors

The “Usage of Gamification techniques in Software Engineering Education and Training: A Systematic Review” is an interesting review and of course an interesting paper.

 In this paper, the authors address the gap in systematic evaluations of gamification’s effectiveness in Software Engineering Education and Training by conducting a comprehensive literature review of 68 primary studies. The authors describe the advantages of gamification, including active learning, individualized pacing, and enhanced collaboration, as well as the psychological drawbacks such as increased stress and responsibility for students.

 The content of the paper is relevant to the journal. This article is well organized, and the authors seem to have a good knowledge of the subject and the areas they refer to. In our opinion, the article is proper for the journal, it is a good scientific work about the games area and answers many questions that innovative and creative educators have.

 Although the topic is in “good taste”, and the authors have an excellent knowledge of their subject, unfortunately, from the point of the subject the authors have not avoided some basic methodological errors which they should review before publishing.

  Below we note some specific comments that we consider important to improve the whole picture of the research.

 From the typical part of the paper

 1)    In the title:  the word techniques should be capitalized in the title.

2)    We do not use two references at once without explaining exactly what we refer to! Rewrite and explain!!! Why do the authors refer to each reference anywhere in the article? What is the meaning of each reference? Explain why some references are together, for example, “…immediate feedback [3,4]. What is the meaning of these two references?  Correct similar behavior.  The use of “engagement and motivation [8–12]”, is out of the question! Please correct! Also “…leadership [17,18].”

3)    At the end of the first section, “…filtered through the PRISMA methodology.”, which is presented in chapter 4. From the point of view of the reader, this is a very serious mistake, because produces a lot of confusion.  But, from the point of view of documentation the most important is that an author cannot “give” an abbreviation without the meaning of each word that is included as you do with the “SEET”? Please explain!

4)    Also in the introduction, we must inform the real meaning of “Software Engineering Education and Training”. It is not clear what the authors mean by these words. The problem is not only the “education game”, but the education game in “Software Engineering Education and Training”. It would be easier for the reader to know what the meaning is because the interpretation is not the same for everyone. In particular, the concept of Training the ”skills” is in no case defined the same for all levels, all degrees, or all directions.

5)    “…like AC-contract leverage” please explain! Not clear

6)    2. Background and 2.1: give a clear message for what is all these about, what is the meaning of this chapter? Why 2. And 2.1? Is it a kind of form? It is better to start with Figure 1! It is not clear.

7)    “Related work” to what?

8)    “Our review also aims to understand how the effectiveness of gamification in SEET is measured”: it is too late for aims declarations!!!

9)    “Research methodology”: As the article is essentially a qualitative review of games for education, we believe that the methodology should concern, if not the first, at least the very first chapters. We do not mention this out of scholasticism but from the search for meaning in the scientific content of the article.

10)                        For “Research Questions” must be organized quality weighting processes before measurement. Use tools like  Taguette, Nvivo qualitative data analysis, Atlas, etc.

11)    What is the meaning of Table 1. (Comparison between related works and our review)?

12)  PRISMA search methodology: please explain in detail!!!

13)   (PICO) approach is another research methodology. Is it for the article, the information, or the research?

14)   Inclusion and Exclusion Criteria about what?

15)  Figures 3, 4, 5 etc., please explain which “tool” produces those figures!  

16)    Please do not use “Finally” unless it is a final part. It is not useful to put "finally" in the middle of the article!  

17)  What is the meaning of Table 2? Main contribution for each relevant paper?

18)   RQ5: In which countries is gamification mostly analyzed? I am not sure how article information compares geographical distribution. Please explain.

19)  What is the meaning of “disadvantages”?

20)  “Our research, which analyzed 68 papers’….is that correct? Is that the meaning and the purpose of the article?

21)   We suggest using pore pure statistical methods and statistical objectives and their meaning.

22)   For a systemic analysis, we need qualitative tools (for example ΝVIVO, ATLAS, R+, etc. First, define the variables from the scientific questions and then present the results, otherwise, they are not clear the meaning for the reader. Because the content of the article in the qualitative research part, use a conceptualization tool for this. You can leverage various tools (https://cloud.rstudio.com/tags/rstudio-cloud/, https://doc.atlasti.com/ManualWin.v9/ATLAS.ti_ManualWin.v9.pdf, https://lumivero .com/products/nvivo/

23)  Finally, all the questions must be answered not only a few or a part of them, otherwise delete them.

Comments on the Quality of English Language

 Moderate editing of English language required

Author Response

Thank you very much for taking the time to review this manuscript. Please find the detailed responses below and the corresponding corrections highlighted in the re-submitted files. 

Comment 1: In the title: the word techniques should be capitalized in the title.
Response 1: We corrected it.

Comment 2: We do not use two references at once without explaining exactly what we refer to! Rewrite and explain!!! Why do the authors refer to each reference anywhere in the article? What is the meaning of each reference? Explain why some references are together, for example, “…immediate feedback [3,4]. What is the meaning of these two references?  Correct similar behavior.  The use of “engagement and motivation [8–12]”, is out of the question! Please correct! Also “…leadership [17,18].”
Response 2: We modified the references accordingly to the given suggestions.

Comment 3: At the end of the first section, “…filtered through the PRISMA methodology.”, which is presented in chapter 4. From the point of view of the reader, this is a very serious mistake, because produces a lot of confusion.  But, from the point of view of documentation the most important is that an author cannot “give” an abbreviation without the meaning of each word that is included as you do with the “SEET”? Please explain!
Response 3: Thank you for the comment. We inserted acronyms the first time they appear in the paper, we added refs to PRISMA in Section 1, and we listed all the used acronyms at the end of the paper.

Comment 4: Also in the introduction, we must inform the real meaning of “Software Engineering Education and Training”. It is not clear what the authors mean by these words. The problem is not only the “education game”, but the education game in “Software Engineering Education and Training”. It would be easier for the reader to know what the meaning is because the interpretation is not the same for everyone. In particular, the concept of Training the ”skills” is in no case defined the same for all levels, all degrees, or all directions.
Response 4: Thank you for the comment. We added an explanation of SEET in the Introduction section.

Comment 5: “…like AC-contract leverage” please explain! Not clear
Response 5: We modified the paragraph for better clarity.

Comment 6: 2. Background and 2.1: give a clear message for what is all these about, what is the meaning of this chapter? Why 2. And 2.1? Is it a kind of form? It is better to start with Figure 1! It is not clear.
Response 6: We moved Figure 1 at the top of Section 2.1, and we defined the scope of Section 2 with introductive sentences.

Comment 7: “Related work” to what?
Response 7: This section shows a comparison with other review papers on gamification and software engineering education. We changed its title in "Comparison with other review papers" for better clarity.

Comment 8: “Our review also aims to understand how the effectiveness of gamification in SEET is measured”: it is too late for aims declarations!!!
Response 8: Thank you for the comment. We corrected the sentence.

Comment 9: “Research methodology”: As the article is essentially a qualitative review of games for education, we believe that the methodology should concern, if not the first, at least the very first chapters. We do not mention this out of scholasticism but from the search for meaning in the scientific content of the article.
Response 9: Thank you for the useful comment. We moved Research Methodology section after the Introduction section.

Comment 10: For “Research Questions” must be organized quality weighting processes before measurement. Use tools like  Taguette, Nvivo qualitative data analysis, Atlas, etc.
Response 10: We added the explanation of used tools in the paper.

Comment 11: What is the meaning of Table 1. (Comparison between related works and our review)?
Response 11: Table 1 compares other review papers with the one proposed in this paper. It summarizes their main contributions and the differences between those findings and ours.

Comment 12: PRISMA search methodology: please explain in detail!!!
Response 12: Thank you for the comment. We inserted a new PRISMA Methodology section.

Comment 13: (PICO) approach is another research methodology. Is it for the article, the information, or the research?
Response 13: PICO is part of the PRISMA methodology. We explain that in the new PRISMA Methodology section.

Comment 14: Inclusion and Exclusion Criteria about what?
Response 14: We explain Inclusion and Exclusion Criteria in the new PRISMA Methodology section. These criteria are used to filter papers in order to obtain relevant studies to be analyzed.

Comment 15: Figures 3, 4, 5 etc., please explain which “tool” produces those figures!  
Response 15: We added the explanation of used tools in the paper.

Comment 16: Please do not use “Finally” unless it is a final part. It is not useful to put "finally" in the middle of the article!  
Response 16: We corrected it.

Comment 17: What is the meaning of Table 2? Main contribution for each relevant paper?
Response 17: This table answers RQ4 by summarizing the main finding carried out by the 68 analyzed papers, We improved it by adding all the 68 rows and by modifying the caption.

Comment 18: RQ5: In which countries is gamification mostly analyzed? I am not sure how article information compares geographical distribution. Please explain.
Response 18: We modified RQ5 for better readability.

Comment 19: What is the meaning of “disadvantages”?
Response 19: We analyze the advantages of using gamification in SEET context, together with its limitations, to understand if advantages are more than disadvantages.

Comment 20: “Our research, which analyzed 68 papers’….is that correct? Is that the meaning and the purpose of the article?
Response 20: We modified the sentence for better clarity.

Comment 21: We suggest using pore pure statistical methods and statistical objectives and their meaning.
Response 21: We added the explanation of used tools in the paper.

Comment 22: For a systemic analysis, we need qualitative tools (for example ΝVIVO, ATLAS, R+, etc. First, define the variables from the scientific questions and then present the results, otherwise, they are not clear the meaning for the reader. Because the content of the article in the qualitative research part, use a conceptualization tool for this. You can leverage various tools (https://cloud.rstudio.com/tags/rstudio-cloud/, https://doc.atlasti.com/ManualWin.v9/ATLAS.ti_ManualWin.v9.pdf, https://lumivero .com/products/nvivo/
Response 22: We added the explanation of used tools in the paper.

Comment 23: Finally, all the questions must be answered not only a few or a part of them, otherwise delete them. 
Response 23: We modified RQs and Results section to answer each RQ in a complete manner.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Thank you for the corrections. The detailed comments are given below:

>Response 1: Thank you for the comment. We divided the text into shorter paragraphs.

Thank you.

>Response 2: Thank you for the comment. We corrected the Figure caption and, by moving it at the top of the subsection, we want to use it as a general starting point for gamified platforms, and not for specialized ones. We also inserted some references to the text.

Thank you.

>Response 3: We updated the section with additional details on related reviews, such as the used databases and the main findings.

Thank you, but Table 2 needs some more description (what exactly is in the respective columns). In my opinion the captions of the last two columns should be made clearer (and/or should have a precise description of their contents in the main text).

>Response 4: We updated the section by adding new related reviews on gamification and SEET.

Thank you.

>Response 5: We updated the sentence to include the used databases. We also updated the PRISMA flow diagram with included databases and number of gathered papers.

Thank you, but "from different databases" is not exactly the phrase used for such a purpose. It should rather say: "The following databases were searched...".

>Response 6: We modified the methodology section to include more details.

Thank you, but regarding "not readable without subscriptions": if you omitted some databases because you did not have paid the subscription, this should be clearly explained when databases are listed (e.g.: "database x (only open content)" or something similar).

>Response 7: We modified the methodology section to include the number of excluded papers per criteria.

Thank you, but the bulk of excluded papers (n = 823) remains without any comment.

>Response 8: We corrected it.

Thank you.

>Response 9: Thank you for the insight. We changed RQ1 to better suit the given answer.

Thank you, but "seeks to uncover patterns in publications" suggests a more thorough analysis, and there is none such. Why won't you simply define the RQ1 as aiming to investigate the trend in publication quantity and the structure of publication venues?

>Response 10: Thank you for the comment. We updated Table 2 and included all the 68 papers.

Thank you.

>Response 11: We corrected it.

Thank you.

>Response 12: Thank you for the comment. We changed "countries" into "continents" and we added an explanation on "hybrid": this field includes papers where authors have affiliations from two or more continents, to avoid inconsistencies in the data extraction.

Thank you.

>Response 13: We updated RQ6 results by including the analyzed papers that explicitly evaluate advantages or disadvantages of gamification in SEET.

Thank you.

Author Response

Thank you very much for taking the time to review this manuscript. Please find the detailed responses below and the corresponding corrections highlighted in the re-submitted files.

Comment 1: Table 2 needs some more description (what exactly is in the respective columns). In my opinion the captions of the last two columns should be made clearer (and/or should have a precise description of their contents in the main text).
Response 1: Thank you for the comment, we modified the last two column names in Table 2, and we added a textual description of the various fields.

Comment 2: "from different databases" is not exactly the phrase used for such a purpose. It should rather say: "The following databases were searched...".
Response 2: Thank you for the comment, we corrected it.

Comment 3: regarding "not readable without subscriptions": if you omitted some databases because you did not have paid the subscription, this should be clearly explained when databases are listed (e.g.: "database x (only open content)" or something similar).
Response 3: Thank you for the comment. We updated the databases list.

Comment 4: the bulk of excluded papers (n = 823) remains without any comment.
Response 4: Thank you for the comment. We added the specification of excluded papers in the exclusion criteria list. Figure 1 only reports the total of excluded reports, accordingly to PRISMA Flow Diagram statement (https://estech.shinyapps.io/prisma_flowdiagram/).

Comment 5: "seeks to uncover patterns in publications" suggests a more thorough analysis, and there is none such. Why won't you simply define the RQ1 as aiming to investigate the trend in publication quantity and the structure of publication venues?
Response 5: Thank you for the comment. We modified RQ1 description accordingly to the suggestion.

Reviewer 3 Report

Comments and Suggestions for Authors

The paper can be accepted without any further changes.

Author Response

Thank you very much for taking the time to review this manuscript.