User-Centered Software Design: User Interface Redesign for Blockly–Electron, Artificial Intelligence Educational Software for Primary and Secondary Schools

Round 1

Reviewer 1 Report

The abstract does not have an introduction to the topic of artificial intelligence used in education and should incorporate a sentence or two in this regard to contextualize the research

 

Both the Introduction and Literature Review sections should contain a greater number of citations. In addition, these must be current and impactful. Artificial Intelligence is being talked about and this is a field of interest that has been taking place in the last 5-10 years. Some statements and statements are made that must be supported by references to other studies and research (for example when artificial intelligence is defined, or the use made of it in China in primary and secondary education). Please complete this correctly.

 

Sometimes there are formatting errors. For example, when mentioning the My safe care software, it should be highlighted either in quotation marks or in italics. The format established by the magazine is not respected. There are times when they cite some authors and then forget to cite the reference correctly. Please review this throughout the document.

 

Congratulations on the design of the methodology used. All the phases carried out are very clear and Figure 1 helps a lot to understand the steps taken. It is a pity that the sample of students is so small, but even so, the research is of great interest and highly relevant.

 

The results are very well presented. All tables and figures are pertinent and very clarifying. It would be interesting if Figure 6 were modified and that the numbers 1, 2, 3 of the three stages of the flow, were put in typographical format and not as they are, which are very informal and unscientific. The same occurs with Figure 8a and the letters A, B, and C.

 

In the Discussion, not a single mention is made of other similar studies, which, even though they are not from the same application-software, can serve as a reference and analysis with the results obtained here. Authors are invited to correct this lack of references to other similar studies with which to compare.

It is appreciated that the small sample size (73) is mentioned among the limitations of this study.

 

Insist that there are few references and that it should be completed with more references, current (from the last five years) and of impact.

Author Response

Point 1: The abstract does not have an introduction to the topic of artificial intelligence used in education and should incorporate a sentence or two in this regard to contextualize the research.

Response 1: Thank you for your valuable suggestion, the lack of discussion of the use of AI in education in the abstract has been revised.

Point 2: Both the Introduction and Literature Review sections should contain a greater number of citations. In addition, these must be current and impactful. Artificial Intelligence is being talked about and this is a field of interest that has been taking place in the last 5-10 years. Some statements and statements are made that must be supported by references to other studies and research (for example when artificial intelligence is defined, or the use made of it in China in primary and secondary education). Please complete this correctly.

Response 2: This is a very important and valuable suggestion. In the newly uploaded manuscript, we have added a large number of citations to the introduction and literature review section, which are all from the research related to artificial intelligence and UCD theory in the past 10 years (2013-2022), and also pay special attention to citing relatively influential materials, such as the Horizon report in the field of education in the past two years, and the high-cited hot articles on artificial intelligence education in the WOS database.

Point 3: Sometimes there are formatting errors. For example, when mentioning the my safe care software, it should be highlighted either in quotation marks or in italics. The format established by the magazine is not respected. There are times when they cite some authors and then forget to cite the reference correctly. Please review this throughout the document.

Response 3: We apologize for the formatting errors in the article, and we appreciate your rigorous scientific attitude and the accurate pointing out of our omissions through solid research literacy. In the revised manuscript, we have carefully revised the above questions.

Point 4: Congratulations on the design of the methodology used. All the phases carried out are very clear and Figure 1 helps a lot to understand the steps taken. It is a pity that the sample of students is so small, but even so, the research is of great interest and highly relevant.

Response 4: Thank you for your evaluation. The lack of sample size is also one of the big limitations of our study, and we have also deeply reflected within our team. So, as written in the article, the next phase of research based on this research will focus on solving the problem of insufficient sample size. Thank you again for your accurate, pertinent and valuable evaluation.

Point 5: The results are very well presented. All tables and figures are pertinent and very clarifying. It would be interesting if Figure 6 were modified and that the numbers 1, 2, 3 of the three stages of the flow, were put in typographical format and not as they are, which are very informal and unscientific. The same occurs with Figure 8a and the letters A, B, and C.

Response 5: This issue is the same as the formatting error mentioned above, and we apologize again for not being rigorous enough. All of the above questions already do carefully revise. Thank you again for your meticulous review of the article and pointing out our shortcomings, which will further help our research to be more rigorous, formal and scientific.

Point 6: In the Discussion, not a single mention is made of other similar studies, which, even though they are not from the same application-software, can serve as a reference and analysis with the results obtained here. Authors are invited to correct this lack of references to other similar studies with which to compare.

Response 6: Thank you for your valuable comments. The horizontal comparative analysis of similar items in the discussion section is indeed missing, and we have added the content of this in the newly uploaded manuscript, while organizing the statements in the discussion section more systematically. 

Point 7: It is appreciated that the small sample size (73) is mentioned among the limitations of this study.

Response 7: Thank you for your appreciation. A clear and serious reflection on the limitation of our research is an important step in taking our future research more scientific and significant.

Point 8: Insist that there are few references and that it should be completed with more references, current (from the last five years) and of impact.

Response 8: Thank you again for your valuable comments. To the revised new manuscripts, we added 25 references from the last 10 years, which were selected as new and impactful as possible.

Reviewer 2 Report

Extensive editing in grammar and english needed - the paper was very hard to read because of grammatical errors -singular/plural; inappropriate tense; fractional/incomplete sentences. Please review the paper and resubmit

Author Response

Point 1: Extensive editing in grammar and english needed - the paper was very hard to read because of grammatical errors -singular/plural; inappropriate tense; fractional/incomplete sentences. Please review the paper and resubmit.

Response 1: Thank you for your important suggestion. We apologize for the problem of language and we already revised and polished the manuscript  through MDPI publication's English editing service department.

Author Response File: Author Response.pdf

Reviewer 3 Report

"usability is how well a product can be used by others effectively"

the same verb, word or lexeme cannot be used in the definition of a term.

 

It is curious to have 73 subjects and that 50 are selected at the end. It is an experimental mortality close to 50%. Also problematic that some questionnaires are selected and 1/3 of them are deleted.

12 of 33 references are outdated, more than 5 years. The theoretical framework is not up to date

 

 

The study is not very original, based on attitudes. There is also no clarity in the intervention. It should be clearly explained what didactic sequence and activities are carried out, for readers and practitioners, so that the article is useful for the educational community.

 

 

It starts from categories (1-7), and the mean (central tendency) is used; also linear regression.

 

All unpublished material, important to replicate the study, is in Chinese. Complex for the scientific community in an international journal

The decision is to reject it due to deficiencies. Although it is a good idea, the description and didactic clarity are scarce; and from the research contributions there are too many shortcomings

 

Author Response

Point 1: "usability is how well a product can be used by others effectively" the same verb, word or lexeme cannot be used in the definition of a term.

Response 1: We appreciate your rigorous and precise advice. In the new manuscript, we have made a more scientific and formal revision of the definition of usability by double-checking the relevant definitions in the ISO standard.

Point 2: It is curious to have 73 subjects and that 50 are selected at the end. It is an experimental mortality close to 50%. Also problematic that some questionnaires are selected and 1/3 of them are deleted.

Response 2: Thank you for your valuable comments. In response to this problem, as we described in the article, when we complete the questionnaire work, before data analysis, we carry out questionnaire cleaning, for the cleaned questionnaire, mainly for two reasons: first, the answer is partially blank or completely blank; Second, there is a clear tendency for regularity in the answers (e.g., all repeated numbers or strong order). Of course, we must deeply reflected on this, and the core reasons for this problem are mainly two: First: we ignore the unique personality of middle school students at the current age, following the rules is what they hate, and expressing flamboyant personality is their tendency, so that some of them have the problem of unwilling or random filling out the scale. Second: the lack of overall sample size search, in the face of the previous problem, we should increase the overall sample size to minimize the impact of the higher loss rate of the questionnaire. Therefore, as written in the article, after being deeply aware of this problem, the next stage of research in this direction will focus on solving the problem of insufficient sample size. Thank you again for your accurate, pertinent and valuable evaluation.

Point 3: 12 of 33 references are outdated, more than 5 years. The theoretical framework is not up to date.

Response 3: This is a valuable piece of advice. This made us aware of the lack of reference count and timeliness. In this regard, in the revised manuscript, we have added 25 influential relevant studies in the past 10 years (63% of which are in the past 5 years) as references to update and improve the theoretical framework of our research. Thank you again for your comments to the point.

Point 4: The study is not very original, based on attitudes. There is also no clarity in the intervention. It should be clearly explained what didactic sequence and activities are carried out, for readers and practitioners, so that the article is useful for the educational community.

Response 4: Thank you for your valuable comments. First of all, for the originality of the research, the main directions of this article such as: user interface design or user-centered design methods are indeed not so original, but our research is to continue to subdivide the field, take the user interface design for artificial intelligence educational software for primary and secondary school students as the specific level of research, and continue to explore the improvement of user-centered design methods on its basis. For example, based on Arnold Lund 2001 proposal for the characteristics of "usability" into four dimensions: "Ease of Learning", "Ease of Use", "Usefulness", and "Satisfaction", and then through empirical testing, we find that the relationship between these four dimensions: "Satisfaction" is significantly influenced by the remaining three dimensions, and in this influence relationship, we find that "Ease of Use" variable has a mediating effect. We think this finding could help researchers or design practitioners provide a direction on how to specifically improve usability. Secondly, regarding the specific implementation or sequence of the teaching work of the research subject Blockly-Electron (artificial intelligence educational software), this part is arranged in the next stage of research. At present, it is planned that the next stage will put the redesigned Blockly-Electron into a wider teaching practice (here we focus on learning the lessons of the current research sample inadequacy), while closely observing the specific impact of the software interface on specific teaching steps and sequences in teaching practice, so as to carry out the "cycle iteration design" mentioned in the article, so as to further improve the software to achieve better teaching results.

Point 5: It starts from categories (1-7), and the mean (central tendency) is used; also linear regression.

Response 5: Yes, here we use the multi-step regression method to verify the mediation effect of the "Ease of use" variable. At first, multiple linear regression analysis was carried out with "Ease of Use", "Usefulness" and "Ease of Learning" as the independent variables and "Satisfaction" as the dependent variable (i.e., regression 1 on Table 4), but it was found that the regression coefficients of the three independent variables were all not significant (p>0.05). And because the correlation test between the four independent variables was previously found to have a very high correlation between "Usefulness" and "Ease of Use" (p<0.05, Pearson=0.802), it is speculated that "Ease of Use" is likely to have a mediating effect. Subsequently, a multi-step regression method was used to test the hypothesis. Thank you for your review.

Point 6: All unpublished material, important to replicate the study, is in Chinese. Complex for the scientific community in an international journal

Response 6: Thank you very much for pointing out our shortcomings. At that time, we did not upload the English version of the supporting materials because they were not published with the article, but after your reminder, we will upload the English version of the supporting materials later to improve the reproducibility of the study.

Point 7: The decision is to reject it due to deficiencies. Although it is a good idea, the description and didactic clarity are scarce; and from the research contributions there are too many shortcomings

Response 7: Thank you very much for your carefully review and pointing out the shortcomings of this study (e.g., small sample size, lack of up-to-date citations to support the theory). But for a description of the steps of our software in specific teaching, actually, this is not the focus of our current research, the focus of this paper is on the redesign of user interfaces of artificial intelligence teaching software in subdivided fields and the further expansion of user-centered design methods, such as specific directions to improve the usability of software artificial intelligence education software. As we said in the article, this software can be combined with our planned courses, exercises, and hardware robots to create a good artificial intelligence teaching environment, but the premise is that the quality of this software first passes the test (here we refer to the usability of the software), so in the current stage of research we first check its usability. Subsequently, when it is more deeply integrated with the curriculum activities, we will specifically study the user experience of its various steps of the curriculum, because this is obviously directly related to the educational effect.

Reviewer 4 Report

Document is well written, depicting an interesting research issue. The paper presents a problematic situation and a solution provided. however, it could be improved if some aspects are considered:

- There are some related works could be interrsting to analyze: Afrianto et al (Implementation of User Centered Design Method in Designing Android-based Journal Reminder Application); Moquillaza et al (Developing an ATM interface using user-centered design techniques)

- About usability evaluation why do not consider to use another technique? cognitive wlakthroughs, Thinking aloud, Nielsen heuristics. check the work of Rusu et al (A Methodology to establish usability heuristics)

- Why do not consider UX(User eXperience) and not only usability aspects? Check the work of Puerta et al (The Thin Red Line Between Usability and User Experiences)

- Improve conclusions

- Include further works

Author Response

Point 1: There are some related works could be interrsting to analyze: Afrianto et al (Implementation of User Centered Design Method in Designing Android-based Journal Reminder Application); Moquillaza et al (Developing an ATM interface using user-centered design techniques)

Response 1：We really appreciate your important recommendation, and we will carefully study the above related articles and add them to the analysis of this article.

Point 2: About usability evaluation why do not consider to use another technique? cognitive wlakthroughs, Thinking aloud, Nielsen heuristics. check the work of Rusu et al (A Methodology to establish usability heuristics)

Response 2：Thank you for your valuable suggestions. The reasons for choosing USE scale research for usability evaluation are follow: first, we agree with the method of subdividing the characteristics of "usability" into four dimensions: "Ease of Learning", "Ease of Use", "Usefulness" and "Satisfaction"; Second, we hope to do further analysis for these four dimensions, and in this paper, we use linear regression to do further impact analysis, and find that the variable "Satisfaction" is significantly affected by the remaining three dimensions, and in this influence relationship, the "Ease of Use" variable has a mediating effect. We think this finding could help researchers or design practitioners provide a direction on how to specifically improve usability. Of course, for many other types of evaluation methods, we plan to reduce the error of a single judgment method in the next stage of research, which is to determine whether the software usability has been significantly improved (and thus determine whether the cycle iteration design described in Figure 1) needs to be performed.

Point 3: Why do not consider UX(User experience) and not only usability aspects? Check the work of Puerta et al (The Thin Red Line Between Usability and User Experiences)

Response 3：Thank you very much for your rigorous scientific research attitude and thoughtful reference of relevant research. Here's our explanation for why we only targeted "usability" and not "UX" in this study. First, as Puerta et al research demonstrates, we agree that "UX" encompasses more scope than "Usability" and that good usability does not necessarily mean a good user experience. But again, puerta et al quotes a point that may answer your question: "Currently different methods for evaluation of Usability and UX are used, but the subjectivity that holds the UX does not allow it to be assessed in the same way that usability is necessary to know how the user feels when interacting with the product .” In our opinion, good UX should be the ideal state that every product ultimately wants to achieve, but in the entire UX system (including emotional state, context, utility, gameplay, and accessibility and so on.), usability is relatively basic, objective and universally measurable, and the rest of the relevant factors need a more specific and targeted approach to judgment because of their strong subjectivity, which may also be when everyone talks about UI optimization always talk about how to improve UX, but when implementing it, you will find that more often focus on using mainstream scales to measure usability (such as: SUS, QUIS, USE, CSUQ, SUMI, PSSUQ, UMUX, PSIUS, etc.) to find usability issues. Second, as we just said, we agree that a good user experience is the ultimate goal of the product rather than usability, but we believe that when our product is still in the design stage that has not been officially used, we should first ensure that its usability is recognized by users (that is, the product is useful in a general sense, but this does not mean that users must have a good experience), and then when the product has passed the specific experience of enough target groups, then further improve the user experience based on feedback. So here we must thank you again for your important advice, because it inspires and reminds us of the important goal of the next stage of research (the judgment of user experience and its specific judgment method).

Point 4:  Improve conclusions

Response 4：Thank you again sincerely for your suggestion. In the revised manuscript, we have sorted out the conclusion section more systematically, and stated our research conclusions and contributions more clearly.

Point 5: Include further works

Response 5：Thank you for your valuable suggestions. In the revised article, we again carried out a detailed review of the introduction, literature review, and overall theoretical framework, adding 25 new and influential references to support our research. At the same time, the discussion and conclusion parts were sorted out in more detail, especially the horizontal comparative analysis of similar types of projects was introduced in the discussion part.

Reviewer 5 Report

Thank you for the opportunity to revise this paper. Overall, I think it is interesting and it addresses an important topic. I have, however, a few comments that I hope the authros find useful.

 

First, I am surprised by the lack of references to support the statemens in the introductions. The first few paragrpahs have none! Also, I would suggest not using expressions like “As we all know” and instead support what is said with relevant references.

 

I would also suggest to highlight in the introduction the specific research question the paper addresses, in the form of a question if possible.

 

Section 2.3. seems quite short, especially compared to the other sections in the literature review.

 

In terms of the methodology, I see you conduct regressions, but I see these regressions do not have a single control variable. Why not? Also, since you are running multiple regressions switching the dv and IV, why not using a technique such as simultaneous equation models?

 

 

I would call limitations, rather than deficiencies. Also, I think this section could be expanded more. I invite the authors to reflect more on this, but one for example could be related to the geographic scope of the stdy and how future research could investigate other countries/regions. 

Author Response

Point 1: First, I am surprised by the lack of references to support the statemens in the introductions. The first few paragrpahs have none! Also, I would suggest not using expressions like “As we all know” and instead support what is said with relevant references.

Response 1: Thank you for your valuable comments. We recognized the problem of under-citation and miswording, and in the revised manuscript, we introduced 25 references from the last 10 years (63% were the last 5 years) to supplement our undercitation. Then, in response to the problem of improper expression of the article, we also made targeted corrections. Thank you again for your careful and rigorous review.

Point 2:  I would also suggest to highlight in the introduction the specific research question the paper addresses, in the form of a question if possible.

Response 2: We are aware of your important suggestions and try to use the form of questions to highlight specific research questions where possible.

Point 3:  Section 2.3. seems quite short, especially compared to the other sections in the literature review.

Response 3: With your prompt, we reviewed the content of the article again and found that what was indeed discussed in section 2.3 was insufficient, and this issue has been revised in the new manuscript. Thanks again for the valuable reminder.

Point 4: In terms of the methodology, I see you conduct regressions, but I see these regressions do not have a single control variable. Why not? Also, since you are running multiple regressions switching the dv and IV, why not using a technique such as simultaneous equation models?

Response 4: Thanks again for your review. Regarding the issue of not setting control variables, this is because according to our review of previous studies, the four variables we studied in this paper (from the four dimensions of usability of the USE scale division) have not been found certain control variables to have clear theoretical basis, important and significantly related to them, so we did not specifically set control variables. Initially, we only planned to conduct multiple linear regression analysis with "Satisfaction" as the dependent variable and "Ease of use", "Ease of Learning" and "Usefulness" as independent variables, but found that all variables in the model the regression coefficients are not significant (p>0.05), so we speculate that there is an intermediary variable among the three independent variables, which affects the significance of other variables. Therefore, a multi-step regression method was carried out to test the mediating effect of the "Ease of Use" variable.

Point 5: I would call limitations, rather than deficiencies. Also, I think this section could be expanded more. I invite the authors to reflect more on this, but one for example could be related to the geographic scope of the study and how future research could investigate other countries/regions. 

Response 5: Thank you again for your valuable advice, especially for the expansion of the research reflection section, which fully inspired our thinking. In the subsequent manuscript, we once again reflected on the overall research, and more of the limitation content in the article was stated, including how to provide reference value for research in different regions, and also gave some possible solutions that we think will be improved, so as to improve this part in the next stage of research.

Round 2

Reviewer 2 Report

N/A

Author Response

Response: Thank you for your careful review and guide. The content of the two sections of "can be improved" has been marked in the newly uploaded manuscript. Appreciate!

Reviewer 3 Report

Please explain: Should you use categories (1-7), and the mean (central tendency)?

Maybe you can use  this references if you want:

Sáez-López, J. M., Vázquez-Cano, E.  & Sevillano-García, M. L. (2022). Scratch and visual block programming in mathematics: perceptions of preservice teachers in Columbia. Scientia Paedagogica Experimentalis, 59(1), 29-50. https://doi.org/10.57028/S59-029-Z1003

Saez-Lopez, J.M., Del Olmo, J., Gonzalez-Calero, J.A. & Cozar, R. (2020). Exploring the Effect of Training in Visual Block Programming for Preservice Teachers. Multimodal Technologies Interact. 2020, 4(3), 65; https://doi.org/10.3390/mti4030065

José Antonio Rodríguez-Martínez, José Antonio González-Calero & José Manuel Sáez-López (2020) Computational thinking and mathematics using Scratch: an experiment with sixth-grade students, Interactive Learning Environments, 28,3, 316-327, DOI: 10.1080/10494820.2019.1612448

 

 

Author Response

Point: Please explain: Should you use categories (1-7), and the mean (central tendency)?

Maybe you can use these references if you want:

……

Response:

Thank you again for your valuable advice and thoughtful reference of important relevant research.

First, the reasons for the use of regression 1-7 are as follows: As we wrote in the manuscript, after further refining the "Usability" metric into four dimensional variables, we initially planned to perform multiple linear regression (category 1) with "Satisfaction" as the dependent variable and "Ease of Use", "Ease of Learning" and "Usefulness" as the independent variables. However, according to the statistical results, the regression coefficients of the three independent variables in the regression model were not significant (p>0.05), and then considering the strong correlation between "Ease of Use" and "Usefulness"、"Ease of Learning" found in the correlation test before the regression analysis (Pearson=0.802, 0.632, p<0.05). We speculated that "Ease of Use" may present a mediation effect, acting as a mediating variable in this influence model. Therefore, we used a stepwise regression method to test our hypothesis（Mediation effect), which is why the regression analysis of Categories 1-8 is used.

Second, for the use of mean, we guessed you are referring to the use in Table 2 for descriptive statistics on the results of the scale. If not here, we would like you to be further specific about where the mean is used. The reason for the use of the mean in Table 2 is that we want to reflect the basic situation of each variable through the central tendency of each item. Of course, we know that if the data is not evenly distributed, extreme values will make the average value not reflect the overall situation, but, as we show in the article, our data is approximately normally distributed, so we use the mean to reflect the overall basic situation.

Finally, thank you very much for the three articles you recommended. We have read two of them carefully and found several things that were very helpful for our research, so we cited them and added them to the references. Unfortunately, for that article in 2022, we did not find the resources for the article, so we could not read it.

Overall, thank you again for your careful review to help us further refine this article, and hopefully our answers will answer your questions.

Reviewer 4 Report

Comments have been considered

Reviewer 5 Report

I have no more comments

Author Response

Response: Thank you again for your meticulous guidance and valuable suggestions during the review process.

Round 3

Reviewer 3 Report

Congratulations