Music to My Ears: Developing Kanji Stroke Knowledge through an Educational Music Game^†

Round 1

Reviewer 1 Report

Dear Authors,

Congratulations for your excellent manuscript. It has been a pleasure to review it. The introduction section is not only deep, but also includes graphic examples. I really liked the methodological section because it’s described in depth and allows replication by future researchers in the field. The fact that you could only have 44 participants in this study is not a major issue to me. One of the few things I miss in the manuscript is the fact that you did not include this as a limitation in your “limitations and future works” section. I am aware of the important restrictions we face as researchers when we conduct this type of studies in education, but I think you should have included it. This is important as your sample is not representative of a large population, so I would definitely include it in your manuscript. In this sense, future researchers may replicate your study in different contexts to re-test this methodology with other students. I also value the fact that you conducted a pretest and that no significant differences were found between groups. To me, this is the key to support all the significant differences you found in the follow-up and all the effect sizes you reported. One last thing. Something is wrong when I click the link in the data availability statement. It takes to a drive folder with a jpg archive. I highly recommend you to use OSF or other open data repositories in the future. It makes your research more valuable, transparent and open.

Congratulations again, you just have some minor things to change, you almost got it.

Author Response

Dear reviewer!

Thank you for your kind words.

• We're including a note about low/unrepresentative number of participants in the Limitations section of the updated manuscript.
• We will also make sure the data link works. 

Reviewer 2 Report

mti-1491842. Music to my ears: developing kanji stroke knowledge through an educational music game

In my opinion, the topic addressed in the article under review is interesting and adequately funded. Likewise, the methodological study is correct and well structured. However, I have some doubts and comments that I believe would help to increase the quality of the work.

Study 1:

Regarding the selection of the participants, (lines 254 and following), are they all foreign language or Japanese language learners?

Has the theoretical model of scales defined by the IEQ been verified to be valid in the sample explored (by some kind of confirmatory analysis)?

It would be useful to indicate some measure of the internal consistency of the IEQ scale on the sample studied (e.g., Cronbach's parameter), although the instrument is validated.

The authors use the t-test, although the sample studied is very small and it is not explained whether the answers have been found to be normally distributed. Can the authors explain why they have opted for parametric tests, instead of using nonparametric tests, for the comparison of means? The same suggestion can be made for subsections 4.2.2, 4.2.3 and 4.2.4.

In line 285, the authors explain that the results on the different scales do not differ between the control and experimental groups, but the t-test statistics show, in fact, that the differences observed are not statistically significant. This should be specified. In the same sense, the authors should specify the mean values of the scales cited for the two groups in the sample. It would also be useful to add the values of the t-statistics of the t-test, and not only the p-values of statistical significance.

It is noted that there are notable differences between the standard deviations indicated in subsection 4.2.4. Have the authors carried out any kind of comparison test of these deviations to explore whether they are due to chance or are they significant differences?

The study shows that there is a large variability in age among the participants. Have the authors explored whether age is a statistically discriminating variable for the results obtained?

Study 2:

Regarding study 2, the same comments can be made as in study 1 regarding the rationale for the choice of parametric tests for comparison of means, the need for additional explanation of the variation in standard deviations, and the inclusion of t-test t-statistics.

In the discussion, the authors should allude to the sample sizes and the great variability of ages that exist in them, in order to explain whether they intuit the existence of latent variables not considered in the study and discuss to what degree the results obtained are universalizable.

Author Response

Dear Reviewer! 

Thank you for your kind words and for the thorough review. We hope the following changes address all of your concerns.

Study 1:

Regarding the selection of the participants, (lines 254 and following), are they all foreign language or Japanese language learners?

None of the participants were language learning majors and all of them reported having no prior experience with Japanese or Chinese. 

We added additional wording to clarify this in section 4.1.2

Has the theoretical model of scales defined by the IEQ been verified to be valid in the sample explored (by some kind of confirmatory analysis)? It would be useful to indicate some measure of the internal consistency of the IEQ scale on the sample studied (e.g., Cronbach's parameter), although the instrument is validated.

The titular IEQ paper (Jennett2008) describes a detailed validation study with 244 participants (section 13). 

The Convergence of Player Experience Questionnaires (Denisova et. al. 2016) reports Cronbach’s α for the IEQ at 0.91 . https://dl.acm.org/doi/10.1145/2967934.2968095

We added this information to the 4.1.3 section of the paper. 

The authors use the t-test, although the sample studied is very small and it is not explained whether the answers have been found to be normally distributed. Can the authors explain why they have opted for parametric tests, instead of using nonparametric tests, for the comparison of means? The same suggestion can be made for subsections 4.2.2, 4.2.3 and 4.2.4.

We performed the Levene’s test for equality of variances and the test didn’t detect unequal variances for any of our outcome variables. Because of that we assumed that it’s safe to use the parametric tests for all the measurements in the experiment. 

We added a note about this at the beginning of the 4.2 section. 

In line 285, the authors explain that the results on the different scales do not differ between the control and experimental groups, but the t-test statistics show, in fact, that the differences observed are not statistically significant. This should be specified. In the same sense, the authors should specify the mean values of the scales cited for the two groups in the sample. It would also be useful to add the values of the t-statistics of the t-test, and not only the p-values of statistical significance.

We modified the wording to clarify that there were no significant differences. We added a table for the scales cited in the 4.2.1 section and added the t-statistics to the existing tables. 

We also added the mean values for scales cited in the second experiment, section 5.2.1

It is noted that there are notable differences between the standard deviations indicated in subsection 4.2.4. Have the authors carried out any kind of comparison test of these deviations to explore whether they are due to chance or are they significant differences?

We performed a Levene’s test for equality of variances. No unequal variances were detected for any of our outcome variables. Specifically for 4.2.4, the significance is 0.053 .

We added a note about the Levene test findings at the beginning of section 4.2

The study shows that there is a large variability in age among the participants. Have the authors explored whether age is a statistically discriminating variable for the results obtained?

Age didn’t appear to be a significant predictor for any of our dependent variables (all p’s > 0.09)

We added this information in section 4.1.2

We performed the same analysis for the second study, and all p’s were above 0.06. We added this information at the end of the 5.2.1 section.

Study 2:

Regarding study 2, the same comments can be made as in study 1 regarding the rationale for the choice of parametric tests for comparison of means, the need for additional explanation of the variation in standard deviations, and the inclusion of t-test t-statistics.

Generally, t-tests are considered robust for sample sizes of 20+, but we also conducted the Levene’s test to validate our choice of parametric tests and to account for the variation in standard deviations. We also added the t-statistics to all the tables in the article. 

In the discussion, the authors should allude to the sample sizes and the great variability of ages that exist in them, in order to explain whether they intuit the existence of latent variables not considered in the study and discuss to what degree the results obtained are universalizable.

We added a paragraph that discusses these points at the beginning of section 7.