High Quality, Equity, and Assessment: An Analysis of Variables Impacting English Learner Standardized Science Test Performance and Implications for Construct Validity
Round 1
Reviewer 1 Report
Abstract section: it will be good if authors explains research method section in abstract section. Who is the target? How is the data collected? How is the data processed? This becomes important information in the abstract. do not use abbreviations in the abstract. What is SES and what is ELP? Main section: There is many abbreviation in the main section, this make the readers difficult to understand Method section: how the authors take research data? Not clear the author also does not provide demographic data. What software do you use to process the data in this study? Not clear
Author Response
A file is attached.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
This research contributes to the field in the area of studying the impact of productive language on learning, especially for English language learners. This is important to consider.
The results do not necessarily lead to the conclusion that the standardized tests are unfair or that they do not test knowledge they purport to test. Vocabulary is an important part of science knowledge; otherwise students cannot be successful in a job in the U.S. where they need to communicate their work with the correct terminology.
Although the results point out this very important issue, the paper should acknowledge that there may be more than one viewpoint of what could be done to resolve the problem. For example, part of the answer may or may not be to remove the vocabulary complexity on the test. It may be that more resources need to be allocated to working with low SES and ELL students for productive language, especially in relation to STEM areas that have specialty vocabulary.
Author Response
A file is attached.
Author Response File: Author Response.pdf
Reviewer 3 Report
I see a lot of merit in a study to bring to attention that measuring English language learners' science content knowledge is not a level playing field for those that have not obtained adequate proficiency in the English language. However, I sort of feel that in the paper it comes off like this is a new idea. I think this is understood by English language researchers that are familiar with teaching learners in English as a second language contexts. I would expected that the authors would have reported some of these previous studies to show how this study builds on this existing knowledge. I, of course see the contribution to the local context of Colorado and also for science content knowledge specifically, but I just expected a more robust review of the literature. I am not expecting something exhaustive but I do think more should have been included to help set the stage and not give the impression that this is a one of a kind study. This study is actually build on a lot of existing knowledge in this area. This is a good study that should be published; however, I have several concerns that I hope the authors will consider when going about the revisions.
Please write out in full first mention of these acronyms in the abstract: ELP, SES
Please provide a citation for the term "English learner".
Please provide a citation to backup the description of productive and receptive language. Actually, you should also consider whether you want to consider using productive and receptive language skills. I say this as the four domains you mentioned are usually referred to as language skills in the English language teaching literature. I will leave this up to the decision of the authors though. It just may come across to those in the field of English language teaching and education to see receptive and productive language without the word skills attached.
ACCESS provides data on general academic language, oral, literacy, comprehension, and overall. Earlier on you indicated in your paper you found evidence for productive language scores being one of the significant predictors in comparison to receptive language scores. Looking at what ACCESS provides data on, I could see you might use the data from the "comprehension" section as comprehension refers to listening and reading. However, there is no "productive" only portion of the ACCESS as you have described it. oral includes both productive and receptive knowledge and literacy includes both productive and receptive knowledge. This will need to be explained and clarified in the paper. Later on I also see you show that you used ELP & R&P knowledge. So then it seems you combined receptive and productive knowledge, so that makes your earlier claims in the abstract confusing to me.
What is the point of the subgroups of ELs? As it is mentioned first in the participants section and not dealt with earlier on in the literature on language proficiency issues, I will feel this is a bit startling here. If this is somehow going to be incorporated in your analyses, then I would have expected it to be dealt with earlier on in the paper. As a reader, this would make it want to see whether this learner variable should be considered.
Non US readers may not understand how SES can be operationalised as free and reduced lunch. Not sure whether you should add a footnote about this.
I think you had better justify why you enter home language as either Spanish or other.
I don't understand your rationale for only including receptive and productive of ACCESS. reading and listening + writing and speaking--not really sure how you did that as your description of ACCESS does not indicate that you can get this data. Why add the overall English language proficiency as measured by ACCESS only to also add some of the subcomponents of this test? Then later on when I kept reading I found a multicollinearity issue (as I guessed would happen). I have to say as you are referring the constructs in one way and then referring to parts of the ACCESS test in another way makes it very confusing to me as a reader. I understand ACCESS is an assessment with it's own labels for things and you also have your own labels as well but it's best to report these clearly when giving the details of how you ran the models. A from ACCESS was considered as X or some similar structure would make it easier for your readers to understand what you did. However, the more concerning issue here is that ACCESS does not seem to split the data in the way you are reporting that it was entered into your models. So it is concerning. You do give some way you went about trying to circumvent this but I see this is a major limitation as you are making some assumptions that their performance is going to be in this way but we have no way to guarantee this. You need to provide more justification. It makes more sense to just have a productive measure used to represent productive knowledge and a receptive measure used to represent receptive knowledge. Please help to convince me. I see this as a limitation and should be clearly mentioned.
After reading the discussion, I feel left wanting for something more. What makes your results about science learning any different than other content language topics?
The issue of productive and receptive English knowledge and its relation to science content knowledge assessment could be strengthened. To understand this finding we also need to understand what learners need to do on the science assessment. How is English used on the assessment? For example, do they need to produce spoken and written English to answer questions? Or do they read questions and then answer multiple choice questions. These require very different types of language skills. If I have a large receptive vocabulary in English and can understand spoken and written English, answering multiple choice questions might be much easier for me to deal with than if say I'm asked to write out short answer questions for problems that are presented to me. There is a lot of research about this in relation to math learning. A lot of research out there that shows the biggest predictor for math scores sometimes is English scores--because the math books are in English (some studies like this have been produced in non-English speaking countries where English textbooks are used as they offered the most up-to-date content). I would expect some deeper insights on the explanation of your results related to science knowledge assessment and what language knowledge/use was required by the learners when completing these assessments.
Author Response
A file is attached.
Author Response File: Author Response.pdf
Round 2
Reviewer 3 Report
I have now been able to read the revised paper and compare it to my original comments and can say that the authors have addressed the comments and the paper can be recommended to be accepted.
