A Cross-Sectional Study Based on Forty Systematic Reviews of Foods with Function Claims (FFC) in Japan: Quality Assessment Using AMSTAR 2

Round 1

Reviewer 1 Report

Dear authors,

Thank you very much for your submission to Nutrients. I have thoroughly evaluated this paper, and although it is of some interest to local food research in Japan, nutrients is a journal with a worldwide scope, so the geographical nature of the paper will not attract the interest of a wider audience. From this perspective, I think this paper would be more appropriate for publication in a local Japanese journal. In addition, the research subjects selected for this paper were based on review articles, which were then analyzed, is this approach representative? Also these results are only suitable for review articles within the scope of the study, and usually, the contents in these review articles have been published for a longer period of time, so the results of the study may not be up-to-date. Moreover, I have been thinking about what is the innovation point of this article? What is the inspiration for me, unfortunately, after reading this work, this problem is still very confusing.

 

Author Response

Response to reviewer 1

I have thoroughly evaluated this paper, and although it is of some interest to local food research in Japan, nutrients is a journal with a worldwide scope, so the geographical nature of the paper will not attract the interest of a wider audience. From this perspective, I think this paper would be more appropriate for publication in a local Japanese journal. In addition, the research subjects selected for this paper were based on review articles, which were then analyzed, is this approach representative? Also these results are only suitable for review articles within the scope of the study, and usually, the contents in these review articles have been published for a longer period of time, so the results of the study may not be up-to-date. Moreover, I have been thinking about what is the innovation point of this article? What is the inspiration for me, unfortunately, after reading this work, this problem is still very confusing.

R: Thank you very much for the peer review and comments on our paper. We greatly appreciate your comprehensive consideration of the international position of Nutrients and its impact on readers. After serious consideration of your comments, we respond below.

As you know, systematic reviews (SRs) have exploded across all healthcare research fields, with more than 12,000 SRs published annually in academic journals. However, it has been pointed out that many SRs have problems with the quality of research [1]. Some researchers suggested that the skyrocketing increase in SR production globally has led to overlap, redundancies, and waste, as well as qualitative problems [2]. The need to pay attention to SRs is also important in the field of nutrition. For example, Zeraatkar et al. identified problems in nutritional review studies using a risk of bias (RoB) assessment tool called ROBIS [3].

We conducted a quality assessment of SRs using the first version of AMSTAR in 2019 [4], and a compliance assessment study of the protocol in clinical trials in 2022 [5]. Both were published in Nutrients, and subsequently have been cited by other studies [6-8].

The Codex Alimentarius Commission (CAC) declared that health claims should be supported by a sound and sufficient body of scientific evidence to substantiate the claim, provide truthful and nonmisleading information to aid consumers in choosing healthful diets, and be supported by specific consumer education [9]. Furthermore, it stipulated that health claims must be based on current scientific substantiation. The level of proof must be sufficient to substantiate the type of claimed effect and relationship to health as recognized by generally accepted scientific review of the data. Furthermore, it noted that scientific substantiation should be reviewed as new knowledge becomes available.

Therefore, we believe that our study will be a guide for future researchers on how to avoid possible problems with presenting evidence for health claims that may be implemented around the world. For these reasons, we are eager to be published in your journal, which is read by many researchers in the field of nutrition around the world. We hope that you understand our justifications.

 

P.S. We submitted this paper based on a recommendation to submit to your journal's special issue, "Databases, Nutrition and Public Health", when we registered the protocol.

References

1. Niforatos, J. D.; Weaver, M.; Johansen, M. E. Assessment of publication trends of systematic reviews and randomized clinical trials, 1995 to 2017. JAMA Intern. Med. 2019, 179, 1593-1594.
2. Siontis, K. C.; Ioannidis, J. P. A. Replication, duplication, and waste in a quarter million systematic reviews and meta-analyses. Circ. Cardiovasc. Quality Outcomes 2018, 11, e005212.
3. Zeraatkar, D.; Bhasin, A.; Morassut, R. E.; Churchill, I.; Gupta, A.; Lawson, D. O.; Miroshnychenko, A.; Sirotich, E.; Aryal, K.; Mikhail, D.; Khan, T. A.; Ha, V.; Sievenpiper, J. L.; Hanna, S. E.; Beyene, J.; de Souza, R. J. Characteristics and quality of systematic reviews and meta-analyses of observational nutritional epidemiology: a cross-sectional study. Am. J. Clin. Nutr. 2021, 113, 1578-1592.
4. Kamioka, H.; Tsutani, K.; Origasa, H.; Yoshizaki, T.; Kitayuguchi, J.; Shimada, M.; Wada, Y.; Takano-Ohmuro, H. Quality of systematic reviews of the Foods with Function Claims in Japan: Comparative before- and after-evaluation of verification reports by the Consumer Affairs Agency. Nutrients 2019, 11, 1583.
5. Kamioka, H.; Origasa, H.; Kitayuguchi, J.; Tsutani, K. Compliance of clinical trial protocols for Foods with Function Claims (FFC) in Japan: consistency between clinical trial registrations and published reports. Nutrients 2022, 14, 81.
6. Karaki, T.; Haniu, H.; Matsuda, Y.; Tsukahara T. Lysophospholipids: A potential drug candidate for neurodegenerative disorders. Biomedicines 2022, 10(12), 3126. doi: 10.3390/biomedicines10123126.
7. Sato, K’; Kodama, K.; Sengoku, S. Optimizing the relationship between regulation and innovation in dietary supplements: a case study of Food with Function Claims in Japan. Nutrients 2023, 15(2), 476. doi: 10.3390/nu15020476.
8. Kamioka, H.; Origasa, H.; Kitayuguchi, J.; Yoshizaki, T.; Shimada, M.; Wada, Y.; Takano-Ohmuro, H.; Tsutani, K. Risk of bias in clinical trials reported for Foods with Functional Claims in Japan: a cross-sectional study on re-search quality. J. Clin. Trials 2022, 12, 1000503.
9. The CODEX Alimentarius Committee. Guidelines for use of nutrition and health claims. https://www.fao.org/ag/humannutrition/32444-09f5545b8abe9a0c3baf01a4502ac36e4.pdf [accessed March 28, 2023].

 

Author Response File: Author Response.docx

Reviewer 2 Report

Introduction. The authors must properly introduce AMSTAR in its first mention, as stated in Ref. [15].

Why do the authors use a range period from April to October 2022 when they could use the entire 2022 year? This is unclear and must be adequately justified. In addition, if there are any other studies from the other months, this should also be mentioned and justified.

Figure 1 is too low quality and amateur. It must be enhanced for quality more scientifically and professionally. I recommend using Lucidchart or Diagrams online tools to present your flowcharts better.

Latin and other scientific names must be presented in italics.

Figure 2 could be enhanced to fit the page with the following adjustments: show the Y-axis title vertically; include the significance of ratings inside the square.

Figures 3 and 4 are also poor quality, and some data are almost unreadable. They must be enhanced. Some graph software allows copy and paste directly into the Word document with higher quality (meta-file).

Author Response

Response to reviewer 2

Thank you for your important and detailed suggestions. We have revised the manuscript accordingly. We appreciate the opportunity to further improve our manuscript.

 

1. The authors must properly introduce AMSTAR in its first mention, as stated in Ref. [15].

R: Thank you very much for your important point. We have added the following language regarding introduction of the first edition of AMSTAR. “A measurement tool for the 'assessment of multiple systematic reviews' (AMSTAR) was developed to assess the methodological quality of SRs, building upon previous tools, empirical evidence, and expert consensus.”

 

2. Why do the authors use a range period from April to October 2022 when they could use the entire 2022 year? This is unclear and must be adequately justified. In addition, if there are any other studies from the other months, this should also be mentioned and justified.

R: Thank you very much for your important point. We have added the following explanation. "In Japan, it is normal for both the government and companies to develop their businesses on an annual basis, so this period was set to collect the latest SRs for FY2022. Therefore, it was adopted from the first in the H series. This is the H series because FY2015, the year of introduction, was the A series, and each subsequent year was identified in alphabetical order, thus, FY2022 was H, or the eighth year.”

 

3. Figure 1 is too low quality and amateur. It must be enhanced for quality more scientifically and professionally. I recommend using Lucidchart or Diagrams online tools to present your flowcharts better.

R: Thank you for pointing this out. As you advised, we have made some modifications.

 

4. Latin and other scientific names must be presented in italics.

R: Following your point, we italicized the relevant parts of Table 1 and the text.

 

5. Figure 2 could be enharnced to fit the page with the following adjustments: show the Y-axis title vertically; include the significance of ratings inside the square.

R: Thank you for pointing out this figure adjustment for ease of viewing. As you advised, we have made some modifications.

 

6. Figures 3 and 4 are also poor quality, and some data are almost unreadable. They must be enhanced. Some graph software allows copy and paste directly into the Word document with higher quality (meta-file).

R: In accordance with your suggestions, we have improved the quality of Figures 3 and 4.

 

Author Response File: Author Response.docx

Reviewer 3 Report

This is a nicely conducted study to reveal the poor methodological quality of SRs being reported to CAA. That is a critical issue because it implies a huge amount of food products can be commercialized without solid research evidence. To better address this issue, there are several comments that authors may consider.

First, authors can list out the outcomes as being examined in the SRs, e.g. cardiovascular risk factors, inflammatory markers, clinical disease outcomes, etc, as well as number of studies being included in each SR. That will help readers to understand if certain area of research may have flawed research evidence, and the scale of this problem.

Second, can authors elaborate how SR authors have done poorly on using a comprehensive literature search strategy? That is a very serious issue because SRs are supposed to have done a comprehensive search. Although SR authors may not do so intentionally, they may have misunderstood some of the technical details. Good to cite the relevant information from the user guide from AMSTAR-2 to clarify how things should be done.

Third, as a stronger support of the research findings, can authors the ROBIS tool as evaluate the included SRs too? When compared to AMSTAR, it is more comprehensive in its assessment of risk of bias, while AMSTAR includes several items that address the construct of reporting quality rather than risk of bias. Authors may refer to the AJCN paper by Zeraatkar et. al. to see how the ROBIS tool can be applied to evaluate the SRs in nutrition field.

Lastly, supplementary information cannot be assessed from the cloud drive. Please upload them to the Nutrients system as zip file.

Author Response

Response to reviewer 3

Thank you for your important and detailed suggestions. We have revised the manuscript accordingly. We appreciate the opportunity to further improve our manuscript.

 

1. First, authors can list out the outcomes as being examined in the SRs, e.g. cardiovascular risk factors, inflammatory markers, clinical disease outcomes, etc, as well as number of studies being included in each SR. That will help readers to understand if certain area of research may have flawed research evidence, and the scale of this problem.

R: In Results Section 3.1, “Study selection and characteristics”, we summarized the types of outcomes as follows: “For each SR, the outcomes were as follows: improvement of bowel movements (n=7), inhibition of rise in triglyceride after meals (n=5), reduction of body fat (n=5), suppression of postprandial increase in blood glucose (n=3), relief of mental stress (n=3), preservation of skin elasticity (n=3), improvement of cognitive function (n=2), decrease in blood pressure (n=2), reduction of fatigue (n=2), improvement of decreased peripheral body temperature (n=2), improvement of contrast sensitivity of the eyes (n=2), improvement of swelling (n=1), improved vascular function (n=1), improved knee joint function (n=1), and improved immune function (n=1).”

 

2. Second, can authors elaborate how SR authors have done poorly on using a comprehensive literature search strategy? That is a very serious issue because SRs are supposed to have done a comprehensive search. Although SR authors may not do so intentionally, they may have misunderstood some of the technical details. Good to cite the relevant information from the user guide from AMSTAR-2 to clarify how things should be done.

R: Thank you very much for your important point. We described several problems with previous literature search strategies in the Results section 3.2.1, “Critical domains”, “Sufficiency in the literature search strategy for critical domains include i) a small number of databases in English and only MEDLINE/PubMed and one other database (DB), ii) no description of how language restrictions were made, and iii) a literature search using only the above DB, and no other search engines, handsearch, citation search, referral, etc.”

 

3. Third, as a stronger support of the research findings, can authors the ROBIS tool as evaluate the included SRs too? When compared to AMSTAR, it is more comprehensive in its assessment of risk of bias, while AMSTAR includes several items that address the construct of reporting quality rather than risk of bias. Authors may refer to the AJCN paper by Zeraatkar et. al. to see how the ROBIS tool can be applied to evaluate the SRs in nutrition field.

R: Thank you for pointing this out. We did not plan to use the ROBIS tool in the protocol. We have added the following language to the section that describes limitations of our research. “This study did not use the ROBIS, which can assess the RoB in SRs with higher sensitivity, and we believe that it can be used in combination with this to better clarify the overall improvement points of SRs. Zeraatkar et al. [44] evaluated 140 reviews in the field of nutrition using the ROBIS and reported the effectiveness of their RoB assessments.”

 

44. Zeraatkar, D.; Bhasin, A.; Morassut, R. E.; Churchill, I.; Gupta, A.; Lawson, D. O.; Miroshnychenko, A.; Sirotich, E.; Aryal, K.; Mikhail, D.; Khan, T. A.; Ha, V.; Sievenpiper, J. L.; Hanna, S. E.; Beyene, J.; de Souza, R. J. Characteristics and quality of systematic reviews and meta-analyses of observational nutritional epidemiology: a cross-sectional study. Am. J. Clin. Nutr. 2021, 113, 1578-1592.

 

4. Lastly, supplementary information cannot be assessed from the cloud drive. Please upload them to the Nutrients system as zip file.

R: We uploaded supplementary information to the system as a zip file.

 

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

The author has given a good response to my concern, it seems to be accepted now

Author Response

Response paper to comment of respective reviewer

 

Manuscript ID nutrients-2316426

 

Original Title： A Cross-Sectional Study based on Forty Systematic Reviews of Foods with Function Claims (FFC) in Japan: Quality Assessment using AMSTAR 2

Authors： Hiroharu KAMIOKA, et al.

 

Response to reviewer 1

Thank you very much for the peer review again. We greatly appreciate your comprehensive consideration.

 

Reviewer 2 Report

I still have some suggestions to improve the quality of the manuscript. 

1) Reduce the image sizes (they are too gigantic);

2) Include more details on the captions of tables and figures, as the ones provided are too general. Those itens must be self-explaining for the readers to understand the content without looking for details on other parts of the manuscript.  

Author Response

Response paper to comment of respective reviewer

 

Manuscript ID nutrients-2316426

 

Original Title： A Cross-Sectional Study based on Forty Systematic Reviews of Foods with Function Claims (FFC) in Japan: Quality Assessment using AMSTAR 2

Authors： Hiroharu KAMIOKA, et al.

 

Response to reviewer 2

I still have some suggestions to improve the quality of the manuscript.

 

R: Thank you for your detailed suggestions. We have revised the manuscript accordingly. We appreciate the opportunity to further improve our manuscript.

 

1) Reduce the image sizes (they are too gigantic);

R: We have downsized all Figures and Tables by 10-20% except Fig.3. We are thinking of asking the MDPI’ officers for the final layout.

 

2) Include more details on the captions of tables and figures, as the ones provided are too general. Those itens must be self-explaining for the readers to understand the content without looking for details on other parts of the manuscript. 

R: We have added each brief explanation on all Figures and Tables except Table 2.

Author Response File: Author Response.docx

Reviewer 3 Report

Authors have addressed my concern adequately, I have no further comments.

Author Response

Response paper to comment of respective reviewer

 

Manuscript ID nutrients-2316426

 

Original Title： A Cross-Sectional Study based on Forty Systematic Reviews of Foods with Function Claims (FFC) in Japan: Quality Assessment using AMSTAR 2

Authors： Hiroharu KAMIOKA, et al.

 

Response to reviewer 3

Thank you for your important suggestions.