Evaluating Scenario-Specific Loading Processes on Mobile Phones
Round 1
Reviewer 1 Report
The study presented in the manuscript focus on the user experience (UX) of loading processes on mobile phones, with a particular interest in loading time. The study is relevant due to the wide-spread use of mobile phones and it can also be advantageous from a UX perspective under some circumstances to concentrate on specific aspects. However, the paper lacks in several senses scientific rigor and quality.
- The presentation of the field and related work are superficial and a more comprehensive exposition is needed, either as an extensive introduction section or in a background section. For instance, user experience needs to be explained and problematized.
- The methodological choices needs to be more clearly explained and motivated. In particular:
o Section 2.1 is hard to follow. What was the purpose of this part? Why did you need this to identify scenarios? What is the relation to this and the scenarios you used? You mention 14 major categories and 58 secondary categories, what is the relation between this and the experiment and/or results?
o In section 2.4 you describe that you screened for addictions. Why? This is really odd, especially since you do not mentioned this as part of the purpose of the study, don’t present any references regarding the importance of having this focus, or that you don’t discuss this in the later part of the paper. The same goes for checking the sleeping quality of the subjects.
- The text in the discussion section is rather an analysis of the result. The discussion section usually consist of putting the findings in a wider context, discussing the probable causes behind the outcome by, e.g., relating the findings to relevant theories that can provide explanations, and consider the possible consequences of the results.
Author Response
Point 1: The study presented in the manuscript focus on the user experience (UX) of loading processes on mobile phones, with a particular interest in loading time. The study is relevant due to the wide-spread use of mobile phones and it can also be advantageous from a UX perspective under some circumstances to concentrate on specific aspects. However, the paper lacks in several senses scientific rigor and quality.
Response 1: From the perspective of user experience, this paper analyses the smoothness of mobile phone usage, and proposes an innovative evaluation method based on multi-dimension and machine interaction by combining objective (loading time recorded by manipulator) with subjective (satisfaction score). We also plan to carry out further research to improve this evaluation method in order to promoting user experience design of mobile phone manufacturers.
Point 2: The presentation of the field and related work are superficial and a more comprehensive exposition is needed, either as an extensive introduction section or in a background section. For instance, user experience needs to be explained and problematized.
Response 2: We have added user experience backgroud in introduction from line 28 to 38.
Point 3: The methodological choices needs to be more clearly explained and motivated. In particular:
o Section 2.1 is hard to follow. What was the purpose of this part? Why did you need this to identify scenarios? What is the relation to this and the scenarios you used? You mention 14 major categories and 58 secondary categories, what is the relation between this and the experiment and/or results?
Response 3: The purpose of Section 2.1 is to make clear how we screen the three scenarios in experiment, which was complained most from 14 major categories and 58 secondary categories by key words(slow, lagging, fluent, freezing, and does not work).
Point 4: In section 2.4 you describe that you screened for addictions. Why? This is really odd, especially since you do not mentioned this as part of the purpose of the study, don’t present any references regarding the importance of having this focus, or that you don’t discuss this in the later part of the paper. The same goes for checking the sleeping quality of the subjects.
Response 4: The screening parameter of addictions is to eliminate fine experimental deviations caused by addictions. We have added datails of the selection of the field study candidates from 132 to 141.
Point 5: The text in the discussion section is rather an analysis of the result. The discussion section usually consist of putting the findings in a wider context, discussing the probable causes behind the outcome by, e.g., relating the findings to relevant theories that can provide explanations, and consider the possible consequences of the results.
Response 5: We have updated discussion taking according to advices of reviewer.
Author Response File: 
Author Response.docx
Reviewer 2 Report
This manuscript highlights an interesting topic about the time perception toward the loading time in mobile apps. The experiment and the approach seems well designed and performed but, I’ll suggest few modifications in order to implement the readability of the text itself and in support of the scientific purpose of this research.
The modifications will be:
- A stronger literature, especially for the scenario, with the aim to strengthen the basis for the research question.
E.g. in the article “Yan, T., Chu, D., Ganesan, D., Kansal, A., & Liu, J. (2012, June). Fast app launching for mobile devices using predictive user context. In Proceedings of the 10th international conference on Mobile systems, applications, and services (pp. 113-126). ACM.” We can find very similar research intents that could apply to the authors’ manuscript too. Therefore, the sentence of being “the first study to investigate human perception in terms of loading time for critical scenarios using a realistic mobile” could be bendable and tricky, but if the authors wants to keep state it they’ll need to support it with a more structured literature review. In that case it will be advisable to add more references and a couple of paragraphs to the introduction/scenario part.
- The paper will benefit a short overview on the users’ journey during the settled experiment, to simply add a lean table or bullet point list of the events that the participants incur during the field study will significantly help to understand the timing tracking sessions.
-In the discussion section, from line 222 to 230, the adopted paradigm should be explained with more details and again, some references could help in strengthening the statements.
-I advice to proof read with a native English speaker or to ask a professional service to review the English and the terms used in the manuscript, e.g.: at the line 230 the term “temporal interval” can be confused in English with a concept part of mathematical logic reasoning very common in Algebra. This is just one example but please, do a more accurate language revision.
-In section 2, would be nice to explain more the parameters who brought up the selection of the field study candidates. It’s not necessary to write up whole new section, but couple of sentences to explain would do the work.
Author Response
Point 1: A stronger literature, especially for the scenario, with the aim to strengthen the basis for the research question.
E.g. in the article “Yan, T., Chu, D., Ganesan, D., Kansal, A., & Liu, J. (2012, June).Fast app launching for mobile devices using predictive user context.In Proceedings of the 10th international conference on Mobile systems, applications, and services (pp. 113-126).ACM.” We can find very similar research intents that could apply to the authors’ manuscript too. Therefore, the sentence of being “the first study to investigate human perception in terms of loading time for critical scenarios using a realistic mobile” could be bendable and tricky, but if the authors wants to keep state it they’ll need to support it with a more structured literature review. In that case it will be advisable to add more references and a couple of paragraphs to the introduction/scenario part.
Response 1: Thanks a lot for reviewer even listing the reference for us. We have changed the sentence of being “the first study to investigate human perception in terms of loading time for critical scenarios using a realistic mobile” and mentioned the above article in updated version.
Point 2: The paper will benefit a short overview on the users’ journey during the settled experiment, to simply add a lean table or bullet point list of the events that the participants incur during the field study will significantly help to understand the timing tracking sessions.
Response 2: We have updated figure 3 to give short overview on the users’ journey during the settled experiment so as to make expriment mote clear.
Point 3: In the discussion section, from line 222 to 230, the adopted paradigm should be explained with more details and again, some references could help in strengthening the statements.
Response 3: We have added details of each scenario, icluding initialization of the dialer; initialization of WeChatTM ; focalization of the camera.
Point 4: I advice to proof read with a native English speaker or to ask a professional service to review the English and the terms used in the manuscript, e.g.: at the line 230 the term “temporal interval” can be confused in English with a concept part of mathematical logic reasoning very common in Algebra. This is just one example but please, do a more accurate language revision.
Response 4: Thanks a lot for advice to English language. Frankly, we have adopted a Wiley editing service to review the English and the terms used in last manuscript. We review again in new version carefully.
Point 5: In section 2, would be nice to explain more the parameters who brought up the selection of the field study candidates. It’s not necessary to write up whole new section, but couple of sentences to explain would do the work.
Response 5: We have added datails of the selection of the field study candidates from 132 to 141.
Author Response File: Author Response.docx
Reviewer 3 Report
I wondered why the mobile phones were not identified, there would be interest on the part of consumers. Overall it appears to be a well-planned and executed project.
Author Response
Point 1: I wondered why the mobile phones were not identified, there would be interest on the part of consumers. Overall it appears to be a well-planned and executed project.
Response 1: The mobile phones selected in this study were on sale in the Chinese mobile phone market at that time. Ten representative mobile phones were selected according to the operating system (iOS, Andriod), price (high, medium and low), and brand factors. In order to avoid suspicion of advertising for manufacturers, this paper did not indicate the brand and type of mobile phones selected in the experiment.
Author Response File: Author Response.docx
Round 2
Reviewer 1 Report
The improvements you have made are relevant, but superficial and insufficient. The clarifications in section 2.4 are enough, although, the screening as such is questionable. For the other parts, the previous comments remains.
Author Response
Response to Reviewer 1 Comments
Point 1: The study presented in the manuscript focus on the user experience (UX) of loading processes on mobile phones, with a particular interest in loading time. The study is relevant due to the wide-spread use of mobile phones and it can also be advantageous from a UX perspective under some circumstances to concentrate on specific aspects. However, the paper lacks in several senses scientific rigor and quality.
Response 1: From the perspective of user experience, this is the first study to investigate human perception in terms of loading time for critical scenarios using a realistic mobile device and proposes an innovative evaluation method based on multi-dimension and machine interaction by combining objective (loading time recorded by manipulator) with subjective (satisfaction score). Though there are much work should do to enhance the results. At least, we thought the paper provide new thinking to promoting user experience design of mobile phone manufacturers.
Point 2: The presentation of the field and related work are superficial and a more comprehensive exposition is needed, either as an extensive introduction section or in a background section. For instance, user experience needs to be explained and problematized.
Response 2: We have added user experience backgroud and problem scope in introduction.
Point 3: The methodological choice needs to be more clearly explained and motivated. In particular:
o Section 2.1 is hard to follow. What was the purpose of this part? Why did you need this to identify scenarios? What is the relation to this and the scenarios you used? You mention 14 major categories and 58 secondary categories, what is the relation between this and the experiment and/or results?
Response 3: The purpose of Section 2.1 is to make clear how we screen the three scenarios in experiment, which was complained most from 14 major categories and 58 secondary categories by key words(slow, lagging, fluent, freezing, and does not work).
We collected 5399 complaints with above keywords from January 1, 2017 to January 1, 2018. Because complaints are subjective and narrative, there is no standard classification. Therefore, according to the function of mobile phone (communication, hardware, system, storage, voice, camera, etc.), we divide the complaints of users into 14 categories, and according to the specific problems of complaints, we divide 14 categories into 58 categories, for example:
Communication: cellular network registration; Wi-Fi connection; positioning; Bluetooth connection; etc.
Hardware: fingerprint identification; physical key; etc.
System: power on; power off: system update: system halts: slackening; etc.
The percentage of the total number of complaints (5399) was calculated, and three scenarios with more than 5% of the total number of complaints were selected as the application scenarios for our investigation.
Point 4: In section 2.4 you describe that you screened for addictions. Why? This is really odd, especially since you do not mentioned this as part of the purpose of the study, don’t present any references regarding the importance of having this focus, or that you don’t discuss this in the later part of the paper. The same goes for checking the sleeping quality of the subjects.
Response 4: The time perception is regulated by specific encephalic region, addictions definitely affect specific encephalic region, there is no evidence no overlap between the two parts. Thus we screen addictions to eliminate deviation.
Point 5: The text in the discussion section is rather an analysis of the result. The discussion section usually consist of putting the findings in a wider context, discussing the probable causes behind the outcome by, e.g., relating the findings to relevant theories that can provide explanations, and consider the possible consequences of the results.
Response 5: We have updated discussion taking according to advices of reviewer.
Author Response File: Author Response.docx
Round 3
Reviewer 1 Report
Dear authors,
I have read you additions in you manuscript, and actually many of my previous concerns remains. However, you have some interesting findings and the revisions you have made have improved the paper, and, thus, I recommend it to be accepted for publications.
