Measuring Efficiency and Accuracy in Locating Symbols on Mobile Maps Using Eye Tracking

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Just a few lines, since the deadline was next week, however, MDPI is forcing me to do it now - or they don't need my review anymore, but I spent some time already...

 

Major - abstract, line 9 - smartphone USAGE frequency - usage is missing, and I thought it is about  frequency of display or something like this...

 

line 30 - "like traditional ones" - what does it mean?

lines 55-60 - hard to understand

line 64 - how you will reveal the potential cognitive mechanism?

RQ3 - is there any theory behind this?

line 114 - mapy.cz are not based on opensource - or are, but outside of Czechia - in their home country, they have their own data

line 115 - website or app?

figure 2 - I do not think that it might work with this setup - participant is looking to the screen, not to the phone, eye-tracker is placed too close to him

Q1-Q4 - are these questions based on the real use of mobile maps?

229 - number of fixations in aoi does not allow you to determine how maoy times respondent RETURNED

you were analysing pupil size, however, according to figure 2, there was a window in the room - so the lighting conditions were not unified and affected the pupil size

Results - I will expect that the structure will be according to RQs

Lines 303-308 / lines 355-363 - how you defined these questions? why it is important that longer saccades are in video 4? it seems to me that you tested everything and presented only results where p<0.05

 

Author Response

Reviewer 1

Just a few lines, since the deadline was next week, however, MDPI is forcing me to do it now - or they don't need my review anymore, but I spent some time already...

Major - abstract, line 9 - smartphone USAGE frequency - usage is missing, and I thought it is about  frequency of display or something like this...

Thank you for pointing out this oversight. We corrected it.

line 30 - "like traditional ones" - what does it mean?

We meant paper maps. We corrected it. 

line 64 - how you will reveal the potential cognitive mechanism?

Indeed, we agree with your thought that revealing the cognitive mechanism is crucial information and requires an explanation. Therefore, we added an explanation after presenting the main goal of the study:

Especially, we would like to reveal the potential cognitive mechanism or at least expose elements of it, such as characteristic eye movement features. This could be possible thanks to the application of eye tracking technology, which allows precisely monitoring and analyzing users' eye movements and gaze patterns, helping to understand gaze duration, saccade patterns, or pupil dilation.

RQ3 - is there any theory behind this?

Thank you for this question. Following it, we added a paragraph to the section "Related Works" which underscores the pupil size differences as a crucial factor in the cognitive workload during various visual tasks such as visual search or route planning:

Building upon Skaramagksa et al.'s (2021) findings, it is essential to underscore that an enlarged pupil diameter could signify heightened engagement in cognitive or emo-tional processes. This observation aligns with established literature suggesting that varia-tions in pupil size are reflective of the intensity of mental and emotional activities (Foroughi et al. 2017). In the specific context of map-reading tasks, where cognitive de-mands fluctuate based on factors such as task complexity, spatial information processing, and user engagement, monitoring pupil dilation becomes a valuable metric for gauging the cognitive load or emotional involvement of individuals (Kiefer et al. 2016).

Additionally, differences in eye movement metrics between novice and experts, as well as male and female participants, were studied by Keskin, Merve, Kristien Ooms, Ahmet Ozgur Dogru, and Philippe De Maeyer. 2019. "EEG & Eye Tracking User Experiments for Spatial Memory Task on Maps" ISPRS International Journal of Geo-Information 8, no. 12: 546. https://doi.org/10.3390/ijgi8120546

line 114 - mapy.cz are not based on opensource - or are, but outside of Czechia - in their home country, they have their own data

Thank you for pointing out this error. Mapy.cz actually uses its own data. OpenStreetMap is mainly used outside the Czech Republic and Slovakia as a supplement. We have added information about this in the manuscript.

line 115 - website or app?

App

figure 2 - I do not think that it might work with this setup - participant is looking to the screen, not to the phone, eye-tracker is placed too close to him

We agree with the comments. The photo could be misleading, so we have removed it from the article. It was taken after the test to show the equipment used, not under real test conditions.

Q1-Q4 - are these questions based on the real use of mobile maps?

On the real use of mobile maps.

229 - number of fixations in aoi does not allow you to determine how maoy times respondent RETURNED

Of course this is our mistake. We corrected this word.

you were analysing pupil size, however, according to figure 2, there was a window in the room - so the lighting conditions were not unified and affected the pupil size

As we have already mentioned, the photo was for display purposes only. During the study, the light was the same for all participants (artificial), and the blinds were always closed.

Results - I will expect that the structure will be according to RQs

We chose the second option for splitting the results, i.e. for the questions from the last part of the study related to the RQs. We believe that this is not a major complication.

Lines 303-308 / lines 355-363 - how you defined these questions? why it is important that longer saccades are in video 4? it seems to me that you tested everything and presented only results where p<0.05

The principle of scientific research is to draw conclusions only from statistically significant results, which is why we only consider p < 0.05.

Reviewer 2 Report

Comments and Suggestions for Authors

 

The paper presents comprehensive investigation into the efficiency and accuracy of symbol location on mobile maps, with a focus on the Mapy.cz application and the utilization of eye-tracking technology. The study delves into the impact of smartphone frequency on users' ability to locate symbols within diverse spatial contexts, emphasizing the role of everyday smartphone use in enhancing mapping task efficiency.

General Comments:

• The subject is suitable for publication in this journal. The title accurately reflects the content, and the number of keywords is sufficient.
• The abstract is clear and concise, effectively summarizing the study's key elements. The research question, methodology, and major findings are well-presented. The study's purpose is well-defined.

Introduction:

• The introduction effectively underscores the significance of mobile mapping applications, addressing common challenges, such as difficulties in reading map symbology on smaller screens. However, it would be beneficial for the authors to explicitly state the link between these challenges and the research questions, as well as articulate the expected contribution of the study to the broader understanding of mobile map usability and its implications for design.

Methodology:

• The justification for choosing Mapy.cz is appropriate, and the detailed description enhances the reader's understanding of the application. However, providing more details on the selection criteria for the Old Town of Toruń as the study area would strengthen the rationale for this choice.
• Consider providing more information on how the acceleration of recordings and removal of the interface might impact the interpretation of eye-tracking data.
• Offering a brief explanation of why the study did not require ethics committee approval would enhance transparency.

Results:

The figures effectively complement the textual results, providing a visual representation of statistically significant differences. The division of results based on questions and the clear presentation of correctness and response time enhance comprehensibility.

• Consider discussing the practical implications of the observed correlations in the results section. How do these correlations contribute to understanding user behavior and interaction with mobile maps?

Discussion:

The Discussion section provides a nuanced interpretation of the findings, connecting them to existing theories and literature. Clear explanations are provided for observed patterns, such as the impact of smartphone usage frequency on symbol detection.

• Consider elaborating on potential reasons behind the observed differences in pupil size between men and women as a potential avenue for future investigations.

Conclusion:

• The Conclusion effectively summarizes the findings but could benefit from reiterating the main contributions of the study and how they advance existing knowledge.
• Mentioning potential limitations, such as the sample size or specific characteristics of the chosen map application, would provide a more balanced perspective on the research.

 

 

Comments on the Quality of English Language

The overall quality of English in the paper is quite good. The language is clear, and the text is well-structured. There are a few minor suggestions for improvement to enhance clarity and readability:

1. • Line 217: consider changing "Eye tracking data retrieved from the Gazepoint GP3 HD eye tracker is a time series" to "Eye-tracking data retrieved from the Gazepoint GP3 HD eye tracker consist of a time series."
   • Line 446: consider rephrasing "smaller screens of mobile devices are recognized as a design consideration" to "the design consideration of smaller screens on mobile devices."
2. • Line 465: "the presented study found an intriguing observation of gender differences" might be clearer as "the study observed intriguing gender differences."
3. •  
4. •  

Author Response

Reviewer 2

The paper presents comprehensive investigation into the efficiency and accuracy of symbol location on mobile maps, with a focus on the Mapy.cz application and the utilization of eye-tracking technology. The study delves into the impact of smartphone frequency on users' ability to locate symbols within diverse spatial contexts, emphasizing the role of everyday smartphone use in enhancing mapping task efficiency.

General Comments:

The subject is suitable for publication in this journal. The title accurately reflects the content, and the number of keywords is sufficient.

The abstract is clear and concise, effectively summarizing the study's key elements. The research question, methodology, and major findings are well-presented. The study's purpose is well-defined.

Introduction:

The introduction effectively underscores the significance of mobile mapping applications, addressing common challenges, such as difficulties in reading map symbology on smaller screens. However, it would be beneficial for the authors to explicitly state the link between these challenges and the research questions, as well as articulate the expected contribution of the study to the broader understanding of mobile map usability and its implications for design.

Thank you for these valuable insights. We have added a description of how the challenges relate to the research questions and the expected contribution to understanding the usability of applications and their design implications.

Mobile maps are confronted with many problems, e.g. difficulties in reading the map symbology correctly. For tourist maps, it is important to make them attractive to the user by finding the original graphic design [2]. Navigation features such as routes and point symbols should be higher up in the visual hierarchy (van Tonder & Wesson 2009). However, poorly designed symbols can be difficult for users to interpret, especially considering that such maps usually do not include a legend (Robinson et al. 2013). Inadequately selected graphics can also overwhelm the user, making the map difficult to use. The problem is exacerbated when you consider the smaller screens of smartphones and tablets. When designing the user experience (UX), it is important to take these limitations into account and design applications for mobile devices first, and only then for larger screens [3]. It is possible that the frequency of smartphone use has an impact on the ability to find symbols on mobile maps. There is therefore a need to investigate this effect and determine whether more experienced users are better able to perform map-related tasks. Analysing eye movements can be helpful in evaluating this impact, as it can be used to examine aspects such as the speed of scanning and finding symbols on the map, as well as the degree of cognitive load. This study will allow us to learn more about the differences in the way people with different smartphone experience use the map. This will allow you to design applications that are more user-friendly and take users' habits into account.

Methodology:

The justification for choosing Mapy.cz is appropriate, and the detailed description enhances the reader's understanding of the application. However, providing more details on the selection criteria for the Old Town of Toruń as the study area would strengthen the rationale for this choice. (WR)

Consider providing more information on how the acceleration of recordings and removal of the interface might impact the interpretation of eye-tracking data.

We have added information about the criteria for selecting the Old Town of Toruń as a research area and the effects of the acceleration of the recordings and removing the application interface on the interpretation of the eye-tracking data:

Five one-minute videos recorded on a smartphone while using the app were used for the study. Each video consists of three fragments, each showing a different section of the route covered on foot (Figure 1). A part of the Old Town of Toruń was chosen as the study area. The factor that influenced the choice of this area is that this area is highly urbanized and rich in various objects represented with cartographic symbols. In a less urbanized area with a lower density of objects, the user would have fewer symbols to consider. The difficulty of the task would therefore be significantly lower. As a result, the process of scanning by sight would take less time and the subject would have more time to look at the symbols more often and for longer. This could blur the differences in the users' visual strategy. Furthermore, this type of application is used much more frequently for navigation in urban areas. The routes of the march were chosen so that different types of symbols were presented in different spatial situations. Some of the symbols were located on the street, others in the polygons marking the outlines of the buildings. The symbols varied in color and shape. During later processing, the recordings were accelerated and deprived of an interface. Increasing the speed should improve the dynamics and smoothness of the recordings so that the user has the impression of being in motion. The degree of acceleration was chosen so that the movement appears natural. The app's interface has been removed so as not to distract the user from the cartographic content. In this way, we can be sure that the people being examined do not focus their eyes on elements that are not the subject of the examination.

Offering a brief explanation of why the study did not require ethics committee approval would enhance transparency.

We have clarified this statement:

The study complied with the applicable regulations of the Ethics Committee of the Adam Mickiewicz University (non-invasive - observation with equipment) and did not require additional consent.

Results:

The figures effectively complement the textual results, providing a visual representation of statistically significant differences. The division of results based on questions and the clear presentation of correctness and response time enhance comprehensibility.

Consider discussing the practical implications of the observed correlations in the results section. How do these correlations contribute to understanding user behavior and interaction with mobile maps?

We included these correlations and their implications in our conclusions to answer the first research question.

Discussion:

The Discussion section provides a nuanced interpretation of the findings, connecting them to existing theories and literature. Clear explanations are provided for observed patterns, such as the impact of smartphone usage frequency on symbol detection.

Consider elaborating on potential reasons behind the observed differences in pupil size between men and women as a potential avenue for future investigations.

Thank you for this suggestion. Following it, we have elaborated on potential reasons behind the differences in pupil size between sexes:

For future investigations, a user-centered design approach can be integrated into the study design, especially when considering maps for mobile devices (Bartling et al. 2022; Dillemuth, 2005; Nivala & Sarjakoski, 2007). Due to the fact that the size of the pupil is determined by physiological as well as emotional factors (Skaramagksa's et al. 2021), there are potential research paths using EEG (electroencephalography) that could integrate biometric user factors in the context of changing cartographic content (Keskin & Ooms 2018).

Conclusion:

The Conclusion effectively summarizes the findings but could benefit from reiterating the main contributions of the study and how they advance existing knowledge. (PC)

We appreciate your perspective, and consequently, we have revised the conclusions to reinforce the primary study questions and illustrate how they contribute to the advancement of existing knowledge:
The research delves into specific findings related to users' performance in map-related tasks and addresses posed research questions. Firstly, it acknowledges the impact of habitual smartphone usage on users' efficiency and accuracy in locating symbols within diverse spatial contexts on mobile maps. The study reveals that frequent smartphone use significantly enhances users' performance in these tasks, suggesting a link between usage habits and improved cognitive processing during map-related activities.

Secondly, the research explores the correlation between average daily smartphone usage time and task outcomes, specifically in terms of scanning speed and symbol detection. The findings demonstrate a moderate correlation, indicating that increased smartphone usage is associated with improved performance in map-related tasks.

Finally, the study investigates gender differences in pupil size during map-related tasks and explores how these differences relate to variations in cognitive workload. The consistent observation of larger pupil diameters in women suggests a higher cognitive workload for female participants during map-related activities, providing insights into the potential cognitive demands for different genders.

The research significantly advances our understanding of map-reading tasks by providing valuable insights into how smartphone usage frequency influences users' efficiency and accuracy. This information is crucial for designing user-friendly map applications that consider users' habits and enhance navigational skills. By exploring the cognitive mechanisms underlying the relationship between smartphone usage and map-related tasks, the study sheds light on the mental processes involved. Additionally, the identification of gender differences in pupil size contributes to our understanding of the cognitive demands of map-based activities for men and women.

Mentioning potential limitations, such as the sample size or specific characteristics of the chosen map application, would provide a more balanced perspective on the research.

However, it is worth remembering the limitations of the study, which include the selection of the sample and the specificity of the selected map application.

Reviewer 3 Report

Comments and Suggestions for Authors

A very nicely prepared article, and easy to understand.

On the pupil size differences and cognitive load, you mention previous research on this but don't review it - or the context in which it was previously observed.

Author Response

Reviewer 3

A very nicely prepared article, and easy to understand.

On the pupil size differences and cognitive load, you mention previous research on this but don't review it - or the context in which it was previously observed.

Thank you for this suggestion. Based on this, we extended the "Related Work" section to include previous research on cognitive load in the context of pupil size in individual users:

Building upon Skaramagksa et al.'s (2021) findings, it is essential to underscore that an enlarged pupil diameter could signify heightened engagement in cognitive or emotional processes. This observation aligns with established literature suggesting that variations in pupil size are reflective of the intensity of mental and emotional activities (Foroughi et al. 2017). In the specific context of map-reading tasks, where cognitive demands fluctuate based on factors such as task complexity, spatial information processing, and user engagement, monitoring pupil dilation becomes a valuable metric for gauging the cognitive load or emotional involvement of individuals (Kiefer et al. 2016).

Reviewer 4 Report

Comments and Suggestions for Authors

Advantages:

The utilization of eye-tracking technology is a commendable approach to address challenges in symbol interpretation. This technology holds the potential to significantly enhance user experiences in mobile map applications.

The article appropriately recognizes the importance of designing applications for mobile devices first, considering the limitations posed by smaller screens. This user-centric approach aligns with best practices in mobile application development.

The research questions presented are well-structured and directly address the impact of smartphone usage frequency, average daily usage time, and gender differences on users' efficiency and accuracy in map-related tasks.

The inclusion of insights from Krassanakis and Cybulski as well as Roth adds credibility to the study, grounding it in the existing body of knowledge related to visual perception, cognition, and mobile map design.

Disadvantages:

The article acknowledges the significance of smartphone capabilities but does not delve deeply into potential accessibility issues. The dependence on advanced devices for optimal eye-tracking performance might exclude users with older or less sophisticated smartphones.

The study focuses on Mapy.cz, and while it provides valuable insights, the use of the findings to other mobile map applications might be limited. The specific features and design choices of Mapy.cz may not be representative of the entire mobile mapping landscape.

Discussion:

The discussion section provides a thorough analysis of the research findings. The observation regarding challenges in locating points within buildings adds practical insights, and the correlation between smartphone usage frequency and efficiency in symbol detection is a notable contribution.

Conclusion:

The conclusion effectively summarizes the study's key findings and their implications. It reinforces the importance of mobile map applications, underscores the role of eye-tracking technology, and introduces valuable metrics for usability studies in cartography.

Overall Assessment:

The article offers a comprehensive overview of the study, presenting a balanced exploration of its strengths and potential limitations. It successfully bridges the gap between the importance of mobile map applications and the integration of eye-tracking technology, contributing to the field of cartography and user experience research.

Author Response

Reviewer 4

Advantages:

The utilization of eye-tracking technology is a commendable approach to address challenges in symbol interpretation. This technology holds the potential to significantly enhance user experiences in mobile map applications.

The article appropriately recognizes the importance of designing applications for mobile devices first, considering the limitations posed by smaller screens. This user-centric approach aligns with best practices in mobile application development.

The research questions presented are well-structured and directly address the impact of smartphone usage frequency, average daily usage time, and gender differences on users' efficiency and accuracy in map-related tasks.

The inclusion of insights from Krassanakis and Cybulski as well as Roth adds credibility to the study, grounding it in the existing body of knowledge related to visual perception, cognition, and mobile map design.

Disadvantages:

The article acknowledges the significance of smartphone capabilities but does not delve deeply into potential accessibility issues. The dependence on advanced devices for optimal eye-tracking performance might exclude users with older or less sophisticated smartphones. (WR)

Both people with more and less advanced smartphones took part in the study. Each of these people was valuable to us in terms of the data obtained. Those with older hardware and less user experience also provided us with information about their visual strategy when using the product. We did not require respondents to have any knowledge of the Mapy.cz application. This allows us to observe differences in the approach to using the app by different groups of people. This enables a user-centred design that takes into account the people who will be potential users in the future.

The study focuses on Mapy.cz, and while it provides valuable insights, the use of the findings to other mobile map applications might be limited. The specific features and design choices of Mapy.cz may not be representative of the entire mobile mapping landscape. (WR)

We agree that the results of our Mapy.cz app study cannot be projected to the entire mobile map landscape. We believe that mobile maps require further research that takes into account other factors and other design solutions. The aim of our study was to show new suggestions for criteria for the studied applications and to present the possibility of using an eye-tracking tool when analysing users' visual strategy when using a particular cartographic product. In the future, we intend to conduct a more comprehensive study on other applications, taking into account other aspects that were not considered in this work.

Discussion:

The discussion section provides a thorough analysis of the research findings. The observation regarding challenges in locating points within buildings adds practical insights, and the correlation between smartphone usage frequency and efficiency in symbol detection is a notable contribution.

Conclusion:

The conclusion effectively summarizes the study's key findings and their implications. It reinforces the importance of mobile map applications, underscores the role of eye-tracking technology, and introduces valuable metrics for usability studies in cartography.

Overall Assessment:

The article offers a comprehensive overview of the study, presenting a balanced exploration of its strengths and potential limitations. It successfully bridges the gap between the importance of mobile map applications and the integration of eye-tracking technology, contributing to the field of cartography and user experience research.

Reviewer 5 Report

Comments and Suggestions for Authors

The paper explores the influence of smartphone usage frequency on the effectiveness and accuracy of symbol location in mobile maps, utilizing eye-tracking technology with Mapy.cz as a case study. The research is interesting, but it has a few flaws that should be addressed to make this article compelling and to improve the reader’s understanding.

 

Comment 1:

This paper proposed three research questions, including the frequency of smartphones, scanning speed, and gender differences in the use of mobile maps. There is little summary of these three related works that affect map symbols or map elements, so it is recommended to add more.

 

As for the first part, this study considers smartphone frequency as an important factor in the effectiveness and accuracy of symbol location, in Section 4.3, for the frequency of use of navigation on a smartphone, authors classify the category into three types, every day, once a week, and once a month or less. This division method is not reasonable, if 2, 3, or 4 times a week, which one should the user choose?

 

For the scanning speed, in Section 4.2 and Figure 5, it can be noticed the correlation coefficient r<0.4, indicates that there exists a weak positive correlation, there is no strong correlation between the time the user uses the mobile phone and the first fixation of the AOI.

 

For the gender pupil diameter, authors present the p<0.005, please add the related data to prove it. In the participant’s selection, including 40 men and 18 women, the proportion is not equal, will this affect the experimental results? The observation of gender differences might require a more in-depth interpretation and comparison with existing research.

 

Comment 2:

Q3 Symbol 2 or Symbol 3 appears many times in the text, but the reader does not know what these symbols are, please add the symbol in details.

 

Comment 3:

Lines 13-14 mention eye tracking as a valuable tool for testing the usability of map products, how it can help evaluate users' visual strategies in location symbols in this study? As shown in Figure 1, the symbols on the map are relatively small, which greatly affects the experimental results.

 

Comment 4:

Line 199, e) Do you know Mapy.cz application? Do you evaluate the knowledge of the Mapy.cz application on users through this question is enough?

Author Response

Reviewer 5

The paper explores the influence of smartphone usage frequency on the effectiveness and accuracy of symbol location in mobile maps, utilizing eye-tracking technology with Mapy.cz as a case study. The research is interesting, but it has a few flaws that should be addressed to make this article compelling and to improve the reader’s understanding.

Comment 1:

This paper proposed three research questions, including the frequency of smartphones, scanning speed, and gender differences in the use of mobile maps. There is little summary of these three related works that affect map symbols or map elements, so it is recommended to add more.

We agree with your opinion that there is a lack of a related work description in the main text. Therefore, we have extended the appropriate section to follow your recommendation and extend the summary according to it:

In related work

Usability testing for personalized user characteristics such as age, gender or experience, has been postulated many times (Bartling et al. 2021; Griffin et al. 2017; Montello 2002; Sarjakoski & Nivala, 2005). Scan path speed during cartographic tasks on satellite images presented through saccadic amplitude referred to more demanding scanning processes in the peripheral search in the study of Krejtz et al. (2017). Saccadic amplitude was also analyzed by Putto et al. (2014) while participants were searching and selecting different geometrical objects on elevation visualization, and in that study, the largest saccadic amplitude were observed for contour lines. Based on saccadic velocity Kiefer et al. (2013) were able to recognize participants activities on cartographic background.

Differences between males and females in map-use were studied some time ago by Gilmartin and Patton (1984). Their findings concerns children and adults such as route planning or symbol identification. The only differences were found among children were boys performance were significantly better than girls. Montello et al. (1999) presented differences between males and females on various map-based tasks. They showed that male participants were better on newly acquired spatial information. On the other hand female participants outperformed males on static object/location memory tasks. Spatial orientation in wayfinding tasks based on 3D maps were studies by Liao & Dong (2017) in the context of sex differences. They found that male participants’ fixation duration and fixation count distribution were more platykurtic that female participants.

In summary

The research delves into specific findings related to users' performance in map-related tasks and addresses posed research questions. Firstly, it acknowledges the impact of habitual smartphone usage on users' efficiency and accuracy in locating symbols within diverse spatial contexts on mobile maps. The study reveals that frequent smartphone use significantly enhances users' performance in these tasks, suggesting a link between usage habits and improved cognitive processing during map-related activities suggested by Meng (2003).

Secondly, the research explores the correlation between average daily smartphone usage time and task outcomes, specifically in terms of scanning speed and symbol detection. The findings demonstrate a moderate correlation, indicating that increased smartphone usage is associated with improved performance in map-related tasks. This aligns with the broader literature on usability testing for personalized user characteristics, where factors such as saccadic amplitude or fixation count (Krejtz et al. 2017; Putto et al. 2014) play crucial roles in influencing user performance during cartographic tasks.

Finally, the study delves into sex differences in pupil size during map-related tasks and examines their relationship with variations in cognitive workload. The consistent observation of larger pupil diameters in women suggests a higher cognitive workload for female participants during map-related activities. This insight adds to the existing knowledge on sex differences in map use, as demonstrated in prior research by Gilmartin and Patton (1984), Montello et al. (1999), and Liao & Dong (2017). These studies highlight the role of sex differences in influencing performance aspects such as spatial information acquisition, and symbol identification on mobile device. The incorporation of these findings enriches our understanding of the cognitive demands associated with map-based activities.

 

As for the first part, this study considers smartphone frequency as an important factor in the effectiveness and accuracy of symbol location, in Section 4.3, for the frequency of use of navigation on a smartphone, authors classify the category into three types, every day, once a week, and once a month or less. This division method is not reasonable, if 2, 3, or 4 times a week, which one should the user choose?

In this question, the choice was made by the participant himself. But we must agree that it is worth considering more choices for this question in the next study.

For the scanning speed, in Section 4.2 and Figure 5, it can be noticed the correlation coefficient r<0.4, indicates that there exists a weak positive correlation, there is no strong correlation between the time the user uses the mobile phone and the first fixation of the AOI.

For the gender pupil diameter, authors present the p<0.005, please add the related data to prove it. In the participant’s selection, including 40 men and 18 women, the proportion is not equal, will this affect the experimental results? The observation of gender differences might require a more in-depth interpretation and comparison with existing research.

Following your suggestion to include relevant data to demonstrate statistical significance, we have taken two actions. First, we have shared all data collected with eye tracking equipment via Harvard Dataverse (an open data repository), enabling the analysis of study data by anyone. Secondly, we have incorporated a figure presenting the results of pupil diameter in the text, allowing potential readers to gain a more in-depth interpretation.

Indeed, participant selection is not uniform, and it may be subject to debate. Following your suggestion, we have added this point as a study limitation:

The participant selection process, comprising 40 men and 18 women, lacks equal gender representation, potentially introducing a gender-related bias into the study results. This imbalance may limit the generalizability of our findings. Addressing this limitation, future research endeavors should strive for a more balanced and representative participant composition to ensure broader applicability of study outcomes.

According to your suggestion, which highlights the issue of comparing existing literature on sex differences, especially in pupil size, is considered significantly important. Additional research items and explanations have been incorporated into Related work section (also in accordance to your Comment #1):

Building upon Skaramagksa et al.'s (2021) findings, it is essential to underscore that an enlarged pupil diameter could signify heightened engagement in cognitive or emotional processes. This observation aligns with established literature suggesting that variations in pupil size are reflective of the intensity of mental and emotional activities (Foroughi et al. 2017). In the specific context of map-reading tasks, where cognitive demands fluctuate based on factors such as task complexity, spatial information processing, and user engagement, monitoring pupil dilation becomes a valuable metric for gauging the cognitive load or emotional involvement of individuals (Kiefer et al. 2016).

 

Comment 2:

Q3 Symbol 2 or Symbol 3 appears many times in the text, but the reader does not know what these symbols are, please add the symbol in details.

We have added a figure with all the symbols involved in the study.

Comment 3:

Lines 13-14 mention eye tracking as a valuable tool for testing the usability of map products, how it can help evaluate users' visual strategies in location symbols in this study? As shown in Figure 1, the symbols on the map are relatively small, which greatly affects the experimental results.

Thank you for pointing out this important issue. For the first part of the question we explored this issue in the introduction:

Eye tracking metrics have consistently served as valuable tools in various research studies, where their presentation and correlation with performance measures contribute to a comprehensive understanding of efficiency and usability. Numerous investigations across diverse fields have utilized eye tracking technology to unravel intricate patterns of visual behavior and glean insights into cognitive processes underlying human-computer interactions (Goldberg & Wichansky, 2003). Researchers have employed fixation and saccade-based metrics to assess the effectiveness of animated and interactive maps, the allocation of visual attention, and the cognitive load associated with different tasks (Çöltekin et al. 2009; Dong et al. 2014).

In the second part, involving relatively small symbols in Figure 1, we would like to reference the studies conducted by Ooms et al. (2018) and Ooms et al. (2015), which evaluate the quality of eye trackers with an even lower sampling frequency than the one used in this study:

In light of prior cartographic research, encompassing studies that have validated the appropriateness of low-cost eye tracking equipment (Ooms et al. 2015; 2018), the choice of a 150 Hz sampling frequency with a 0.5-degree spatial accuracy can be deemed sufficient for conducting scientific experiments involving the display of symbols on mobile devices.

Comment 4:

Line 199, e) Do you know Mapy.cz application? Do you evaluate the knowledge of the Mapy.cz application on users through this question is enough?

Your comment is a good starting point for further research on this topic. In the future, we would like to ask users more detailed questions about their knowledge of the Mapy.cz application and see how the results differ from those in this article.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

I do not aggree with your assumption that 

"The principle of scientific research is to draw conclusions only from statistically significant results, which is why we only consider p < 0.05."

The fact that you did not find statistically significant difference is also a result - variable X does not have influence on variable Y.