Improving Performance and Quantifying Uncertainty of Body-Rocking Detection Using Bayesian Neural Networks
Round 1
Reviewer 1 Report
Body-rocking is an often-undesired stereotypical motor movement performed by some individuals, and its detection is essential for self-awareness and habit change. Authors envision a pipeline that includes inertial wearable sensors and a real-time detection system for notifying the user for awareness. For this task, false detection may prevent continuous engagement, leading to alarm fatigue. Authors present a pipeline using Bayesian Neural Networks with uncertainty quantification for jointly reducing false positives and providing accurate detections. Authors show that increasing model capacity does not consistently yield higher performance by itself, while pairing it with the Bayesian approach does yield significant improvements. Disparities in uncertainty quantification are better quantified by calibrating them using deep neural networks. This paper has a certain degree of innovation. The following suggestions could be of help to further improve the quality of this paper.
Q1: It is recommended to add to the abstract the previously addressed human sway detection.
Q2: It is proposed to add to Introduction the limitations of the methods already available to address human sway detection.
Q3: It is proposed to add the use of Bayesian neural networks in Introduction to address the motivation of human sway detection.
Q4: It is proposed to add the advantage of using Bayesian neural networks to solve human sway detection in Related Work over previous methods.
Q5: It is suggested to add a summary description of all formulas in the chapter Bayesian Neural Networks.
Q6: It is suggested to add a brief description of the arrangement of the experiment.
Q7: It is proposed to add a discussion of the shortcomings of the work done in the summary chapter.
Q8: Suggest adding more discussion on future work in the summary chapter.
Author Response
Reviewer 1
Q1: It is recommended to add to the abstract the previously addressed human sway detection.
Thank you for the comment. We added a note by the end of the abstract that summarizes the main contribution as well as mentions that this work is part of a continuous effort.
Q2: It is proposed to add to Introduction the limitations of the methods already available to address human sway detection.
Thanks for the suggestion. We added this information starting on line 84 of the introduction and the text has been highlighted to ease the revision. Other methods were concerned mostly with high-performance detection but not necessarily the number of false positives that a system with such implementation may have. False positives represent a key factor to allow continuous use by patients and thus, the uncertainty quantification provided by means of a Bayesian neural network is highly valuable.
Q3: It is proposed to add the use of Bayesian neural networks in the Introduction to address the motivation of human sway detection.
We appreciate the suggestion. The fact that Bayesian neural networks can perform the sway detection along with estimating the uncertainty makes them very good candidates for the task. This has been further clarified in the introduction starting at line 82, the text has been highlighted to ease the revision.
Q4: It is proposed to add the advantage of using Bayesian neural networks to solve human sway detection in Related Work over previous methods.
Thanks for the feedback. We have added a couple of observations by the end of section 2 Related Work. There, after describing the main methods found in the literature for sway detection, we clarified that none of the works were focused on avoiding false positives. Additionally, the use of Bayesian neural networks is encouraged since the best results obtained so far were obtained with deep learning based approaches as references [27, 28] show for instance.
Q5: It is suggested to add a summary description of all formulas in the chapter Bayesian Neural Networks.
Thank you, that is very appreciated. At the end of section 3 Materials and methods, we have added a summary table with the main equations which tell the story that starts from the neural network until obtaining the loss functions of the Bayesian approach, as well as the equations that allow the estimation of uncertainty.
Q6: It is suggested to add a brief description of the arrangement of the experiment.
Thanks for the suggestion. We have further described our setup under section 3.1 Datasets, we describe more details about the system that allows one to reproduce our setup with further details.
Q7: It is proposed to add a discussion of the shortcomings of the work done in the summary chapter.
We appreciate the suggestion. We have added several points that describe our sincere judgments about the limitations of the manuscript and also of the Bayesian neural networks as a method for sway detection. These points have been added to the end of section 5 Discussion. To briefly cite them, we believe it is worth mentioning: (1) run time constraints, since estimating uncertainty relies on the ensemble of several predictions. (2) Limited data set for evaluating the uncertainty quantification methods. (3) Our methods have not been fine-tuned thus the performance shown could be improved. (4) We lack prior sensitivity analysis but we recognize its importance, however, a large dataset may alleviate the effects of the prior on the predictions. Finally, (5) non-dynamic dropout probability in the model, which is a factor that limits its performance. We believe that by stating these points, the reader will have a clear picture of the deficiencies of the paper as well as directions for future research, thanks again for this comment.
Q8: Suggest adding more discussion on future work in the summary chapter.
Thanks again for the great suggestion. The comments on future work were also added along with the discussion of the limitations mentioned above.
Reviewer 2 Report
In the peer-reviewed manuscript, the authors present a comprehensive comparative study of methods to classify the body-rocking activity. The methods were evaluated considering a Bayesian approach. It was observed that a shallower model tends to not take advantage from the Bayesian approach. Additionally, the Bayesian approach was shown to provide superior performance benefits when applied to higher capacity models.
The paper brings original novel information in the domain of the journal’s thematic focus. Bayesian DL is still a growing research area for which new insights are being shared. I foresee that the performance observed in this paper can be further improved by not only comprehensively evaluating deep architectures, but also exploring the effects of different priors for body-rocking classification and new ways of obtaining posteriors. The research results are clearly distinguished from results adopted and used literary resources are mentioned properly. Credibility of published results is documented (experiments - simulations). Text readability and its linguistic correctness (even English texts, especially in the case of the technical terminology) is on the appropriate level.
I have several comments on the content of the article:
1. The abstract needs to be adjusted: focus on the goal and contribution of the work.
2. line 29 (not “…in table 1…”, but “…in Table 1…”).
In general, after appropriate corrections and additions, I approve publication of this manuscript.
Author Response
REVIEWER 2
1. The abstract needs to be adjusted: focus on the goal and contribution of the work.
Thanks for pointing that out. We have added a summarizing sentence at the end of the abstract that describes our main contribution along with the goal.
2. line 29 (not “…in table 1…”, but “…in Table 1…”).
Thanks for your detailed reading. We have fixed that.
Round 2
Reviewer 1 Report
Q1: It is recommended to present the challenge of Body-rocking detection in the abstract.
Q2: It is suggested to add more motivation for the method used for human oscillation detection in the introduction.
Q3: It is suggested to change the text color in Figure 1, for example, from orange to purple, to increase the recognition.
Q4: It is recommended to check the format of the table in the text and change it to a three-line table.
Q5: Why is there a font bold in the text? It is recommended to double check the font format.
Author Response
Q1: It is recommended to present the challenge of Body-rocking detection in the abstract.
We appreciate the suggestion. We have added a sentence to summarize the challenge in distinguishing body rocking signals from other similar ones and how that induces false positives. With that, we believe that a concise but complete message is being passed to the reader. The change is in purple and we kept the previous ones in yellow for a full context.
Q2: It is suggested to add more motivation to the method used for human oscillation detection in the introduction.
Thanks for further clarifying this suggestion, we see the point here. On top of further justification for using BNNs for body rocking detection, starting at line 86 we added a paragraph explaining how this model approach is especially useful for oscillation detection. The change is in purple and we kept the previous ones in yellow for a full context.
Q3: It is suggested to change the text color in Figure 1, for example, from orange to purple, to increase the recognition.
The text has been altered for better recognition, thank you.
Q4: It is recommended to check the format of the table in the text and change it to a three-line table.
Thank you. We have removed that last column and added that information as a footnote in the third line of the table. The visualization of the table is cleaner than before.
Q5: Why is there a font bold in the text? It is recommended to double-check the font format.
The texts in bold are only used in the Discussion section and they are intended as visual shortcuts to the contents in the text as well as to aid the review by our peers when trying to do the first skimming through that section. The latex formatting with the command “\hl{}” makes it a bit messy, but as soon as we remove the highlighting, the text will look better and will have such aid for the reader.
Finally, we preferred to remove the reference
[32] Akbari, A.; Jafari, R. A Deep Learning Assisted Method for Measuring Uncertainty in Activity
Recognition with Wearable Sensors. In Proceedings of the 2019 IEEE EMBS International
Conference on Biomedical Health Informatics (BHI), 2019, pp. 1–5. https://doi.org/10.1109/
BHI.2019.8834505.
Since we already had enough examples of references utilizing uncertainty for activity recognition using wearable sensors. The intention was to show how flexible the solution with uncertainty quantification is and also provide instances of success in the literature.
