Next Article in Journal
A Simplified Kinematics and Kinetics Formulation for Prismatic Tensegrity Robots: Simulation and Experiments
Next Article in Special Issue
Evaluating the Performance of Mobile-Convolutional Neural Networks for Spatial and Temporal Human Action Recognition Analysis
Previous Article in Journal
Design and Evaluation of an Intuitive Haptic Teleoperation Control System for 6-DoF Industrial Manipulators
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Robotics
Volume 12
Issue 2
10.3390/robotics12020055
robotics-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Heart Rate as a Predictor of Challenging Behaviours among Children with Autism from Wearable Sensors in Social Robot Interactions

by
Ahmad Qadeib Alban
1,
Ahmad Yaser Alhaddad
2,
Abdulaziz Al-Ali
1,3,
Wing-Chee So
4,
Olcay Connor
5,
Malek Ayesh
1,
Uvais Ahmed Qidwai
1 and
John-John Cabibihan
2,*
1
Department of Computer Science and Engineering, Qatar University, Doha P.O. Box 2713, Qatar
2
Department of Mechanical and Industrial Engineering, Qatar University, Doha P.O. Box 2713, Qatar
3
KINDI Computing Research Center, Qatar University, Doha P.O. Box 2713, Qatar
4
Department of Educational Psychology, Faculty of Education, The Chinese University of Hong Kong, New Territories, Hong Kong
5
Step by Step Centre for Special Needs, Doha P.O. Box 47613, Qatar
*
Author to whom correspondence should be addressed.
Robotics 2023, 12(2), 55; https://doi.org/10.3390/robotics12020055
Submission received: 24 January 2023 / Revised: 19 March 2023 / Accepted: 21 March 2023 / Published: 1 April 2023
(This article belongs to the Special Issue Towards Socially Intelligent Robots)
Browse Figures
Versions Notes

Abstract

:
Children with autism face challenges in various skills (e.g., communication and social) and they exhibit challenging behaviours. These challenging behaviours represent a challenge to their families, therapists, and caregivers, especially during therapy sessions. In this study, we have investigated several machine learning techniques and data modalities acquired using wearable sensors from children with autism during their interactions with social robots and toys in their potential to detect challenging behaviours. Each child wore a wearable device that collected data. Video annotations of the sessions were used to identify the occurrence of challenging behaviours. Extracted time features (i.e., mean, standard deviation, min, and max) in conjunction with four machine learning techniques were considered to detect challenging behaviors. The heart rate variability (HRV) changes have also been investigated in this study. The XGBoost algorithm has achieved the best performance (i.e., an accuracy of 99%). Additionally, physiological features outperformed the kinetic ones, with the heart rate being the main contributing feature in the prediction performance. One HRV parameter (i.e., RMSSD) was found to correlate with the occurrence of challenging behaviours. This work highlights the importance of developing the tools and methods to detect challenging behaviors among children with autism during aided sessions with social robots.

1. Introduction

Autism Spectrum Disorder (ASD) is a developmental and neurological disorder that causes impairments characterised by difficulties in social communication and restricted behaviours [1]. Children with ASD exhibit challenging behaviours frequently at varying intensities and in different forms, such as meltdowns, tantrums, property destruction, and aggression [2,3,4]. The prevalence rate of challenging behaviours and aggression among children with ASD is high [5,6,7]. It is reported by the Autism Society of Minnesota that one in every 68 people have ASD [8]. Individuals with ASD exhibit recurrent challenging behaviours, and these constitute a source of concern for parents and caregivers due to their sudden occurrences. Challenging behaviours vary in frequency, intensity, duration, and may put the child or others in an unsafe situation. These behaviours may appear in many forms, which include [9]:
  • Withdrawal: The child may zone out, gaze into space, or performs repetitive actions.
  • Outward distress: This includes crying in an uncontrolled manner, stomping, screaming, or curling up into a ball.
  • Physical and verbal aggression: Destruction of surrounding objects, outbursts, and self-damaging. These aggressive behaviours include scratching their faces or hands and pulling their own hair, as well as with skin pinching.
Many children with autism experience distress prior to meltdown. This period that precedes the meltdown is called the rumble stage [10,11]. Specific behaviours, such as challenging behaviours, appear at this stage just before a meltdown. These lead to actions such as head banging (e.g., against an object or using their hands), hand flapping, kicking, throwing objects, hand biting, screams, and others [5,12]. Being frustrated and the presence of new stimuli are some of the contributing factors that increase the occurrence of challenging behaviours [13].
Early intervention can help in managing challenging behaviours [14,15]. The advances in technology are being integrated in the screening of ASD and in therapy sessions to improve the outcomes [16,17]. Social robots are examples of adopted technologies in therapy that have reported positive outcomes (e.g., improved communication, motor, and social skills) among children with autism [18,19,20].
A social robot represents a new stimulus into a child’s environment that might trigger an unwanted behaviour. Hence, investigating the perception of a social robot is essential [21,22]. The occurrence of aggression toward social robots was reported during interaction sessions [23,24]. Potential harm could occur due to the manifestation of challenging behaviours, for example, due to the throwing of a social robot [25]. Improved safer designs and hardware optimisations approaches are needed to mitigate any potential harms [26,27,28]. Furthermore, solutions based on machine learning techniques, especially personalised, are needed to anticipate such behaviours and to provide reactions through a therapist or the companion social robots [9,29,30,31,32].
Physiological changes of the human body can provide indicators about health and the current emotional state using wearable devices and machine learning techniques. A previous study found that challenging behaviours are influenced by the physiological arousal of children with autism [33]. Another study reported a difference in the heart rate between adults with ASD and normal adults during public speaking [34]. Research that detects challenging behaviours among children with autism during interactions with social robots is still limited [9,35,36].
To notify parents or caregivers to the occurrence of these behaviours, the vital signs and physiological changes should be investigated accordingly to assist in the detection of challenging behaviours. One approach is to measure the child’s vital signs using wearable sensors while leveraging machine learning and deep learning algorithms to construct an accurate detection model. In this study, we conduct experiments using data acquired from five children with ASD and machine learning techniques to detect challenging behaviours during interaction sessions with toys and social robots. The contributions of this work are summarised as follows:
  • Integration of machine learning techniques and wearable sensors to detect challenging behaviours.
  • Evaluation of physiological and kinetic features in identifying challenging behaviors.
  • Analysis on the HR and HRV roles in supporting the detection.
This paper is organised as follows. Section 2 presents the background. Section 3 describes the methods used. Section 4 presents the results, while Section 5 provides the discussion. Section 6 concludes the study.

2. Background

Prior studies investigated the possibility of integrating wearable sensors and machine learning techniques to interpret the physiological and kinematic properties of the human body to predict or detect specific affective patterns of emotions or behaviours [37,38,39,40]. The considered modalities and detection objectives using wearables and machine learning techniques in autism research have varied. Some of the sensors and modalities considered were photoplethysmogram, electrodermal activity or the galvanic skin response, heart rate, temperature, acceleration, and skin conductance level. There is a growing interest in incorporating wearable sensors in autism therapy. For example, Fazana et al. presented a framework that incorporates a set of existing programs for augmentative and alternative communication with wearable sensors to improve the communication skills, enhance behaviours, and promote health monitoring of children with autism [41].
Goodwin et al. [42] investigated the detection of stereotypical motor movements in children with autism using three-axis accelerometers worn on different parts of the body (i.e., wrists and torso). The data were collected from six individuals with autism. The recognition performance of the two employed classifiers (i.e., decision tree and support vector machines) achieved accuracies ranging from 81.2% to 99.1%. Rad et al. [43] used the same datasets to implement a convolutional neural network and long short-term memory algorithms. Their results demonstrated that applying deep learning techniques on the acceleration data would improve the detection of stereotypical motor movements in real-time conditions. Another work also used accelerometer data to classify challenging behaviors based on simulated behaviours by a specialist [44]. When tested on a child with autism, their best machine learning model achieved an accuracy of 69.7%. Another study explored techniques to detect common motor movements for children diagnosed with autism [45]. The study investigated the impact of these motor movements on learning and social interactions using deep learning approaches.
Pollreisz et al. [46] established an emotion recognition system using an Empatica E4 watch to collect electrodermal activity, heart rate, and temperature values from ten young adults. The reported success rate for emotion recognition was 65% using a decision tree algorithm. Another study investigated the different emotions encountered throughout meltdowns to develop an emotion recognition system using machine learning methods [47]. The study concluded that the random forest performed the best (91.27%) when feature selection techniques were employed. Heart rate variability parameters were also considered (Table 1). Lee et al. [48] measured both the heart rate variability and galvanic skin response to identify emotions using neural networks. They analyzed the data collected from participants in the frequency and time domains. They found that changes in some of the HRV parameters (i.e., RMSSD and SDNN) might lead to elevated activity in the sympathetic nervous system, which could be interpreted as a sign of fear. Their reported accuracy of the best trained model was 80.2%.
Yap et al. [50] investigated the impact of listening to music on the heart rate and anxiety levels of children with autism to identify which music genre could calm the children. They devised a mobile application that was connected wirelessly to a pulse sensor to measure the heart rate. The goal of their application was to improve communication and learning skills, support emotion regulation, and monitor the heart rate while listening to music. Lydon et al. [51] focused on investigating the correlation between the heart rate and challenging behaviours experienced by children with autism. They analyzed the heart rate data before, during, and after the instances of challenging behaviours in three children with ASD. They found that such behaviours might increase arousal for some children with autism. The prediction of challenging behaviours has also been proposed by J. Nuske et al. [52]. They investigated the possibility of applying the heart rate to predict such behaviours in children with autism based on the statistical analysis for the acquired data. Forty-one children diagnosed with autism were recruited in their experiments. The participants wore an electrocardiograph monitor and low-level stress was stimulated. Considering the intrusive nature of the device and to avoid pulling the ECG electrodes, vests with pockets were used to house the device while placing the electrodes on the backs of the children. Their results demonstrated that heart rate changes could be an early sign for the occurrence of challenging behaviours.

3. Materials and Methods

3.1. Participants

Five male children with autism, with ages ranging between 7 and 10 years old, participated in this study. The participants attended a local centre for special needs in Doha, Qatar. Parental consent was obtained by the centre. The sessions were conducted with each child individually with the supervision and assistance of a teacher or their caregiver. The procedures for this work did not include invasive or potentially hazardous methods and were in accordance with the Code of Ethics of the World Medical Association (Declaration of Helsinki).

3.2. Stimuli

Social robots and regular children’s toys were used as stimuli in this study. The social robots were the humanoid Nao robot (SoftBank Robotics, Tokyo, Japan) and a white furred robotic seal (PARP robots, Itasca, IL, United States). These social robots are shown in (Figure 1b). The toys consisted of a squishy green rubber ball, multi-colour train, brass cymbals, and wooden letter blocks that are placed on a toy truck. Further details regarding the stimuli used can be found in [23].

3.3. Wearable Device

A wristband wearable sensor (Empatica E4, Milano, Italy) was used to obtain the data readings from the children during the experiments (Figure 1a). The E4 wearable sensor contains a real-time clock and it is capable of recording physiological data signals to an internal memory (Figure 1c). The physiological signals considered are as follows:
  • Acceleration (ACC): measures wrist’s motion changes in terms of the acceleration changes in the x, y, and z directions.
  • Electrodermal Activity (EDA): determines the change in skin conductance and the skin’s electrical properties.
  • Temperature (TEMP): determines the temperature of the skin.
  • Heart Rate (HR): the number of beats per minute.
  • Blood Volume Pulse (BVP): determines the changes in the blood volume.

3.4. Algorithms

The considered supervised machine learning algorithms were as follows:
  • Support-Vector Machine (SVM): non-probabilistic binary linear supervised learning model that can solve and classify both linear and non-linear problems.
  • Multilayer Perceptron (MLP): learning technique inspired by the biological brain that consists of layers of artificial neurons that can learn from data.
  • Decision Tree (DT): an algorithm that predicts the output by moving through the different discrete decision options that are represented in a tree-like structure until a conclusion is reached.
  • Extreme Gradient Boosting (XGBoost): an ensemble supervised machine learning technique which utilises regularised gradient boosted decision trees to improve the performance and classification speed.

3.5. Procedures

3.5.1. Annotation

Manual annotation was carried out for each of the five children’s behaviours. This was conducted with the help of a free annotation software (BORIS, v. 7.10.2, Torino, Italy). The behaviours were annotated as either ’Challenging’ or ‘Non-challenging’. A challenging behaviour is considered to be any action that is interfering, repetitive, stimming, and might inflict harm on oneself or others. Challenging behaviours also included head banging, arm flapping, ear pulling, kicking, and scratching. The total number of challenging behaviour instances in all the sessions was 17. The respective percentage of challenging behaviours in each session was in the range of 1.74% to 18.21%.

3.5.2. Data Preprocessing

To ensure consistency, the data acquired from the wearable device were preprocessed and the sampling frequency of every acquired data signal was set to 64 Hz. This is crucial since the different sensors obtain data at different sampling rates. The preprocessing stage included the outliers’ removal and resampling the training data to ensure that the classes are equally balanced. A portion equal to thirty percent of the original dataset was used as the unseen testing set. The initial experiments with the dataset indicated that the extracted features produced a better performance as compared to the raw features alone. For this reason, only the time-domain extracted features (i.e., mean, standard deviation, min, and max) were considered throughout this study.

4. Results

4.1. Machine Learning Models

Four machine learning algorithms were examined based on the evaluation metrics and prediction speed (Table 2). In the results, challenging behaviors were considered to be the positive class. The models were developed using Python libraries (i.e., Sklearn [53] and XGBoost [54]). The depth of the DT algorithm was set to dynamic and the Gini function was used for the splitting criteria. SVM used a radial basis function kernal with a regularisation parameter of 0.1, and a gamma parameter was set to the scale. As for the MLP, it contained one hidden layer that consisted of 100 neurons with weights adjusted using a stochastic gradient descent at a 0.0001 L2 regularisation. The XGBoost was trained with the logistic objective and a max depth of 6, with a aplha equal to 1, learning rate of 0.3, and 100 estimators.
XGBoost showed a better overall performance compared to other classifiers in terms of precision (0.88), recall (0.99), F1-Score (0.93), and accuracy (0.99). Additionally, it achieved the fastest time (i.e., 0.24 s) to predict the test samples. The second best performing algorithm was DT followed by MLP. SVM achieved the lowest performance and took the longest time to predict the test samples, which was around 2.5 s. Due to its performance, the XGBoost has been considered in the upcoming experiments.

4.2. Feature Effects

To measure the contribution of each sensor to the prediction performance, sensor features were added gradually to the overall feature vector and the results were compared for the individualised models and combined model (Table 3). With the ACC alone, the personalized models performed poorly in all five participants and so their combined model. Set 2 considered the effect of adding the HR sensor reading to the feature vector that has led to a large increase in performance for all participants individually and their combined model. As for Set 3, adding BVP had little effect on all the models. Adding TEMP improved the performance of the individual personalised models and their combined model slightly. Finally, adding EDA in Set 5 has led to a further increase in the overall performance for most of the models.

4.3. Kinetic vs. Physiological

To understand which category of features are most significant, the kinetic, physiological, and a combination of the two were investigated. The evaluation metrics results for the two categories and their combined features are depicted in Figure 2. The results demonstrated that kinetic features alone performed poorly with respect to the physiological and combined features. The physiological features were found to perform similarly to the combined features. In spite of this, the overall best performance comes from using the combined features.
To further investigate how each individual feature contributes to the performance of the machine learning model, the importance of each individual feature with respect to the F-score were plotted using the built-in XGBoost tool [54]. The individual plots for each participant revealed a discrepancy between the importance of each features (Figure 3). However, HR appears to be the most important factor for the majority of the participants, followed by either EDA or TEMP. The combined plot for the generalised model revealed that the most important feature was HR then followed by EDA, TEMP, ACC, and BVP (Figure 4).

4.4. HRV and Challenging Behaviours

To further investigate the importance of heart rate parameters, the heart rate variability (HRV) based on calculating the RMSSD was considered. The RMSSD was derived from the interbeat interval signal of the wearable device using a sampling frequency of 64 Hz. The HRV changes for one of the children during different states were investigated (Figure 5). The HRV values appear to be highest during a rest state while the lowest during the occurrence of a challenging behaviour (Figure 5c).
A machine learning model was trained that contained an additional feature, called HRV. The results demonstrated that the importance of HRV for a child that exhibited more challenging behaviours (Figure 6a) is higher compared to a child that exhibited less instances of challenging behaviours (Figure 6b). Furthermore, the contribution of HRV outweighed that of HR in the child exhibiting challenging behaviours and vice versa in the child experiencing less challenging behaviours. Hence, the detection of challenging behaviours appears to depend on the changes in HRV.

5. Discussion

To properly understand the occurrence of challenging behaviour, it is vital to analyze the interactions of the participants throughout their sessions. The children displayed different levels of engagement with the presented stimuli. The participants showed fascination in the colourful train that produced bubbles, which encouraged their engagement. The fascination and interaction came in various forms that included both facial expressions and physical movements. Most of the participants did not prefer the white robotic seal, which could be due to its animal-like appearance. Hence, that might have led to participants exhibiting challenging behaviours. Furthermore, the robotic humanoid caused confusion and curiosity as some of the sudden movements produced by it have led to negative reactions by some of the participants.
In this paper, we investigated the detection of challenging behaviours among children with autism using wearable sensors to acquire data and machine learning techniques. While there are many machine learning approaches, not all are suitable to be considered in such an application. In addition to the prediction accuracy, a system must also make predictions fast enough for the timely intervention. In our evaluations, the XGBoost algorithm fulfilled these two criteria. This promising performance and robustness of XGBoost could be attributed to its ability in recognizing intricate data patterns, identifying subtle physiological changes, dealing with outliers, handling overfitting, and supporting parallel processing.
The findings demonstrated that the heart rate (HR) was the most significant contributing feature on the performance of the classifying model for almost all the participants. Our interpretation is that challenging behaviours are usually accompanied by higher stress levels, which lead to an increase in the HR above the baseline [55]. Studying the contributions of both kinetic and physiological features in behavioural classification helped us to better understand the nature of challenging behaviours. More precisely, it was observed that most challenging behaviours tend to involve some sort of specific hand movements. And these hand movements were distinguished from regular hand motions through the accelerometer. Considering HR along with other modalities can offer a valuable decision support during moment-to-moment treatment planning for individuals with autism.
Another experiment was conducted to elucidate the relationship between the heart rate variability (HRV) and challenging behaviours. We found that HRV decreased during stress and stimulating episodes and increased during rest states. With the exception of one participant, the HRV analysis for the participants demonstrated a strong correlation between the fluctuations of HRV and the occurrence of challenging behaviours. A possible explanation for these disparate findings is that children with autism may not have a stable system for regulating emotions [52]. The children initially interacted with the social robots with fear at different levels and intensities. This variation in emotions has led to distinct representations of the HRV signal. Hence, HRV can be used as an indicator for the occurrence of challenging behaviours when it is associated with high reactive interactions in children with autism. Nonetheless, further research is required on a larger number of participants to outline the psychological and physiological changes during the exhibition of challenging behaviours.
Employing wearable sensors allows for lower costs, non-invasive, and less restraining methods in tracking motor movements and physiological stress for children with autism. Based on the findings, it is promising to derive HRV parameters from the wearable sensors to acquire extra information. Dedicated warning techniques that are activated due to an increase in challenging behaviour episodes would provide a valuable support for children with autism. The benefits of such systems are magnified for nonverbal individuals who have restricted means to express their stress to their parents or caregivers [56]. Hence, parents or caregivers, or even a social robot could intervene early to remove the stimuli causing that challenging behaviour [9].
One of the main limitations in this study was the low frequency of the observed challenging behaviours compared to non-challenging ones. While resampling techniques to balance the challenging and non-challenging behaviours can be used during the training phase, collecting more data that contain more instances of challenging behaviours is essential to capture the full spectrum of such behaviours. The differences due to gender was not investigated in this study. Future work will explore ways to collect more data, including, but not limited to a longer observation period and larger and more diverse participants’ pool. Another limitation was that data collection was restricted to utilizing a wrist wearable device within controlled environmental conditions. Children with autism might not tolerate the wrist wearable device. Hence, they might attempt to remove it, throw it, or even hurt themselves with it (e.g., in head banging). Future work should investigate different body locations that are less intrusive to place one or multiple wearable devices that can recognize different patterns of behaviours at the same time. Additionally, acquiring data should be conducted under less controlled conditions that are closer to their daily living scenarios to generalize the observed challenging behaviours in children with autism throughout the day.

6. Conclusions

The advances in technology can be exploited to help target challenging behaviours among children with ASD. The combination of wearable sensors to detect behaviours and social robots to respond have a great impact on the outcomes of therapy sessions. In this study, we have conducted several investigations using wearable sensors and machine learning techniques to detect challenging behaviours among children with autism. The wearable sensors acquired different physiological and kinetics signals from five children. Annotated video sessions and time-extracted features were considered to evaluate the detection models. Four machine learning techniques were evaluated and the best, based on XGBoost, was considered in further tests. Of all the evaluated machine learning modes, the XGBoost showed better precision (0.88), recall (0.99), F1-Score (0.93), accuracy (0.99), and achieved the lowest (i.e., best) time (i.e., 0.24 s) in predicting the test samples. Features tests were conducted to evaluate the effects of adding each feature to the existing pool of features. In terms of feature importance, heart rate (i.e., importance score = 405), followed by electrodermal activity (i.e., importance score = 265), and temperature (i.e., importance score = 238) were found to be the most affecting features on the performance of the prediction model. Testing the categories of features revealed that physiological-based features provided more useful information to the machine learning model as compared to kinetic features, hence, improving its performance considerably. The heart rate variability changes based on the RMSSD parameter was also derived and investigated. This parameter was found to correlate with challenging behaviours and to be a major contributor to the prediction performance.
The outcomes of this work pave the way toward the development of methods and tools based on machine learning techniques and wearables technologies that can be used to detect challenging behaviours and be integrated into social robot-aided sessions.

Author Contributions

A.Q.A. and M.A. annotated the video sessions and analyzed the data; A.Y.A. and A.Q.A. wrote the first draft; W.-C.S. and O.C. established the annotation protocol; A.Q.A., A.Y.A., A.A.-A. and J.-J.C. produced the figures; A.A.-A., W.-C.S., O.C., U.A.Q. and J.-J.C. oversaw the work and edited the manuscript. All authors have read and agreed to the published version of the manuscript.

Funding

The work was supported by a research grant from QU Marubeni Concept to Prototype Grant under the grant number M-CTP-CENG-2020-4.

Institutional Review Board Statement

The procedures for this work did not include invasive or potentially hazardous methods and were in accordance with the Code of Ethics of the World Medical Association (Declaration of Helsinki).

Informed Consent Statement

Parental consent for each participant was obtained by the centre.

Data Availability Statement

The data presented in this study are available on request from the corresponding author. The statements made herein are solely the responsibility of the authors.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders (DSM-5®); American Psychiatric Pub.: Washington, DC, USA, 2013. [Google Scholar]
  2. Estes, A.; Munson, J.; Dawson, G.; Koehler, E.; Zhou, X.H.; Abbott, R. Parenting stress and psychological functioning among mothers of preschool children with autism and developmental delay. Autism 2009, 13, 375–387. [Google Scholar] [CrossRef] [PubMed]
  3. Matson, J.L.; Gonzalez, M.L.; Rivet, T.T. Reliability of the autism spectrum disorder-behavior problems for children (ASD-BPC). Res. Autism Spectr. Disord. 2008, 2, 696–706. [Google Scholar] [CrossRef]
  4. Murphy, O.; Healy, O.; Leader, G. Risk factors for challenging behaviors among 157 children with autism spectrum disorder in Ireland. Res. Autism Spectr. Disord. 2009, 3, 474–482. [Google Scholar] [CrossRef]
  5. Kanne, S.M.; Mazurek, M.O. Aggression in children and adolescents with ASD: Prevalence and risk factors. J. Autism Dev. Disord. 2011, 41, 926–937. [Google Scholar] [CrossRef]
  6. Baghdadli, A.; Pascal, C.; Grisi, S.; Aussilloux, C. Risk factors for self-injurious behaviours among 222 young children with autistic disorders. J. Intellect. Disabil. Res. 2003, 47, 622–627. [Google Scholar] [CrossRef]
  7. Bodfish, J.W.; Symons, F.J.; Parker, D.E.; Lewis, M.H. Varieties of repetitive behavior in autism: Comparisons to mental retardation. J. Autism Dev. Disord. 2000, 30, 237–243. [Google Scholar] [CrossRef]
  8. Christensen, D.L.; Braun, K.V.N.; Baio, J.; Bilder, D.; Charles, J.; Constantino, J.N.; Daniels, J.; Durkin, M.S.; Fitzgerald, R.T.; Kurzius-Spencer, M.; et al. Prevalence and characteristics of autism spectrum disorder among children aged 8 years—Autism and Developmental Disabilities Monitoring Network, 11 sites, United States, 2012. Morb. Mortal. Wkly. Rep. Surveill. Summ. 2018, 65, 1–23. [Google Scholar] [CrossRef]
  9. Cabibihan, J.J.; Chellali, R.; So, C.W.C.; Aldosari, M.; Connor, O.; Alhaddad, A.Y.; Javed, H. Social Robots and Wearable Sensors for Mitigating Meltdowns in Autism—A Pilot Test. In Social Robotics; Springer International Publishing: Berlin/Heidelberg, Germany, 2018; pp. 103–114. [Google Scholar]
  10. Liu, W.; Yu, X.; Raj, B.; Yi, L.; Zou, X.; Li, M. Efficient autism spectrum disorder prediction with eye movement: A machine learning framework. In Proceedings of the 2015 International Conference on Affective Computing and Intelligent Interaction (ACII), Xi’an, China, 21–24 September 2015; pp. 649–655. [Google Scholar]
  11. Linstead, E.; German, R.; Dixon, D.; Granpeesheh, D.; Novack, M.; Powell, A. An application of neural networks to predicting mastery of learning outcomes in the treatment of autism spectrum disorder. In Proceedings of the 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA), Miami, FL, USA, 9–11 December 2015; pp. 414–418. [Google Scholar]
  12. Matson, J.L.; Nebel-Schwalm, M. Assessing challenging behaviors in children with autism spectrum disorders: A review. Res. Dev. Disabil. 2007, 28, 567–579. [Google Scholar] [CrossRef]
  13. Myers, S.M.; Johnson, C.P.; Council on Children with Disabilities. Management of children with autism spectrum disorders. Pediatrics 2007, 120, 1162–1182. [Google Scholar] [CrossRef] [Green Version]
  14. Blair, K.S.C.; Lee, I.S.; Cho, S.J.; Dunlap, G. Positive Behavior Support Through Family–School Collaboration for Young Children With Autism. Top. Early Child. Spec. Educ. 2010, 31, 22–36. [Google Scholar] [CrossRef]
  15. Neitzel, J. Positive Behavior Supports for Children and Youth with Autism Spectrum Disorders. Prev. Sch. Fail. Altern. Educ. Child. Youth 2010, 54, 247–255. [Google Scholar] [CrossRef]
  16. Ennis-Cole, D.L. Technology for Learners with Autism Spectrum Disorders; Springer: Berlin/Heidelberg, Germany, 2015. [Google Scholar]
  17. Cabibihan, J.J.; Javed, H.; Aldosari, M.; Frazier, T.W.; Elbashir, H. Sensing Technologies for Autism Spectrum Disorder Screening and Intervention. Sensors 2016, 17, 46. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  18. Martinez-Martin, E.; Escalona, F.; Cazorla, M. Socially assistive robots for older adults and people with autism: An overview. Electronics 2020, 9, 367. [Google Scholar] [CrossRef] [Green Version]
  19. Alabdulkareem, A.; Alhakbani, N.; Al-Nafjan, A. A systematic review of research on robot-assisted therapy for children with autism. Sensors 2022, 22, 944. [Google Scholar] [CrossRef] [PubMed]
  20. Cabibihan, J.J.; Javed, H.; Ang, M., Jr.; Aljunied, S.M. Why robots? A survey on the roles and benefits of social robots in the therapy of children with autism. Int. J. Soc. Robot. 2013, 5, 593–618. [Google Scholar] [CrossRef] [Green Version]
  21. Alhaddad, A.Y.; Mecheter, A.; Wadood, M.A.; Alsaari, A.S.; Mohammed, H.; Cabibihan, J.J. Anthropomorphism and its negative attitudes, sociability, animacy, agency, and disturbance requirements for social robots: A pilot study. In Proceedings of the Social Robotics: 13th International Conference, ICSR 2021, Singapore, 10–13 November 2021; Springer: Berlin/Heidelberg, Germany, 2021; pp. 791–796. [Google Scholar]
  22. Katsanis, I.; Alhaddad, A.Y.; Cabibihan, J.J.; Moulianitis, V. Children Perceived Perception of a Mini-Humanoid Social Robot Based on a Psychometric Scale: A Pilot Study in Greece. In Proceedings of the Social Robotics: 14th International Conference, CSR 2022, Florence, Italy, 13–16 December 2022; Springer: Berlin/Heidelberg, Germany, 2023; pp. 13–22. [Google Scholar]
  23. Alhaddad, A.Y.; Javed, H.; Connor, O.; Banire, B.; Thani, D.A.; Cabibihan, J.J. Robotic Trains as an Educational and Therapeutic Tool for Autism Spectrum Disorder Intervention. In Robotics in Education; Springer International Publishing: Berlin/Heidelberg, Germany, 2018; pp. 249–262. [Google Scholar]
  24. Boccanfuso, L.; Barney, E.; Foster, C.; Ahn, Y.A.; Chawarska, K.; Scassellati, B.; Shic, F. Emotional Robot to Examine Differences in Play Patterns and Affective Response of Children with and without ASD. In Proceedings of the Eleventh ACM/IEEE International Conference on Human Robot Interaction, Christchurch, New Zealand, 7–10 March 2016; pp. 19–26. [Google Scholar]
  25. Alhaddad, A.Y.; Cabibihan, J.J.; Bonarini, A. Head Impact Severity Measures for Small Social Robots Thrown during Meltdown in Autism. Int. J. Soc. Robot. 2019, 11, 255–270. [Google Scholar] [CrossRef] [Green Version]
  26. Cabibihan, J.; Javed, H.; Sadasivuni, K.; Al Haddad, A. Smart Robotic Therapeutic Learning Toy. WIPO Patent WO2018033857, 22 February 2018. [Google Scholar]
  27. Alhaddad, A.Y.; Cabibihan, J.J.; Hayek, A.; Bonarini, A. Influence of the shape and mass of a small robot when thrown to a dummy human head. SN Appl. Sci. 2019, 1, 1468. [Google Scholar] [CrossRef] [Green Version]
  28. Alhaddad, A.Y.; Cabibihan, J.J.; Hayek, A.; Bonarini, A. Safety experiments for small robots investigating the potential of soft materials in mitigating the harm to the head due to impacts. SN Appl. Sci. 2019, 1, 476. [Google Scholar] [CrossRef] [Green Version]
  29. Rudovic, O.; Lee, J.; Dai, M.; Schuller, B.; Picard, R.W. Personalized machine learning for robot perception of affect and engagement in autism therapy. Sci. Robot. 2018, 3, eaao6760. [Google Scholar] [CrossRef] [Green Version]
  30. Alhaddad, A.Y.; Cabibihan, J.J.; Bonarini, A. Real-Time Social Robot’s Responses to Undesired Interactions between Children and their Surroundings. Int. J. Soc. Robot. 2022, 1–9. [Google Scholar] [CrossRef]
  31. Alhaddad, A.Y.; Cabibihan, J.J.; Bonarini, A. Influence of Reaction Time in the Emotional Response of a Companion Robot to a Child’s Aggressive Interaction. Int. J. Soc. Robot. 2020, 12, 1279–1291. [Google Scholar] [CrossRef] [Green Version]
  32. Alhaddad, A.Y.; Cabibihan, J.J.; Bonarini, A. Recognition of aggressive interactions of children toward robotic toys. In Proceedings of the 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), New Delhi, India, 14–18 October 2019; pp. 1–8. [Google Scholar]
  33. McDonnell, A.; McCreadie, M.; Mills, R.; Deveau, R.; Anker, R.; Hayden, J. The role of physiological arousal in the management of challenging behaviours in individuals with autistic spectrum disorders. Res. Dev. Disabil. 2015, 36, 311–322. [Google Scholar] [CrossRef] [PubMed]
  34. Jansen, L.M.C.; de Wied, C.C.G.; Wiegant, V.M.; Westenberg, H.G.M.; Lahuis, B.E.; van Engeland, H. Autonomic and Neuroendocrine Responses to a Psychosocial Stressor in Adults with Autistic Spectrum Disorder. J. Autism Dev. Disord. 2006, 36, 891–899. [Google Scholar] [CrossRef] [PubMed]
  35. Alhaddad, A.Y.; Al-Ali, A.; Pandey, A.K.; Cabibihan, J.J. A Framework for Assistive Social Robots for Detecting Aggression in Children. In Proceedings of the Social Robotics: 14th International Conference, ICSR 2022, Florence, Italy, 13–16 December 2022; Springer: Berlin/Heidelberg, Germany, 2023; pp. 74–84. [Google Scholar]
  36. Alban, A.Q.; Ayesh, M.; Alhaddad, A.Y.; Al-Ali, A.K.; So, W.C.; Connor, O.; Cabibihan, J.J. Detection of challenging behaviours of children with autism using wearable sensors during interactions with social robots. In Proceedings of the 2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN), Vancouver, BC, Canada, 8–12 August 2021; pp. 852–857. [Google Scholar]
  37. Alcañiz Raya, M.; Chicchi Giglioli, I.A.; Marín-Morales, J.; Higuera-Trujillo, J.L.; Olmos, E.; Minissi, M.E.; Teruel Garcia, G.; Sirera, M.; Abad, L. Application of Supervised Machine Learning for Behavioral Biomarkers of Autism Spectrum Disorder Based on Electrodermal Activity and Virtual Reality. Front. Hum. Neurosci. 2020, 14, 90. [Google Scholar] [CrossRef] [PubMed]
  38. Masino, A.J.; Forsyth, D.; Nuske, H.; Herrington, J.; Pennington, J.; Kushleyeva, Y.; Bonafide, C.P. M-Health and autism: Recognizing stress and anxiety with machine learning and wearables data. In Proceedings of the 2019 IEEE 32nd International Symposium on Computer-Based Medical Systems (CBMS), Cordoba, Spain, 5–7 June 2019; pp. 714–719. [Google Scholar]
  39. Sarabadani, S.; Schudlo, L.C.; Samadani, A.A.; Kushski, A. Physiological detection of affective states in children with autism spectrum disorder. IEEE Trans. Affect. Comput. 2018, 11, 588–600. [Google Scholar] [CrossRef]
  40. Williams, R.M.; Gilbert, J.E. Perseverations of the academy: A survey of wearable technologies applied to autism intervention. Int. J. Hum.-Comput. Stud. 2020, 143, 102485. [Google Scholar] [CrossRef]
  41. Fazana, F.; Alsadoon, A.; Prasad, P.; Costadopoulos, N.; Elchouemi, A.; Sreedharan, S. Integration of assistive and wearable technology to improve communication, social interaction and health monitoring for children with autism spectrum disorder (ASD). In Proceedings of the 2017 IEEE Region 10 Symposium (TENSYMP), Cochin, India, 14–16 July 2017; pp. 1–5. [Google Scholar]
  42. Goodwin, M.S.; Haghighi, M.; Tang, Q.; Akcakaya, M.; Erdogmus, D.; Intille, S. Moving towards a real-time system for automatically recognizing stereotypical motor movements in individuals on the autism spectrum using wireless accelerometry. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, Seattle, WA, USA, 13–17 September 2014; pp. 861–872. [Google Scholar]
  43. Rad, N.M.; Kia, S.M.; Zarbo, C.; van Laarhoven, T.; Jurman, G.; Venuti, P.; Marchiori, E.; Furlanello, C. Deep learning for automatic stereotypical motor movement detection using wearable sensors in autism spectrum disorders. Signal Process. 2018, 144, 180–191. [Google Scholar]
  44. Plötz, T.; Hammerla, N.Y.; Rozga, A.; Reavis, A.; Call, N.; Abowd, G.D. Automatic assessment of problem behavior in individuals with developmental disabilities. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing, Pittsburgh, PA, USA, 5–8 September 2012; pp. 391–400. [Google Scholar]
  45. Rad, N.M.; Furlanello, C. Applying Deep Learning to Stereotypical Motor Movement Detection in Autism Spectrum Disorders. In Proceedings of the 2016 IEEE 16th International Conference on Data Mining Workshops (ICDMW), Barcelona, Spain, 12–15 December 2016. [Google Scholar]
  46. Pollreisz, D.; TaheriNejad, N. A simple algorithm for emotion recognition, using physiological signals of a smart watch. In Proceedings of the 2017 39th annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Jeju, Republic of Korea, 1–15 July 2017; pp. 2353–2356. [Google Scholar]
  47. Jarraya, S.K.; Masmoudi, M.; Hammami, M. A comparative study of Autistic Children Emotion recognition based on Spatio-Temporal and Deep analysis of facial expressions features during a Meltdown Crisis. Multimed. Tools Appl. 2020, 80, 83–125. [Google Scholar] [CrossRef]
  48. Lee, C.; Yoo, S.; Park, Y.; Kim, N.; Jeong, K.; Lee, B. Using neural network to recognize human emotions from heart rate variability and skin resistance. In Proceedings of the 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference, Shanghai, China, 17–18 January 2006; pp. 5523–5525. [Google Scholar]
  49. Park, H.; Dong, S.Y.; Lee, M.; Youn, I. The role of heart-rate variability parameters in activity recognition and energy-expenditure estimation using wearable sensors. Sensors 2017, 17, 1698. [Google Scholar] [CrossRef] [Green Version]
  50. Yap, C.Y.; Ng, K.H.; Cheah, Y.; Lim, S.Y.; Price, J.; De Vries, M. App4Autism: An Integrated Assistive Technology with Heart Rate Monitoring for Children with Autism. In Proceedings of the International Visual Informatics Conference, Bangi, Malaysia, 19–21 November 2019; Springer: Berlin/Heidelberg, Germany, 2019; pp. 498–512. [Google Scholar]
  51. Lydon, S.; Healy, O.; Dwyer, M. An examination of heart rate during challenging behavior in Autism Spectrum Disorder. J. Dev. Phys. Disabil. 2013, 25, 149–170. [Google Scholar] [CrossRef]
  52. Nuske, H.J.; Finkel, E.; Hedley, D.; Parma, V.; Tomczuk, L.; Pellecchia, M.; Herrington, J.; Marcus, S.C.; Mandell, D.S.; Dissanayake, C. Heart rate increase predicts challenging behavior episodes in preschoolers with autism. Stress 2019, 22, 303–311. [Google Scholar] [CrossRef] [PubMed]
  53. Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; et al. Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res. 2011, 12, 2825–2830. [Google Scholar]
  54. Chen, T.; Guestrin, C. XGBoost: A Scalable Tree Boosting System. In Proceedings of the KDD ’16: The 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, 13–17 August 2016; ACM: New York, NY, USA, 2016; pp. 785–794. [Google Scholar]
  55. Goto, M.; Nagashima, M.; Baba, R.; Nagano, Y.; Yokota, M.; Nishibata, K.; Tsuji, A. Analysis of heart rate variability demonstrates effects of development on vagal modulation of heart rate in healthy children. J. Pediatr. 1997, 130, 725–729. [Google Scholar] [CrossRef] [PubMed]
  56. DiStefano, C.; Shih, W.; Kaiser, A.; Landa, R.; Kasari, C. Communication growth in minimally verbal children with ASD: The importance of interaction. Autism Res. 2016, 9, 1093–1102. [Google Scholar] [CrossRef] [PubMed]
Robotics 12 00055 g001 550
Figure 1. An overview of the adopted methodology in this study. (a) The Empatica E4 wearable device. (b) One of the children interacting with the social robots (see https://youtu.be/sGNslV2Yuks (accessed on 20 March 2023)). (c) A sample of the acquired data using the wearable device.
Figure 1. An overview of the adopted methodology in this study. (a) The Empatica E4 wearable device. (b) One of the children interacting with the social robots (see https://youtu.be/sGNslV2Yuks (accessed on 20 March 2023)). (c) A sample of the acquired data using the wearable device.
Robotics 12 00055 g001
Robotics 12 00055 g002 550
Figure 2. The evaluation metrics results for the three categories using the best performing algorithm (i.e., XGBoost).
Figure 2. The evaluation metrics results for the three categories using the best performing algorithm (i.e., XGBoost).
Robotics 12 00055 g002
Robotics 12 00055 g003 550
Figure 3. The contributing features on the performance of the best prediction algorithm (i.e., XGBoost) for each child. (a) Child 1. (b) Child 2. (c) Child 3. (d) Child 4. (e) Child 5.
Figure 3. The contributing features on the performance of the best prediction algorithm (i.e., XGBoost) for each child. (a) Child 1. (b) Child 2. (c) Child 3. (d) Child 4. (e) Child 5.
Robotics 12 00055 g003
Robotics 12 00055 g004 550
Figure 4. The contributing features on the performance of the best prediction algorithm (i.e., XGBoost) for the combined model.
Figure 4. The contributing features on the performance of the best prediction algorithm (i.e., XGBoost) for the combined model.
Robotics 12 00055 g004
Robotics 12 00055 g005 550
Figure 5. The changes in the HRV (i.e., RMSSD) corresponding to different states. (a) The child is overwhelmed and stimulated by the bubble gun toy. (b) The child is in a rest state. (c) The child experiences a challenging behaviour.
Figure 5. The changes in the HRV (i.e., RMSSD) corresponding to different states. (a) The child is overwhelmed and stimulated by the bubble gun toy. (b) The child is in a rest state. (c) The child experiences a challenging behaviour.
Robotics 12 00055 g005
Robotics 12 00055 g006 550
Figure 6. The importance of HRV in the performance of the machine learning model (i.e., XGBoost). (a) Represents the feature importance for one of the participants whose challenging behaviours were more frequent and intense. (b) The feature importance for another participant who displayed less challenging behaviours.
Figure 6. The importance of HRV in the performance of the machine learning model (i.e., XGBoost). (a) Represents the feature importance for one of the participants whose challenging behaviours were more frequent and intense. (b) The feature importance for another participant who displayed less challenging behaviours.
Robotics 12 00055 g006
Table
Table 1. Some of the common heart rate variability (HRV) parameters that were previously considered in autism research [49].
Table 1. Some of the common heart rate variability (HRV) parameters that were previously considered in autism research [49].
ParametersUnitsDomainDescription
RMSSDmsTimeSquare root of the mean squared differences between successive RR intervals
SDNNmsTimeStandard deviation of NN intervals
LFHzFrequencyPeak in low frequency range (0.04 to 0.15 Hz)
HFHzFrequencyPeak in high frequency range (0.15 to 0.4 Hz)
Table
Table 2. The evaluation metric scores for the four algorithms and their test times (in seconds) needed to evaluate the test samples.
Table 2. The evaluation metric scores for the four algorithms and their test times (in seconds) needed to evaluate the test samples.
PrecisionRecallF1-ScoreAccuracyTesting Time (s)
XGBoost0.880.990.930.990.24
MLP0.670.980.800.970.36
SVM0.240.910.380.852.48
DT0.870.920.890.980.29
Table
Table 3. A summary of the results showing the precision, recall, and F1 scores for the experiments considering the impact of adding each feature to the feature set for the personalised models of each participant and their combined generalised model.
Table 3. A summary of the results showing the precision, recall, and F1 scores for the experiments considering the impact of adding each feature to the feature set for the personalised models of each participant and their combined generalised model.
Set12345
FeatureACCSet 1 + HRSet 2 + BVPSet 3 + TEMPSet 4 + EDA
MetricPrecRecF1PrecRecF1PrecRecF1PrecRecF1PrecRecF1
Child 10.350.870.500.6310.780.6210.770.790.970.870.790.970.87
Child 20.530.890.670.7210.840.710.970.820.770.990.860.9210.96
Child 30.510.750.610.690.900.780.690.890.780.960.990.970.960.990.98
Child 40.370.720.490.750.980.850.750.980.8510.980.9910.980.99
Child 50.260.640.370.430.910.580.580.970.730.5410.700.5710.73
All0.360.660.460.620.860.720.630.860.720.820.980.890.880.990.93
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Alban, A.Q.; Alhaddad, A.Y.; Al-Ali, A.; So, W.-C.; Connor, O.; Ayesh, M.; Ahmed Qidwai, U.; Cabibihan, J.-J. Heart Rate as a Predictor of Challenging Behaviours among Children with Autism from Wearable Sensors in Social Robot Interactions. Robotics 2023, 12, 55. https://doi.org/10.3390/robotics12020055

AMA Style

Alban AQ, Alhaddad AY, Al-Ali A, So W-C, Connor O, Ayesh M, Ahmed Qidwai U, Cabibihan J-J. Heart Rate as a Predictor of Challenging Behaviours among Children with Autism from Wearable Sensors in Social Robot Interactions. Robotics. 2023; 12(2):55. https://doi.org/10.3390/robotics12020055

Chicago/Turabian Style

Alban, Ahmad Qadeib, Ahmad Yaser Alhaddad, Abdulaziz Al-Ali, Wing-Chee So, Olcay Connor, Malek Ayesh, Uvais Ahmed Qidwai, and John-John Cabibihan. 2023. "Heart Rate as a Predictor of Challenging Behaviours among Children with Autism from Wearable Sensors in Social Robot Interactions" Robotics 12, no. 2: 55. https://doi.org/10.3390/robotics12020055

APA Style

Alban, A. Q., Alhaddad, A. Y., Al-Ali, A., So, W. -C., Connor, O., Ayesh, M., Ahmed Qidwai, U., & Cabibihan, J. -J. (2023). Heart Rate as a Predictor of Challenging Behaviours among Children with Autism from Wearable Sensors in Social Robot Interactions. Robotics, 12(2), 55. https://doi.org/10.3390/robotics12020055

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop