Assisted Behavior Control with Wearable Devices: A Case Study for High-Functioning Autism ^†

Abstract

This study aims to analyze the efficacy of wearable and mobile systems to assist people with High-Functioning Autism (HFA) in their emotional self-regulation learning process compared to the proven efficacy of this technology with individuals with autism in the low functioning area of the spectrum. For that purpose, we carry out an experiment with a smartwatch system (Taimun-Watch) that had been tested previously with individuals in the low-functioning range. This experiment involves two (N = 2) individuals with HFA and we compare their performance to the obtained in the prior experiment by monitoring their activity and observing their behavior during 7 and 9 labor days, respectively. The results evidence that, although it takes more time to find and customize effective self-regulation strategies in comparison to the low-functioning autism individuals due to their sharper, more complex cognitive abilities and perception, they were able to use the system to recover from stress episodes as well using the system and tolerated suitably the devices in their daily activity.

Published: 15 November 2019

1. Introduction

Autism as a clinical phenomenon has evolved throughout recent history in terms of classification of symptoms, features and co-morbid disorders [1,2,3,4]. In the early stages of autism research, the description of this phenomenon did not grasp the complex and multifaceted nature of autism: individuals with very different characteristics were classified under the umbrella term of “autism” [5,6,7,8], regardless of their particularities and needs. When assistance and intervention paradigms started to come into the psychopedagogical scene of autism [9], its generality became a problem that hindered the possibility to meet the individual needs in a broad level. Although the majority of cases of autism have such a particular set of characteristics that does not justify a specific nosology, researchers agreed that it required a definition that allowed a broader approach rather than the classic diagnosis of a single disorder. Because of this, the term “autism spectrum disorders” (ASD) was created [6,10]. As a spectrum, it became possible to outline different sets of problems that aimed at characteristics of groups of individuals within the autism perspective, so that new sub-descriptions such as Asperger’s syndrome or High Functioning Autism (HFA) appeared. This channeled the research workforce into more specific approaches, and therefore the intervention strategies and assistive methodologies became more effective [11,12,13].

Assistive technologies also took advantage of this paradigm shift. There are multiple platforms, systems, frameworks and applications that address specific target populations within the autism spectrum. For instance, REACT [14] is an authoring system for HFA individuals to create instructional support for enhancing social skills, and El-Seoud [15] proposes a mobile application to implement the PECS [16] communication system into a digital platform for individuals with autism and severe communicative problems. Mintz [17] focuses on the use of smartphone technology on those ASD individuals who are ready to incorporate into a mainstream education class. Specific populations demand specific solutions.

However, a different approach on assistive technologies for ASD considers customization and adaptability as the main goal. Even if a system is designed to meet the needs of a specific target population within the autism spectrum, its content and functionality still has to adapt to the individual necessities [18]. Due to this, an increasing number of technological proposals to assist ASD individuals (and with other cognitive impairments) rely on the idea of becoming as flexible and customizable as possible, instead of targeting specific populations. This way, these systems pretend to be suitable for the individual as well as for other subgroups of the spectrum. Due to this phenomenon, there is a corpus of tools and platforms that target autism spectrum disorders, in general, whose content can be integrally crafted by the stakeholders of the person with ASD. Thus, we have SWARM [19], a wearable affective system focused on modular actuation components to adapt to each user’s capabilities, the work of Sano [20] about a universal annotation tool to track the people of different autism disorders within the spectrum, or the use of wearable cameras by Marcu [21], that focuses on obtaining data from the parent in order to adapt the assistance provided to the individual afterwards.

Despite the aforementioned benefits of this holistic approach, when it comes to evaluate the effectiveness of a system designed under this philosophy, a partial test in terms of subjects does not ensure its global success. In other words, the more ambitious is the target of the system, the more types of subjects have to take participation in the study that claims its effectiveness.

Following this idea, this paper presents a user experience of a smartwatch-based system that had been already tested with individuals in the low functioning area of the autism spectrum [22,23]. The purpose of this study is to check whether it is feasible for HFA individuals as well and analyze the kind of differences we may expect when it comes to apply wearable assistance to these individuals’ issues in terms of interaction with the devices, content presentation and performance.

In the previous experiment [22], the system had been evaluated with 2 subjects who belonged to the low-functioning area of the spectrum, and its conclusions only applied to that subgroup. However, since its design intended to cover most of the spectrum in terms of possibilities and customization, evaluating its effect with subjects outside the low-functioning autism area also became necessary.

This system is called Taimun-Watch, and consists of a smartwatch-based application that aims to help individuals with ASD to learn emotional self-regulation strategies using the wearable sensors of heart rate to infer stress episodes (which some studies state to be a decent stress marker for these purposes [24]) and provide previously crafted instructions to regain calm. Educators create the sets of instructions, which are arranged into strategies and sent to the smartwatches via device synchronization. These strategies can be composed of images, animations, text and pictograms organized into sequences than can be navigated step-by-step by touching or swiping on the screen (see Figure 1). This data model has been designed in order to allow the implementation of most of the evidence-based practices about visual support on the literature [9,11,25,26], since these practices share a common formal structure, which can be summarized as “sequences of instructions composed of alternative and augmentative communication resources”. In this respect, we did not find relevant methodological differences regarding HFA compared to rest of the autism spectrum. Hence, we relied on the same kind of paradigm for intervention strategies and made no changes to the core functionality of the system.

In the previous publication about Taimun-Watch, we explained the design principles that led us to choose commercial smartwatches among other wearables to implement this system. The smartphone side of the application is an authoring tool to create the strategies and customize them in order to meet the user’s needs: timers, interaction, text format, backups, and content arrangement. In a more recent version, data visualization and multi-user functionality were added to the system.

2. Materials and Methods

In the original experiment [22], 2 subjects (N = 2), namely subject A and subject B wore the smartwatch with Taimun Watch during 9 days in the center “Alenta” in Madrid, Spain. Both subjects belonged to the lower-functioning area of the spectrum, that is to say, their communicative abilities were severely impaired or non-existent, and they suffered numerous outbursts and meltdowns during the day due to several stimuli. Each smartwatch contained customized strategies for their emotional self-regulation in stress episodes and outbursts. These strategies consisted of simple sequences of instructions through visual support that told them what to do when they felt stressed or anxious. This assistance was shown on the smartwatch’s screen when the smartwatch sensor’s detected a significant peak in their heart rate signal, which was monitored as long as they wore the smartwatch. The results were satisfying, as both subjects were able to follow the strategies by themselves yet on the first half of the study. When this happened, their heart rate signals returned to average values, and their affective state improved substantially, as it was showed by both the observer’s notes and the logs from Taimun-Watch. In this paper, we replicated the study under mostly identical conditions and a similar methodology, in order to ascertain whether this technology also works for subjects in the high functioning area of the spectrum.

The experiment was carried out between June and July 2017 in the center “Fundación Juan XXIII Roncalli” of individuals with special needs. It is a private and non-profit entity dedicated to the labor and social inclusion of people with cognitive impairment. This center has resources to encourage working, creative and domestic activities of its students. The experiment lasted 4 weeks, of which the subjects used the system in 2 days/week due to the observer and educators’ availability and the scheduling of the classroom activities that were interesting in terms of emotional self-regulation.

Compared to the original experiment with low-functional subjects, some aspects of this experiments were managed differently due to the nature and complexity of the autistic features of these subjects. In this case, sessions were not supervised by the researchers directly, since the experts argued that these subjects are more sensitive to intrusiveness: the environmental perception of individuals with HFA is sharper and the influence of unknown people in their space during the experiment could have an impact in the results. Due to this, the observers had a different profile: they were selected from within a group of psychology students specialized on structured behavior record that were doing their internship in the center. They had expertise on experiment observation and had had previous interaction with the subjects of the experiment.

Thus, analysis was performed after the experiments, and the research team received the observation notes and the smartwatch and smartphone logs from the educators and observers. Although researchers did not take part in the very record of the behavior of the subjects, these conditions enhance the verisimilitude of the experiment, which reproduces a case in which the system is used in a strictly realistic context.

2.1. Subject Profiles

The experiment was carried out with 2 users (N = 2), hence subject C and subject D (following the nomenclature used in the experiment described in [22] with subjects A and B, since there will be comparisons between their respective performances). Both of them belong to the high functioning zone of the autism spectrum. This means that their communication capabilities are elevated and nearly neurotypical. Also, their executive dysfunction has a lesser impact on their social skills. The emotional expression and behavioral manifestation of frustration, anger, sadness or stress are also similar to the ones associated to neurotypical individuals. Therefore, there is no difficulty on inferring the inner state of the subject, unless the features of his ASD affect directly to these expressions of emotion. Their affect manifestations related standardly to their physiological signals, so the affective opacity that characterized subjects A and B did not jeopardize the extraction of conclusions in this case.

The activities in this center tackled varied competences that were not only oriented to enhance labor aspects of the students but also other fundamental aspects of their lives in order to encourage self-determination and autonomy. Hence language, creativity and social interaction are main goals of their formative agenda. Additionally, these features are issued following several strategies that are adapted to each student’s capabilities. They followed a role-play or mutual mentorship education model, that is to say, the very students were arranged into work teams in order to complete small creative projects or academic tasks that involved transversally the several subject areas that each group was covering. Given that each group included individuals with different disabilities such as motor, communicative, learning and development disorders, these roles or tasks were carefully assigned so that each individual were able to get the most out of the group activity. As for subjects C and D, whose disabilities fell into the high zone of the autism spectrum, they were able to act as leaders, mentors or supporters of the other members of their groups.

2.2. Materials

Regarding hardware, we used the same smartwatches as in the original experiment, LG Watch Urbane. These devices have 3 9-axial kinetic sensors (3-axial accelerometer, 3-axial gyroscope and 3-axial compass), as well as PPG (heart-rate monitor) and barometer. The smartphone used for editing the strategies was a Nexus 5. Both devices were linked and paired accordingly through the Android Wear interface. We kept the same technical decisions regarding heart rate threshold and stress inference as in the original experiment as well. However, regarding the strategy configuration, we acted differently. Educators, as well as many studies in the literature [16,27], argued that the customization of the visual support plays a main role in the assistance of individuals with HFA. When it comes to ASD in the low-functioning area of the spectrum, distractors had this importance, because stressing stimuli tend to be elements that are unavoidably part of their environment; however, concerning HFA, the stimuli that jeopardize their affective state are subtle, so the assistance should be flexible enough to adapt rapidly and accordingly to the changes that may take place in the set of affective reactions of HFA individuals. Due to this, one strategy per user for the whole experiment did not seem a suitable decision. Educators agreed on creating these strategies along the way from the smartphones, then they would synchronize themselves the strategies with the smartwatches. This way they would be able to check the effect of the strategies on their students and, if the effect was not the expected one, they would be free to change it again. This experiment protocol actually resembles a real use case of the system.

2.3. Methodology

The experiment was carried out during the whole class time of the subjects, from 10 a.m. to 2 p.m., adapting ourselves to their availability, since some activities took place outside the classroom (workshops, gym, etc.). Subject C used the system for 7 days, and subject D used it for 9 days. As we did in the original experiment, heart rate thresholds were adjusted empirically after running the system for three days prior to the experiment.

This way, the daily process of data collection and system launching was carried out by the very subjects and their educators, a simple manual was provided so everything was made accordingly. It contained a few instructions for the system configuration so that the telephone was paired with both smartwatches and a user tutorial for the authoring tool to create, edit and synchronize strategies that would appear on the subjects’ smartwatches when their heart rate exceeded the aforementioned threshold. Every day they would launch the system in the smartwatches before the first class and shut it down, get the data, run a backup copy and put the devices to charge before going back to home so they were ready for the following day. During the sessions, they were given a template to write down the observations in a structured manner and were told to change the strategies freely in mid-session if they considered so.

3. Results

We followed the same qualitative analysis approach as we did in the original experiment. In this case, there is not a fixed structure for each subject’s schedule, and they engaged in varied activities throughout the morning: in the observation notes we can find the activities that may be relevant for the results. The vibration was activated in both smartwatches.

In Appendix A we can see the heart rate and behavior log of subject C, and Appendix B contains the same data from subject D.

Table 1. System activations per subject.

Subject	Activations	Successful	Assisted	Interaction Not Possible	High Intensity
C	66	21	16	10	19
D	38	13	9	4	12

shows the number of times the system activated for each subject on the smartwatches during the sessions. In these sessions we will analyze the relationship between the tracked behavior of the subjects, the activations of the system and the reasons behind them in comparison to the results obtained from subjects A and B in the original experiment.

In the graphics of the Appendices we can see the heart rate signals with more noticeable increases and decreases than the ones of the original experiment, and it happens both in the broad scope and in the detailed framing of the signal, if we selected reduced intervals of time. This is due to the ability of these subjects to be more engaged in physical activity, which raises their heart rate easily and results in a higher variability of the signal. In other words, their heart rate signals are more similar to the ones obtained from neurotypical subjects, if we did so. Even so, the value of their signals tends to be more stable and stay within a lower range of values than subject A and B’s. This phenomenon highlights the peaks of the signal and makes them easily associable to the behavior log events that are shown in the notes.

The affective state of subject C tended to destabilize due to conflicts with his classmates. During days 1, 3 and 5 (Appendix A) observers noticed some problematic behaviors that hindered his ability to complete normally his assigned tasks. The content of his strategies was created and edited from his educator’s smartphone, who made changes on days 3, 4 and 5. The two first days he ignored the smartwatch and they decide to change the strategy on day 3, but the subject shows difficulty to understand and follow the new one, so it had to be changed once again. This caused confusion in the subject, moreover when the system activates when he starting to wear the smartwatch on his wrist and its sensor is still calibrating. After this moment, a stress episode starts, and it causes the subject to ignore the watch during the following ~15 min. This happens again on day 4, with another change of strategy. In this case, the subject shows interest on the new strategy but ends up ignoring the smartwatch again when it activates. On day 5, the educators manage to create a strategy that actually works with the subject and makes him complete the sequence of steps and follow the instructions. In these cases, subject C regained a state of calm and recovered his ability to work with his classmates and incorporate into the flow of activities of the center. This behavior remained until the last day of the experiment, where the subject had to perform physical activity, which causes his heart rate to raise and more smartwatch activations to happen. Thus, the system was truly effective in those cases where the subject payed attention to the smartwatch when it got activated by his heart rate signal, and this attention depended entirely on the content of the strategies that were shown on its screen. The interactive workload and the intensity of his stress episodes did not seem to jeopardize the human-computer functioning of the system. The amount of autonomy that seemed to be achievable by subject C by means of the system is high as long as the content is carefully crafted for him and keep him engaged. It would be interesting to analyze how long a strategy remains valid for a certain subject, that is to say, the number of times that a subject would repeat the strategy every time it activates in his smartwatch before he gets bored and ignores it. Educators and experts of the center argued that reaching the point in which a certain emotional-self-regulation strategy becomes tedious, we would be in a situation where the subject has completed it so many times that its sequence of steps has been assimilated and integrated into the subject’s daily routine, so abandonment would not be necessarily dangerous in this scenario but a natural consequence of the learning process.

Quantitatively, subject C received 66 system activations of their self-regulation strategies (see Table 1). On the first days, in which he tended to ignore the device or did not understand the strategies, was assisted by the educators to complete the sequence of steps. There were 16 assisted activations. On the days that has been described as more conflictive for the subject, we registered 19 activations that belong to occasions where his stress and frustration levels were so high that the subject refused any interaction with other people or the device. Of the 31 remaining activations, 10 happened during physical activities (therefore, false positives). Finally, we have 21 activations where the subject was able to perform the strategies completely, without help, and recovering from a stress episode, as the lowering of his heart rate shows.

Subject D suffered behavioral problems related to the presence of stimuli that provoked changes in his environment: unknown people around, visits, strong noises, etc. He was also prone to anger episodes when a classmate refused to give him a hug or argued with him, or even when he perceived conflict between people in his environment.

In the observers’ notes we did not find any strategy change, so we assume that the same one was used along the whole experiment. There were similar phenomena as with subject C: physical activity and device calibration provoked some false positives. Furthermore, on day 2 (Appendix B) we can see several system activations that seem to be false positives but, however, increase gradually along the day. Educators and observers ended up finding out that they are due to his reaction to the music that was being played in the classroom that day. Although excitement tends to classified as a positive reaction to external stimuli in neurotypical subjects, actually its detection and regulation is also necessary, since uncontrolled episodes of joy or euphoria have a high risk of ending in a meltdown that lead to further episodes of sadness or frustration when the exciting stimuli disappears.

Although in the first half of the experiment subject D tended to ignore the smartwatch, on days 7, 8 and 9 he started perceiving and noticing the watch alerts and completed the strategies, recovering the calm and showing lower heart rate values afterwards. Additionally, on day 7, the subject performed a self-evaluation of his inner state when the smartwatch activated. The educators pointed out that this was particularly valuable, since having an individual with ASD read his own affective state entails a significant contribution that mitigates the difficulties of the subject derived from the alexithymia, inherent to the very autism disorder. Regarding autonomy, this individual did not have any problem to understand the content that was showed to him in the smartwatch, so it seems feasible for an autonomous use; additionally, he did not require mid-experiment changes to his strategies, which would have required the intervention of his educators, although this might be an arbitrary case in which the first strategy just turned out to be the optimal for him.

In total, subject D received 38 system activations (see Table 1). Educators assisted 9 of them, although the reported that they were due to false positives, not misunderstanding of the system’s functioning. There were 12 activations that belong to episodes of highly intense stress or anger. However, there were 13 activations that led to successful, effective self-regulation strategies. After these activations, subject D recovered his state of calm and was ready to continue his activities. This subject did not perform a lot of physical activity that might have interfered with the experiment: we only found 4 false positives due to heart rate increased by physical activity.

4. Discussion

The results for both subjects, in comparison to the ones obtained in the original experiment were similar in terms of efficacy and interaction. The subjects were able to interact with the smartwatch, and they understood the interactive paradigm underlying: vibration and screen light meant that the strategy was launched, they navigated through screen touches and swipes throughout the steps of the strategies and did not get distracted by the fact of wearing it, nor their classmates noticed anything in them that could lead to stigmatization or differentiation. In terms of learning, the subjects lasted less time to use the system without help that the subjects of the prior experiment, but tended to ignore the device’s alerts on more occasions. Vibrations and the screen turning on were not enough sometimes to catch the subject’s attention even though they were aware of the activation.

Thus, from the technological, device-wise point of view, wearables are feasible and effective for providing systematic assistance for issues like behavior control and emotional self-regulation. We did not observe any difference on the way subjects with HFA interacted with this technology: they needed 2–3 days of previous learning, but after that they managed to navigate without help through the interactive guidance to regain calm and we observed how their heart rate went back to normal, calm-related levels, which allowed them to continue the daily set of activities in the special education center.

Regarding instructions, strategies and the representation of the information on the devices, they understood the pictograms and the images that represented the steps for their behavior control. However, more design work was needed to obtain helpful strategies, in comparison to the subjects of the previous experiments. Their tendency to ignore the device when it got activated is related to this phenomenon: due to their higher cognitive capacity, they remembered better the content of the strategies, and that made them less interested on watching them again on the device’s screen. From the technological side, this leads us to recommend either implement more varied notification mechanisms to catch the user’s attention; however, from the psychopedagogical perspective, it is advisable to look after the design of the visual support in order to avoid monotony and boredom. This is a crucial difference towards low-functioning autism, since mechanic and systematic support worked better in their case. Certainly, the tools for content creation (in the case of this study, a smartphone with an authoring tool) must provide the stakeholders with maximum customization capacity in order to be able to implement these sorts of support.

In this same direction, we observed that the affective state of HFA subjects was prone to be altered by circumstances that were not strictly confined to the context of the educational center nor the very day on which the subject was using the system: they more sensitive on certain days because of family problems or because they remembered something bad that happened some days ago. This did not happen with low-functioning autism subjects because their cognitive framing is more reduced and their stimuli, generally, fall within a more local scope: events that happen during the day they are on and in the place they are. This is, again, due to the cognitive level difference between these individuals under the spectrum. Thus, strategies should also take into account the context of the HFA individuals in a broader sense, so that the support also addresses the individual’s problems and appeal to the aspects in which they have to work on in order to improve their ability to emotionally self-regulate themselves. Hence, the ability to customize and individualize the strategies and visual support becomes even more important, and wearable systems ought to be able to provide the necessary functionality for that purpose.