Effect of Sensory Feedback on Turn-Taking Using Paired Devices for Children with ASD

Abstract

Most children can naturally engage in play and by this, develop skills while interacting with their peers and toys. However, children with Autism Spectrum Disorder (ASD) often show impairments in play skills which result in limited opportunities for interaction with others and the learning of social skills. In this regard, robotic devices/toys that can provide simple and attractive indications are advantageous to engage children with ASD in play activities that require social and interaction skills. This project proposes a new interaction method using paired robotic devices called COLOLO to facilitate a fundamental exchange of intention in communication so-called turn-taking. These tangible devices are designed to sense the user’s manipulation, send a message to the paired device, and display visual cues for assisting children to achieve turn-taking through play. On the sessions with COLOLO there are two devices, one held by the therapist and one by the child, and they take turns to manipulate the toys and change their colors. In this article, two experimental conditions or interaction rules: the “two-sided lighting rule” and the “one-sided lighting rule" were introduced. The two interactions rules differ from each on the way the devices used the visual cues to indicate the turn-holder. The effect of each interaction rule on children’s turn-taking behaviors was investigated through an experimental study with four children with ASD. From the results, we found that with the one-sided lighting rule participants tended to shift their gaze more and to decrease the failed attempts of turn-taking. The discussion covers the possibilities of using paired devices to describe participants’ behaviors related to turn-taking quantitatively.

1. Introduction

Play is one of the ways children learn and develop skills to engage and interact with the world around them [1,2]. However, children diagnosed with Autism Spectrum Disorder (ASD) have shown impairments in play skills [3,4,5]. They are known to have difficulties in understanding social cues that are necessary to cooperate and interact with others during play, and usually opt to isolate themselves from others. This impairment is reflected in their motor skills, in the way they manipulate toys, and in social and pretend play [6,7].

Different studies have made efforts to develop information and communications technology (ICT)-based solutions for interventions for children with ASD [8]. Specifically, robotic toys can potentially be used as social mediators, engaging children with ASD in social play styles that involve others [9]. As opposed to the use of only social cues, toys with sensory feedback, like flashing lights or sounds, might elicit more play behaviors [10]. Robotic toys can be defined as tangible devices with embedded sensors and actuators that can perceive certain aspects of the environment, such as the behavior of children, and automatically react to them.

In general, robot-based interventions are carefully designed to elicit, facilitate, or train a particular behavior, by adapting the hardware and interaction rules according to the goal of the intervention. Among the different interventions designed to address social skills, some specifically target the training of turn-taking skills. Turn-taking is a back and forward exchange between two or more individuals, and it is considered a fundamental skill for different aspects of social interaction such as being patient, being able to recognize others’ intentions, and collaborating with others [11]. Through turn-taking children are able for example, to play and let others play or to talk and listen others talking in a conversation. Considering that children with ASD has significant impairments in turn-taking behaviors, different approaches explore improving this condition [12,13,14]. Moreover, studies discussed the possibility of reducing the severity of ASD and impairments in social communication, by training and improving turn-taking behaviors [15,16]. To achieve turn-taking, individuals have to be able to switch the initiative during interaction spontaneously. In conventional turn-taking interventions, the therapist supports this back and forward structure by responding to the child in a specific way according to the rules of the activity [13]. Robotic toys can be used as tools for the therapist to: (1) Provide cues that engage children in the activity, and (2) Describe children’s behavior during the activity.

1.1. Robotic Toys to Engage Children in Interaction

The use of robotic toys to facilitate turn-taking training has been explored, and positive effects have been observed [17,18,19,20]. Robotic toys can respond to children’s behavior and provide repetitive and engaging stimuli to guide children through turn-taking play. These studies share a common aspect, which is having a single robot/robotic toy interacting with the child. The role of the robot is to mediate the interaction between the child and others [17,18], the robot taking turns with a child to chase each other [19], or the robot as the child’s partner in an activity mediated by the therapist [20].

However, one robot can provide contingent feedback to one person at a time. On the other hand, a different approach using more than one robotic devices to mediate turn-taking between the child and the therapist has been proposed. Multiple robotic devices can deliver contingent feedback due to the actions from both, the child and the therapist. Moreover, guiding the participant’s attention to different locations/objects can be easily achieved with multiple devices. These characteristics are advantageous in terms of facilitating turn-taking during play activities. A multi-agent platform with simple visual was developed cues in the form of blinking lights in a turn-taking intervention [21,22]. They assigned meaning to the colors (e.g., blue blocks means water) and simulated the interaction among different blocks (e.g., when a block representing an animal is thirsty, it has to be placed next to the water block. The changes of states are represented by blinking frequency or intensity). To be able to play with them, the participant should be able to understand the abstracted meaning of the color patterns and the relationship between them. The results showed that children were able to understand metaphoric meanings through abstract and simple geometrical objects, and the interactive toys were appealing for the participants. However, it was reported as well that the user group should have the ability to understand metaphoric meaning in order to understand the game. Considering that children with ASD have impairments in understanding metaphors and figurative language [23,24], the population of children who can participate in the game is limited. Additionally, the turn-taking behavior of participants was investigated on the context of the designed game scenario; hence the effect of sensory feedback policy on turn-taking behavior independent from the play context has not been discussed.

1.2. Robotic Toys to Describe Children’s Behaviors

Considering that the treatment of autism is adapted to each child’s needs, it is important then to evaluate the interventions objectively. Traditionally, the evaluation is performed based on observations and post analysis by video coding, with which the therapist identifies those relevant behaviors according to the purpose of the intervention [12,13]. This is not only a demanding task, but it can also lead to different conclusions based on the therapist’s criteria. In this context, technology can work to provide therapists with quantitative data related to the child’s performance during the intervention [25]. This can potentially facilitate the adaptation of the activity to stimulate the child’s learning process adequately.

In this field, studies have explored the use of computer vision approaches to automatically collect and describe children’s behavior, thereby reducing the human effort required in analyzing the interventions [26]. Computer vision was used to analyze and assess autism, and it was used a robot to engage with children in face to face interactive scenarios [27]. However, solutions using computer vision have limitations such as higher costs, occlusion, and fixed setups. To overcome them, different studies explored the use of wearable devices to describe children’s behavior. A combination of three accelerometers has been explored to detect stereotypical movements related to ASD [28]. Wearable devices were used to detect children’s smiles as an indicator of enjoyment and positive disposition [29], or to sense the orientation of the facial region, to quantitatively understand children’s focus of attention [30]. A framework that combines a robot as engaging element together with wearable sensors and computer vision was used to analyze children performance during the therapy, and based on this personalize the content of the lessons [31].

One of the limitations of the use of wearable devices is the dependence on the child’s disposition to wear them. Moreover, wearing something unusual might have an additional effect on their behavior. This is one of the motivations for the approaches using sensors embedded in familiar objects and toys. Robotic toys can follow the child’s performance based on the way in which the device is manipulated during play. One study used embedded sensors in rattles to identify early signs of autism [32]. Blocks-like toys were designed to detect developmental delays from patterns of motion while manipulating the toys, time to accomplish a task and accuracy [33]. Toys made with smart textiles were designed to sense and monitor child-toy interaction during playtime [34]. These works identified the benefits of the information extracted from the manipulated toys and proposed alternatives to analyze the sensor’s data for describing children’s behavior.

1.3. Proposal and Contribution

To facilitate turn-taking on children with ASD, we proposed a system using paired devices COLOLO [35,36]. This system was used to facilitate a fundamental exchange of the intention to communicate such as turn-taking. Following an approach similar to that of [22], we aim to use robotic toys with visual cues, using colored lights to engage children in turn-taking activities. However, in this study we target individuals with low-functioning autism, and in order to investigate the context-independent effect of visual stimuli on the turn-taking of children with ASD, the interaction rules of the robotic toys were simplified and used to indicate the exchange of turns and represented using paired devices.

In our previous study, the effect of the intervention with COLOLO on children’s play and social behavior has already been investigated using a psychological approach [37]. We compared the effect of COLOLO’s visual feedback condition against no-feedback condition and found that visual feedback was related with an increased number of contact with toys and an increased number of gaze shifting toward the therapist’s toy. However, in the previous study no investigation was done regarding the effect of different COLOLO interaction rules on play/social behavior of children and its potential for the facilitation of turn-taking. Moreover, the use of COLOLO as a measurement device to describe the behavior of children automatically and quantitatively during the intervention has not been discussed.

In this study, we, therefore, propose two different interaction rules for the intervention using COLOLO, the “Two-sided lighting rule” and the “One-sided lighting rule”, in order to clarify the effect of different sensory feedback methods (i.e., interaction rules) on children’s behavior. These rules are implemented based on the results of a pilot experiment with a previous interaction rule of COLOLO [35] to overcome its limitation and be beneficial for facilitating turn-taking behavior. Thus, the purposes of this study are: (1) To evaluate how the devices influence turn-taking behaviors of children by comparing the effects of the two different interaction rules; and (2) To explore the potential for and the limitations of describing children’s play behavior in turn-taking interventions using quantitative data from the robotic devices. To investigate this, an experimental study was performed with a therapist and children with low-functioning autism to compare the data obtained from the human analysis with the data obtained from COLOLO. Using this data, the effect of the two rules was explored by quantifying the behavior of children related to turn-taking.

The contributions of this work are as follows:

• The introduction of different interaction rules using paired devices for turn-taking.
• Investigation of the effect of different interaction rules using paired devices on the turn-taking behavior of children with ASD.
• Investigation on the capacity and viability of using COLOLO for automated quantitative measurement of children’s and therapist behavior during therapy sessions.

2. Methodology

2.1. COLOLO: Paired Robotic Devices

COLOLO is a spherical robotic device developed for enhancing remote communication by conveying abstract messages using colored lights (Figure 1 (Left) [38]). As shown in Figure 1 (Center), each COLOLO device is a 10 cm diameter spherical case that contains: a tilt sensor using a ball switch, an XBee WiFi module, full-color LEDs, an electric motor, and a microcontroller. A counter mass is attached to the rolling axis of the motor, enabling COLOLO to “wiggle” when the motor is powered. When a user makes contact with a COLOLO device and it is detected by the tilt sensor (manipulation), the manipulated COLOLO sends a message to an external server through wireless communication. Then, the external server transfers the message to the paired COLOLO and it is represented by colored lights and wiggling movements. The message consists of the ID of the sender with its LED color, and all messages are recorded with time stamps as shown in Figure 1 (Right).

COLOLO holds several advantages when used as an assistive device for children with ASD. The spherical shape with no sharp corners provides a safe, soft, and friendly appearance that is advantageous when the device is being manipulated [39]. Studies on activities with children with ASD using a ball-shaped robot [40] showed the potential of these robotic toys as an attractive element in the intervention. In addition, the visual cues in the form of colored lights and wiggling movements stimulate the child, and the ability to communicate with the paired device provides a simple way to include others in the game. Based on these considerations, we used COLOLO devices for the intervention aimed at facilitating the exchange of intention in play activities that involve turn-taking.

2.2. COLOLO in Turn-Taking Interventions

Turn-taking is an important and fundamental skill that involves the spontaneous exchange of initiative during social interaction. Different approaches particularly targeted training turn-taking skills and using robotic toys for this purpose has shown positive effects [17,18,20,22]. Robotic toys can respond to the child’s actions with engaging and constant stimuli, which is beneficial to guide the child during a game that involves turn-taking.

Following this, this study explores the use of paired robotic toys COLOLO in facilitating turn-taking. On the sessions with COLOLO, there are two devices/modules, one held by the therapist and one by the child, and they take turns to manipulate the toys and change their colors. The change of color indicates the event of “taking a turn”, and the color itself has no specific meaning. Manipulation of the toys includes any action that activates the tilt sensor inside the device, including shaking, kicking it, rolling it, among others. When the user manipulates the device during the user’s turn, the device sends a message to the paired device and “passes the turn”. The therapist uses these two devices to assess and train turn-taking skills, based on the timing the child manipulates the toy. The child must learn when to interact with the ball by realizing when it is his/her turn aided by the cues delivered from the devices. The way these cues are delivered is defined by interaction rules.

2.3. Interaction Rules and Hypotheses

The interaction rules of COLOLO are illustrated on Figure 2. The leftmost of Figure 2 represents the previous COLOLO interaction rule in which both devices always respond when manipulated (indicated by black arrows) and provide feedback through the change of light color and movements. However, this rule does not include the concept of a “turn-holder", and it was used only for the pilot experiment (Section 3.2).

Figure 2A represents the “Two-sided lighting rule” in which the feedback with colored lights is always enabled for both devices and indicates the turn-holder through wiggling movements. A “turn” or “turn holder” refers to the device that is enabled to send a message to the paired device. When a device is manipulated while it is holding the turn, the device sends a message to the server to provide the associated visual feedback, and then the other device will hold the turn for the next manipulation. On the other hand, when the device that is not holding the turn is manipulated, the devices will not provide any feedback, and the manipulation will be recorded as a “failed attempt”. Since LEDs of both devices are always enabled, this rule would be engaging to children with ASD who are known to be sensitives as well as positively responsive to visual stimuli.

Figure 2B represents the “One-sided lighting rule” in which the colored-lights feedback is disabled (i.e., “off”) while the device is not holding the turn. In this rule, the device that is holding the turn is identifiable by lights and wiggling movements. Removing the feedback made by lights from the receiving device might be beneficial to indicate the turn-holder and reduce distractions.

On both rules, the turn-holder is indicated by the wiggling device, and the changes of turns are indicated by changes of colors. The difference between these two rules is the role of the feedback made by lights, as it can be used to also indicate the turn-holder or not. Based on the considerations above, we hypothesized the relationships between the two interaction rules and children’s behaviors related to turn-taking as follows:

Hypothesis 1 (H1).

Lighting up both devices (lights do not indicate the turn holder), will elicit a higher number of manipulations because of its relatively higher amount of visual stimuli.

Hypothesis 2 (H2).

Lighting up one device (lights indicate the turn-holder) will reduce the number of the child’s manipulations during the therapist’s turn (i.e., reduce the number of failed attempts of turn-taking).

Hypothesis 3 (H3).

Lighting up one device (lights indicate the turn-holder) will lead to a higher occurrence of turn-taking by the child.

2.4. Turn-Taking Behavior Analysis Using Data from COLOLO

On the interventions with COLOLO, turn-taking is described based on the manipulations of the toys and timing. Using COLOLO, the history of messages exchanged between the participant and the therapist can be recorded as shown in Figure 1 (right), to describe and analyze the participant’s behavior during a turn-taking intervention. Each message contains the following information:

• Time: the time a message from any of the toys is received by the server.
• Label: “RECEIVE” indicates that the tilt sensor in a device detected a manipulation and then the server received a notification. If the manipulation was made during its turn, a new line will be added in the log with the label of “CHANGE”, and the server then forwards the message to the paired device.
• ID: the identification number of the sender device. There is a number for the therapist’s device and for the participant’s device.
• Color: the information related to the light color of the sender.

Figure 3 illustrates an example of how COLOLO data are visualized. Each dot represents a manipulation detected by the device. Turn-taking, on the other hand, is a combination of different behaviors (e.g., taking actions, impulse control, and gaze orientation among others) and it cannot be represented using only the data from COLOLO. However, since the turn-taking behavior is observed as a time series of manipulations, the temporal feature such as the interval between manipulations ($\Delta t$) can be used to parametrize the turn-taking in the COLOLO data. Turn-taking described by the data from COLOLO will, therefore, be constituted by the messages exchanged within the time interval ($\Delta t$) between the paired devices. Considering that the sessions using COLOLO involve the therapist and the child exchanging turns to change the color of the devices, it is essential to select an appropriated time interval that allows the system to exclude those exchanges that less likely will be turn-taking.

3. Evaluation

In this section, a pilot study conducted for establishing the two-sided lighting rule and one-sided lighting rule is first explained, followed by the experimental study for verifying the hypotheses regarding the effect of sensory feedback using the two proposed rules on children’s behaviors.

3.1. Experimental Setup

Experiments were held at the playground in Keio University, which consists of a carpeted room with a two-way mirror behind which the parents could observe the sessions. In the sessions with COLOLO, the therapist and the child played in the center of the room most of the time, and the session was video-recorded using two cameras. The data from COLOLO were collected by a computer placed in a separated room. In addition to the therapist that facilitated the intervention with COLOLO, two other observers analyzed the sessions. Considering that the visual feedback from the devices was difficult to see in a fully illuminated room, the room’s main lights were switched off, and two lamps were placed in two corners of the room.

At the beginning of every session, the therapist introduced COLOLO and showed the child how to play with them. The child was then given the opportunity to initiate playing with COLOLO. The game consisted of taking turns to change the color of the balls by manipulating them. Since COLOLO looks like a ball, children manipulated it as one (rolling it on the floor, shaking it, throwing it or pushing it against the other). When the child intended to manipulate COLOLO during his turn, he was praised by the therapist (e.g., exclamations such as “great!”). When the child was playing with his device during the therapist’s turn, the therapist gave indications to guide his attention back to the activity (e.g., “look at me, here I go!”). When the child lost complete interest in the toys, the therapist attempted to capture his interest by showing him how to play (e.g., by rolling the ball in front of him and saying, “let’s play!”).

This study was approved by Keio University, Faculty of Letter’s Institutional Review Board and was, therefore, completed in accordance with the ethical standards established in the 1964 Declaration of Helsinki. Written informed consent from the parents of all participants was obtained. All participants had a diagnosis of autistic disorder, PDD-NOS, or ASD by an external medical doctor. Diagnosis of Pervasive Developmental Disorders (PDDs) was further confirmed using the Pervasive Developmental Disorders Autism Society Japan Rating Scale.

3.2. Pilot Study

One participant was selected for this intervention, a male (6 years and 11 months old) diagnosed with autism and Attention Deficit Hyperactivity Disorder (ADHD). The purpose of the pilot study was to observe the way the participant interacted with COLOLO under the previous interaction rule while playing with the therapist.

Through this pilot study, we observed that the visual sensory feedback delivered from COLOLO engaged the participant in the activity. However, it was difficult to elicit play behaviors related to turn-taking because in the previous interaction rule, COLOLO provided visual feedback every time the participant manipulated the device regardless of who the turn-holder was. The therapist used different instructions in an attempt to try to guide the participant’s gaze away from his device, and give him the opportunity to look at the therapist taking his turn to change the color of the devices. But since the participant observed that the device always changed color every time he manipulated it, it was difficult to use the device feedback to elicit turn-taking.

Based on the findings from the pilot study, the two-sided lighting rule and the one-sided lighting rule were designed by implementing the concept of the turn-holder explicitly (i.e., only one COLOLO could send a message at a time), and indicating the turn-holder via visual feedback (i.e., colored lights and/or wiggling movements).

3.3. Experimental Study: Comparison of Two Interaction Rules

Four male participants were recruited as volunteers through the Department of Psychology at Keio University. Their profiles can be found in

Table 1. Participants’ profile.

Participant	Chronological Age (Year; Month)	Developmental Quotient	PARS	CARS
P1	4; 7	43	46	44
P2	4; 5	39	21	36
P3	5; 8	70	26	32.5
P4	3; 8	44	21	N/A

. Informed consent was obtained from their parents before the children were included in the study. All participants tried each interaction rule in separated sessions on different days. They tried the two-sided lighting rule on the first session and the one-sided lighting rule on the second session. Figure 4A shows a participant in a session with the two-sided lighting rule, and Figure 4B shows a participant in a session with the one-sided lighting rule.

The interventions lasted 5 min, but in those cases where the therapist could not engage the child in the activity, the sessions were terminated before that. On both conditions the turns were mediated by the devices, allowing one user, either the therapist or the participant, to take a turn each time. The purpose of this experiment was: (1) To evaluate how the devices influence children’s behaviors related to turn-taking by comparing the two modified rules, and (2) To explore the potential and limitations of describing turn-taking interventions using the data from the robotic toys.

3.3.1. Effect on Children’s Behavior during Turn-Taking Interventions

The sessions were analyzed using both, the data from human coders and the data from COLOLO. Three human coders (C1, C2, and C3) watched the videos from the intervention and were asked to count the number of gaze shifting and manipulations of the toy. During the interventions with COLOLO, the participant and the therapist were exchanging turns to manipulate the toy. While taking turns to manipulate the toys and change their color, the participant is expected to be able to look at the therapist while waiting his turn. This skill is of importance for turn-taking interventions, as looking at the peer helps to understand context and intentions. The role of COLOLO is to facilitate this exchange by delivering visual cues to guide the child’s gaze and to indicate the moment to manipulate the toy.

For counting the number of gaze shifting instances, the videos from the sessions were divided into segments of 10 s, and the human coders were asked to judge if gaze shifting occurred or not during each segment by assigning a numerical value such as 0 or 1. Scoring intervals is a common practice for analyzing social behaviors in interventions for children with ASD [41,42]. Note that, more than one instance of gaze shifting in the same segment still counted as 1. The instructions given to the human coders for identifying gaze shifting were as follows:

• Condition: During the therapist’s turn and when a message is being exchanged. Indicators: Each time the child directs his gaze from the therapist’s facial region or the therapist’s device to his device.
• Condition: During the child’s turn and when a message is being exchanged. Indicators: Each time the child directs his gaze from his device to the therapist’s facial region or the therapist’s device.

For counting the number of toy manipulations, the human coders were asked to use the definition presented below. Moreover, identified manipulations were categorized as a “successful” or a “failed” attempts of turn-taking, according to the result of the action of manipulating the toy. Indicators of clear attempts to manipulate the device were described as observing the participant’s gaze on one of the devices or the therapist’s face while directly manipulating the toy (i.e., shaking, throwing, pushing, or rolling with either hands or feet).

• Successful attempt of turn-taking: described by clear attempts to manipulate the device that result in a device changing color (the child manipulates his toy during his turn)
• Failed attempt of turn-taking: described by clear attempts to manipulate the toy that did not result in a change of color (the child manipulates his toy during the therapist’s turn)

3.3.2. Turn-Taking Analysis by COLOLO

In our previous study, two human coders (C4 and C5) analyzed the videos interventions to identify when turn-taking was observed [36]. The therapist provided a definition of turn-taking that was suitable for interventions with COLOLO. It was defined as a social exchange mediated by the device’s visual cues and it was counted as an occurrence:

• Each time the child took his turn: described by clear attempts to send a message by manipulating COLOLO, with the gaze on one of the devices or the therapist’s face (Pattern A).
• Each time the child waited without manipulating COLOLO during the therapist’s turn: the child is looking at the therapist or paired device, showing no intention of manipulating the device (Pattern B).

Using COLOLO’s log file, it was possible to extract the number of exchanged messages and the time interval between each of them. However, not all the messages recorded by COLOLO represent turns exchanged between the child and the therapist. For example, during the sessions, there were cases where the child started playing with his toy without looking once to the therapist. According to the definition, even if COLOLO is recording the child’s manipulations of the toy, these do not represent turn-taking. For this reason, to compare the data from the human analysis with the data from COLOLO, it is necessary to exclude as much as possible, those exchanges that are not turn-taking. Since the information collected by COLOLO is the time interval between the exchanged messages, this is the information that we are going to extract from the human analysis. The process to extract the time intervals from the human analysis had two steps: (1) The selection of a time interval (Tmin–Tmax) that contains the majority of observed turn-taking, and (2) Filter COLOLO data by excluding the messages outside the selected time interval, and compare COLOLO data to the human analysis.

To select the time interval (Tmin–Tmax), two human coders (C4 and C5) counted the number of turn-taking instances using the previously mentioned definition. They counted from the moment they observed the child’s intention to manipulate the toy until the paired device was manipulated as a response (Pattern A), and also from the moment the therapist intended to manipulate the toy until the child’s manipulation of the toy (Pattern B). From this information, we obtained the time intervals between all the exchanged messages that were considered turn-taking by the coders. It was observed that both coders followed a similar pattern that formed the curve represented in Figure 5. The selected time interval Tmin–Tmax was between 3 and 13 s, which is where coders C4 and C5 observed the majority of turn-taking.

The second part of the analysis involved asking different coders (C1, C2, and C3) to do the same analysis and count turn-taking instances based on the definition. The coders analyzed the sessions of the 4 participants with both interaction rules (Human data). With the previously selected interval Tmin–Tmax, we filtered the data from COLOLO as follows: those exchanges of messages within the interval of 3 s and 13 s, were discarded (COLOLO data). With this, it was expected that the number of messages exchanged that were less likely to be considered turn-taking by human analysis would be reduced. The results from this analysis compare the human data with COLOLO data to explore the agreement between these, and to understand how much it is possible to describe turn-taking using the data from the toy’s manipulations.

4. Results

In this section, results from the video analysis by human coders and COLOLO data are presented. Three coders were asked to count the occurrences of the participant’s manipulation and gaze-shifting behavior based on the definitions explained in Section 3.3.1. To compensate for the differences in the sessions’ duration, the number of occurrences of each behavior was normalized by dividing it by the session length (in minutes). The results presented in Section 4.4 refer to the use of the data collected from the interaction using COLOLO to describe the exchanges between the child and the therapist.

4.1. Effect on Manipulation

Figure 6 shows the number of manipulations per minute for each participant during the sessions with two-sided lighting and one-sided lighting, counted by the human coders (Figure 6A) and the COLOLO devices (Figure 6B). In the case of the human analysis, the number of total manipulations refers to the sum of the number of successful and failed attempts of turn-taking.

Markers and error bars in Figure 6A represent the average of the three coders and maximum/minimum values, respectively. For the results obtained by the human coders, a two-way repeated measures ANOVA was applied to investigate the statistical difference in the number of manipulations between the two-side lighting rule and the one-sided lighting rule. The results indicate that no significant difference was observed regarding: the rule factor ($F(1,9)=0.03,p>0.05$, ${\eta }^2=0.003$), and the interaction between the rule factor and the coder factor ($F(2,9)=0.83$, $p>0.05,{\eta }^2=0.16$). The result from COLOLO data shows that for all participants there is a slight decrease in the number of manipulations during the sessions with one-side lighting.

4.2. Effect on Gaze Shifting

Since gaze shifting is not as straight-forward for human coders to identify as manipulation is, Fleiss’ Kappa was calculated to evaluate the inter-rater agreements before the analysis. The Fleiss’s Kappa is a typical statistical measure to assess the agreement among more than two raters. Since there is no ground-truth data for gaze shifting behavior, Fleiss’s Kappa was calculated to evaluate the reliability of human coders’ observation. The 75th, 50th and 25th percentile of the distribution of Fleiss’s Kappa values were 0.43, 0.35 and 0.1 respectively. According to this result, in most of the sessions, the result from three coders showed moderate agreements.

Figure 7 shows the number of gaze shifting instances per minute for all participants counted by human coders. Markers and error bars represent the average of the three coders and maximum/minimum values, respectively. Two-way repeated measures ANOVA was applied and a significant increase of gaze shifting in one-sided lighting compared to two-sided lighting rule was observed ($F(1,9)=30.94,p<0.05,{\eta }^2=0.775$). The interaction effect between factors was not observed ($F(2,9)=2.043,p>0.05,{\eta }^2=0.312$).

4.3. Effect on Failed Attempts of Turn-Taking

The results of the number of failed attempts (the child manipulates the toy during the therapist’s turn) per minute are shown in Figure 8. Figure 8A,B represent the results from human coders and COLOLO data, respectively. It was represented as the ratio of failed attempts to the total of manipulations of the toy. Markers and error bars in Figure 8A represent the average of the three coders and maximum/minimum values, respectively. A significant reduction in failed attempts for the one-sided lighting rule was observed by Two-way repeated measures ANOVA ($F(1,9)=7.145$, $p<0.05,{\eta }^2=0.443$), without the interaction effect between factors ($F(2,9)=0.426,p>0.05$, ${\eta }^2=0.086$). This reduction could also be observed in the analysis made with COLOLO data from most of the participants (Figure 8B).

Finally, it was evaluated the effect of different rules on the number of turn-taking behaviors observed during the interventions. Results presented in Figure 9 show the number of turn-taking instances per minute counted by the coders for the four sessions with both modified rules. Markers and error bars represent the average of the three coders and maximum/minimum values, respectively. Two-way repeated measures ANOVA was applied and a significant increase of turn-taking with the one-sided lighting rule compared to the two-sided lighting rule was observed ($F(1,9)=10.511$, $p<0.05,{\eta }^2=0.539$). The interaction effect between factors was not observed ($F(2,9)=2.341$, $p>0.05,{\eta }^2=0.3412$).

4.4. Turn-Taking Analysis by COLOLO

Figure 10 shows the average of the analysis of precision and recall between the data from the coders (C1, C2, and C3), and the data from COLOLO filtered with the T-value described in the Section 3.3.2. Taking the data from one participant’s session as a sample, we overlapped the data from COLOLO with the human analysis (Figure 11). Each mark represents the toy’s manipulation by the participant and the therapist after being filtered using the selected T-value, and the shadowed areas represent turn-taking instances (pattern A and B) observed by one of the coders. Exchanges of messages (two marks from two devices) that overlap within the areas can be considered as a moment when COLOLO could indicate turn-taking as observed by the coder.

5. Discussions

5.1. Effects of Sensory Feedback on Turn-Taking Behaviors

To answer the hypotheses regarding the number of manipulations, and failed and successful attempts of turn-taking (H1∼H3 in Section 2.3), the results from three human coders were analyzed with two-way repeated measures ANOVA with two factors (i.e., independent variables); “coder” and “rule”. Throughout the analysis, the “coder” factor did not have a significant effect on the number of coded behaviors.

Hypothesis H1: Lighting up both devices (lights do not indicate the turn-holder), will elicit a higher number of manipulations because of its relatively higher amount of visual stimuli. As shown in Figure 6A, no significant difference was found between the effect of the two-sided lighting and one-sided lighting on the number of manipulations. Based on the results, H1 was not supported, as coders did not observe a significant difference between the two conditions in the number of times the participants manipulated COLOLO. However, the data from COLOLO showed that all the participants reduced the number of manipulations during the session with the one-sided lighting rule. This difference between the human analysis and the system is understandable, as COLOLO register an event each time the sensor is activated (by tilting the toy) regardless it is a toy manipulation as it was previously defined. The data from COLOLO while lacking context, it provides an image of the level of engagement of the child to play with the toy. It was not clear that the level of engagement to play with COLOLO was due to a learning outcome or as a consequence of the novelty of the toys, which is a limitation of the study. The current results compare the effect of each rule on the manipulations of the toys, which for the therapist is an indicator of “engagement to play with the toy”.

Hypothesis H2: Lighting up one device (lights indicate the turn-holder) will reduce the number of the child’s manipulations during the therapist’s turn (i.e., reduce the number of failed attempts of turn-taking). This assumption was supported by the results presented in Figure 8A. We consider that since children with ASD tend to show a strong interest in visual stimuli, illuminating only one device at a time was effective to lead the gaze shifting. This could be related to the reduction of the number of failed attempts in the sessions with the one-sided lighting rule as hypothesized in H2. Because in the one-sided lighting condition, the participant looked at the therapist’s device more often than during the two-sided lighting condition, as it was described by the results on Figure 7, this was helpful to clarify who was holding the turn at a time. A similar tendency was displayed by the data collected from COLOLO (Figure 8B). Occurrences of failed attempts of turn-taking can indicate both that the child is engaged enough to play with the toys but can also be a negative indicator of a child not being able to wait for his partner to take the turn. The human analysis and the device analysis followed a similar trend, a reduction of failed attempts at turn-taking under the one-sided lighting rule, that implies the reliability of the effect.

To facilitate the number of turn-taking instances during the sessions with COLOLO, both increasing the number of successful attempts of turn-taking and reducing the failed attempts are important aspects. We hypothesized that more colored-lights feedback would elicit more manipulation (H1) based on the insights from our previous study [37]. However, in the sessions with one-sided lighting, the participants manipulated the device as much as they did during the sessions with two-sided lighting. Moreover, indicating the turn-holder by both lights and movements was beneficial for reducing the failed attempts (H2) which also resulted in an increased number of turn-taking instances observed by the human coders (Figure 9). For this reason, we assumed that H3: Lighting up one device (lights to indicate the turn-holder) would lead to a higher occurrence of turn-taking by the child. The results in Figure 9 supported this hypothesis, and overall the one-sided lighting rule was more effective than the two-sided lighting rule for facilitating turn-taking behavior of children with ASD. On interventions with COLOLO, switching off one of the toys to make emphasis on the turn-holder resulted in more turn-taking behaviors. This effect was also reflected on the previous results, as the one-sided rule elicited less failed attempts of turn-taking (the child waited for his turn without manipulating the toy) and more gaze shifting (the child looked at the therapist during the therapist’s turn).

For children with ASD that also has difficulties with verbalization, this proposed system worked to facilitate the exchanges between them and the therapist, and also to describe aspects of their play behavior. This paired devices configuration using simple visual cues could be incorporated into different styles of games that involve taking turns. Although limited interaction rules and their effect on turn-taking behavior was studied with COLOLO, the findings suggest an insight to design robotic toys for facilitating turn-taking. To facilitate a targeted behavior, it is necessary to design how to display the visual cues carefully. We consider that the proposed interaction rules for paired devices can be applied to other paired devices that communicate and provide feedback using colored lights and movements.

5.2. COLOLO as a Descriptor of Turn-Taking Behaviors

The selected interval Tmin–Tmax was from 3 to 13 s (Figure 5), and with this value the data from COLOLO was filtered, leaving only the exchange of messages within Tmin and Tmax. This filtered data set was tested by comparing it with the analysis of different coders (C1, C2, and C3) obtaining in average an agreement of 0.72 between the human analysis and the analysis with COLOLO (Figure 10). These results have different implications. The data from the toys cannot describe turn-taking, but it can be used as an indicator of turn-taking. For example, the plot in Figure 11 shows that the data from the toys can follow the human analysis to a high degree, but not all the turn-taking indicated by COLOLO was also indicated by the human coder. To describe turn-taking-like behavior more accurately by computers, one possible approach is the combination of different sensors. However, with this approach, it was possible to describe children’s behavior during play activities that involve the manipulation of toys. This type of data is: (1) Difficult to obtain by traditional evaluation methods such as video coding, and (2) Valuable to understand how children play with the toys. While the information obtained from interactive systems such as COLOLO will not replace the therapist’s judgment, providing quantitative measurements with good reliability will facilitate the understanding of children’s responses in different social situations.

6. Conclusions

In this project, we initially investigated how robotic toys are being designed and used in ASD interventions. We separated them according to two main features: toys for stimulating behaviors and toys for describing behaviors. We considered these two characteristics as the main functions of the proposed system COLOLO. The purpose of the study was: (1) to evaluate how the devices influence children’s turn-taking behavior by comparing the effect of two different interaction rules, and (2) to explore the potential for and the limitations of describing children’s play behavior in turn-taking interventions using quantitative data from the robotic devices. We followed a similar approach to the ones using multiple robotic toys for turn-taking [22], but it was simplified into a system using paired devices. The devices work with interaction rules meant to help children to identify turns, which can be only achieved by a paired/multiple devices configuration. The approach of using paired devices benefits from being able to deliver sensory feedback based not only on children’s actions but also on the therapist’s actions. This creates new opportunities for interventions. Then, we described the hardware characteristics and introduced simple interaction rules for facilitating turn-taking behaviors. Four hypotheses were proposed based on the relationships between the interaction rules and children’s behaviors related to turn-taking.

The evaluation consisted of a pilot study to observe and decide which modifications of the previous rule were necessary. This was followed by an experimental study to compare the effect of the two modified rules: the two-sided lighting rule and the one-sided lighting rule. Moreover, we explored the use of COLOLO’s data as an indicator of turn-taking behaviors. Results showed that regarding the number of manipulations, there was no significant difference between two-sided lighting and one-side lighting. However, the number of failed attempts of turn-taking and gaze shifting were reduced in the sessions with one-sided lighting. These results are related to the increased number of turn-taking instances in sessions with one-sided lighting. Following this, we evaluated the use of the data from the toys’ manipulation to indicate those exchanges that have a higher probability of being classified as turn-taking. We obtained an agreement of 0.72 between the human coder and the system. These results indicate the potential for and limitation of using this approach to describe children’s play behavior in turn-taking interventions. Further studies to evaluate the effect of feedback delivered by paired devices should include larger samples. Moreover, they must seek to eliminate ordering and novelty effects through blocked and longitudinal study designs.