Hand Gestures in Virtual and Augmented 3D Environments for Down Syndrome Users

Featured Application

This work provides the keys to design interactive apps based on virtual and augmented reality using suitable Mid-Air gestures according to considerations of User Experience.

Abstract

Studies have revealed that applications using virtual and augmented reality provide immersion, motivation, fun and engagement. However, to date, few studies have researched how users with Down syndrome interact with these technologies. This research has identified the most commonly used interactive 3D gestures according to the literature and tested eight of these using Oculus, Atheer and Leap Motion technologies. By applying MANOVAs to measurements of the time taken to complete each gesture and the success rate of each gesture when performed by participants with Down syndrome versus neurotypical participants, it was determined that significant difference was not shown for age or gender between these two sample groups. From the results, a difference was only demonstrated for the independent variable Down syndrome when analysed as a group. By using ANOVAs, it was determined that both groups found it easier to perform the gestures Stop, Point, Pan and Grab; thus, it is argued that these gestures should be used when programming software to create more inclusive AR and VR environments. The hardest gestures were Take, Pinch, Tap and Swipe; thus, these should be used to confirm critical actions, such as deleting data or cancelling actions. Lastly, the authors gather and make recommendations on how to develop inclusive 3D interfaces for individuals with Down syndrome.

1. Introduction

The use of technologies by individuals with Down syndrome (DS) is an emerging field of study [1]. Throughout mainstream society, devices such as tablets and smartphones have become increasingly popular. They have been widely adopted across all areas of our lives because of just how useful they have proven to be in many cases. In recent years, there has been a growing trend towards [2] using Augmented Reality (AR) and Virtual Reality (VR) devices, and as such, new applications are now being developed that are of use to society. According to the experts, this is yet another piece of new technology that is probably here to stay [3].

In the literature, there are currently few studies on interactions with VR and AR, and even less on users with Down syndrome. Kumin et al. [4] mention that studies do exist demonstrating how adults and children with Down syndrome use the computer keyboard and mouse without problems, although these contradict observations made in other scientific publications that describe difficulties and challenges relating to fine motor skills. What is clear is that further research is needed on 3D gestures and interactions within AR and VR environments to identify whether devices currently being manufactured suit all user needs and adapt to all types of users.

It is important to point out that Shin [5] reports about an increase in the number of babies born with Down syndrome in the United States from a prevalence of 9.0–11.8 per 10000 live births in ten US regions in the period from 1979 to 2003. Meanwhile in European countries, such as Spain, there has been a drop in this number in recent years [6]. These figures prove to be very different for different countries across the globe, for example in Canada the number of births per year stands at 330000, with the prevalence of DS being 14.41 per 10000 births; in the Czech Republic that number stands at 110000, with the prevalence of DS being 21.03; in Finland, 60000 with a prevalence of 29.94; Hungary, 100000 with a prevalence of 17.40; and Sweden, 100000–120000 and a prevalence of 28.91 [7]. In Mexico, the World Health Organisation estimates that on average there is one DS birth for every 1000–1100 live births.

Despite these changing figures, it is nonetheless important to identify those characteristics that prove significant in order to ensure the creation of inclusive technology. Cáliz [8] is of the opinion that research involving usability testing with DS and similar disabilities is on the increase, and will continue to increase in the future.

As a result of their cognitive skills, individuals with Down syndrome struggle with abstract thinking, short term memory problems, and short attention spans [9]. They have stronger social skills, whereas their language and perception skills are typically poorer. Most noticeably, visual memory and spatial memory are much stronger than verbal skills [10]. In terms of technology use, the three main skills that influence computer usage are cognitive skills, motor skills and perceptual skills [11].

2. Related Work

2.1. Use of Technologies by Individuals with Down Syndrome

Few relevant scientific contributions exist relating to technology usability studies that have selected individuals with Down syndrome as users. The recent study by Nacher [12] analyses touch gestures on interactive touch screens performed by children with DS. The study by Feng [13] mentions the main obstacles that parents of children with DS report when their children use computers, these being: frustration when navigating and troubleshooting, lack of patience, and design (flaws), among others [9]. Lazar [13] points out the importance of working specifically with individuals who have Down syndrome, and not just with “people with cognitive impairment” in general. Lazar argues that each group has its own unique cognitive traits and thus its own sets of strengths and weaknesses. Cortés et al. [14] analysed the usability of applications that are used by children, however the general recommendations focus more on software programming than on the hardware itself. Feng [9] studied the use of computers by adolescents and children, observing that although children encountered problems interacting with these primarily as a result of the software not being designed for their specific needs, results gathered from expert users shows that, with practice, teenagers gain the skills needed to work with word processing, data entry, and communication software [15].

Other works performed with users who have DS have attempted to stimulate the cognitive abilities of children [16] using computer software designed to teach mathematics [17]. Miyauchi [18] designed a system for training the tongue that includes exercises to facilitate movement. Nascimento and Salah [19,20] provide recommendations on serious games that are accessible for individuals with DS. González-Ferreras [21] proposes a video game for improving verbal skills, in particular prosody, and focused on the design and evaluation of the educational video game, from point of view how appealing is. Alonso-Virgós et al. [22] propose recommendations for web design based on the experiences of users with DS.

In the case of mobile devices, Mendoza [23] studies how easy it is for first-time users to learn how to use gestures on mobile devices. Del Rio et al. [11] suggest a methodology for evaluating the ease with which touch gestures, body movements and eye movements can be performed by individuals with DS. Bargagna [24] proposes using a robotic kit to promote education and collaborative learning in a play setting. Felix et al. [25] studied how to improve reading and writing skills using a multimedia tool, with significant improvement being found.

2.2. Studies on Fine and Gross Motor Movements

The literature offers several pieces of research on fine and gross motor movements in individuals with DS. Gross motor skills coordinate the movement of the arms, legs, feet, and entire body (when running, crawling, walking, swimming, and any other movements that require the use of the body’s large muscles). Fine motor skills coordinate the movements of the hands, feet, fingers, toes, lips, tongue (e.g., picking up objects using the thumb and index finger, or movements involving any of the body’s small muscles that occur on a daily basis). In this paper, our work centres on those that relate to gestures.

Although children with DS follow the same motor development as children without DS (neurotypical), they take twice as long to acquire the abilities of their neurotypical counterparts [26]. Research exists on hand tracking and gesture recognition [27]. Torres et al. [28] worked on an emotional development assessment tool using the platform Kinect. Alt et al. [29] studied Mid-Air gestures for large interactive displays, and the means by which users receive feedback. Cabreira [30] reviewed the most widely used gestures on three platforms: Kinect, Leap Motion and MYO, and identified a total of fifteen recurrent gestures. Raisbeck and Diekfuss [31] analysed the differences in fine and gross motor skills by measuring performance during tasks execution. Additionally, they proposed the game Beesmart, developed for Kinect, to improve users’ day-to-day motor skills [32]. Capio [33] studied the fundamental movement skills that show delayed development in children with DS. This development was documented by Winders et al. [34]. To increase the autonomy and independence of individuals with DS, Engler [35] proposes the Poseidon Project, that involves the use of a technical assistance. Silva [36] demonstrated that the use of exercises on the Wii could improve physical condition, functional mobility, and motor proficiency in adults with DS. Beerse [37] revised the skill needed by children to hop on one leg on the same spot, and concluded that they would need to be at least seven years old in order to successfully perform the action. Another author, Macías [38] reported the differences that could be observed between neurotypical participants and those with DS. Puspasari [39] found the fine motor skills of children with DS who used Wii consoles to be inferior to neurotypical children, whereas Berg [40] found that the use of Wii consoles could help DS children improve their postural stability, limits of stability, and Bruininks-Oseretsky Test of Motor Proficiency.

2.3. Studies on Virtual Reality and Augmented Reality with Users Who Have Down Syndrome

Hovorka et al. [41] ran a study based on VR and AR technologies in which users with DS had to perform everyday tasks. Abdel Rahman [42] state that motor training with virtual reality during therapy sessions promised encouraging results in this population. Del Ciello [43] provides a revision of existing literature on rehabilitation therapies using AR apps.

Ramli et al. [44] propose the usability factors that need to be taken into account when developing applications for users with DS, especially when using AR. In the field of education, a number of different studies have been performed involving AR and VR. McMahon [45], for example, successfully used AR to teach scientific terminology. Lopes [46] analysed brain activity when children with DS entered virtual reality using VR technology. For their part, Salah [47] performed a study on individuals with DS to study the cognitive difference acquired through the use of educational games using a computer and AR, and Martín-Sabarís [48] also studied the use of AR applied to users with DS using the game Pokémon Go.

The research presented in this paper studies the use of interactive gestures in AR and VR in order to identify the most suitable gestures that should be taken into account when designing apps for all users.

3. Materials and Methods

3.1. 3D Gesture Selection

To determine which gestures should be included in this study, the authors reviewed the literature and existing VR/AR applications to establish which gestures are most commonly used or mentioned, although they may not be identical or referred to by these same names. These gestures are presented in

Table 1. Gestures identified from bibliography review.

Gesture	Name	Jang, Elmqvist, Ramani [49]	Jégo, Paljic and Fuchs [50]	Buchmann et al. [51]	Albertini et al. [52]	Manders et al. [53]	Caro et al. [54]	Bernardes, et al. [55]	Bai et al. [56]	Kin et al. [57]	Alkemade et al. [58]	Choi; Kim [59]	This Study
	Stop	X				X	X	X		X		X	X
	Pinch								X	X			X
	Grab and Drop using fingers									X	X
	Point	X	X		X	X		X			X	X	X
	Rotate			X		X				X
	Arm/hand rotation	X	X			X					X	X
	Take												X
	Hold and Swipe										X
	Grab and Drop using hand			X					X		X		X
	Pan	X			X								X
	Swipe (Move with 2 fingers)								X	X			X

. From this list, eight were selected for this study (

Table 2. Vocabulary of 3D gestures for Leap Motion, Atheer and Oculus Rift.

Gesture	Image of Gesture	Activity	Description of Activity	Device Used
Stop		Stop the vehicle	A vehicle is driving towards the screen and the user. To stop the vehicle the user must raise their hand and hold it up with the palm facing outwards towards the screen. The test is completed when the vehicle is stopped.	Leap motion
Pinch		Place marbles in box	The user must place marbles inside of a box. This test does not have an associated gesture, instead the purpose is to identify which gesture is chosen by the user. The objective was to observe which gesture is most frequently used. No specific instructions were given to users on how to perform this task.	Leap motion
Grab and drop		Move coloured blocks	The user must place blocks on circles of the same colour.	Oculus Rift with Leap motion
Pan		Push ball towards goal	The user pushes three different coloured balls towards a red line.	Oculus Rift with Leap motion
Take		Pick apple from tree	The user has to pick an apple from a tree.	Oculus Rift with Leap motion
Point		Search for cat	The user has to point to the cat that appears in different locations in the game.	Oculus Rift with Leap motion
Tap		Press button to move character	The user has to press a button to make the character advance on the base.	Atheer
Swipe		Move ball over obstacle	The user has to select the ball using two fingers to move it over an obstacle.	Atheer

).

3.2. Experimental Study

The purpose of this work is to analyse a set of 3D gestures and evaluate their usability for individuals with DS when used with Augmented Reality and Virtual Reality technologies.

This study includes individuals with DS from different age groups and different genders who also possess different technological competencies. In order to be able to discern whether the motor development of DS participants affects their ability to interact with AR and VR devices or not, we use a control group containing neurotypical participants (referring to participants who do not suffer any developmental disability), and record any differences that emerge between these two groups.

The activities of both groups were examined to establish whether significance difference exists. Any differences arising as a result of gender and age (as independent variables) were also analysed. The time taken to complete a gesture and the percentage of successful attempts were the independent variables for each of the tasks.

The hypotheses formulated:

H_1A:

The gesture has an effect on the success rate percentage of the task.

H_1B:

The gesture itself has an effect on the time taken to complete the task.

H_1C:

Gender has an effect on the success rate percentage of the task.

H_1D:

Gender has an effect on the time taken to complete the task.

The alternative hypotheses are defined as: H_0i = –H_1i

The first step involved establishing which gestures to measure, and elaborating the vocabulary of gestures shown. As proposed by Abraham et al. [27] a set of 3D gestures was defined: Stop, Pinch, Grab and drop, Pan, Take, Point, Tap and Swipe (drag with two fingers) (Table 2).

3.3. Tasks

Tasks were created in the form of games for the eight aforementioned gestures. One task/game was designed for each gesture. When designing the tasks, the main prerequisite was to ensure that activities should not require any cognitive workload that could distract from the performance of the task, as suggested by Mendoza et al. [23]. To ensure this, simple games were created that used less colours, and avoided the use of abstract images and ambiguity. Images were presented on white backgrounds to show objects clearly and accompanied by as little text as possible, none if possible.

To determine which game would be best suited to a given gesture, we surveyed 100 computer science students and sector professionals, all of who are familiar with User Experience design. In this survey, respondents were asked to name activities that could relate to the gesture in question; for example, the pinch gesture feedback included activities such as plucking the petals from a flower, moving marbles from one spot to another, or adding a pinch of salt to a plate of food. Once the feedback was collated, a tag cloud was generated and the most mentioned games were selected. The activity ascribed to each gesture is detailed in Table 2 below; the table also lists the device on which a gesture would be tested.

Once the games were designed and developed, a pilot was run to check there were no issues with performing the gestures. This pilot was run using two individuals with DS. Based on the observations made during the pilot, a few minor adjustments were made. The two individuals involved in the pilot did not participate in the study used to measure the success of a task.

3.4. Equipment

The Oculus Rift Development Kit Dk2 with a built-in latency tester was used to perform tasks in VR. The Oculus has a low-latency positional tracking, and a low persistence OLED display (to eliminate motion blur and judder). The Leap Motion sensor and Atheer Air Android 4.4 goggle was used for tasks in AR. The Leap Motion is a sensor measuring less than seven centimetres and is capable of capturing both the finger movements and hand movements of users. The advantages of this piece of equipment include: detects hands and fingers rapidly; integrates with other devices, such as the Oculus Rift and Atheer Air; and is compatible with Android 4.4.

3.5. Participants

A total of 24 participants performed the augmented reality tasks using Atheer and Leap Motion, and 23 of them completed the virtual reality tasks using the Oculus Rift. The participants’ ages ranged from 14 to 46 years old, with a mean of M = 24.22 and a standard deviation of SD = 7.69. The gender of participants was evenly balanced with 11 men and 12 women. Participants were recruited from a Down syndrome support centre where individuals with Down syndrome receive support helping them to develop motor skills, cognitive skills and problem-solving skills.

A second group consisting of 25 neurotypical participants was established as a control group. Participants in this control group were students recruited from the same university at which the Down syndrome support centre is hosted. This second group contained 17 men and 8 women. Ages ranged from 18 to 29 years old, with a mean of M = 20.28 and a standard deviation of SD = 2.25. All participants gave their informed consent for inclusion before they participated in the study. The study was conducted in accordance with the Declaration of Helsinki, and the protocol was approved by the Ethics Committee of University of Monterrey.

3.6. Procedure

For each task, participants were accompanied by a moderator, an observer, and a caretaker employed by to the Down syndrome support centre. The moderator was responsible for demonstrating the gesture to the participant, after which the participant would have to repeat the gesture without assistance. Once the participant understood how to execute the gesture s/he would then have to complete the task designed for said gesture. The app provided feedback in the form of an audible sound when the gesture was completed correctly. The system recorded the start time and end time of each interaction. In the event a gesture was not completed within a given time it was logged as not completed. Each participant took around 5 min to put on the equipment, understand and complete tasks on the Oculus with Leap Motion, (average to do the tasks: DS group 41.08 s, and neurotypical group 17.97 s), 5 min to put on the equipment and complete tasks on Leap Motion, (average to do the tasks DS group 61.3 s, and neurotypical group 18.98 s), and between 7–10 min on the Atheer (average to do the tasks: DS group 81.35 s and neurotypical group 47.34 s). As these proceedings took place, an observer logged the session and noted down any problems that arose.

The same procedure was used for the control group containing neurotypical participants. The results from both sample groups were then compared and contrasted.

4. Results

In order to know which are the gestures that can provide a better execution experience for all users, the proposed gestures in Table 2 were analysed according to the success and execution times.

Table 3. Gestures: Descriptive statistic.

Gestures		Stop	Pinch	Grab and Drop	Take	Pan	Point	Tap	Swipe
DS Group	Mean success (St.Desv)	0.754 (0.265)	0.041 (0.085)	0.073 (0.058)	0.022 (0.024)	0.391 (0.367)	1.957 (1.637)	0.03 (0.03)	0.037 (0.109)
	Mean Time (St.Desv)	6.261 (5.328)	116.348 (12.194)	89.478 (32.946)	3.609 (2.888)	(41.043) (30.022)	30.174 (33.107)	57.087 (30.716)	106.727 (30.363)
Neurot. Group	Mean success (St.Desv)	0.693 (0.258)	0.240 (0.133)	0.141 (0.071)	0.062 (0.026)	0.820 (0.372)	1.760 (0.723)	0.072 (0.056)	0.080 (0.085)
	Mean Time (St.Desv)	3.080 (2.943)	34.880 (15.865)	48.640 (32.140)	3.360 (1.497)	10.760 (6.180)	9.120 (6.948)	19.920 (29.508)	74.760 (57.523)

show the descriptive statistics for each gesture in each experimental group.

4.1. Analysis of Success Rate

The first hypothesis to test is whether the task itself determines the success rate. In other words, we must establish whether there are tasks that are more difficult to perform due to their nature. Firstly a Kolmogorov-Smirnov test was performed to identify the distribution of the sample regarding the success rate. Both samples show normal distribution, so a parametric test is applied to find out the presence of significant differences between groups. To test hypothesis H_1A: The gesture has an effect on the success rate percentage of the task, the success rates of both groups were compared calculating successes/attempts and analysed using two sample Student’s T-test. As

Table 4. Comparison of tasks by success rate.

Action	Stop the Vehicle	Place Marbles in a Box	Move Coloured Blocks	Pick the Apple from the Tree	Push Ball Towards Goal	Search for the Cat	Press a Button to Move a Character	Move Ball over Obstacle
Gesture	Stop	Pinch	Grab and drop	Take	Pan	Point	Tap	Swipe
Down syndrome vs. Neurotypical	P = 0.485	P = 0.000	P = 0.001	P = 0.322	P = 0.000	P = 0.600	P = 0.002	P = 0.143

reveals, it is the tasks involving the gestures Pinch, Grab and drop, Pan and Tap that present significant difference between the two groups. Thus, as they prove to be more difficult for one group to perform than the other, these gestures should be avoided when trying to create inclusive technology. When analysing the mean of these activities, it is possible to see that the neurotypical participants had a much higher success rate than individuals with DS. The H_1A hypothesis is accepted for the tasks: Stop, Take, Point and Swipe, because there is no significant difference in the success of execution by both experimental groups.

4.2. Analysis of Tasks by Completion Time

To test the second hypothesis, H_1B: The gesture itself has an effect on the time taken to complete the task, the successful gestures of both groups were compared to see which times were faster; these were analysed using a two sample Student’s T-test. As can be seen in

Table 5. Comparison of tasks by completion time.

Action	Stop the Vehicle	Place Marbles in a Box	Move Coloured Blocks	Pick the Apple from the Tree	Push Ball Towards Goal	Search for the Cat	Press a Button to Move a Character	Move Ball over Obstacle
Gesture	Stop	Pinch	Grab and Drop	Take	Pan	Point	Tap	Swipe
Down syndrome vs. Neurotypical	P = 0.016	P = 0.000	P = 0.702	P = 0.714	P = 0.000	P = 0.007	P = 0.000	P = 0.055

, there is significant difference in the following gestures: Stop, Pinch, Pan, Point and Tap. These are more difficult for individuals with DS to perform. However, individuals with DS can perform the gestures Grab, Take and Swipe with equal dexterity, which means these gestures should be given preference. Once again, in trying to be inclusive, we recognise that these tasks require more time to complete. The H_1B hypothesis is accepted for the four tasks: Grab and Drop, Take and Swipe, because there is no significant difference in the time of execution by both experimental groups.

4.3. Analysis of Success Rate by Gender

An ANOVA was performed to compare the success rate of gestures in tasks completed by individuals with DS versus neurotypical participants, with the independent variable being gender. The results, presented in

Table 6. Comparison of success rate by gender.

Action	Stop the Vehicle	Place Marbles in a Box	Move Coloured Blocks	Pick the Apple from the Tree	Push Ball Towards Goal	Search for the Cat	Press a Button to Move a Character	Move Ball over Obstacle
Gesture	Stop	Pinch	Grab and Drop	Take	Pan	Point	Tap	Swipe
Gender and Down syndrome	P = 1.000	P = 0.879	P = 0.470	P = 0.494	P =0.943	P = 0.902	P = 0.205	P = 0.452
Gender and Neurotypical	P = 0.941	P = 0.642	P = 0.498	P = 0.414	P = 0.161	P = 0.243	P = 0.087	P = 0.098

below, show us that there is no difference based on gender in either of the sample groups. As such, the third hypothesis H_1C: Gender has an effect on the success rate of the task in the current study has been tested. The H_1C hypothesis is rejected, there is no significant difference in the success performing any of the gestures/tasks by gender in the current study.

4.4. Analysis of Completion Time by Gender and Age

An ANOVA was performed to compare the completion time of gestures in tasks completed by individuals with DS versus neurotypical participants, with the independent variable being gender. Based on the results that are presented in

Table 7. Comparison of completion time by gender.

Action	Stop the Vehicle	Place Marbles in a Box	Move Coloured Blocks	Pick the Apple From the Tree	Push Ball Towards Goal	Search for the Cat	Press a Button to Move a Character	Move Ball on Obstacle
Gesture	Stop	Pinch	Grab and drop	Take	Pan	Point	Tap	Swipe
Gender and Down syndrome	P = 0.697	P = 0.137	P = 0.476	P = 0.216	P = 0.701	P = 0.989	P = 0.501	P = 0.728
Gender and Neurotypical	P = 0.739	P = 0.478	P = 0.398	P = 0.601	P = 0.42	P = 0.95	P = 0.739	P = 0.468

below, it is possible to conclude that there is no difference in completion times based on gender in either of the sample groups. Following this, a MANOVA was performed using gender, Down syndrome or neurotypical and the co-variable age as the independent quantitative variable. As seen in

Table 8. Comparison of age, gender by success rate/ number of attempts (between groups).

Age	Gender
P = 0.548	P = 0.645

below, age and gender do not affect the performance of a gesture.

The hypothesis H_1D is rejected as gender does not affect the time taken to perform gestures or complete tasks. Alternative hypothesis is accepted (H_0D).

4.5. Comparing Tasks

Once gestures were analysed, a post hoc test was applied using the Tukey’s method. Having done so, it was possible to establish how easy or difficult gestures proved to be and tasks could be clustered based on gesture difficulty. As can be seen in

Table 9. Groupings of gestures by completion times.

Task	Gesture	Mean	Groups
Place marbles in box	Pinch	116.348	A
Move ball over obstacle	Swipe	102.087	A
Press button to move character	Tap	57.087		B
Move coloured blocks	Grab and drop	52.957		B	C
Push ball towards goal	Pan	41.043		B	C
Search for cat	Point	30.174			C	D
Stop car	Stop	6.261				D	E
Pick apple	Take	3.609					E

, the gestures that proved easiest to perform for both sample groups were Stop and Take, based on their lower average scores. A scale using the letters A to E has been applied to record the difficulty rating of gestures: (A) Most complicated to perform-(E) Least complicated to perform. The most complicated gestures (A), which also have the largest mean scores, are the gestures Swipe and Pinch.

The comparison of completion times using a test for equal variances results in P = 0.000, as such there are significant differences in the completion times for all gestures performed by individuals with DS. In Figure 1 it is possible to observe that there are noticeable differences for the gestures Stop and Take when compared against the other gestures.

4.6. Geometric Distribution. Probability of Individual with DS Correctly Performing a Gesture

From the data recorded for ‘number of attempts’ and ‘number of successful gestures’, it is possible to establish the mean time for completing a gesture on the first attempt (successes/attempts) by users with DS.

Table 10. Geometric distribution of probability of performing gesture on first attempt for individuals with DS.

Gesture	Average n° Attempts	Success Rate of Attempts	Probability of Performing Gesture
Stop	1.57	1	63.69%
Point	2.04	1	49.02%
Pan	14	2.78	19.86%
Grab and drop	25.35	1.39	5.48%
Take	33.09	1	3.02%
Pinch	39.57	1.04	2.63%
Tap	61.17	1	1.63%
Swipe	41.65	0.3	0.72%

shows us that the gesture that is least likely to be successfully completed is Swipe (by joining the index finger and middle finger); whereas the gesture that is most likely to be completed successfully is Stop (by raising the hand).

4.7. Observational Findings. Problems Generated by Physiological Traits

4.7.1. Microcephaly

Participants with microcephaly needed to readjust their glasses on several occasions, and this even led to some having to ask to stop tasks (Figure 2a).

4.7.2. Clinodactyly

A common physical trait encountered in users with Down syndrome is different finger angles. Finger angles and curves proved a limitation for some gestures, as the device could not correctly identify the gesture being made (Figure 2b).

4.7.3. Inability to Join Index and Middle Fingers

The majority of users found it difficult to perform the Pan gesture, which required holding up and joining the index finger and middle finger. Performing the gesture correctly proved impossible for many (Figure 2c).

4.7.4. Dizziness When Using Oculus

Eight users complained of feeling dizzy when using the Oculus Rift. However, only three of these asked to remove the headset (Figure 2d). Three said that although they felt dizzy they were ok. One experienced a headache, and another user’s eyes watered.

4.7.5. Atheer Detected Pan Gesture as a Single Finger

There were users who performed the gesture despite experiencing difficulty in doing so. However, the Atheer glasses detected both fingers as a single finger.

5. Discussion

5.1. Usability Issues and Design Considerations

People with DS usually encounter problems with the fine motor skills required to interact with gestural interfaces. From our experiences and studies exploring 2D gestures on touchscreens that have involved individuals with DS [11] we can confirm that some 3D gestures (Mid-Air) prove easier and straightforward for users with DS to perform than the 2D gestures used on touch screens. This should be taken into account when designing interfaces otherwise user performance will diminish.

The easiest gestures for users with DS to perform in AR y VR environments are Stop and Point out. Overall, the Swipe gesture was the hardest to perform.

Mendoza [23] reported the Tap gesture as being one of the easiest gestures to perform with touch interfaces; however, in this study the Tap gesture proved to be one of the hardest to perform when using Atheer and Leap Motion in an AR.

Having a better understanding of the difficulties encountered by individuals with DS is the first step towards explaining and addressing any problems that arise that arise with these technologies.

In reviewing the results obtained for hypotheses H_1A and H_1B that are detailed in Table 4 and Table 5, no statistical significance differences have been identified between either of the study groups for the successful execution of the gestures Stop, Take, Point and Swipe, nor the execution time of the gestures Grab and drop, Take and Swipe. Based on these findings, it appears that when it comes to the design of virtual reality or augmented reality applications the aforementioned gestures prove to be those best suited for all users. Our recommendation would be to give priority to gestures that provide the greatest guarantee of success; therefore, we would argue that these findings form part of basic guidelines for developing apps and should be taken into account when establishing the functions required during a task (precision or speed of execution).

When selecting gestures there will be pros and cons involved in selecting one over another. For example: Stop, Take or Point may take longer to complete, but there is a higher guarantee of success for all users; Grab and drop and Take completion times are the same time for all users, but nonetheless these gestures prove more difficult for all users to consistently perform correctly; and the Swipe gesture (move with two fingers) is the gesture par excellence as it can be correctly performed by all users.

In this study, the authors have found that any usability testing for gestures must be conducted for each environment independently.

Below is a list of recommendations that serve as design guidelines for apps that require Mid-Air gestures:

• Basic recommendations regarding cognitive psychology are very useful, as are those proposed by Mendoza [23].
• For 3D gestures, try to use gestures that involve the whole hand and avoid gestures that require the use of fingers and precise movements.
• Regardless of environment, whether it be virtual reality or augmented reality, it is advisable to design apps so that they use gestures similar to those used in real life since the goal is to simulate within an synthetic environment those actions that are performed in the physical setting.
• Ensure devices have a greater margin of error for gesture recognition.
• Avoid the use of objectives to touch that are too small.
• Do not assume that gestures for a touch interface should be similar to those in a 3D environment or headset.
• When design with 3D gestures, give preference to: stop, point, pan and grab.
• Gestures that are much harder to perform can be used to confirm critical tasks, such as deleting data. For critical tasks, such as Delete, use an appropriate gesture for the action that is being performed and a second gesture to confirm the action.
• Avoid gestures using two fingers.

Specifically, it is important to take into consideration the design of specific elements for users with disabilities, thus we would propose the following:

• Using voice recognition when developing AR and VR apps for individuals with Down syndrome to compensate for users’ low literacy skills.
• Prioritising the use of gestures that suit the motor skill development of users with disability over other gestures when selecting suitable gestures for an app.

According to Table 10 of this paper. That said, however, the true goal is to ensure that all applications are inclusive, not just those developed specifically for users with Down syndrome.

Given the observations made regarding the morphology of individuals with DS, we would argue that AR and VR devices need to be better adapted to all people. This means they should be better at detecting gestures and capable of fitting difference-sized heads. Individuals with DS have clear physical limitations when it comes to wearing VR and AR headsets, as revealed by this research.

5.2. Threats to Validity

The study relied on the support of individuals with DS involved in a labour inclusion program. All these individuals had basic computer skills and used smartphones. It is important to repeat this study with participants who are not familiar with technology and compare the results to be sure of our findings.

6. Conclusions and Future Research

There is an increasing amount of legislation emerging covering inclusion policy. For some time now, organisations have been founded to design strategies, standards and resources aimed at providing universal access to individuals living with disabilities. One such organisation, the Web Accessibility Initiative (WAI), has developed guidelines that are regarded as the international standards for Web accessibility [60]. Nevertheless, these guidelines focus on physical disabilities (sight, hearing, etc.) and do not include cognitive disabilities such as Autism, Asperger or Down syndrome. We would like to see similar guidelines put in place that include Down syndrome. We propose designing guidelines that will assist hardware manufacturers in selecting more adequate gestures for Down syndrome users when performing tasks related to everyday activities in VR or AR environments. We think that in the future we will probably see the emergence of regulations within this legislation for hardware that includes individuals with motor, cognitive, or physical disabilities and technological handicaps.

In the literature, we came across one recommendation that we did not test ourselves: computer training is beneficial for users with DS. Facial recognition software and voice recognition software are making using devices easier and easier.

One interesting line of work that requires further study within the field of voice recognition research is that of voice recognition assistive technology for individuals with speech or voice impairments [61]. Verbal instructions might be an appropriate manner through which to interact with machines via software, and as such it is something that should be taken into account during UX design, as discussed in this paper.

When designing tasks to evaluate in usability testing, it is important to remember that participants will not always follow the instructions given. As such, one should expect these users to propose new gestures for performing day-to-day tasks, and that these should be included as valid new gestures.

Research has shown that individuals with Down syndrome can effectively use digital devices [23], and this study proves that AR and VR technology can be part of their daily interactions.