How children with autism spectrum disorder behave and explore the 4-dimensional (spatial 3D + time) environment during a joint attention induction task with a robot

https://doi.org/10.1016/j.rasd.2014.03.002Get rights and content

Highlights

  • Using ICT, we built a system that employs a Nao robot to elicit joint attention (JA) with children and to capture social engagement cues.

  • Interaction with Nao depends on the partner: better in typically developing (TD) children than children with ASD.

  • Multimodal JA induction is more efficient in both TD and ASD.

  • The 3D spatial world gaze exploration shows less accuracy and the trunk position shows less stability in ASD compared to TD.

Abstract

We aimed to compare, during a joint attention (JA) elicitation task, how children with autism spectrum disorder (ASD) and children with typical development (TD) behave and explore their 4 dimensional (meaning spatial 3D + time) when interacting with a human or with a robotic agent.

We built a system that employed a Nao robot and a perception system based on a RGB-D sensor (Kinect) to capture social engagement cues. A JA induction experiment was performed in which children with ASD (N = 16) and matched TD children (N = 16) had a 3-min interaction with the robot or with a therapist. Nao induced JA by gazing; by gazing and pointing; and by gazing, pointing and vocalizing at pictures. Both groups of children performed well with the therapist. However, with Nao, both groups had lower JA scores, and the children with ASD had a significantly lower score than the TD children. We found that (i) multimodal JA induction was more efficient in both groups; (ii) the 3D spatial world gaze exploration showed less accuracy; and (iii) the trunk position in ASD showed less stability in the 4 dimensions compared to TD controls.

We conclude that, in ASD, JA skill depends on the interaction partner, and implies a higher motor and cognitive cost.

Introduction

The purposes of the work presented in this paper is: (1) to compute a robotic platform able to elicit joint attention (JA) during an interaction task; (2) to compare, during the JA elicitation task, how children with autism spectrum disorder (ASD) and children with typical development (TD) behave when interacting with a human or with a robotic agent; and (3) to assess how children with ASD explored their 4 dimensional (meaning spatial 3D + time) environment compared to children with TD.

ASD is a developmental syndrome that implies impaired social interaction, communication and language as well as stereotyped and/or restricted behaviors. Despite evidence that some symptoms of ASD are present early in life (Guinchat et al., 2012, Saint-Georges et al., 2011), autism diagnosis is generally made between 3 and 5 years of age (Saint-Georges et al., 2013, Cohen, 2012). Achieving efficient interaction between humans and autistic children is a difficult task for their families as well as for well-trained therapists (e.g. Saint-Georges et al., 2011, Cohen et al., 2013). Although ASD remains a devastating disorder with a poor outcome in adult life (Roux et al., 2013, Howlin et al., 2013), there have been important improvements in the condition with the development of various therapeutic approaches. The literature on interventions in ASD has become quite extensive, with increasing convergence between behavioral and developmental methods (Matson et al., 2012, Ospina et al., 2008). The focus of early intervention is directed toward the development of skills that are considered to be “pivotal”, such as JA and imitation as well as communication, symbolic play, cognitive abilities, sharing emotions and regulation (Toth, Munson, Meltzoff, & Dawson, 2006) (e.g. the Early Start Denver Model; Rogers & Dawson, 2009). One of the main problems when interacting with children with ASD is their deficit of social interaction. While playing a game or conducting other activities with a social partner, these children tend to not concentrate on what they are actually doing, switching to other, repetitive, stereotypical behaviors that are of interest to them but that usually have no or few relations with the actual social context. In other words, children with ASD can display concerted attention to toys or objects that they like, but they have difficulties in sharing attention or interests with others (Rogers & Dawson, 2009). For example, maintaining eye contact with the caregiver is especially complicated (Maestro et al., 2005, Saint-Georges et al., 2010). Specifically, they lack JA, which is a key element of social cognition. JA teaches us much about social relationships, and it is a critical precursor of theory of mind (Premack & Woodruff, 1978) and language acquisition (Dominey & Dodane, 2004). Emery defined JA as a triadic interaction that showed that both agents focus on a single object (Emery, 2000). Agent 1 detects that the gaze of agent 2 is not directed at him/her and, therefore, follows the direction of the gaze to look at the “object” of attention of agent 2. This definition highlights a unidirectional process, unlike shared attention, which appears to be a coupling between mutual attention and JA. In shared attention, the attention of both agents concerns not only the object but the other agent as well (“I know that you are looking at the object, and you know that I am looking at the object”). Some authors (e.g., Tomasello, 1995) have argued that JA implies viewing the behavior of other agents as intentionally driven. In that sense, JA is much more than gaze following or simultaneous looking. By 12 months of age, TD infants display all aspects of JA (Carpenter, Nagell, & Tomasello, 1998).

In children with ASD, JA has been studied mainly by the annotation of video-recorded interaction: in a natural context (e.g., home movies, Saint-Georges et al., 2010) or in a laboratory context that uses JA induction during interactive play (e.g., early social communication scales, (Mundy et al., 2003). Children with ASD showed impairment in social orienting compared to children with intellectual disability (ID) and with TD (e.g., Dawson, Meltzoff, Osterling, Rinaldi, & Brown, 1998). They also showed impairment in sharing and proto-imperative JA (such as requesting) (Sigman, Mundy, Sherman, & Ungerer, 1986). Additionally, JA abilities at preschool ages predict language ability at the age of four years (Toth et al., 2006). In more recent years, social attention was explored using more sophisticated methods, including Information Communication Technology (ICT): 2-year-old toddlers with ASD showed the absence of preferential looking into the eyes of approaching adults, which predicted the level of social disability (Jones, Carr, & Klin, 2008), and there was a limited attention bias for faces (Chawarska, Volkmar, & Klin, 2010). Additionally, Klin, Lin, Gorrindo, Ramsay, and Jones (2009) showed that toddlers with ASD preferentially oriented visually to non-social contingencies rather than to biological motion.

ICT-based approaches and methods have been used for the therapy and special education of children with ASD. ICT research has explored several approaches for the treatment of persons with ASD, which are: (i) counteracting the impact of autistic sensory and cognitive impairments on daily life (assistive technologies, e.g., Crittendon, Murdock, & Ganz, 2013); (ii) trying to modify and improve the core deficit in social cognition (cognitive rehabilitation/remediation, e.g., Serret, 2012); and (iii) bypassing ASD impairments to help children acquire social and academic skills (special education, e.g., Lanyi & Tilinger, 2004). Nonetheless, much has yet to be improved to attain significant success in treating individuals with ASD. From a practical perspective, many of the existing technologies have limited capabilities in their performance, which limits the success of ICT treatment in persons with ASD. Clinically, most ICT proposals have not been validated outside the context of proof of concept studies (Boucenna et al., 2014a). Because most ICTs have limitations (e.g., the interaction is not natural, intuitive, or physical), emerging research in the field of autism is aimed at the integration of social robotics (Diehl et al., 2012, Kozima et al., 2009, Welch et al., 2010). Social robots are used to communicate, display and recognize the “emotion” and develop social competencies and maintain social relationships (Fong, Nourbakhsh, & Dautenhahn, 2003).

In recent years an increasing number of studies have focused on the use of robots with individuals who have ASD (Diehl et al., 2012, Scassellati et al., 2012). These studies involve the robots mainly in two roles of intervention: practice and reinforcement (Duquette, Michaud, & Mercier, 2008). Some studies have attempted to evaluate the reaction of children involved in interaction with robots, according to robot-like characteristics (Pioggia et al., 2005, Pioggia et al., 2008, Feil-Seifer and Mataric, 2011a, Feil-Seifer and Mataric, 2011b), and to emotional stimuli (Nadel, 2006, Nadel et al., 2006). Other studies have attempted to use robots for diagnostic purposes (Scassellati, 2007, Tapus et al., 2007) or as a tool to elicit behaviors (Feil-Seifer and Mataric, 2011a, Feil-Seifer and Mataric, 2011b). In other cases, robots have been used as simplified tools that can facilitate social interactions (Duquette et al., 2008) and thereby teach or practice certain skills (Dautenhahn, 2003). Finally, robots have been also used as an agent that can provide feedback and encouragement (Dautenhahn, 2003, Picard, 2010) thus acting as a social mediator during activities between children with ASD and their partners.

In this paper, we used a small humanoid robot, Aldebaran's Nao, in conjunction with an intelligent perception system, during a trial of experiments that involved both children with ASD and matched children with TD. In these experiments, the robot acted as an autonomous interactive partner, proposing to each participant a small set of activities that focus on stimulating JA. The perception system makes it possible to track and register the behaviors of the child, allowing their off-line analysis. To perform the experiments, we decided to use Aldebaran's Nao because it is a humanoid robot with a toy-like simplified shape, it has been used previously with children with ASD and it appeared to be attractive to them (Boucenna, Anzalone, Tilmont, Cohen, & Chetouani, 2014). Also, to explore JA as a cognitive activity per se, we decided to focus on older children with ASD assuming that they developed JA abilities through development and treatment. Exploring them would prevent comparison with TD children to be a consequence of poor JA abilities as show in young children with ASD. We had the following 3 main hypotheses: (H1) our system which employed a Nao robot to interact with children and a perception system based on a RGB-D sensor (Kinect) to capture social engagement cues should be able to elicit JA during interaction with a child. (H2) JA performance of older children with ASD should be close or similar to performance of TD children when interacting with a human partner and with Nao. (H3) Despite similar JA performance, we expected differences between children with ASD and children with TD in the way they explore their 4D (spatial 3D +time) environment during the JA induction task. More specifically, we expected less time spent on target and less movement smoothness during interaction.

Section snippets

Materials and methods

The aim of the system presented here was to provide a set of relevant measures about the social engagement capabilities of children. The collected information should help the understanding of how children with ASD explore their interactive world during JA activities with the Nao partner. A Nao robot was used in conjunction with a perception system based on a RGB-D sensor as a playground mate for children. During the interactions the system could track the position of the child and of his/her

Performance of children during the JA task

For the experiment, we recruited 16 children with ASD and 16 children with TD as the control group matched for developmental age and sex. Each subject performed the JA induction with the therapist during a play session first and then with the robot (see methods). Because of technical issues with recording, two recordings were not exploitable in the TD group. We therefore compared 16 children with ASD and 14 TD children. The characteristics and JA annotation scores of the participants are

Discussion

In this paper, we built a 4-D (spatial 3-D and time) interactive environment system to explore the responses of children during a JA task. This system was mainly composed of a small humanoid robot, called Nao, that could arouse empathy and interact with children, and a perception system that could track and model in real-time a child's body and head movements. The performances shown by the perception system assure that the information is extracted with a high level of reliability, especially

Conflict of interest

The authors declare that they have no competing interests regarding this manuscript.

Authors’ Contributions

MC, DC, and the MICHELANGELO Study Group designed the study; SMA, EA, SB performed the experiment; ET, JX and DC assessed the children; SAM, SB, MC performed the computational analyses; NB, KM and DC performed the statistical analyses; and SMA and DC wrote a preliminary draft. All of the authors read, critically modified and approved the final manuscript.

Acknowledgments

The authors would like to thank all of the patients and families who participated in the current study. They are also very grateful for the support from the hospital staff, in particular Julie Brunelle, MD. The authors would like to thank to A. Carbone and T. Luiz for their kind collaboration.

The current study was supported by a grant from the European Commission (FP7: Michelangelo under grant agreement No. 288241), the fund “Entreprendre pour aider”, the “Fondation Serge Dassault” and

References (55)

  • C. Saint-Georges et al.

    Signes précoces d’autisme: D’où vient-on? Où va-t-on?

    Neuropsychiatrie de l’Enfance et de l’Adolescence

    (2013)
  • E. Bekele et al.

    Pilot clinical application of an adaptive robotic system for young children with autism

    Autism

    (2013)
  • S. Boucenna et al.

    Information Communication Technology (ICT) and autism: Overview and Focus on Early Developmental Issues and Social Robotics

    Cognitive Computation

    (2014)
  • S. Boucenna et al.

    Extraction of social signatures through imitation learning between a robot and a human partner

    IEEE Transactions on Autonomous Mental Development

    (2014)
  • M. Carpenter et al.

    Social cognition, joint attention, and communicative competence from 9 to 15 months of age

    Monographs of the Society for Research in Child Development

    (1998)
  • A. Catarino et al.

    Task-related functional connectivity in autism spectrum conditions: An EEG study using wavelet transform coherence

    Molecular Autism

    (2013)
  • K. Chawarska et al.

    Limited attentional bias for faces in toddlers with autism spectrum disorders

    Archives of General Psychiatry

    (2010)
  • D. Cohen et al.

    Do motherese prosody and fathers’ commitment facilitate social interaction in infants who will later develop autism?

    PLoS ONE

    (2013)
  • J. Crittendon et al.

    Use of an ipadplay story to increase play dialogue of preschoolers with autism spectrum disorders

    Journal of Autism and Developmental Disorder

    (2013)
  • K. Dautenhahn

    Roles and functions of robots in human society: Implications from research in autism therapy

    Robotica

    (2003)
  • G. Dawson et al.

    Children with autism fail to naturally orient occurring social stimuli

    Journal of Autism and Developmental Disorder

    (1998)
  • M.W. Doniec et al.

    Active learning of joint attention

  • A. Duquette et al.

    Exploring the use of a mobile robot as an imitation agent with children with low-functioning autism

    Autonomous Robots

    (2008)
  • D. Feil-Seifer et al.

    Automated detection and classification of positive vs. negative robot interactions with children with autism using distance-based features

  • D. Feil-Seifer et al.

    Toward socially assistive robotics for augmenting interventions for children with ASD

    In Experimental Robotics Springer Berlin

    (2011)
  • I. Fujimoto et al.

    Mimicking and evaluating human motion to improve the imitation skill of children with autism through a robot

    International Journal of Social Robotics

    (2011)
  • P. Howlin et al.

    Social outcomes in mid- to later adulthood among individuals diagnosed with autism and average nonverbal IQ as children

    Journal of the American Academy of Child and Adolescent Psychiatry

    (2013)
  • Cited by (123)

    View all citing articles on Scopus
    1

    The members of the MICHELANGELO Study Group are listed in Appendix A.

    View full text