Keywords

1 Introduction

The assessment of the listening and speaking skills of students who use cochlear implants (CIs) is an essential task that should take place frequently and accurately. Typically, the speech therapists and audiologists use assessment tasks that include different ways to stimulate the students’ auditory comprehension. Beyond the spoken instructions given to the child, a task may involve the use of sounds from the environment such as, the sound of a closing door or vocalizations of emotions; the child must derive meaning from the sounds and pronounce the respective word(s) [1]. The “auditory-verbal method” is a characteristic method for the development of verbal communication for users of CIs; it is a very strict approach which does not allow the use of lip reading (i.e. receiving visual information from the lips of the others [2]). During the therapy sessions, speech therapists often cover their mouths in order to make the child use their amplification device (hearing aid/cochlear implant) to hear the speech sound [1]. Emerging technologies of the new era, might be able to assist such processes and assessment methods.

This exploratory study investigated the potential of using the humanoid robot, NAO, as a tool for assessing the listening and speaking skills of seven hearing-impaired students, through play. NAO does not have a human mouth and therefore, students cannot do lip-reading; we considered this to be a unique characteristic of the technology that can help make the assessment of listening and speaking skills efficient and accurate. Further, NAO has been successfully used in previous studies for play [3], assessment, therapy and education of children with special educational needs [46]. Yet, there is virtually nothing on the use of humanoid robots for the assessment of listening and speaking skills of hearing impaired children. As opposed to using a humanoid robot, [7] for example, used a robotic voice simulator for rehabilitation of the speech of hearing impaired people [7]. Also, a couple of studies have focused on teaching sign language using NAO [8] or other robotic platform [9]. The present study appears unique in the arena of listening and speaking assessment.

2 Method

Participants. Seven students participated, after consent forms by their parents were obtained. This was an exploratory investigation; therefore, participants were children of various ages and different levels of listening and speaking skills. The profile of the participants is presented in Table 1. All students had one or two CIs. Based on input from the school experts, P1, P2, P4, P5 had a successful CI(s) surgery and could potentially hear, learn and understand the verbal language even though, they chose to communicate in sign language. Yet, there were cases with unsuccessful CI surgery (P6 and P7) or delayed surgery (P3) (e.g., the recommended time for the surgery is between 12 and 18 months of age) resulting to failure in the efficient use of the CI. Overall, the sign language was the preferred method of communication for all the participants, leaving their speaking skills underdeveloped.

Table 1. Students’ Profile

Activities and Setting. The study was conducted at the School for the Deaf in Nicosia-Cyprus with the permission of the school director. The robot used for the study was NAO by Aldebaran Robotics, a 58 cm tall humanoid robot that exhibits human-like features. Among others, it has the ability to recognize images, faces and objects as well as respond to speech and other sounds of the environment. NAO was positioned on a table and the participant was seated facing NAO (see Fig. 1). The sign language interpreter was present in the room, in case the children needed sign language or verbal language support for instructions. Upon responding to each activity, the sign language interpreter provided feedback to the participant, as needed.

Fig. 1.
figure 1

Student selects the blue square according to NAO’s instructions (up-left); NAO yawning – child saying which is the demonstrated emotion (up-right); NAO needs to brush teeth – child placing products in the cart (bottom). (Color figure online)

Three game-like applications were developed and deployed on NAO for the purpose of this work. Each game involved a series of tasks to help assess students’ listening, understanding of the verbal language and speaking skills:

Shapes. This game involved listening and following spoken instructions. Two shape-images were positioned in front of the student for each piece of instruction. The participant listened to NAO’s instructions, selected an image and positioned it within NAO’s hands (see Fig. 1). Instructions consisted of two pieces of information: shape and color (e.g. “Give me the shape which has four equal sides and has the color of the sea”).

Emotions. This game involved the recognition of everyday emotions, derived from sounds. NAO asked “How do I feel now?” while he played the sound of an emotion (e.g. yawning) and demonstrated the respective gesture or bodily movement (see Fig. 1). For each emotion, two images were positioned in front of the student as visual clues to help them decide on the correct emotion and say it loud (pronunciation).

Shopping. The game involved understanding of spoken instructions; recognition of everyday sounds in the environment; familiarity with vocabulary linked to everyday routines; and speaking (pronunciation). For each piece of instruction, a total of six different products were placed in front of the student so that she/he could select the ones NAO needed and say them loud (pronunciation). For the first part of the game, two instructions were given; NAO played two sounds (sounds of running shower and brushing teeth) and demonstrated the respective bodily movement e.g., teeth brushing; in this case the student put the toothbrush, toothpaste and dental floss in the cart (see Fig. 1). For the second part of the game, three instructions were given with NAO requesting some products e.g., “I love honey cake, so please place in my shopping cart the products I need to cook it, and tell me their names”; in this case, the student selected the correct products, said their names loud, and placed them in the shopping cart (e.g., honey, flour, sugar and a cake box).

Data Collection. Data were collected in three forms: (i) an attitudinal questionnaire administered to the students right after the completion of the activity; (ii) video recordings of the activity; and (iii) interview sessions with the special education teachers of the School, the sign language interpreter and the technology (IT) teacher who had the opportunity to observe the study.

3 Results

In this poster, we elaborate on students’ performance data as well as, the attitudinal questionnaire on their experience. The results from the full data corpus are presented elsewhere (please check http://cyprusinteractionlab.com/).

In regard to students’ performance, all students responded to the activities, often beyond the expectations of their teachers. Video analysis focused on recording the number of correct answers (on their 1st try), the correctly pronounced emotions or products, and times the participant sought help, either in the form of sign language (SL) interpretation or verbal language (VL) repetition of instructions (see Tables 2, 3 and 4). The results provided evidence of the most and least competent students in performing each activity.

Table 2. Performance on Shapes game – Pick the correct shape (N = 7)
Table 3. Performance on Emotions game – Tell me how NAO feels (N = 7)
Table 4. Performance on Shopping game – Fill in NAO’s shopping cart (N = 7)

With regards to the attitudinal questionnaire data on the participants’ experience, the use of NAO and associated game-activities were positively endorsed by the participants (see items 1–4 of Table 5), while there was some variation in students’ responses based on items 4–7 of Table 5 about hearing and understanding. The questionnaire also included an open-ended question for additional comments; here, two high performing students (P4, P5) reported that they would like NAO to speak at a slower pace, in order to clearly hear his instructions.

Table 5. Questionnaire results on participants experience (N = 7)

4 Discussion and Conclusion

The assessment of the listening and speaking skills of students who use CIs is an essential task that should take place frequently and accurately. Emerging technology, such as humanoid robots, might hold promise in this area. This exploratory study assumed the potential of NAO as a playful tool for assessing the listening and speaking skills of hearing-impaired students; the absence of a human mouth on NAO prevents lip-reading and therefore makes him an ideal tool for this task. Findings from this work, suggest that NAO was successfully used in this context, with all participating students responding to activities, even beyond the expectations of their teachers. Moreover, NAO was positively endorsed by the participants, which confirms previous research findings on the use of humanoid robots with children [3, 4]. Future research should continue to examine the unique characteristic of this technology that can help make the assessment of listening and speaking skills efficient and accurate in addition to fun. A full report of our results, including findings from our interview data are presented elsewhere. Our results, although preliminary, should encourage future research in the area of assessment as well as speech enhancement via play with social robots.