Keywords

1 Introduction

Role-play provides an effective way for participants to “dive in” to a particular situation, enabling them to learn to adopt other perspectives, often that of another person, in a simulated situation [1, 2]. In social- and medical-study activities, role-play is recognized as a suitable technique for the study of social situations including dynamic, complex, and non-routine situations such as decision-making training. Crisis management training [3, 4], nursing or medial situations [5, 6], and classroom teaching [7] are good examples that adopt role-play for decision-making training. Decision-making in a crisis is critical; however, it is ill-structured, involving communication among multiple people or parties and considerable information from various sources. Nursing and medical training often adopt case-based simulations in real complex situations. For quality service, nursing and medical staff should learn methods for dealing with hysterical patients or other stressful situations during bedside care through role-play. Pre-service teacher training often uses “microteaching” as a teaching simulation for learning teaching techniques before practical training in actual schools. Teachers work in ill-structured, dynamic, and non-routine environments, focusing on communication with their students, in classroom teaching. They need to estimate the student capacities, and contrive and decide teaching methods to enable better student understanding and satisfaction.

One interesting aspect of these kinds of role-play in complex situations is that they often emphasize on improvisations than the use of scripted scenarios to enable the participants change their fundamental belief system. Improvisation in role-play is a particularly powerful learning strategy because it provides the participants a safe opportunity to undertake character roles or parts, in a particular situation [8]. Such role-play can be structured to be counter-attitudinal such that biases can be overcome through the development of the new beliefs fostered during role-play. The improvisations created during role-play are reflections of the agency’s daily life, based on the role players’ experience and require the participants to pay attention to all forms of feedback available in the role environment [9].

The importance of such improvisation in role-play for learning decision-making and communication in complex situations can be explained in terms of Bakhtin’s theory of dialogism. According to Bakhtin [10], all utterances can be seen as replies to the voice of the preceding person, as the speaker takes into account his listener’s background knowledge, previous utterances, gestures, etc. in predicting the listener’s likely refutation, and, in order to avoid the same, will fashion his argument accordingly [11]. Hence, one’s utterances will be formed with the anticipated words of the listener in mind; at the same time, the follow-up response of the listener will also be foreseen. In this sense, utterances in themselves, can be said to constitute dialogue. Bakhtin termed this essential character of utterances as multivoicedness. In this dialogic view, decision-making and negotiations in complex situations are the process of forecasting the reactions of the actors in the situation, engaging in hypothetical dialogues, and incorporating the results into the method of dealing with the situation. From this point of view, involving improvisation in role-play is crucial because the situation is multivoiced and role-play requires the ability to vividly imagine diverse actor voices that are rooted in the individual values and background of each participant. Therefore, engaging the participants in imagining the situation from the standpoint of the actors, as “others”, is central to role-play.

However, it is not easy, particularly for novice students or citizens, to envision the reaction of the actors to ones’ dealing strategy, in the situation. Even though role-play is more effective in enabling shy students to participate in learning activities than the usual lectures, some of the participants cannot play their roles effectively due to self-consciousness [12] or evaluation apprehension [13]. For example, microteaching role-play in pre-service training requires the participants to play their roles collaboratively as a teacher and young pupils, and to simulate an actual classroom themselves, in order to demonstrate his/her teaching in a real context, as far as possible. However, reactions and feedback provided by the colleagues acting as pupils are sometimes out of context, although they play the role of considerably younger people. Further, their reactions and feedback cannot be ensured to be serious, honest, or realistic due to embarrassment or hesitation. Such ineffective microteaching cannot directly improve a pre-service teacher’s ability to imagine possible learner reactions.

For scaffolding the participants’ dialogic imagination in role-play, we argue that puppetry can be a catalyst for enabling people to engage in the face-to-face conversation and for eliciting various realistic reactions or responses (multivoices) from the participants, enabling the perspective-taking of a wide variety of people during role-play. Puppetry allows each participant to obtain a participant-observer balance by creating a clear separation between the self (puppeteer) and non-self (puppet), as well as the character (puppet) and observer (puppeteer), while playing a puppetry story; therefore, participants can use informal/irregular discourse more in puppetry, similar to a realistic situation, than in self-performed role-play [14].

This paper describes a tangible puppetry-based role-play system, which decreases the participants’ anxiety of acting each role sufficiently, enabling them to reflect the role-play effectively. The design and features of the system, and its applications in practical education are presented.

2 System Design and Implementation

2.1 Overview of the EduceBoard System

The EduceBoard system was designed such that users can achieve and dramatize the possible insights of the various characters in a role-play with puppets. The system has two functions: (1) engaging-in and recording the actions and conversations of puppetry-play on a tabletop system, and (2) playing the recording online to enable the participants to reflect upon their performance [15].

The system comprises of puppets, which represent characters (such as students, patients, etc.) on a transparent table that is the arena for the puppetry, a small microphone to recode the puppetry conversation, small lights to adjust the lighting, and a server to store the puppetry data and generate a puppetry animation movie for reflection. Figure 1 shows the overview and architecture of the system; Figs. 2 and 3 depict the actual positioning of the system and the puppets, and Fig. 4 displays the players playing puppetry using the tabletop system collaboratively with the small puppets.

Fig. 1.
figure 1

Tangible puppetry simulation system overview

Full size image
Fig. 2.
figure 2

Tangible puppetry simulation system

Full size image
Fig. 3.
figure 3

Tangible puppet interface (Color figure online)

Full size image
Fig. 4.
figure 4

Tangible microteaching puppetry (Color figure online)

Full size image

The system records the actions and conversations of the participants (hereafter, the “characters”) on a transparent table. Figure 2 depicts the system, when it is ready to conduct the puppetry role-play. Each puppet or prop is attached to a transparent box with an augmented reality (AR) marker on the bottom. Each character can express his or her puppet’s condition by manipulating a switch to change the color of the LED to red or blue (Fig. 3); in microteaching role-play, for instance, a red LED may represent a sleeping or careless student and a blue LED an attentive/note-taking student. We can assign various statuses for the characters, as per the situation played by the participants (for example, if they play nurses and patients in a hospital, these statuses might be illness, anger, hunger, etc.). A web camera and microphone, under the table, record the puppet movements and conversations (i.e., the behavior of the characters), by detecting the AR markers. After role-play (Fig. 4), the system sends all the recorded data, including the puppet movements, conversation, and conditions, to the web server to enable the participants to view the recorded puppetry and reflect upon it (Fig. 5). The webpage displays the role-play in an animated form, from a bird’s-eye view. The users can stop and resume the animation, as desired and can also manipulate the seek-bar to find appropriate points to discuss or to include comments for reflection, on the timeline located next to the animation.

Fig. 5.
figure 5

Web-based reflection-support system

Full size image

The application programming languages are Visual C++ 2008, Action Script 3.0, and Javascript; Microsoft Visual Studio 2008 and ARToolKit were used for the development environment. MySQL was used for the database and PHP was implemented as the interface for connection with the database. We discuss the implementation of the EduceBoard system in detail, in the following sections.

2.2 Functions for Capturing and Recording the Locations of the Puppets and Furniture

We have adopted a tabletop interface [16] for implementing the technical support of the puppetry-based role-play simulation. This interface enables all the players to access to the puppetry arena such that the players can interact with each other without physical barriers, offering an open collaboration environment for fruitful collaboration [17]. Although various implementations for a tabletop computing interface have been proposed, vision tracking is a promising implementation, which enables the system to identify the locations of the objects on the table. Markers and the image processing technology for vision tracking that identify each marker location, enable the identification of the various puppet locations on the tabletop; thus, many studies for tabletop collaborative learning systems adopt this solution [18, 19].

We have adopted the ARToolKit [20] as the tool for vision tracking, attaching a marker at the bottom of each puppet box (see Fig. 3). Each marker represents a character in the puppetry; for example, in microteaching puppetry, boy student A, boy student B, girl student C, girl student D, female teacher E, etc. In addition, some of the markers represent furniture. Each piece of furniture is distinguished such that all the characters and furniture are depicted correctly in the reflection animation. All the data regarding the assignment of the characters and markers are stored in a database, in the webserver in which the reflection animation is generated.

The tracking system requires the calibration and establishment of a coordinate system defined by locations with known coordinates within the acquisition region. Hence, users must locate a puppet with a standard marker on the four corners of the tabletop, the first time, at the start of the puppetry; with this, the system can know the fixed points and edges to identify the size of the stage for calculating the relative positions of the characters to generate the reflection animation. The ARToolKit records the coordinates of the markers with the bottom left corner of the captured image as the origin, parallel to the x-y plane in which the moving markers are located; no data regarding the z-coordinates is used. In addition, the attitude or direction of each puppet or furniture is captured from −180° to 180°. at a negative clockwise and counterclockwise positive with reference to the direction facing the top edge of the captured stage. Note that we suspect certain unexpected error recognitions by the ARToolKit; thus, we have implemented a procedure to omit impossible movements, such as the short appearance of a certain marker and immediate distant movements.

2.3 Functions for Fostering and Capturing the Non-verbal Representations in Puppetry

Users need to express and recognize non-verbal representations also to achieve a more realistic communication among the characters, during puppetry. In addition, replaying such non-verbal communication is crucial in recognizing realistic communication and for discussing user participation in such communication, in order to promote participant reflection, after the puppetry, to achieve effective experiential participatory simulation.

Hence, we have included another function that lights-up colors on the markers; each human character puppet has LED color lights that are projected onto the marker at the bottom of the transparent box and there is a switch on top of the box to manipulate the light status. This allows users to manipulate the non-verbal status of the puppet, such as an emotion or concentration, as a visible expression, both for the puppetry players and for the camera under the tabletop for capturing the location and status of each puppet (see Figs. 3 and 4, for example). The transparent box permits the projection and reflection of the LED color on the back of the AR marker, enabling the observation of the non-verbal status of each puppet during puppetry and displays the color of the status for the camera. Each box has red and blue LEDs to enable the users to express three different statuses, for the abovementioned purpose.

The allocations of the color and status are stored in the database in the webserver; hence, the reflection animation can include the non-verbal representation of each human character. For example, for microteaching puppetry, we have assigned three typical pupil reactions during class hours: distraction (such as sleeping), concentration (such as note-taking), and a normal reaction. We believe that these reactions can assist the teacher in distinguishing student attitude, during class hours. This color allocation must be defined based on the context of each role-play situation (i.e., microteaching, nursing, or crisis management), similar to the character definition in the database to generate the effective reflection animation described in the next section. Before starting the puppetry simulation, the facilitator should explain this function and the non-verbal expression indicated by each color.

The client software of the EduceBoard system, with the ARToolKit processes, includes the following: It detects the color information in the RGB colorimetric system within each square area of the AR marker pattern captured by the camera and records the state of each character, as well as its location. In this procedure, the software distinguishes two image systems, the AR marker system and the visible light identification system; the former processes the location information of each AR marker, whereas the latter, the non-verbal representation information of each character. For processing the visible light identification image, the image processing system detects the number of pixels that fall within the threshold value of the red or blue in the RGB colorimetric system, when the red or blue LED is turned on. If the ratio of the blue or red color exceeds a certain value, the system judges and records it as the lighting-up of the respective color, else it is recognized as an off-state (i.e. normal). This threshold value can be configured in the client software, based on the influence of ambient light at the installation location, such as a classroom. While playing puppetry using the EduceBoard system, the software captures, analyzes, and integrates the information from both the image systems into a single file, and then sends and records it to a table in the database, as shown in Table 1.

Table 1. Marker information table
Full size table

2.4 Interface for Reflection by Playing the Recorded Puppetry Role-Play

After the role-play, reflection confirming what the participants experienced as players, discussions on their actions and how they would change it in the next trial, including the self-performed role-play, is critical for the participants to acquire skills, competencies, attitudes, and strategies that can be learned through role-play simulation [12, 21].

Video-recording is the most common way to support such reflection, for role-play simulation. However, capturing the puppetry on a small tabletop can be difficult because there are several obstacles in capturing the puppet movements correctly, including the players around the tabletop, their hands and arms, when moving the puppets or switching the LEDs, etc.

Instead of video-recording the actual puppetry, the EduceBoard’s Web system provides a bird’s-eye view of the animation of the puppetry simulation, using the data and voices captured by the camera and the microphone in the tabletop system. This system captures the locations and non-verbal expressions of each puppet character so that the animation can provide trajectories, including the verbal and non-verbal actions of each character, to effectively foster player reflection [22]. Further, the system also provides review and comment functions for mutual asynchronous reflection outside the classroom or school, particularly for learners who were immersed in acting during the simulation. This animation is generated using Adobe Flash technology.

Figure 5 shows a sample of the Flash animation, reflecting the movements of the puppets; the recorded sounds can be played simultaneously. When a user opens the Web reflection interface by selecting his/her experienced simulation, the Flash module calls two PHP modules with the simulation information (i.e., session number and trial number described in Table 1) via HTTP. Then, one of the PHP modules retrieves the recorded puppetry information such as the time, location, attitude, and status of each puppet from the database and organizes them into an XML; whereas, the other retrieves the captured voice data of the corresponding puppetry on the server. The Flash module loads and caches this data and waits for the user’s action.

When the user clicks the play button, the Flash module starts to generate the animation, presenting images of the characters and furniture assigned to each marker, according to the database. As described above, the database stores the allocations of the markers and characters or furniture images, which include all the status/attitudes of the character/furniture. Figure 6 depicts examples of the character images according to the non-verbal information and directions, in the case of microteaching puppetry. The Flash module depicts these character images according to the information in the timing, described as timestamps in the XML, thereby, generating an animation. When the information indicates that a character’s status or direction changes, the Flash module starts to use the appropriate image, when necessary. If a character’s status does not change for a while, the Flash module displays the same image for that character, according to the information on the XML. To present a natural trajectory, the Flash module uses the motion tween function to generate a smooth movement, adjusting the timestamps provided by the ARToolKit.

Fig. 6.
figure 6

Character images according to the non-verbal information and directions of the figures

Full size image

Users can review this animation by manipulating a seek bar on the bottom of the Flash movie interface in order to confirm/replay the puppetry role-play. The seek bar is generated based on the length of the puppetry voice data and the Flash animation is linked to the seek bar so that the Flash module can reload the XML data and reallocate the character images according to user manipulation on the seek bar. Thus, the users can replay the animation correctly, when required, and reflect upon a specific part of their play.

Reflecting upon the puppetry by watching the animation movie requires more time than the actual time of the experienced puppetry. For educational purposes in schools or classes, providing an opportunity to reflect upon the conversations and actions later at a set time period is crucial for considering the communication and strategies, in accordance with the simulation. The comment function beside the Flash animation field enables users to add comments interactively in the timeline. While watching the animation, users can write a comment with a time tag. When reviewing the animation, all the comments are listed as per the attached time and the comment linked to the current time being played in the animation blinks to attract user attention. Users can also observe the viewpoints of the other participants in the puppetry simulation, deepening their reflection.

2.5 3D Animation for Promoting Deeper Perspective-Taking

Transfer to self-performed role-play or actual practice, after tangible puppetry, is an important issue. For self-performed role-plays, we determined that the transfer of multi-voiced perspective-taking was not effective, even immediately after the tangible puppetry role-play [6]. One promising way of fostering deep perspective-taking may be a complete shift of viewpoint. Lindgren [23] indicated that experiencing a first-person perspective in a virtual world can generate person-centered learning stance and perspective-taking, discussing the possibility and efficacy of inter-identity technology (IIT) for learning from the viewpoint of a virtual learning environment such as the MUD or MR (mixed realities). They discussed that the IIT enables the learner to see through the avatar’s point of view and this blurs the boundaries between the self and the other; hence, the players can gain novel perspectives.

We consider this to be a potentially effective method for eliciting more discussion not only on the performance of the various characters in the role-play, but also on the content of the reflection and on the conversation regarding this performance from the viewpoints of the diverse characters.

Therefore, we developed an additional system function, which allows participants to reflect upon their role-play by combining the wide and thorough (bird’s-eye view) views of all the dialogues, and the various participant views (character points of view), using Unity3D technology (Fig. 7). From the bird’s-eye view (Fig. 3), the participants can examine the overall situation, whereas, from the character points of view, they can consider the possible reactions (communication and behavior) of specific characters. The participants can switch the interface, while watching the role-play animation; hence, they can consider the first-person perspective of each character, when necessary.

Fig. 7.
figure 7

Character’s point of view for reviewing tangible puppetry role-play in microteaching

Full size image

3 Applications for Educational Purposes

We have conducted and plan to conduct evaluation studies in several fields, in which the EduceBoard system can be beneficial for learning various communication strategies.

3.1 Microteaching Role-Play in Teacher Education Program

One promising application is the microteaching role-play in teacher education. Microteaching, in which teachers are trained to engage in communication and decision-making in the classroom, requires the participants to play the roles of a teacher and young pupils, realistically. However, as described in the first section, the reactions and feedback provided by the student teachers acting as young pupils tend to be unrealistic, during self-performance, as they often disproportionally play honest students, who follow the teacher’s instructions.

We have conducted several preliminary evaluations of the EduceBoard system by comparing with self-performed role play in the context of a university teacher training course with microteaching practices; this demonstrated that the system enables the participants to play various roles in the role-play and assisted them in reflecting upon their role-play from several perspectives [15].

3.2 Nursing Education

Another promising application is in nursing education, particularly for nursing involving multiple duties with interruptions. This training is usually conducted with a scenario-based scripted simulation [24], enabling nurses to reflect upon their own actions and observe optimal nursing practices including the realization of the importance and difficulty of priority management, efficient operations, and collaborative problem solving. However, this training lacks the various viewpoints of patients, which cause interruptions. Understanding these viewpoints through tangible puppetry is a promising way to understand the patients’ feeling regarding the nurse’s decisions and to reflect upon such practices.

We have developed new types of characters and puppets that are appropriate for nursing education (Fig. 8). Before implementing these characters on the system, we plan to conduct a case study of the puppetry role-play in a situation involving multiple duties with interruptions; this kind of role-play training entails significant costs and efforts such as room reservations, patient players, facilitators for reflection, considerable time for reflection by reviewing the video recording, etc. We believe that even by using only puppets, the training can be achieved, with reduced costs.

Fig. 8.
figure 8

Patient and nurse puppets for nursing education

Full size image

4 Conclusion

This paper describes the implementation of a tangible puppetry role-play support system that fosters perspective-taking and discusses several system applications. The use of puppets in role-play as transitional objects that prompt a projection of the self (puppeteer) on the non-self (puppet), elicited a variety of informal discourse, rarely used in self-performance, but useful for studying the various perspectives in a possible scenario. The EduceBoard system is a tangible puppetry role-play simulation system that assists improvisational role-play such as microteaching by enabling the students to play using puppets. It also provides web animation and comment functions for reflecting upon the role-play recorded in the server.

The issues to be addressed in future are as follows. (1) Auto adjustment for ambient light in various learning environments. Due to the nature of vision tracking, ambient light significantly affects image recognition accuracy. An auto adjustment function for the ambient light is required, in this system, in a general setting in the dissemination phase. (2) We need to consider the use of the tabletop as the display in the reflection animation movie to support timely trial-and-error simulation, during puppetry. This can achieve a different type of reflective learning experience in role-play simulation, enabling the participants to stop the simulation and discuss, and conduct another simulation from the interrupted point. This is not yet included in the scope of our research, but would be challenging from the technical point of view.