Jigsaw: Authoring Immersive Storytelling Experiences with Augmented Reality and Internet of Things

3.1.1 Benjamin Franklin Kite Experiment.

The Benjamin Franklin Kite Experiment is a story of Benjamin Franklin and his son taking a kite out during a storm to see if a key attached to the string would draw an electric charge (as seen in Fig. 2a). To play the Benjamin Franklin story, three users are required to be co-located with each user holding a phone. At the beginning of the experience, users coordinate according to the app and assign each user a different role: a narrator, an actor of Benjamin Franklin, and an actor of Benjamin Franklin Jr. After assigning the roles, users can see the two actors’ bodies augmented with virtual costumes on them. The app then asks the narrator to narrate the story by reading a physical book out loud. As the narrator reads the story on the physical book, the app uses voice recognition to transcribe what the narrator says and automatically detect keywords in the transcription. A list of keywords that are selected based on the physical book and their corresponding scenes are predefined in the app. As one keyword is detected, the app will jump to its corresponding scene and wait for the next trigger. In this story, we defined a total of six pairs of triggers and scenes, with the last trigger being a touch event between users’ hands and a virtual key in AR.

In this experience, we focused on using IoT devices as background augmentation, having human actors interacting with the story, and having a narrator reading from a physical book. The augmentation is considered as in the background since IoT devices add to the story’s ambience, such as dimming the lights to simulate cloudy weather. The visualization of costumes and interaction of touching the key are designed to facilitate the sense of participation. Lastly, the design of reading a physical book is to increase the sense of physicality, where the physical interactions of flipping pages and reading printed text can be meaningful in the experience.

3.1.2 The Wind and The Sun.

The Wind and the Sun story is from the famous Aesop’s Fables, where the two characters quarrel about which of them is stronger (as seen in Fig. 2b). Any number of users can play the story with each user holding a phone. The system narrates the story through the smart speaker. Users can tap on the screen to make the story proceed. In this story, the IoT devices act as the driving actors of the story with visual augmentations from the phone’s AR, where the system superimposes the Wind on the smart fan and the Sun on the smart light.

Compared to the previous story, this experience is similar to a 4D movie with less focus on interactivity. The users take a passive role, and the primary user who would have been the narrator before now only has control of moving to the subsequent scene.

3.1.3 Goodnight Moon.

We recreated Goodnight Moon [16], an American children’s story written by Margaret Wise Brown in 1947. The narrator greets objects around the room (e.g., “Goodnight, red balloon. Goodnight lamp. Goodnight fan”). The greeting causes an appropriate change in the greeted AR or IoT device (e.g., the balloon goes down; the smart lamp turns off; and the smart fan stops blowing wind). The story progresses with keyword detection as the narrator speaks, similar to the Benjamin Franklin story. However in this experience, the story is shown on the screen rather than on a physical book (as seen in Fig. 2c).

Because the narrator explicitly greets the IoT devices, the devices take on more of a foreground role. While the audience still has a passive role, the narrator is now an active, interactive participant in driving the story. The one-to-one mapping of each narration line with an effect on some object (AR or IoT) creates a simple but magical experience.

3.1.4 Implementation.

We implemented the three stories using Lens Studio¹, which provides AR tracking, multi-user co-located experience, body pose detection and speech-to-text/text-to-speech capabilities. These features are necessary to enable the rich set of interaction techniques of the system, and motivated our choice for Lens Studio. We connect and interact with the smart light using Kasa Smart’s API [10]. We use an IR-controlled fan, along with Bond Bridge [8] for networked control of IR devices. Lastly, we use Amazon’s Echo smart speaker [9] connected to the narrator’s phone via Bluetooth.

3.2.1 Scene-based Authoring and In-situ Editing.

Jigsaw enables novices to author immersive stories by creating a sequence of scenes in the editor, which draws inspiration from prior work on scene-based authoring (e.g.,  [18]) and presentation tools such as Google Slides.

In Jigsaw, users are able to define the following primitives of immersive stories via Jigsaw’s authoring tool:

In Jigsaw, three trigger types are available: tapping on the screen, keyword recognition via speech, and touching a virtual object in AR. Keyword triggers, as those in the Goodnight Moon story, allows users to define custom keywords for entering each scene. This is incorporated since verbal narration has been a key component of storytelling experiences. The touching trigger enables users to enter a new scene when their hands collide with an AR object. This is one of the basic interaction in AR applications in order to add to the interactivity of the stories. Lastly, tapping is the default trigger for stories that are narrated by the system where the user enters a new scene by tapping on the screen.

In addition to defining triggers to different scenes, Jigsaw allows novices to edit behaviors of AR content and IoT devices within each scene directly in AR. This is aligned with the spirit of What-You-See-Is-What-You-Get that numerous authoring tools employ to lower the barrier for end-users to create immersive content [25, 62]. An interface for importing 3D assets is shown on the screen when the user edits the scene. By saying the name of the 3D asset that they want, users can add the 3D asset to the AR scene. They can then adjust the placement of the asset by tapping and dragging the asset on the screen; they can also adjust the depth dimension by moving the phone in the physical environment while dragging the 3D asset. Jigsaw also enables in-situ editing of IoT devices including smart lights, smart fans, and smart speakers. A corresponding editing interface is popped up when the distance between the user’s phone and the IoT device is below a pre-defined proximity threshold. For the smart light, users can edit its color, brightness, and special effect. The color is visualized in a semi-circular palette and the user can change its color by tilting the phone. Users can change the brightness using the same control, where the value of 0 will turn off the light. Lastly, users can add special effect of smart lights by saying the name (e.g. flickering). We envision delegating the low-level implementation of special effects to advanced developers and only expose the effect name to novice creators. Similarly, users can edit the on/off and intensity of the smart fan, and the on/off, volume, and special sound effects of the smart speaker.

3.2.2 System Walkthrough: Building the Goodnight Moon Story.

We present a system walkthrough involving a general end-user who has little to no technical background and is motivated to use Jigsaw to create the Goodnight Moon story that end-users can play with their families and friends (as seen in Fig. 3).

Opening Jigsaw’s editor, the user first selects the narrator of the story to be users rather than the system. After selecting the narrator, the user then enters the initial scene which is an empty scene. The first scene that the user is going to build for the story is dimming and changing the color of the smart lights when the user says “In a small, cozy room...” From the initial scene, the user can create a new scene. Before entering the new scene, the system prompts the user to define a keyword; in this case, the user speaks out the keyword “small" and confirms. After the trigger is defined, the user can edit the new scene by defining behaviors of the smart light. As the user walks closer to one of the lights, an editing interface of the smart lights is shown in AR based on the user’s proximity to the lights and allows the user to edit the status of the lights. The user can toggle between changing different attributes of the smart light including the brightness and the color. They can then turn down the brightness and change the color of the smart light. Jigsaw allows users to define multiple behaviors in a scene (e.g., they can walk up to other IoT devices to edit additional behaviors) though in this specific story the scene only contains the behavior of one device, i.e. the smart lamp. In this way, the first pair of trigger (i.e., the keyword “small, cozy room”) and scene (i.e., changing the brightness and color of smart lights) is recorded in the system.

Upon finishing the first scene, the user can add a new scene, which saves the existing scene and jumps to the definition of a new trigger. The second scene that the user is going to build for the story is making a red balloon appear when the user says “There was a red balloon.” The user then defines the corresponding keyword as the trigger and starts editing the new scene. In addition to changing the status (e.g., on/off and intensity) of IoT devices, Jigsaw also allows users to create, manipulate, and delete 3D models or animation. In this case, to add a balloon model, the user can tap on the corresponding button and say the model name (e.g., “red balloon") directly to the system. The system then dictates the model name spoken by the user and shows the corresponding model in AR. The user then places the content in the physical space by simple drag-and-drop interaction; alternatively, they can delete it by dropping it to the trash icon on the screen. The system can create a fade-in, fade-out, and moving animation by default. For example, when scene 1 does not contain a balloon and scene 2 has one, the system can create a fade-in effect for the balloon in scene 2 during playback. Similarly, the system automatically applies a translation animation to a 3D object by moving from the position of the previous scene, to the position of the next scene during playback. In this way, the user can author scenes such as a moon model appears and disappears. After creating the two scenes, the user can create the following scenes in a similar manner (note that the Goodnight Moon story contains 11 pairs of triggers and scenes in total) as the story proceeds. Jigsaw allows users to navigate and edit previous scenes. For example, the user can navigate to the first scene and change the color of the lamp. Finally, using Jigsaw, the user can play the storytelling experience immediately in the authoring environment.

Figure 4:

View Figure

3.2.3 Generalizability of the Authoring System.

A key factor that makes our authoring system generalizable is that the the triggers and scenes that users create are open-ended and not coupled with any specific story. For example, a user can define any keyword or touching with any AR object as the trigger and use the brightness of smart lights for any kinds of story. To demonstrate the generalizability of the authoring system, we use replicated examples [41] by replicating the other two immersive stories using the same system.

To replicate the Benjamin Franklin story, users can define pairs of triggers and scenes like they do in creating the Goodnight Moon story. The unique challenges of creating this particular story include allowing multiple behaviors in one scene and defining touch triggers. Different from the Goodnight Moon story, the Benjamin Franklin contains multiple behaviors triggered by one keyword in a scene. For example, when the user says “when the clouds first passed over”, the smart fan will turn on and the AR cloud model will appear. To achieve that, users can walk up to the smart fan and adjust its state, and then import the 3D model of a cloud while staying in the same scene. After editing all the behaviors in a scene, the user can proceed to add a new scene, which indicates finishing the current scene. The Benjamin Franklin story also incorporates touch triggers, where the user touches the virtual key model and triggers the flickering effect of smart lights, as seen in Fig. 4cbesides keyword recognition when adding a new scene. After selecting touching as the trigger type, the user can then select the object for collision detection by touching it directly in AR.

The Wind and the Sun story is different from the Goodnight Moon example since it requires the system instead of the user to narrate the story. Jigsaw enables this by allowing users to select narrators at the very beginning of the authoring experience, as seen at the beginning of Fig. 3. After selecting the system as the narrator, the user can add a narration to each scene by saying it directly to the system. The system can then transcribe the narration and play it through the speaker during the playback, as seen in Fig. 4a. When the system is assigned as the narrator, the system will omit trigger selection when adding a new scene and the trigger type goes to default, i.e. tapping on the screen. Users can import 3D assets such as the Wind character and place it in the physical environment such as anchored to the smart fan. They can also edit the intensity of the smart fan as seen in Fig. 4b

3.2.4 Implementation.

The authoring system is implemented using Lens Studio for displaying and interacting with AR content. The connection with IoT devices is built using the same technique as the implementation of the immersive stories mentioned above. We used the built-in voice recognition and hand tracking component in Lens Studio to enable users to define keyword and touch triggers. The playback of the final experience among multiple users is implemented using the built-in connected lens component.

4.3.1 Experience session.

In the first session, we asked the participants to play the three immersive stories that we created in the order of Goodnight Moon, the Wind and the Sun, and Benjamin Franklin. We started with Goodnight Moon for its simplicity, where each trigger links to one behavior, and concluded with the more complex Benjamin Franklin story involving multiple behaviors and interactions. We chose this story order to progressively familiarize users with the immersive storytelling landscape, aligning with our goal of exploring this emerging paradigm of immersive stories. Before playing each story, participants were debriefed about the story background and their roles in the story. Since the Benjamin Franklin story requires three people to experience, one member from our study team acted as the Benjamin Franklin Jr. In this story, one participant acted as the narrator holding the physical book and the other participant acted as the character listening to the story and touching the key. Our team member filled this audience role their role was only listening to the story, which overlapped with existing participant roles, ensuring all participation aspects were experienced. After experiencing the three stories, we conducted a semi-structure interview with both participants to ask about their experiences and the benefits and challenges of the immersive stories.

4.3.2 Authoring session.

In the second session, we spent the other half an hour evaluating the authoring system with one randomly picked participant from the group. We decided to ask participants to first experience and then recreate as we believed it is essential for them to understand this new paradigm of immersive stories (i.e. “what" immersive stories are) before coming up ways to craft them (i.e., “how" to create immersive stories). The participant was first given a 10-minute tutorial introducing the functionalities of Jigsaw’s authoring tool, including defining triggers, creating scenes, and editing behaviors of 3D models and IoT devices. After ensuring the participant understood how the authoring system works, we then asked the participant to complete the task of recreating the Goodnight Moon story that they experienced in the first session. We asked participants to replicate existing stories rather than creating their own since having one consistent story across all participants can help us determine whether participants were able to create immersive stories using Jigsaw’s authoring tool. In the evaluation of Jigsaw’s authoring tool, we also emphasized the creation process rather than the story content. During the task, the participant was provided with the complete script on a piece of printed paper including keywords and their corresponding behaviors. An example of the first three scenes in the script can be seen in Table 2. We provided this so that the participant could focus on how to create the story using the system instead of what to create. During the task, the study team was not allowed to provide any help unless the participant explicitly asked for help. Any question that the participant asked during the task was noted down. The task included setting up 11 pairs of keywords and behaviors and took 10 minutes in total. Finally, we conducted a 10-minute semi-structure interview with the participant to ask about the usability and utility of the authoring system.

5.1.1 IoT devices made the experience immersive, engaging, and memorable.

Immersion is often associated extensiveness (i.e., the range of sensory modalities accommodated) and surroundings (i.e., the extent to which the experience is panoramic rather than limited to a narrow field) of the experience [55]. Our results suggested that, in Jigsaw, the augmentation via IoT devices increases the extensiveness to make the storytelling experience more immersive by invoking more senses such as haptics from the smart fan.

“if you just had AR simulated wind, there’s something that’s not as exciting about the fact of having real wind and having that effect come together...the wind being real because you can feel it."-P9B

In line with our motivation to create AR experiences beyond just “pixels" on the screen, participants also reported the storytelling experience being immersive as it extends from the screen to the IoT devices in the surroundings .

“it’s not only are you seeing it on the screen, it had panned out a little bit further and not only do you see it on the screen, but you’re feeling it too, so it’s 4D as opposed to just the AR experience of seeing it.” -P11B

With the enhanced immersion, the storytelling experiences became memorable and engaging to participants due to the various neural stimuli involved in the experiences.

“I think it (without IoT) would’ve been less of an immersive experience and also probably little less memorable. So you remember, "Okay, at some point the moon went to sleep, the lights went out," and it’s a bit easier to remember visual cues than it is just reading it off. And then also staying with the story was a lot easier because things are changing around and keeping me engaged." -P2A

5.1.2 Users’ participation and interaction improves the sense of immersion and agency.

We also found that allowing users to act as different roles (e.g., narrators or characters in the stories) in the stories enhanced the sense of immersion. This is because when users act as characters in the stories and their interaction with the story elements triggers certain effects, it reinforces the impression that they are embodied as the character; when users act as narrators, they tend to focus more on the storyline and feel immersed.

“It makes you feel like you are in a theatre with the whole background noise, the light changes, and the fact that you can touch things virtually... I was very involved in everything going on, so I was definitely very immersed." -P5A

“I think being the narrator helped me be immersed in the story more. I think maybe because I had to focus on the story more... Since I’m narrating, I have to think of what’s going to happen next or think what I have to say next. Whereas when I’m an audience member, I think it’s a little easier to get distracted by what’s going on and then lose what the computer’s saying." -P7A

Agency refers to “the satisfying power to take meaningful action and see the results of our decisions and choices" [45]. In addition to prior digital storytelling experiences that explore the sense of agency via questions and answers [64], we found that being the narrator in the immersive stories can give participants the sense of agency since they feel that they have control over how the story goes, by closely examining the mapping between triggers and scenes based on their own paces.

“I definitely enjoyed being a narrator more because I felt more in control of the actions that I was committing throughout the story. So let’s say I was saying the story and then something came on the AR screen, I was able to look around and say, okay, I’m done looking at that. I’m ready for the next thing." -P7A

5.1.3 Sensory overload is the primary challenge in immersive stories.

While Jigsaw’s immersive stories are immersive, engaging, and memorable, we found that our design of immersive stories also comes with the challenge of sensory overload. We found that participants’ attention was occupied by participation as different roles, AR effects on the phone, and IoT effects in the surroundings, making it difficult for them to catch up with all the changes. For example, participants reported missing the IoT behaviors easily when they were focusing on the AR effects on the phone.

“I think if the experience doesn’t happen at the same time... that would make sense, because I feel like if you’re already dealing with something flashing in your phone and you’re going to blink and miss what’s in your world, because you’re really focusing on what’s on your phone there." -P11A

In addition to the tension between AR content on the screen and IoT behaviors in the surroundings, participants also expressed sensory overload of both reading the physical book and paying attention to the changes on the phone.

“I love reading books. I love reading to children. I didn’t need to have a phone. The reader in that case doesn’t need to have any devices. I don’t need to be a part of the immersive experience. I can just be the reader reading to the people in the room. So I didn’t really like having the phone sitting beside me because I’m just like, now I’m tempted to pick up the phone, but I’m trying to read this story in a really nice way. So it just felt like there was this weird divide and I felt like I was being pulled in two different directions and not really being able to fully be in the moment." -P4B

5.1.4 Immersive stories are versatile for various population and scenarios.

Lastly, we found that Jigsaw’s immersive stories are versatile for both ambient and interactive scenarios, and for both kids and adults. This corresponds to our motivated design explorations of using IoT devices for both foreground and background augmentation (Table 1). For example, participants commented that the foreground augmentation of IoT devices could be used for stimulating children to keep them engaged and for offering interactive gaming experiences such as escape rooms for adults.

“because children have low attention spans, they’re all looking at screens all day long, watching those 15-second unboxing YouTube videos... they need to be stimulated to be engaged, so that’s why I see this being working wonders with children" -P4A

“I can’t tell you of how many escape rooms I know where you trigger one thing happen in the room, like flickering lights. That would be perfect for it." -P11B

In addition, we found that when Jigsaw’s immersive stories can also be used for ambient scenarios, when IoT devices were used as background augmentation to set the mood of the experience.

“I think one thing that I would love to see, so maybe for the adult story part, would be more horror type of stories. I could see a potential there because you have lights flickering, you have sort of sounds that come out of nowhere or wind that starts blowing." -P8A

“I have a set bedtime and 30 minutes before that I’m supposed to do this end of the day meditation, where I watch a sunset. And I just thought it reminded me of that, because it would be really cool to see that in AR and then maybe you power down your devices, because I’m supposed to be off my phone until bedtime. You power down your TV and electronics and screens and stuff, lower the lights... that type of transition." -P10A

5.2.1 The authoring system has a low barrier of entry.

Participants commented that the authoring system is easy to learn, partly because the in-situ interface that Jigsaw’s authoring tool offers allows users to directly edit behaviors of AR and IoT devices in AR with little technical literacy such as imperative programming.

“I think if I was a non-technical person, it was pretty easy to use because you just point at an object and then you can interact with it. So that mechanic, rather than me specifying things through a complicated user interface or a bunch of text, makes it a lot easier to just as a lay person to be, "Oh, I want the lights to turn on or to be red," and then you can just do it...I’ve actually made a lens that controls Internet of Things before, but I had to do all of this through script and code, and it’s not simple to use in that sense." -P8A

This is in line with the benefits of prior authoring tools for immersive experiences that take advantage of directness (e.g.,  [42, 62]). In addition, participants reported that the authoring system makes it available to settings other than theaters due to less production cost to create immersive storytelling experiences.

“For me, I almost immediately thought of preschools and kindergartens... especially at younger ages, you don’t have that much production... so, just having, I would say, devices take cares of a lot of that."-P2A

5.2.2 Open-ended triggers make the system generalizable and fun to use.

Participants commented that the keyword triggers were open-ended, which made the authoring tool generalizable and fun to use. The ability to define custom keyword triggers, powered by Jigsaw’s speech recognition capabilities, made users feel like they could “do anything with it" (P10B).

“I guess the benefits are with the keyword part that’s super open-ended versus just on a Google Slides, you know, have only preset options, and so you’re somewhat limited to the options that they provide you with. But the keyword element being something where you could just really make it any word that you want to say, it makes that part fun for the interactions themselves"-P9A

The flexibility to specify any number and any types of behaviors triggered also enhanced the open-ended nature of the triggers.

“It’s actually a story so that I can see this stuff can put me into a situation where I can probably trigger multiple things following the story as long as the story is intuitive and easy to follow. So that’s definitely a lot add on there." -P5A

“It would really be able to do a whole lot for each keyword or throughout the entire story that it’s really going through a full experience. I like that it’s not just one thing, so not just one keyword equals one action. One keyword can equal ten actions if you have it set up correctly." -P3A

5.2.3 Creating complex stories is challenging using Jigsaw.

We also found that the expressiveness of our authoring system is limited when participants wanted to author special animation and more complex behaviors of objects such as making a shaking animation or setting the duration for the status of objects’ behaviors. Authoring these behaviors typically requires being able to edit lower-level attributes of the experience such as time, mesh, and position.

“Maybe if you wanted to make it feel like the ground was shaking or something like that, that would be hard."-P9A

“If you wanted to have a flicker or setting a duration of the fan so it’s like, oh, it’s on for 10 [seconds] off for 10 [seconds], back on for another 30 [seconds]. Something like that, having more complex actions for each object would be really interesting." -P3A

5.2.4 Authoring long stories is challenging due to the limit of creation support..

Finally, we found that Jigsaw’s authoring tool is suitable for authoring short stories rather than long stories due to the lack of support for duplicating triggers, scenes, or behaviors, and for navigating longer sequence of scenes.

“it would be too much work to copy from one scene to the next and let’s say you want to have duplicated scenes or something like that. But yeah, I feel like children stories because they’re kind of simple to create." -P7A

“I think this one, like I said, was a bit more tedious because editing on for example, Adobe Premier Pro, I was able to just pick exactly what parts I want and put them next to each other whereas with this one, it’s based on the keyword’s is the next line. The more I have, the more work it’s going to be, whereas the more clips I have might not necessarily be a linear growth at the time." -P2A

Participants also commented that they might encounter challenges of coming up with pairs of triggers and scenes if they were going to create their own stories and would like more creation support during this process.

“I feel like it should have said something like, "Okay, now do you want to trigger the lights, the audio, or the fan?" and then you could decide. You already gave me the script and the script told me exactly what to do, but if I was trying to just create my own story from scratch for the very first time, at least once when you first make your first story there should be more prompts." -P4A

6.1.1 Adding immersion via physical augmentation and user participation.

We found that the augmentation of IoT devices and user participation enhanced the sense of immersion during storytelling experiences. This extends prior effort of bringing in physical augmentation and participation for immersive experiences such as 4-D theatres and participatory plays [6]. Our study indicates that enhancing readily available IoT devices, instead of using specialized, costly equipment, can effectively promote immersive storytelling experiences. Our results also align with prior exploration of dimensions of immersion including extensiveness and surroundings [55]. We found that Jigsaw’s immersive stories can effectively increase the extensivenss of immersion by adding in sensory augmentations and the surroungdings by extending the storytelling experience beyond just the content on the screen. We encourage future researchers and practitioners exploring immersive stories integrate elements of physical augmentation and user participation to add to the immersion of stories. For example, future immersive stories might explore how to extend the experience beyond visual displays on the scree via more IoT devices such as rombots and projectors which can be used for diverse tasks and content. They may also explore immersive stories that involves more human senses such as haptics [11] and tastes [15]. They can also explore more means of participation beside being narrators, actors, and audience and investigate how they can affect users’ sense of immersion during storytelling experiences. For example, prior work has explored the involvement of generative AI for storytelling experiences [20, 64] and future research can explore how these new roles of user collaboration with AI could affect their sense of immersion in storytelling experiences. In addition, another key insight that we gained from the results is that participants faced challenges of catching up with everything in the immersive stories due to sensory overload. Therefore, future research should explore how to design immersive storytelling experiences that could balance various sensory augmentations and modalities of participation.

6.1.2 Authoring immersive stories via open-ended triggers and behaviors.

Our findings show that Jigsaw’s open-ended approach to trigger and behavior definition enables it to be widely adaptable for authoring diverse stories. Users can assign a single keyword (e.g. "balloon") or behavior (e.g. smart light brightness change) to serve different narrative functions across multiple stories. We found that this flexibility can empower participants to get creative and develop novel ways of linking triggers with behaviors for diverse immersive experiences. This aligns with prior work in creativity support tools that aims to enable people to concentrate on creative designs by removing low-level implementation and supplying appropriate building blocks [33]. We suggest future research and design in authoring immersive storytelling to consider a range of triggers and behaviors that are essential and open-ended for building immersive stories. For example, in the context of authoring context-aware IoT applications, prior research has explored triggers such as the user’s location and the state of IoT devices to activate events of IoT devices [58]. While being intuitive and versatile for authoring immersive stories, Jigsaw’s authoring tool also has challenges of authoring long stories. Participants also reported challenges coming up with the appropriate triggers and their corresponding behaviors. Future research could thus explore ways to automate the process of defining triggers, scenes, and behaviors. For example, one can imagine narrating the story verbally for the first time and the system could intelligently detect and record triggers and behaviors on the fly.

6.1.3 Supporting end-user creation of complex behaviors of AR and IoT.

While being generalizable for creating various stories, one limitation we found about Jigsaw’s authoring tool is its expressiveness for defining complex behaviors such as interactivity. Such complex behaviors typically require coding to accomplish. We encourage future researchers to explore ways to raise the ceiling of expressiveness of the authoring tool for immersive stories while keeping the low barrier of entry for end-users. For example, in the context of editing animation, visual programming platforms such as Alice [22] and Scratch [51] could be incorporated into authoring tools like Jigsaw to enhance its expressiveness. In-situ visual programming approaches (e.g., [42, 62, 65]) can also be considered as a way to enhance the expressiveness while augmenting the existing intuitive in-situ controls of IoT devices in Jigsaw. For example, FlowMatic [63] utilized a visual flow-based diagram that enables end-users to create complex behaviors of virtual objects in VR. In addition, future researchers can consider ways for reusing or remixing [24] to make the authoring experience even more expressive for adapting to different storytelling contexts. The functionality of remixing and reusing could also foster a broader and more vibrant community for exchanging storytelling ideas.