“Oh My God! It’s Recreating Our Room!” Understanding Children’s Experiences with A Room-Scale Augmented Reality Authoring Toolkit

2.1 AR in Education

AR has been studied for its educational impact across various K-12 age groups for two decades. It has been applied in many contexts, such as humanities and arts, science, vocational, and programming education[4, 55].

Radu identified three main potential benefits of AR in education: Aligned representations present learning contents in 3D spaces and provide better contextualization, resulting in a better understanding of complex phenomena and visual reasoning[26, 47, 51, 70]. Transformed representations enhance original representations with more media forms, visualize the invisible, reduce cognitive load, and enable a transition from presentation to active exploration and knowledge construction[26, 47, 51, 70]. Embodiment encourages learners to physically and tangibly interact with the learning content, leveraging presence, immediacy, and immersion for a deeper learning outcome[26, 47, 70]. In the K-12 age group, AR designs have successfully targeted many aspects of learning and education, e.g. as interactive textbooks, educational toys, or embodied play[3, 26, 71].

While mobile-based AR has been increasingly available through the support of low-cost smartphones, research in AR design for education is still at an early stage[32]. At this point, a majority of design studies with mobile-based AR in education still use marker-based AR, focusing on presenting digital resources[32, 55]. As a result, AR designs generally under-utilize the spatial properties of the physical environments, leading to children focusing more on digital content instead of real-world surroundings. For example, a design study by Malinverni et al. found that children aged 9-11 only followed markers during usage, leaving the rest of the physical space unexplored. The finding corroborates with an earlier literature review, where learners may pay too much attention to virtual information, making the AR technology intrusive in learning settings[4].

AR’s current landscape in K-12 education contrasts with what children expect when co-designing AR headsets with researchers[68]. Beyond visually augmenting the physical environment, children expected that AR systems should be able to create immersive environments at a room scale by transforming their physical surroundings, where virtual and physical elements can interact with each other. Informed by the study, we propose the design of NetLogo AR that could transform children’s physical environments into room-scale AR experiences.

2.2 AR for Spatial Thinking and Exploration

Educational research has assigned huge importance to spatial thinking skills, as they serve as a gateway or barrier to STEM learning[57, 58]. Previous literature has identified three main functions of spatial thinking[15]: A descriptive function, where people capture and preserve the appearance of and relationships among objects, and then convey them to others; an analytic function, where people integrate, relate, and structure spatial information into relational representations (i.e. structures); and an inferential function, where people generate answers to questions about the evolution and function of spatial objects. Spatial exploration, which refers to exploration behaviors (e.g. body movement aimed at revealing more information) during children’s construction of spatial knowledge, is known to support the development of spatial abilities[22]. Children’s practice of whole-body movements supports their understanding of spatial structures, a concept coined as "body syntonic learning"[43]. Children start spatial exploration early on[22], and its importance is not diminished for STEAM education aimed at K4-12 children[16, 50]. In certain learning scenarios, such as museum exhibits and botanical gardens, spatial exploration is also important in its own right, as design goals often include familiarization and exploration of physical spaces[27].

While many AR designs attempted to support spatial thinking through the visualization of spatial structures[8, 10], some projection-based AR designs did it through spatial exploration and whole-body movement. An early study positively associated the spatial learning gains of AR with physical movement[52]. Price and Rogers’s design framework for augmented spaces sees exploration and awareness of the physical world as key benefits of AR. Several later studies used AR for participatory simulation, which allows players to function as different, individual parts of complex systems, usually in a physical space[34, 67]. In such environments, spatial movement is a natural outcome and a crucial resource for disciplinary learning[30]. For example, Enyedy et al. develops learners’ deep understanding of forces and motion, two fundamental physical concepts, through supporting learners’ spatial movement and reasoning with AR. Besides projection-based AR, geospatial AR could also introduce spatial movement, as evidenced by Pokemon Go[13]. In education, while college freshmen successfully navigated the physical world with abstract representation (maps)[33], students sometimes struggled with cognitive overload, which was introduced by the demand for geospatial navigation and collaborative problem-solving together[19].

It has been less clear what design factors in AR may lead to spatial exploration. Projection-based AR may lead to spatial movement, but device-based AR (e.g. children holding a tablet) could also produce it. The problem is that movement does not automatically lead to exploration. For example, Malinverni et al. noticed that both device-based AR and production-based AR failed to produce spatial exploration among children. In the study, children aged 9-11 moved across space, following one marker to another. On the other hand, a markerless prototype design in the early phase of the same study stimulated children’s interest in exploring their physical surroundings[40]. While markers provide easy access to digital content, they also direct children’s focus and interest, limiting children’s interaction with physical spaces. Thus, Malinverni et al. suggests that markerless design has the potential to encourage children’s spatial exploration by opening up all spaces as potentially valid for interaction. However, it is unclear if the markerless AR technology is the deciding factor for spatial exploration. While some studies corroborated the hypothesis (e.g.[28, 38], the marker-based NatureAR design successfully fostered spatial exploration activities among 6-12 years kids by intentionally placing nature markers (e.g. leaves, pinecones) in a nature park for "seek and find"[2].

Recently, with the capability to augment physical surroundings at room scale, AR headsets such as Microsoft HoloLens open up new spatial learning opportunities[37]. For example, Mathland MR facilitated physical, spatial perception of users through spatially linking the physical world with a digitally augmented space, thus supporting spatial understanding of physical rules[31]. AR headsets were also used to support collaborative robot programming, where introducing spatial AR improved users’ spatial task performance[5]. So far, little literature explored headset-based AR’s potential for spatial exploration, and even fewer studies explored device-based, room-scale AR.

To better understand the impact of technology and design choices on children’s spatial exploration, in this study, we compare children’s spatial-related reactions and usage preferences to two markerless, device-based AR modalities supported by NetLogo AR: plane-based AR, which anchors AR content on a wall, floor, etc; and room-scale AR, which anchors AR content in entire rooms.

2.3 AR Authoring Toolkits for Children

While researchers or professionals still create the majority of AR experiences for learning, recent research started to recognize the importance of empowering K-12 teachers and/or students to create AR experiences[53, 59]. As the adoption of AR in educational settings has been hindered by limited time and technical expertise, supporting learning participants to customize AR experiences with a lower threshold becomes more important[53].

Moreover, having children create AR experiences could contribute to their deeper learning of spatial thinking and AR technology. As the constructionist learning theory builds on the connotation of learning as “building knowledge structures”[44], it argues that learning happens felicitously when a learner is consciously engaged in constructing a “public entity”. Following the constructionist tradition, a few toolkits were introduced for children as designers and/or developers of AR experiences[17]. For example, AR Scratch enabled children to project their Scratch program onto pre-designated markers with live preview[48]; ExposAR enabled children to collaboratively create AR experiences through image and plane recognition[39]; StoryMakAR brought together physical prototyping and AR authoring for children to create stories in their spatial surroundings[24].

However, most toolkits either treat the world as a blank canvas or require precise mapping with real-world environments, making them unable to work with dynamic surroundings meaningfully[62]. For example, a space-agnostic design may project a 3D object in front of the viewer, without connecting to the real world; a marker-based design has to be triggered by the exact marker item. While a plane-based system might recognize arbitrary planes, but could hardly assign meanings for them beyond sizes and heights. Professional design teams also report the lack of tool support, as they struggle to iterate on AR designs that would be applied to a remote and distinct environment[35],

Several approaches have been recently developed to tackle this challenge. Building on the LIDAR and 3D reconstruction technology, DistanciAR captured the appearance of physical environments for remote authors to author AR contents in a specific site[62]. Going beyond site-specific authoring, ScalAR captures the semantic relationships between physical surroundings to create AR contents that potentially work in multiple physical locations[46]. Story CreatAR also focused on site-agnostic AR authoring, with the potential to support authors’ spatial thinking[54]. While those studies informed our technology design, no studies have leveraged room-scale spatial semantics for AR authoring.

NetLogo AR is made available by recent computer vision and machine learning developments, which empower more mobile devices with room-scale spatial capabilities. Recently, all iOS devices with LiDAR sensors acquired the capability to recognize building structures and furniture[6]. Instead of relying on predefined markers, images, or dynamic but crudely approximated planes, the RoomPlan SDK makes it possible for adults and children to author room-scale dynamic AR contents on many commercially available mobile devices[1]. Unlike the direct 3D reconstruction in DistanciAR, RoomPlan focuses on semantic and spatial properties of the physical world (i.e. where / what kind of a physical item is).

No longer treating the world as strictly scripted (pre-located markers) or as a blank canvas (space-agnostic), it is now possible to have AR experiences that meaningfully interact with ad-hoc, unpredictable real-world spaces and obstructions. However, few studies have explored the affordances brought by RoomPlan or any similar device-based, room-scale AR technology at the time of writing. No studies, to our knowledge, have explored room-scale AR authoring toolkits and/or for children.

3.1 Design Objectives

Encouraging Spatial Exploration through Markerless AR. Spatial exploration is important both for the learning of spatial thinking and often in its own right (see 2.2). To encourage children’s spatial exploration behaviors, we leveraged two modalities of markerless AR: room-scale and plane-based. We chose markerless AR because 1) our goal is to combine diverse real-world spaces with augmented reality, yet marker-based AR needs intense preparation for each space; 2) we were informed by a recent design framework that markerless AR would generate spatial exploration[41]. Since only a small portion of devices currently available in classrooms (incl. our implementation site) support room-scale AR, we decided also to support plane-based AR, which is available to all ARKit or ARCore-compatible mobile devices today. Plane-based AR provides more accessible but less powerful capabilities: it can scan certain spatial properties of the physical world (the shape, dimension, and height of planes). However, due to its limited ability to take in real-world information, Plane-based AR provides less connection with the real world. Thus, we hypothesized that children would explore less with, and would be less inclined to interact with this AR modality.

Authoring AR with Body Syntonicity in Complex Physical Surroundings. Body syntonicity was the key design principle underlying the Logo programming language, where children could learn abstract knowledge with their body senses[43]. Instead of thinking about abstract coordinate systems, children could situate themselves in the perspectives of a turtle, acting out both in their bodies, minds, and programming commands. As we attempt to incorporate spatial properties of the physical world into augmented reality experiences, we leverage the idea of body syntonicity to help ease the threshold of authoring in diverse and potentially unexpected physical spaces. Instead of thinking abstractly about how the spaces could be, children could use their bodies to experience the current space, thinking about localized features and rules (e.g. should my agent turn right if something is blocking it). As children consider rules at the individual level more, their design might become more adaptive in different physical environments. Encouraging Spatial Thinking through Easy-Switching Modalities. The immersive, room-scale AR experience can potentially encourage children to think from the embodied perspectives of individual agents[65]. Any physical obstacle - walls, doors, chairs, and tables - could block children’s immersive AR view, leading to children’s ad-hoc, spatiotemporal, and concrete understanding of computational models, artwork, or games. On the other hand, there is also a need for a more abstract, macro-level, unobstructed perspective of emergent phenomena, where the top-down 2D modality has an advantage. Through an easy switch design that instantly converts between AR and non-AR modalities, we hope to create spatial learning opportunities where children can mentally map between the physical world and abstract 2D views, which was known to be difficult in previous learning studies[58].

3.2 Design Overview

NetLogo AR supports three modalities: room-scale AR; plane-based AR; and non-AR. To introduce NetLogo AR to children and stimulate their design ideas, we presented the Ants (AR) activity to children during the first session. Here, we used this activity to showcase the design and interaction flow of NetLogo AR. (Note that the technical details are in a separate paper, anonymized for review)

Figure 1:

View Figure

Building on the classical computational model Ants[64], Ants (AR) (Fig 1) simulates a colony of ants. As simple computational rules drive each ant, ants collaboratively forage for food, manifesting swarm intelligence. However, there are also notable differences between the two versions: 1) Instead of living in an ideal virtual world, digital ants now share the same physical world with users; 2) digital ants have to navigate around physical obstacles; and 3) users are invited to play as an ant with body movement such as turning and running around. The merging of physical and computational worlds creates many possibilities unseen in the original model. For instance, while ants in the original model always succeed in fetching food, “physical ants” could be blocked by obstacles and stuck in loops.

When children enter any AR-infused model, the mobile version of NetLogo[11], a menu will show up with options for room-scale and plane-based scans based on the device’s capability. On laptops which some children would use to design and develop their projects (for the activity, see 4.2), researchers could also load a pre-scanned room. Otherwise, children would then hold the device to finish the scanning process. No limit exists on how large the plane or the room could be. Through network synchronization, children could create, edit, and play individual projects across laptops, tablets, and phones and in different modalities.

Room-scale AR Modality If the child chooses room-scale AR (Fig 1 A), their physical position will be mapped into the computational world. Then, the “player ant” (represented as an arrow) will match its position and overlap in the physical world with other ants, creating a mirroring relationship with the child. Physical surroundings, once recognized, will be visualized as semi-translucent boxes. As the child moves around the world, the “player ant” will always follow the tablet’s position, as if it follows the child’s feet.

Plane-based AR Modality If the child chooses plane-based AR (Fig 1 B), a scanned plane (e.g. the floor, a wall, or a table) will serve as the stage for the computational model, using the rough, scanned boundary of the physical plane as the boundary of the computational model. The child will control the “player ant” by a virtual “crosshair”[49]: the “player ant” will attempt to stay at the center of the tablet. Crosshair is not the only supported interaction mode; virtual joysticks were used in some children’s designs.

Non-AR Modality In either AR mode, children could switch to and from the non-AR mode at any moment. The “player ant” will follow the center of the screen, and children will use touch interactions to move around the perspective and thus the “player ant”. As shown in Fig 1 C, the non-AR modality still keeps properties from the scanned physical space: walls or boundaries are rendered as white lines; blockades (furniture, etc) as colored blocks, with each color representing a type.

Authoring the Experiences In Figure 1, the bottom-right buttons allow children to author the model: the first button for editing input parameters; the second button for block-based programming; and the third button for text-based programming. A dedicated open-source extension was built for NetLogo to incorporate and visualize the spatial information from AR scans ( GitHub link). Each element (e.g. a segment of wall or a desk) of the scanned space is automatically imported as a turtle (agent) with physical properties. Some primitives are added or adapted, so digital agents can easily interact with imported real-world agents. For example, the primitive “can-move?” checks if a turtle can move forward or not. Its AR counterpart “ar:can-move?” will also check if any real-world obstacles stand in the way. As a result, both the block-based and text-based versions of the adapted AR (Ants) model can be written or read similarly to the original model. Once children finish editing any aspect, the changed model will come into effect immediately.

4.1 Research Site and Participants

We convened a participatory design group[18] in an existing after-school program about AR creation. A PreK-12 social enterprise school in the U.S. Midwestern area hosts the program. We collaborated with the school to conduct the study. All seven children aged 11-13 in the program gave oral assent. The facilitator and all parents gave written consent. Four researchers in the team facilitated the sessions with the teacher’s assistance. The school provides children’s demographics (Table 1). Most children had some prior experience in block-based coding and only one with text-based coding. A university institutional review board approved our study protocol.

4.2 Procedure and Data Collection

Before the study, four researchers observed the first two weeks of the same after-school program where children designed AR projects with CoSpaces Edu, an AR authoring toolkit. Then, we designed our session plans based on what we learned (Table 2). For eight weeks, we implemented weekly one-hour design sessions. The sessions started in a classroom, but children often went to other places on the same floor for design and play. After researchers introduced NetLogo’s models library, children were invited to find inspiration and generate their project ideas. Children designed, coded, and iterated on their individual projects, with inputs from peers and adults. Researchers and the facilitator offered support in design and programming on a needed basis. While we initially planned on having researchers implement NetLogo code for children’s design ideas, most children eventually learned some NetLogo and edited code with support from researchers and the facilitator. In the final showcase week, we had around 20 to 30 5th-8th grade children and their teachers as the audience in a different classroom.

Our activity design was inspired by literature in participatory design[7, 61, 72]. Children collaborated with adults as testers, informants, designers, and developers for their and peers’ projects[18]. Children carried out the entire cycle of the design process[23, 29], while adults learned from observing, interacting with, and supporting children. Following the example of[25], we adapted our activity plan within and between sessions using children’s and facilitator’s feedback. For example, when children expressed interest in seeing their video data, researchers edited two-minute recap videos weekly and played them at the beginning of each session. Honoring children’s requests motivated children to offer more feedback and participate more enthusiastically, enabling us to build a deep, playful working relationship with children.

During the activity, we collected three types of data:

4.3 Data Analysis

After completing the eight design sessions, four researchers iteratively applied a grounded theory approach to the entire dataset that includes video streams and design artifacts (e.g. design sketches). For video data, researchers coded 48 hours for two rounds[14]. During the first round:

Based on topics that repeatedly emerged from the significant events, researchers decided on three main themes related to 1) children’s reactions to room-scale AR; 2) AR modalities’ impact on children’s engagement patterns; 3) AR modalities’ impact on children’s design decisions. During the second round:

Most of our analysis focuses on a qualitative understanding of what emerged from children’s interactions with NetLogo AR in a more naturalistic setting. More rigid measurement is used in two cases: first, when measuring children’s preference of modalities, we looked at each incident when the child asked for, or turned down AR-capable devices; and which modality they elected to switch to or stay in, as they frequently re-initiated AR scanning and could choose from room-scale or plane-based AR. Second, to understand the different impacts of modalities, we marked out three types of incidents:

5.1 Children’s Interest and Exploration of Room-scale AR

5.1.2 Playful Exploration: Children’s Collaborative Investigation.

Children’s strong interest in room-scale AR led to nine collaborative investigations across four sessions. After the first demonstration, children were intrigued by the AI-driven technology, which seemed capable of recognizing and visualizing the location and dimensions of their physical surroundings. Children were particularly curious whether AI could recognize their bodies as still objects. They collaboratively formed and tested hypotheses about how to “trick” AI. During each investigation, one child was responsible for scanning the room and reporting who got scanned. Other participants tested and adjusted their strategies based on that child’s feedback.

Figure 2:

View Figure

Fig 2 presents three representative episodes of such investigation. In each episode, one child constantly held a tablet and reported whether other children were scanned. Other children quickly adapted their hypotheses and tried new strategies. Children’s hypotheses became more sophisticated as time passed, reflected through their body movements. During the first group investigation, children theorized that the technology might only include non-moving objects, so they sat on a couch or stood still in front of a wall. When children realized those strategies did not work, they gradually layered additional conditions to their hypotheses. By the seventh group investigation, children formed more complex hypotheses. They theorized that technology might detect non-moving people by recognizing breathing movements, so they held their breath while keeping still. Children also hypothesized that AI might scan people if they completely blend in with furniture, so they lay on a couch underneath pillows or on the bay window behind curtains with a chair in front. Such playful group investigations furthered their interest in the AR technology and the AI algorithm that drives the room scan.

Overall, children showed strong interest and engagement in the room-scale AR technology. They were eager to try it out repeatedly, and launched a series of collaborative investigations to explore its capability by forming, testing, and adjusting hypotheses.

5.2 Children’s Spatial Exploration to Spatial Thinking with Room-scale AR

Across eight design sessions, children engaged in spatial activities by playing with and designing room-scale AR experiences. Children’s initial interest-driven spatial exploration sometimes led to spatial thinking activities in all three functions: descriptive, analytical, and inferential[15].

5.2.1 Children’s Design Ideas and Playful Exploration in Physical Spaces.

Most children incorporated dynamic, physical, and spatial configurations in their initial design ideas (Table 3). Five out of six design ideas situated the digital content in physical surroundings: the ants or cars would avoid physical obstacles (Sullivan, Michael, Jayden); the laser beams or guards would limit the player’s movement together with physical obstacles (Sophia, Vihaan). Children’s design ideas necessitate spatial exploration: one has to move across the room to find ants, treasure, or navigate the car. The only exception (Keisha) was not directly connected with spatial configuration, but still centered around body movement.

Table 3:

Student	Project Name	Initial Idea	Spatial Exploration?
Sullivan	Ants Art	The player moves around the room. The ants follow the player, leaving traces that form an artwork. If two ants collide, they disappear. If one ant stays on another ant’s trace, the trace will be erased.	Yes
Sophia	Stealing Treasure	The player needs to find the treasure box randomly placed in the room, and avoid or disable laser beams that might appear on the screen. The player will lose health points if Touched by laser beams.	Yes
Vihaan	Stealing Mullah	The player needs to find treasure in the room and bring it back to home base while avoiding randomly stationed guards. If players get caught by guards, they will say mean words to players.	Yes
Jayden	Driving Virtual Car	The players take the room as a driveway, and steer a car through heavy traffic to reach the end.	Yes
Keisha	Dance!	The player dances while a virtual character follows the player’s body movements	No
Michael	Ants Battle	Two teams of ants fight for resources. The player will be the lead ant of red ants and guide them to fight with blue ants and lead them to food sources.	Yes
Adam	Absent	Absent, would later build on Sophia’s idea.	N/A

View Table

Table 3: Children’s Initial Design Ideas

We found that children’s spatial design ideas did lead to spatial exploration. For example, children constantly attempted to scan novel places to play with the same project. Almost all children initiated a room scan in physical spaces beyond the classroom, with curiosity about exploring different spaces. For example, in Session 3, when Jayden invited Sophia and Vihaan for a collaborative room scan, Vihaan proposed: “Scan both rooms!” All three children stood up and went to the other room. In Session 4, Jayden attempted to scan the street outside the classroom window to project his car racing project onto a physical road.

Figure 3:

View Figure

As scanning and exploring different places brings novel spatial constraints to the virtual world in room-scale AR, digital agents (e.g. ants) often behave differently between one room and another. Because of that, children not only walked around, but often used their whole body to explore the nuances of the space. For example, in Session 5, Sophia was testing her stealing treasure project. As the treasure was blocked from her perspective by physical obstacles, she squeezed between a desk and a wall, a part of classroom space not designed for students to pass, to check whether the treasure was in between (Fig 3). Similar spatial-body interaction also happened in Session 2, when Vihaan attempted to find digital agents under a sink, and in Session 6, when Sullivan and Jayden walked around and tilted their bodies to find the treasure in Adam’s project.

5.2.2 Spatial Thinking Through Playing, Ideation, and Iteration Activities.

The increased spatial exploration, in turn, led to increased spatial thinking opportunities for children. We observed all three functions of spatial thinking: descriptive, analytical, and inferential [15].

Figure 4:

View Figure

Descriptive function. Children leveraged the descriptive function when communicating their project ideas to facilitators. They sketched 2D spatial representations to convey the spatial arrangement of digital objects and the player’s spatial experiences. For example, by illustration, Sophia explained her thinking process and game rules stealing treasure to facilitators. She began by drawing the top-down view of the Ants AR model (room-scale), transforming her real-world experience into 2D illustrations (Fig 4a

Analytical function. Children utilized the analytical function by leveraging the 2D view to understand the spatial structure of their physical surroundings augmented with digital agents[58]. For example, in Session 2, when Vihaan, Sullivan, and Adam interacted with the AR-infused ants model, they walked around a room but could not find any ants. Vihaan switched to the 2D non-AR modality and realized that the ants and food piles were not generated yet because the model was paused. He unpaused the model to generate them and switched back to AR. However, he still could not find any ants and switched to 2D again. One child in the group realized that “Oh, wait, they are in the other room” (Fig 5a). Vihaan switched back to the room-scale view, and found the digital ants in the other room (Fig 5b).

Figure 5:

View Figure

Inferential function. Children further used the inferential function to plan their next spatial move. In the same episode, when the three children finally found the digital ants, they stopped and considered their next steps. Vihaan switched to 2D again and pointed at the screen: “So we’re right here” (Fig 5c). Then, he pointed to another food pile and said, “and I’m going to go over there and lead them there” (Fig 5d) Children actively switched between the more concrete room-scale AR and the more abstract non-AR modalities to plan out their movements (we should move in this direction to the food pile) and infer the possible outcomes (ants would follow there).

The inferential function was more salient when children iterated their spatial design. In particular, children had to consider the mapping between the physical world and the digital, computational world as a two-way lane. In Session 4, Adam tested and iterated on Sophia’s project, stealing treasure. In the original game, users who touched lasers would lose but could restart from the same position. Adam sketched his first idea on the paper: the user’s position would also be reset. He highlighted the starting point of the game (Fig 6a) and the user’s hypothetical location when they touched laser beams (Fig 6b). Soon, Adam realized that the idea from non-AR games might not be readily transferable to a physical world. He pointed at the location in the room (Fig 6c

Figure 6:

View Figure

5.3 Room-scale, Plane-based, and Non-AR Modalities

To find the main contributing factors to children’s spatial engagement, we compared children’s interest levels and spatial interactions of the same projects across three modalities: room-scale, plane-based, and non-AR (more on the methodology: 4.3). Children’s in-depth spatial engagement was more present in room-scale AR. We further investigated why some children shifted in their preferred modalities.

5.3.1 Different Modalities, Different Interaction Patterns.

We observed less body movement, spatial exploration, and self-immersion when children engaged with plane-based AR or non-AR modalities. Here, we excluded cases where children switched back and forth between the room-scale AR modality, as children always treated room-scale AR as the main modality in those cases.

Body Movement. With room-scale AR, children, in almost all cases, walked around the space with rich body movements. With plane-based AR or non-AR modality, aside from one exception, children always stand or sit still (e.g. Fig 7). In plane-based AR (Fig 7a), children mostly tilted tablets up and down towards different angles. They did not walk around or explore different spaces. The only exception was Vihaan walking to find a large enough plane to accommodate his spatial model. After he found the spot, he no longer moved. The body movement pattern with non-AR mode was similar to that with other tablet or laptop-based technology (Fig 7b): children only moved their fingers (touch-screen) or hands (mouse and keyboard).

Figure 7:

View Figure

Spatial Exploration. While we reported many instances of spatial exploration for room-scale AR, children never explored physical spaces using plane-based AR or non-AR modalities. For example, after trying room-scale scans several times, Jayden tried to scan the street outside for his car racing project. Once a researcher told him that the tablet did not support the room-scale scan, Jayden immediately gave up the plane-based scan and returned to the classroom. For Jayden, there was no meaning in exploring (or scanning) physical spaces, if the underlying technology could not recognize the properties of physical spaces and blend them with his virtual world.

Self-Immersion. Children self-immersed themselves in individual agents’ perspectives, e.g. as one (leading) agent in the modeling world, when engaging with models across modalities. With room-scaled AR, children’s self-immersion was embodied and verbal when using the room-scale modality. For example, Sullivan designed and developed his project in non-AR mode, mostly at his table. When he first got a chance to test it in room-scale AR, he not only expressed his excitement by shouting: “Hooray!”, but consciously avoided digital ants projected around the ground by running around. Asked by the facilitator, Sullivan explained: “Because I don’t want them (the ants) to die.” In Sullivan’s project, all ants (including the ant played by himself) leave traces around the world. Other ants will die of too many traces in the same place. As a comparison, children’s self-immersion behaviors in plane-based and non-AR modalities were verbal only. For example, when Vihaan was playing his game in plane-based AR, he said to himself: “Where is home… see you boys. Oh, there is home!” However, as he played, his body stayed in his chair, without any indicator of alignment with his thoughts.

6.1 An AR Design Framework for Spatial Exploration

NetLogo AR was designed to promote children’s spatial movement and exploration. While we initially focused on room-scale AR, not all devices in our implementation site support it. Informed by Malinverni et al.’s conceptual framework, which suggests a link between spatial exploration behaviors and markerless AR, we supported plane-based markerless AR. Different from the hypothesis, while both modalities present similar digital content (agent-based models, games, or artwork), children only engaged with spatial movement and exploration in room-scale AR. Previous studies also showed cases where marker-based AR leads to spatial exploration[2]. These findings motivated us to propose a new AR design framework. In this framework, we examine where AR content is linked with markers or open-ended spaces, which influences the users’ goals of movement; and the certainty of AR content’s location, which influences users’ destination of movement.

Figure 8:

View Figure

Similar to Malinverni et al.’s framework, our goal is to provide designers with practical suggestions that might inform their design decisions. As shown in Fig 8, we situate the perceived spatial properties of AR content in a two-dimensional space, categorize existing AR design from our study and previous literature, and then describe the corresponding user behaviors. The horizontal dimension is related to AR content’s location: whether the AR content could be certainly triggered in a given place(s) or not. The vertical dimension indicates to which the AR content is linked with: with one or more markers; or with an open-ended space. Both dimensions are continuous. We made an implicit assumption: users are already interested in augmented content. Below, we discuss each quadrant with examples:

Contents linked with markers; Location certain. The two studies reported in Malinverni et al. both match this dimension. Despite the differences in presentation (device-based vs. projection-based), the AR content in both designs is triggered by markers that learners can easily identify and recognize. As a result, learners moved from one marker to another, with the sole goal of triggering the digital content[40, 41]. Similarly, when finding in-game locations such as "gyms" or "PokeShops", players of the location-based game Pokemon Go (11-56 years old) moved between destinations, as incentivized by the game[13].

Contents linked with markers; Location uncertain. A marker-based design, NatureAR successfully triggers learners’ (6-12 years old) spatial exploration by placing hidden markers all over a yard[2]. The AR content is still linked with markers, but the location becomes uncertain. While children were motivated to explore the natural environment, they constantly focused on finding markers, leading to reduced interactions with other elements in physical environments[2].

Contents linked with space; Location uncertain. AR Magic Lantern employed a similar design idea: hiding triggers of AR content in a heritage site[27]. Different from NatureAR, triggers of AR content were not marked from the environment, making the entire space legitimate for visitors to explore. As such, cross-generational participants were motivated to explore different aspects of the physical surroundings while trying to trigger AR content[27]. Our findings around room-scale AR also fit into this dimension. Children started with uncertainty about the location of augmented content. Nor did they have an idea about what kind of physical items would be augmented. Consequently, children iteratively refined their hypotheses by exploring different parts of the physical world with their bodies (see 5.1). During the design and play-testing, digital ants were scattered across physical rooms in unknown places, triggering children’s playful exploration behaviors (see 5.2.1).

Contents linked with space; Location certain. While digital ants behave similarly to room-scale AR, the location of the “playground” was certain: plane-based AR gives a full, unobstructed picture of the computational model. The digital contents are much less related to the physical world. As a result, children’s interactions with the physical world are fewer and less related to exploration. While we did not design the plane-based AR activity to use the same controlling method as the room-scale one (i.e., use children’s bodies to control the digital agent), children used at least one more method (virtual joystick) besides crosshair. In both cases, even when children moved around, their behaviors were more about controlling the game than exploring the physical space (see 5.3). Children do not need to explore because all digital content has a specific and known location. Similar patterns could be seen in another AR design study focusing on spatially manipulating virtual objects. Even when AR content covered a large swath of the room, as the content stayed unobstructed and predictable in the same location, users preferred seated and avoided large body movements[60].

To summarize, in our proposed framework, the vertical dimension (certainty of location) is connected to the destination of spatial movement, while the horizontal dimension (linked with) is connected with the goal of it. When the destination is uncertain (i.e. the content is at uncertain locations), users’ spatial movement becomes a spatial exploration of the physical environment. When the goal is also unspecified (i.e. the content is linked with the space), the entire space becomes legitimate for open-ended spatial exploration. In all cases, users were motivated by the desire to reveal augmented content and incentivized by the design, yet the different incentive structures led to different patterns of spatial movement and exploration[2, 13, 27, 41, 60]. Therefore, we suggest designers first evaluate the intended spatial movement and exploration patterns relevant to the specific context. Then, make design decisions about the location and link of AR content accordingly.

6.2 Room-scale AR and Spatial Activities

We reported several cases where children moved from spatial exploration into spatial thinking activities. While all of them are directly related to room-scale AR, the technology alone does not generate the desired activities directly. Rather, the design decision to include a non-AR modality that is easy to switch back and forth (3.1) contributed to the transition, as evidenced in several episodes in our findings. Here, we discuss four moments that future designers should consider working on:

6.3 Room-scale AR and Children’s Design

With NetLogo AR, it is possible to transform children’s physical surroundings to create immersive, room-scale experiences. Virtual elements can interact with the real world in a dynamic and programmable way that welcomes children’s open-ended exploration and design ideas. Our findings showed that most children passionately played, designed, and iterated with room-scale AR technology. Children successfully leveraged spatial thinking skills to iterate on their spatial design, sometimes with sketches, and sometimes with coding - with a programming language they first learned during the activities.

NetLogo AR’s capability to materialize computational design at room scale stimulated children’s wild imagination beyond what is possible now. For example, Keisha’s initial design idea asked for virtual characters to dance in the room with the player; Jayden attempted to situate his car racing game on the street outside the school; during the final sessions, a child and a teacher tried to physically “stride across” the laser line in Sophia’s design, sparked a discussion about wearable devices that could capture more nuances of whole-body movements. Unfortunately, many of those ideas were not yet supported by the current iteration of NetLogo AR, leading to two children switching design ideas. Still, the enhanced awareness of bodily interaction and the capability to work with larger spaces are certainly directions we should continue to work toward.

On the other hand, the limitations of current technologies still negatively affected the design sessions. Two children in our study had to switch to a non-spatial design idea after knowing we could not implement their initial spatial idea quickly. Some children struggled with the misalignment of AR contents and overheating devices, leading to a shift of preferred modalities. To achieve Eisenberg’s[20] future of room-scale children-computer interaction, where children could control, design, and program digital artifacts all over physical surroundings, it seems that more design and development efforts are yet to be made.