DrawTalking: Towards Building Interactive Worlds by Sketching and Speaking

2.1 Natural Language-Adjacent Interfaces

Systems such as SHRDLU [46] and Put That There [3] pioneered the vision of employing natural language to communicate with computers. Due to recent advances in speech recognition and natural language understanding, the popularity of this interaction modality has exploded, and has been used in a wide range of domains. For example, VoiceCut [16] and PixelTone [18] allow users to speak short phrases or sentences to perform desired operations in image editing applications, but these applications are heavily domain-specific. Tools like WordsEye [5], Scones [12], and CrossPower [47] enable scene generation or content editing via language, and interfaces such as Visual Captions [25], RealityTalk [22], and CrossTalk [48] use language structure to make content appear during talks or conversations. However, many of these interfaces tend to assume that the user knows what they want to create in advance — i.e. an end-product with an initial goal. They require an initial phase in which the user must define content and behavior up-front. This could limit open-ended exploration during the creative process, when the user does not necessary have an end-goal in mind. Further, the majority of such interfaces use language input to generate or spawn content without the user in the loop. An alternative is to enable greater interactive control and definition of the behavior of the content. Our approach emphasizes flexibility and user-control during the creative process. The user can define and iterate on content at any point. Our prototype specifically supports definition of behaviors within an interactive simulation. Within this prototype, we explore language input, combined with direct manipulation, as a way of empowering the user to program and control scenes interactively.

2.2 Dynamic Sketching Interfaces

HCI researchers have extensively explored sketching interfaces for dynamic and interactive visualizations ever since the first graphical user interface (GUI) SketchPad [40] and William Sutherland’s thesis, the forerunner of the visual programming language [41]. Many works use direct manipulation and sketching techniques to help users craft interactive behaviors and toolsets for illustrated animation, UI, and visual-oriented programs. For example, works by Kazi et al. [14, 15], Landay et al. [17], Saquib et al. [33], and Jacobs et al. [13] focused on mixing illustration, programming, and prototyping. Programming by-demonstration is featured in works such as K-Sketch [7] and Rapido [21]. Scratch [31] is a well-known visual programming environment mixing game-like interactions with user-provided content and images for a playful experience. [39] supports the user in forming connections between texts and concepts to learn via active diagramming. Our interface is framed in a complementary way around correspondences between sketches and language elements, but we use sketch-language mapping as a control mechanism, enabling the user to create interactive simulations and behaviors for open-ended exploration.

Prior work also explored supporting development of pre-programmed simulations and domain specific behaviors to craft interactive diagrams. In Chalktalk, for example, the system uses sketch recognition to map a user’s hand drawn sketches into corresponding dynamic, pre-programmed behaviors/visualizations [30]. More domain-specific tools like MathPad2 [19], Eddie [34], PhysInk [35] and SketchStory [20] use hand-drawn sketches and direct manipulation interactions to create interactive simulations in physics, math, and data visualization.

2.3 Programming-Like Interfaces

The customizability and flexibility of such a more general interface implies a need for programmability. Examples of these include real-time world simulation systems and programmable environments such as the SmallTalk programming language[11], Scratch [26, 31] (and similar AI voice-enabled explorations like StoryCoder [8]), Improv [29], ChalkTalk[30], and creative world-building games like the Little Big Planet series [23, 24, 32] and Dreams [9]. These encourage interactive building of scenes, games and stories. They combine with elements of interactive visual programming and drawing/sculpting with many types of content (2D, 2.5D, 3D, images). But all use explicit interfaces for programming or programming-like functionality (nodes, wires, text). We depart from explicit UI for sketching+programming-like capability, and largely replace many programming-like functionality with the use of language. Our direction explores the use of verbal, descriptive story narration together with other input modalities (i.e., touch and pen input) to create animated and interactive graphics through sketching.

4.1 Sketching & Language Interface Elements

DrawTalking’s interface has a standard pen+multitouch tablet app design with a manipulable infinite canvas. Sketches are independently-movable freehand-drawings, text, or numbers created by the user.

DrawTalking receives a subset of English as input. We built-in primitives for verbs, adjectives, and adverbs for playful interactions, e.g. procedural transformations and movement, edit operations, collisions, user input. These are composable into more complex behavior. These primitives are intended to be a small sample of possible functionality that demonstrate our working concept. (They are partly inspired by existing software and design spaces [9, 31, 36].) We do not account for all of natural language or claim that our implementation is the only solution. Rather, in the scope of our research our main contribution is the interactive way of controlling elements, independent of the fidelity and comprehensiveness of the supported behaviors. Authoring completely new behaviors sans programming is out-of-scope, but is a complementary future direction. That said, we provide several ways to build on-existing behaviors. We note that the result was just one possible instantiation of our concept. It was most convenient to prototype an all-in-one interface we could fully customize. Other interface implementations could use a similar control mechanism, while integrating more advanced visuals and ways to author new behaviors to control.

The user labels their sketches to make them controllable: nouns (names) for unique identification and adjectives and adverbs (properties) for modulating sketch behavior. For flexibility, we offer two direct ways to label (or unlabel):

The labels display adjacent to sketches, similar to whiteboard annotation, and are quickly-removable by touch.

Consistent with all design goals, the interface (Figure 2) exposes a transcript view and semantics diagram (2b ) to make received input transparent, quickly editable and accessible, and error-robust. The former lets the user visualize speech input, link sketches, and stage/confirm commands. Speech recognition is continuous for interactive use. The design places user-control first, so the user decides when speech is ready as a command. The user taps ‘language action’ to stage a command. Then the diagram visualizes the machine’s understanding of the input and provides a way to reassign objects within the command, regardless of location. The user confirms with the same button to execute the command, or taps the discard button to cancel.

Figure 2:

View Figure

4.2 Language Commands

DrawTalking interprets the structure of language input into commands built from primitives.

Verbs are actions performable by sketches and the system, either built-in or user-defined in terms of other verbs. Examples of implemented verb primitives include:

Verb behavior changes based on other parts of the sentence:

Special verbs include "become", which modifies the sketch’s labels, and "transform into", which also instantly replaces a sketch’s visual representation with another’s. This can be used to support state changes, e.g. "the sun transforms into a moon" or "the frog transforms into a prince".

Nouns, pronouns, and deixis refer to object labels and are used to pick specific objects or specify types of objects. Using deixis while selecting an object will select the object immediately; pronouns can refer-back to objects, enabling more natural sentences as commands. For choosing specific sketches in a command:

There are a few special cases. The pronoun "I" is reserved; it allows the user to take-part in the narrative of a command (e.g. "I destroy the wall"). "Thing" is also a reserved noun that can refer to any object regardless of label. Plural nouns with no modifiers (e.g. as in "blades rotate") refer to labels used to define the interactions between objects with that label. Specifically, this is used to construct rules, as described below and in Figure 4.

Adjectives and adverbs are usable as labels that define properties on objects. They’re interpreted by verbs continuously to modulate effects (e.g. magnitude, speed, size, distance, height). "fast", "slow," "excited", for example, impact jump height and/or movement speed. Adjectives also disambiguate like-noun labeled-objects (e.g. "first house" vs. "second house").

Adverbs heighten adjective effects multiplicatively – e.g. "very" and "slightly" – and are chainable – e.g. "very, very." Special adjectives offer system-control: e.g. "static" fixes sketches to the screen like a GUI element, useful for buttons, d-pads, score displays, etc.; "visible"/"invisible" toggle sketch visibility.

Rules are conditionals that run commands in the future when objects with certain labels satisfy the condition. This enables the user to specify automated commands for the future when, as, or after an event has completed, without needing to know what they want in advance, e.g., “When arrows collide with balloons arrows destroy balloons.” Rules allow definition of new verbs in terms of existing ones e.g. if eat is not defined: “When things eat objects things move to objects and then things destroy objects.”

4.3 Putting It All Together by Example

We demonstrate DrawTalking with illustrated examples showing multiple procedural and programming-like capabilities.

4.3.2 Dog and boy’s infinite game of fetch.

This example of looped sequences highlights a key design decision: being able to iterate on a scene and play with it anytime restarting or knowing the desired content in advance. The user begins by sketching a boy, dog, water, and ball (Figure 3). They can then interactively move the objects to influence the simulation. For additional effect, the water could rise upon collision with the ball, with a command like: "When water collides with balls water moves up for 0.2 seconds and then water moves down for 0.2 seconds." The user can also move the objects to affect their behavior.

Figure 3:

View Figure

4.3.3 Windmill simulation.

This example demonstrates rules, custom object saving, and spawning (Figure 4). The user begins by drawing a windmill base and attaches a blade sketch. Then, they draw their own wind spiral and save it. They then command the blades to rotate on collision with the wind and immediately see the result by moving their wind object over the blades. They would notice the blades don’t stop. They may quickly fix that by adding an "after" stopping condition as shown. Now, they create a switch sketch and a wall, and finally configure the switch to spawn rightward-moving wind that causes the windmill blades to spin on-contact.

Figure 4:

View Figure

4.3.4 Creating the game "Pong" and turning it into "Breakout".

We now describe a more complex example of iteration demonstrating instant playtesting functionality reuse: quickly turning Pong [1] into Breakout [2] (Figure 5). For clarity, some detail is omitted. Create a ball and walls, then paddles, goals, and points for each player. For points, say "I want the number 0" and touch+pen the number to spawn a number object. Label with "This is the first/second score, goal.". Make points screen-space UI with "This thing is static" and set-up rules for collisions between the ball and goals: "When balls collide with first/second goals second/first scores increase". Last, add collision logic: "when balls collide with walls walls reflect balls", "when balls collide with paddles paddles reflect balls." Now we have a playable touch-based version of Pong.

We transform this into Breakout by rotating the canvas, deleting the second-player-related sketches, and simply adding a blocks mechanic. To speed-up our drawing process, we can sketch a temporary rectangular ‘region’ and say "I pack the region with blocks" to fill the region with a sketch e.g. a custom ‘block’ sketch. To make the blocks destructible and increase the point count, create a rule "When balls collide with blocks balls destroy blocks and then the score increases." We are done. In a few steps we have been able to build-on the same scene to try multiple variants of the game.

Figure 5:

View Figure

4.4 Implementation in-Brief

We implemented DrawTalking natively on the iPad Pro M2. Speech recognition is on-device. A local NodeJS+Python serverside for NLP runs on a macbook pro. Text is continuously sent to the server where 1) a dependency-tree is created via Spacy 3.1.4 [27], 2) sent to the client and compiled into a key-value format for semantic roles (e.g. AGENT, OBJECT), 3) either interpreted in real-time by our custom engine to drive interactive simulations or used to find objects by deixis.

5.1 Results

All participants understood and learned the mechanics, and produced thoughtful discussion. They tried their own ideas for how to make things or test their own understanding out of curiosity. All became comfortable. They identified use cases include educational applications, rapid videogame / rapid prototyping, paper prototyping, user interface design, presentation, language learning, and visual-oriented programming.

5.2 Qualitative Findings

Below, we explore the qualitative experiences of participants using DrawTalking.

Labeling. We observed labeling by speech (deixis) was unanimously preferred over linking. Users called it intuitive, direct, and similar to how people talk. Linking could be a useful fallback when freer grammar is preferred, e.g. when giving talks and presentations.

Semantics diagram. People understood it and found it useful to introspect commands: P7: "[It’s] showing what it’s going to do. It’s taking the verbs and cutting all the other stuff out. P2: "You might say something wrong and it interprets it in a different way and you can correct it." P8: "I immediately understand what happens when I read it."

Interactivity/Physicality. Users reported a strength of DrawTalking was a sense of physicality: P5: "I can execute different rules of things that are happening. scene is playing out here AND physical thing is happening." P1: "It’s like language and Legos put-together." P2: "When we make games there’s something called a paper prototype where we make a bunch of pieces of the players and the objects. We just move it around by hand to kind of simulate what it would be. [DrawTalking] is kind of like that but on steroids a bit. So it’s very nice to be able to kind of have those tools to help you with it without needing to manually make it."

As a generic toolset, Re-applicability. Users saw potential for DrawTalking as generic functionality for integration with other applications in a creative pipeline, e.g. as a playful ideation phase from which to import/export assets or runnable code; attaching speech control to production tools to support faster workflows. P1: "You could sit here and have a conversation and build up just using language your interactions and then you send that out for code." P3: "here, it just takes one second for me to ’say it’ – [this could be a] built-in function for any tool/interactive AI". P8 "just incorporate the language and control interface here. We can easily create animations with the fancy stuff they have." This suggests users perceive DrawTalking as a useful interaction, complementary to others in many contexts.

As a programming-like environment. Several participants immediately drew comparisons between primitives in DrawTalking to constructs like variables and loops. P7, an experienced digital artist, but a non-programmer, appreciated DrawTalking as an accessible tool that could reduce frustration encountered with existing programming tools: "Me instructing a game engine – someone like me who’s not a programmer and who is intimidated by doing C# or... using [a] visual scripting language like blueprints – this is a really clean interface that I think can achieve you can get 90- or 80% the way there. It just makes the user experience cleaner than having to use all these knobs and buttons or things like that or using scripting language or having to actually write code".