«Creative AI: From Expressive Mimicry to Critical Inquiry»

The nascent field of what has come to be known as “creative AI” consists of a range of activities at the intersections of new media arts, human-computer interaction, and artificial intelligence. This article provides an overview of recent projects that emphasise the use of machine learning algorithms as a means to identify, replicate, and modify features in existing media, to facilitate new multimodal mappings between user inputs and media outputs, to push the boundaries of generative art experiences, and to critically investigate the role of feature detection and pattern identification technologies in contemporary life. Despite the proliferation of such projects, recent advances in applied machine learning have not yet been incorporated into or interrogated by creative AI projects, and this article also highlights opportunities for computational artists working in this area. The article concludes by envisioning how creative AI practice could include delineating the boundaries of what can and cannot be learned by extracting features from artefacts and experiences, exploring how new forms of interpretation can be encoded into neural networks, and articulating how the interaction of multiple machine learning algorithms can be used to generate new insight into the intertwining sociotechnical systems that encompass our lives. Recommended citation Forbes, Angus G. 2020. «Creative AI: From Expressive Mimicry to Critical Inquiry». In: Andrés Burbano; Ruth West (coord.) «AI, Arts & Design: Questioning Learning Machines». Artnodes, No.26: 1-10. UOC, Leonardo/ISAST. [Accessed: dd/mm/yy]. https://doi.org/10.7238/a.v0i26.3370 The texts published in this journal are – unless otherwise indicated – covered by the Creative Commons International Attribution 4.0 International license. The full text of the license can be consulted here: http://creativecommons.org/licenses/by/4.0/ Angus G. Forbes University of California, Santa Cruz Issue coedited by: Universitat Oberta de Catalunya and Leonardo/The International Society for the Arts, Sciences and Technology NODE «AI, ARTS & DESIGN: QUESTIONING LEARNING MACHINES» https://artnodes.uoc.edu artnodes Issue coedited by: Universitat Oberta de Catalunya and Leonardo/The International Society for the Arts, Sciences and Technology «Creative AI: From Expressive Mimicry to Critical Inquiry» 2 A UOC scientific e-journal 2020, Angus G. Forbes Artnodes, no. 26 (2020) I ISSN 1695-5951 2020, of this edition by FUOC and Leonardo/ISAST CC CC The Paper:


Introduction
One advantage in using machine learning to extract meaning from data is that it lets the researcher sidestep the need to articulate the low-level details contained in the data, which can be difficult to tease out and hard to define. How do you describe what films you like? It is easier to provide a training set of films that you've rated and let the algorithm discover what features highly related films have in common (Hallinan & Striphas 2016). How do you capture the nuances in meaning when translating a phrase from one language to another? It is more accurate to provide the machine learning system with a vast amount of data in order to infer these subtleties without requiring formal semantics (McCann et al. 2017). How do you best describe the special characteristics of a person so that they can be distinguished from others in an image, no matter where the image was taken, what pose they are in, or what they are wearing? State-ofthe-art recognition systems do not require any description whatsoever, only a sufficient number of examples that the deep learning network extrapolates from and encodes as weights within its hidden layers (Taigman et al. 2014, Sun et al. 2014. What strategy do you use to articulate the rules that define an artist's expressivity? Style transfer algorithms effortlessly let you transform any image or video into an impressionist painting, using even a single image of a painting to automatically find the characteristic elements of the artist's style (Gatys et al. 2016).
For many applications, deep learning neural networks are the most effective method to identify useful features in datasets and to use them to interpret new data with similar content. In addition to choosing the most computationally efficient architecture or parameters, a main focus of the data analyst using them becomes to define the space of interpretation by choosing the dataset that represents that space, by selecting an appropriate loss function for training the network, and by deciding what outputs can be returned when querying the network. Learning to interpret the data occurs through a process of encoding hierarchies of features that indicate whether a particular input (or part of that input) belongs to a particular category. Although there has been much work on trying to make sense of what these features "mean" (Olah et al. 2017, Carter et al. 2019, either individually or in aggregate, understanding is enabled through a process of curation rather than by explicit explanation. In this way, machine learning introduces a new approach to making sense of the world in which choosing examples and defining mappings judiciously enables new applications and new forms of creative expression.
The Creative Coding Lab at University of California, Santa Cruz 1 investigates the use of machine learning algorithms for scientific research and creative explorations across a range of contexts. One effort, Deep Illumination, explores how deep learning can be used effectively in the graphics pipeline, investigating, for example, how to infer complex lighting models from a large dataset of examples, rather than through expensive rendering calculations, and evaluating how such an approach can provide useful trade-offs between time and memory (Thomas & Forbes, 2017, Elek et al. 2019, Alsaiari et al. 2019. Our lab has also investigated the use of machine learning technologies for a range of practical applications. One project, Com-postNet, trains a neural network to classify food waste appropriate for available trash and recycling receptacles (Frost et al. 2019a). Another project uses machine learning to predict biker density at dangerous road intersections so that drivers and bikers can experience improved shared road safety (Dubey et al. 2019a). Researchers in the Creative Coding Lab have also investigated creative applications using machine learning. For example, the Art I Don't Like project used a novel recommender system that introduces users to artists and art genres that they may be unfamiliar with (Frost et al., 2019b), and the Data Brushes art application enables users to interactively paint using specialised brushes that generate output using neural style transfer networks (Dubey et al. 2019b). Much of the architecture for deep learning neural networks was first theorised and implemented in previous decades (Bishop 1995, LeCun et al. 1998, Rumelhart et al. 1996, but the recent explosion of deep learning techniques and applications introduced in the last few years was in part enabled by innovations in GPU technology (LeCun et al. 2015, Krizhevsky et al. 2012. Neural networks are loosely modelled on the behaviour of neurons, and the Creative Coding Lab has been exploring models of computational intelligence inspired by other biological processes. One recent project, developed in collaboration with astrophysicists at University of California, Santa Cruz, emulates properties of the Physarum polycephalum (the "many-headed slime mold") in order to infer a simulation of the dark matter filament structure of the Cosmic Web using only a sparse sampling of astrophysical observations (Burchett et al. 2020). 2 The term "creative AI" is increasingly used by artists and designers who utilise machine learning to generate creative outputs, or who treat machine learning algorithms as a medium in and of itself in various ways (McCormick et al. 2020). In recent years, creative AI projects have been featured at the NeurIPS Workshop for Creativity and Design, as well as at other arts and computation venues, such as the ACM SIGGRAPH Art Gallery and Art Papers tracks, the IEEE VIS Arts Program, and the International Symposium on Electronic Art. Broadly 1. https://creativecoding.soe.ucsc.edu/ 2. An overview of projects from the UCSC Creative Coding Lab was presented in late July 2019, as part of the "AI in the Arts and Design" panel discussion with Erkki Huhtamo, Memo Akten, and Max Sims at ACM SIGGRAPH, organised by Ruth West, Victoria Szabo, and Danielle Siembieda.
speaking, creative AI projects involve one or more of the following: mimicking existing data, mapping features found in one dataset onto another, or mapping inputs to outputs in unusual ways, visualising or otherwise probing the inner workings of the algorithm, and analysing or speculating about the societal impact of machine learning systems. These activities can enable new kinds of generative artworks that can either replicate or incorporate existing artworks or create entirely new artistic outputs. They also can be used to design new techniques of more expressively interacting with existing art forms. In doing so, they introduce new ways to analyse and experience cultural artefacts and cultural data. Finally, the machine learning algorithm, its computational architecture, the input it requires, the resulting output, and the analysis framework it is part of can be thought of as a cultural artefact in and of itself, enabling new forms of critical inquiry. In the sections below, I provide an overview of these trends, along with descriptions of related projects, and highlight opportunities for computational artists working in this area.

Creative AI as expressive mimicry
Creating software that automatically generates artworks -either in the style of a particular artist, or in an original voice that does not directly reference existing work-is a perennial pursuit in new media practice and generative art. Well-known early examples include Harold Cohen's robot paintings (Cohen 1995) and David Cope's experiments in musical intelligence (Cope 1996). Often in these projects, the visual or audio outputs, while interesting on their own, are a byproduct of the actual artwork, which is the system itself: in Cohen's case, AARON is the artwork; for Cope, his EMI software is the main creative contribution. A more recent example is introduced by Sougwen Chung, who, as part of her Drawing Operations series, co-improvises drawings in collaboration with a robotic arm that is controlled via a recurrent neural net that has been previously trained on her own drawings (Chung 2019 (Chan et al. 2019) introduce a method to transfer the recorded movements of an expert performer onto a new video featuring an amateur performer, appearing to transform novices into professional dancers. This technology exacerbates difficulties in separating facts from opinions, in thinking critically, and in identifying bias and propaganda (Gebru 2019, Jo & Gebru 2020), but it also potentially presents new avenues for exploring these issues and for new forms of creative work.

Creative AI as interactive mapping
Machine learning enables the creation of tools that map a range of inputs to new outputs, often in a different modality. By definition, all algorithms require an input that is then processed in some way to produce an output. Neural networks, including deep learning networks, are "tuned" through a training process that encodes an effective mapping of inputs to outputs for a particular dataset (the training set). If successful, and if the training set is representative of the kinds of inputs that will be encountered in the future, then the network can be queried nearly instantly to provide a meaningful output given some new, previously unseen input data. Fiebrink's Wekinator tool enables users to quickly train a neural network (or another machine learning algorithm) to recognise, for example, different gestures from a web camera and associate them with sounds or musical instructions (Fiebrink et al. 2016). KIMA: The Wheel is a multimedia performance by the art collective Analema Group that uses machine learning to correlate sound and visual parameters, generating a multimodal mapping between voices and visual outputs (Gingrich et al., 2018). Style transfer networks that encode stylistic features of a source image learn to map any image into a transformed version of that image that incorporates those features. Gatys et al. (Gatys et al. 2016) introduced neural style transfer, which makes use of a convolutional neural network to identify image patterns that represent a particular painter's "style", and can then transfer it onto any other image, making it possible, for example, to transform a photograph into an image that looks like it was painted by Van Gogh or Kandinsky, to use popular examples.

Creative AI as generative art
A range of techniques investigate the neural network as space of possibility. The "deep dream" algorithm, which transforms images into psychedelic quilts was originally created as a tool to highlight which features were being activated when processing an image with a neural network. If a neural network is trained to classify, for example, different species of birds, then a particular patch of a bird image (or an image that contains bird-like objects) will trigger the neurons within the network that have been tuned to respond to that particular bird feature. Often these features, when viewed in isolation, resist easy interpretation, and represent a particular curve or gradient or texture that proved to be useful in detecting a bird within an image (Olah et al. 2017). The Inceptionism project takes these features and iteratively integrates them onto the image, allowing us to see which features are observed in a given input image. To continue the example, even if an input image contains no birds at all, and if the network is trained only to recognise bird features, the technique ends up generating a kind of Boschian hellscape of bird parts (Mordvintsev et al. 2015). Initial breakthroughs in deep learning led to state-of-the-art methods in data classification, identifying items in a photo, automatically tagging people in social media posts, or recommending products or content based on previous interactions or purchases on a website (LeCun et al. 2015, Goodfellow et al. 2016. If a network has been trained to identify particular features in order to, say, decide what category an image belongs to, then that network could also be used to generate new images made up of those features and belonging to that category (Goodfellow et al. 2014). The generative adversarial network (GAN) architecture consists of both a generator network and a discriminator network. During the training process, the generator network gets better at producing output, and the discriminator network gets better at distinguishing a real image from the training data from a generated image. Once the generator is sufficiently trained, any input to the generator network will produce a realistic output, that is, an output that contains features recognised by the discriminator network as a real image. The input to the generator network is a vector of numbers within a particular range of values describing a "latent space", and slightly changing the values of one of more of the numbers in the input vectors produces images that are similar to each other (Bojanowski et al. 2019). Animations of images created by "drifting" through the latent space (i.e. updating the input vector) produces a morphing between images that resemble the training images, sometimes creating a surreal effect. Artists have been inspired by GAN techniques that make it possible to direct the data generation process (Mirza & Osindero 2014, Radford et al. 2015, Karras et al. 2019. For example, a recent iteration of Refik Anadol's Machine Hallucination project uses a GAN trained on 100 million photographic memories of New York City found publicly in social networks to create synthetic representations that envision a possible "near future" (Anadol 2019). Mario Klingemann has created a series of animations using a technique he calls "neural glitch", in which he alters the weights in a trained generator to create intriguing "misinterpretations" that nonetheless retain a coherent style (Klingemann 2018 (Menezes 2019). Memo Akten's Learning to See processes live camera input, composing images that resemble the shape and structure of this input, but replacing the content with data learned through training a network on particular types of images, transforming, for example, keys and wires into flowers and waves, or faces into galaxies (Akten et al. 2019).

Creative AI as critical inquiry
Some recent creative AI projects can be considered as critical inquiries that investigate sociotechnical systems that utilise machine learning. Tom White's influential project Perception Engines creates idiosyncratic images made out of a few simple shapes with solid colors and curved dark lines. While at first glance they seem to be vaguely evocative of a particular object or action, upon reading the title of each print (such as "cello", "cabbage", "hammerhead shark", "iron", and "tick"), it becomes hard to see anything else. While the prints create a kind of visual puzzle, they also function as images that return the highest confidence score on different image classification algorithms (often higher even than photographs of those objects), providing insight into what shape features form a "Platonic ideal" of a category encoded in the image recognition network, and representing the "character" of a class more effectively than any one instance (White 2018). Avital Meshi's Classification Cube features an interactive surveilled space in which multiple machine learning algorithms are used to classify a participant's behaviours, expressions, age, and gender. In addition to making it clear that some expressions and poses are incorrectly categorised, and that a person's age or gender can be misclassified depending on seemingly minor changes, the project provides a space for reflecting on the ubiquitous automated interpretations that permeate our daily lives (Meshi & Forbes, 2020). While machine learning systems are implicated in algorithmic bias (Diakopoulos 2015, Eubanks 2018, bias of course exists prior to being encoded into datasets and deep learning networks trained on those datasets. Creative interrogations of machine learning systems can help to pinpoint aspects of a data analysis pipeline that introduce bias and spark discussions about the ramifications of weaving machine learning into the fabric of public life.

Creative AI opportunities
Novel sophisticated machine learning techniques are presented each year at the International Conference on Computer Vision (ICCV), the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Neural Information Processing Systems (NeurIPS), the ACM Special Interest Group on Computer Graphics and Interactive Techniques (SIGGRAPH), and various other computer science venues.
Researchers often put versions of the articles online at the arXiv. org open-access archive and make the code for these projects available in online repositories, enabling anyone to test out their techniques using popular software frameworks, such as TensorFlow and PyTorch. Given their accessibility, there are many opportunities for incorporating contemporary machine learning techniques into creative projects. For instance, Isola et al. and Park et al. introduce architectures that have been used to make interactive demos that infer a reasonable image from only outlines or coloured rectangles (Isola et al. 2017, Park et al. 2019. A more recent project called GauGAN lets a user easily modify generated images by "painting" particular features on the image (Bau et al. 2019), and an interactive demo by Liu et al. lets a user edit an existing photo by erasing people or objects, automatically "inpainting", replacing them with relevant elements from the surrounding landscape (Liu 2018). Other generative machine learning projects have appeared over the last few years, many of which are geared toward graphics techniques for visual effects in films and games, but have not yet, to the best of my knowledge, been incorporated into media arts projects or to augment interactive performance. Xie et al. (Xie et al. 2018) showed that realistic motion dynamics could be created and shaped interactively by training a neural network on a database of fluids. Their system learns to generate fine details in explosions, water, or smoke from low-resolution inputs, which speeds up computation and enables visual effects artists to quickly create high-quality animations of different fluids. A number of projects have focused on generating realistic human and animal motion and motion planning strategies for navigating specialised environments, including for rock climbing simulations (Naderi et al. 2017), walking through diverse terrain (Zhang et al. 2018), or in crowds (Amirian et al. 2019). For example, work by Holden et al. (Holden et al. 2017) trains a neural network using a database of human movement captured in a motion capture lab, including walking, jumping, climbing stairs, and crouching. This network is then able to determine the most reasonable motions for a virtual character moving through any scene, finding correlations between the motions stored in the networks and the elements within the scene. Even for scenes with arrangements of terrain and objects that are quite different from the data it was trained on, the network produces synthetic motion outputs that are convincingly realistic.
Many creative AI projects differentiate themselves by curating the data and labels they choose for the training set or as inputs into the network. To take just two examples, Chris Rodley uses a style transfer network to create compelling images of dinosaurs composed out of fruit (Rodley 2017) and Pinar Yanardag and Emily Salvador use generative adversarial networks trained on a database of fashion designs to create new dresses and jewellery (Yanardag and Salvador, 2019). Some intriguing machine learning techniques enable crossmodal mapping, in which data from one domain informs or creates the output in another (Baltrušaitis et al. 2018). Recent techniques automatically provide captions from an image or accurately label subregions in an image (Karpathy & Fei-Fei 2015, Gan et al. 2017, or the reverse, generate images from text (Qiao et al. 2019). For example, Zhang et al. generate accurate images (at least at first glance) of birds from simple descriptions, such as "This bird is red and brown in color, with a stubby beak", enabling users to "paint" with words (Zhang et al. 2017). In addition to encapsulating a form of cognitive blending, in which emergent meanings are constructed from mixing together partial matches in two different domains (Fauconnier & Turner 2003), they illustrate that existing cultural artefacts (such as online field guides for bird watchers, photo collections of flowers, or various forms of social media) contain conceptual analogies that define "unseen" relationships that enable new forms of automated reasoning (Peyre et al. 2019. Designing machine learning systems that leverage or investigate cultural artefacts presents opportunities for new creative work and cultural insight. Techniques such as style transfer and inpainting show that there is unexpected information that can be mined from even a small number of input data samples, and which can then be used for creative reinterpretations. Other recent examples include learning 3D information from 2D data, such as a technique to synthesise animations that contain novel views of complex scenes from a set of input images (Mildenhall et al. 2020) and a technique that estimates the depth of elements in an image in order to automatically create a "Ken Burns" animation effect consisting of zooming, panning, and motion parallax (Niklaus et al. 2019). Another technique learns to synthesise frames of future frames from a single image, predicting plausible ways that a scene might change over time (Xue et al. 2018). Techniques such as these can infer unexpected features and relationships between those features, and present many creative possibilities that have yet to be fully explored. Additionally, different types of sensors can expose new features in data. For instance, by using a slow-motion camera with a high temporal resolution, Davis et al. (Davis et al. 2014) were able to recreate sounds in a room by observing the subtle motions of particular objects in that room, such as plants or packaging. In one experiment, they demonstrate that they can retrieve someone singing a nursery rhyme simply by recording and accentuating the vibrational movements on a package of chips. While this has implications for surveillance, it also illustrates how inventive uses of sensors can provide unexpected streams of information in other sensory domains. Incorporating datasets from higher resolution instruments into machine learning systems can lead to new creative applications.

The future of Creative AI
Given the continuing breakthroughs, it is worth thinking about what machine learning is not yet able to achieve, and about what components of an artwork cannot be effectively modelled or mimicked. For example, so far, machine learning approaches have not successfully generated convincing dramatic experiences or engaging multimedia performances. These kinds of experiences require contextual information which we do not yet understand how to encode effectively and thoroughly. Narrative, dance, performance, and cinema are inherently more complex than static images or sound recordings, and require integrating many elements simultaneously, such as lighting, editing, acting, narrative, and sound design. Machine learning makes the assumption that all relevant features can be found within the training data, and even if there were a way to gather and label relevant data from, say, a film or a live performance, we bring our knowledge of the world and our expectations about how to interpret particular genres when experiencing art. Moreover, these experiences are ultimately interior and perhaps ineffable, resonating with a rich personal database of our own experiences and our own thoughts and feelings. That is, machine learning algorithms can effectively identify and utilise features in artworks in increasingly sophisticated ways, but do not model how an artwork is perceived or why it is interpreted in a particular way. Media artists, in addition to using new media forms to create new representations and new experiences, also investigate the nature of media itself, and often foreground concept over or alongside aesthetics and technical craftsmanship (Shanken 2002a, Agüera y Arcas 2017, Ackerman et al. 2018). Creative AI practitioners will continue to identify which concepts resist machine learning approaches and to investigate how machine learning tools can make particular interpretations either inescapable or impossible.
Machine learning technologies can be thought of as a type of measuring instrument. Many sensors include a computational component in which data is filtered or otherwise processed to separate out the noise from the signal. Neural networks measure distinguishing features in data, and can provide insight into the system the data is drawn from, as well as about other systems with which it is entwined. For example, observing transportation patterns or analysing pollution levels can be used to provide insight into the economic health of a city (Washington 2020), and interactions on social media can be used to identify personality traits, and then exploited for targeted advertising or disinformation campaigns (Kaiser 2019). Insight into these auxiliary systems then could allow us to infer patterns from yet other interacting systems. The promise of "big data" is not simply that we can collect higher and higher resolution spatiotemporal data, and not only that we can retrieve and analyse data more and more quickly, but that we can make use of all this data to make sense of how systems interact and integrate with each other (Shanken 2002b, Hassad 2020. How should we design the next iteration of machine learning tools that reason about the world holistically by integrating multiple interpretations encoded in text, mined from image and video databases, perceived by sensors, provided by human-computer interactions, and communicated by yet other machine learning tools? Creative AI will continue to be a space in which artists and researchers create