The intangible material of interactive art: agency, behavior and emergence* NODE: “ART MATTERS II”

This paper presents a conceptual analysis of some of the basic notions for the practice of interactive art and the relations among them. A sound understanding of these notions is essential for the creation of the aesthetics of artistically behaving systems. Interactivity, agency, behavior and emergence are presented as the building blocks of this practice, understanding that they are at least as important as the materials that physically instantiate the pieces and installations that constitute the body of interactive art. Interactivity is defined and confronted to the metaphor of the conversation and to the idea of designing interactive systems with artistic purposes. The notions of agency, behavior and performativity are reviewed through the reading of Andrew Pickering’s account for the ontology of Cybernetics and in relation to interactive art practices. Finally, the concept of emergence and Peter Cariani’s emergence-relative-to-a-model are presented as a theoretical framework with which the analysis and creation of unexpected and non pre-designed behaviors in interactive systems can be based.


Introduction
This paper presents a conceptual analysis of some of the basic notions for the practice of interactive art and of its relation. A sound understanding of these notions is essential for the creation of the aesthetics of artistically behaving systems. Interactivity, agency, behavior and emergence are presented as the building blocks of this practice, understanding that they are at least as important as the materials that physically instantiate the pieces and installations that constitute its body of work.
Interactive art materializes in computer screens, installation spaces, robotic devices and the like. The interactor (the one who dialogues with the piece) relates with it through sensor interfaces: keyboard, mouse, proximity sensors, customized buttons, etc. and is addressed through actuators: loudspeakers, screens, motors, etc. All of these materially instantiated elements are certainly a part of the interactive experience, and it is through them that the artistic experience is brought about. In addition, more often than not, interactive art pieces are characterized through these elements: categories such as net.art, installation art or robotic art are examples of this.
However, the choosing of components and the designed interfaces is only a part of the creative endeavor of interactive art. Arguably, what is essential to this practice is precisely that which is not physically instantiated, even in robotic pieces. Agency and behavior, which form the basis of performativity, are instead the defining and very central aspects of interactive art. They are the intangible material of interactive art. In this context, the concept of emergence appears as an essential piece in order to understand the possibility of non pre-defined behaviors to appear in such pieces.

Interactivity
There is a body of literature that has addressed the definition of interactivity in relation to both general and computer media, mostly since the mid-late 1990s. The general consensus is that there are three main approaches to defining the concept from the media studies point of view, depending on whether the defining efforts situate the focus of the definition on the structure of media, on the user, or on the process of communication that develops between them (Downes and McMillan, 2000;Kiousis, 2002;Quiring, 2009;Mechant, 2012;Weber, Behr and DeMartino, 2014).
A first group of authors is that which focuses on interactivity as a characteristic of media (Durlak, 1987;Baecker and Buxton, 1988;Wilson, 1994;Jensen, 1998;Sundar, 2004;Lee, Park and Jin, 2006). The center of interest within these approaches is on how media is structured in order to afford interactivity: what it offers to the potential user and how, so that interaction can take place. Interactivity is often presented in these cases as a continuum, and different media are placed in different points along that continuum (from less interactive to more interactive media). The types of activities that each of these media offer to users is also read within these parameters. On the other end, some authors place the focus of study on the user, mainly around the concept of perceived interaction (Wu 1999;Liu and Shrum, 2002;Leiner and Quiring, 2008;Quiring 2009). What is most important here is what interactivity means for the person that uses the interactive system and how he or she relates to media on its terms. Interactivity here is an information-based process that is relevant in terms of individual perception (Newhagen, 2004). Finally, there is a third group of authors that center their approach on the communicational process that interactivity represents (Kiousis, 2002;Rafaeli, 1988;Rafaeli and Sundweeks, 1997;Crawford, 2003;Noble 2009;Penny 2000;Green, 2010). The focus in these cases is on the process that develops as the interactive system and the user act and react to one another.
I have defined interactivity as "a series of related actions between two or more agents where (1) at least one of them is an artificial system that (2) processes its responses according to a behavior specified by design and (3) takes into account some of the previous actions executed by them" . This definition aims not at trying to solve the everlasting discussion on how to define the term but at clarifying the concept in the context of interactive art. The definition addresses the three main aspects that should be taken into account when trying to frame such a complex concept within a particular context: 1 first, specify the system under analysis (ie, are we analyzing the computer as a whole, for instance, or a particular piece of software?); second, analyze what the system is capable of doing (ie, how it responds to its environment); third, following one of the classic definitions of the term (Rafaeli, 1988), discern if these responses are merely reactive or are a case of interactivity in terms of acknowledgement of already performed actions.
This definition corresponds to the third category mentioned above. It aims at clarifying the communicational process of interactivity, and it understands it in terms of design and functionality, not of subjective experience by the user or interactor. The point of view for the analysis is on how the systems are designed in order to respond to the users' actions in a way that is read as interactive; on the process of dialogue that is created between the behaving system and the human interactor and not on the subjective experience of the latter.
Within this discussion, the metaphor of the conversation is useful: the dialogue between the human and the machine that inspired Cybernetics and that is the foundation of Human-Computer Interaction. After all, the idea is that computers exchange information with users, as the back and forth of messages advances. User does, computer reacts, then user does again in response to this reaction, and so on and so forth. This conversation can be understood in a very simplistic manner, in terms of the control of systems where the artificial system responds to the commands of a human user, or with a more ambitious interpretation of the metaphor, which aims at generating, through the interactive system, a series of meaningful exchanges.
Gordon Pask's Conversation Theory is an important effort to theorize this (Pask, 1975;1976;1996), although it is a general theory not specifically centered on man/machine conversation. Pask's writings on the subject find its context in his interest in learning devices. Conversation is in his work presented -and interpreted by latter authors -as an advanced form of interaction by systems that have the ability to learn. More recently, Paul Pangaro has elaborated on Conversation Theory in relation to social media and design (Pangaro, 2009;Dubberly and Pangaro, 2009) and to relevant models of interactivity (Dubberly, Pangaro and Haque, 2009). Following Pask, Pangaro and colleagues understand that conversing systems are a sophisticated assemblage of (second-order cybernetic) learning systems that feed on and react to each other. This conversation is, according to them, the most elaborate form of interaction.
In an effort to differentiate systems that are merely reactive from those that afford interactivity, Dubberly, Panagaro and Haque distinguish among static, dynamic-reactive, and dynamic-interactive systems. The first are systems that "cannot act and thus have little or no meaningful effect on their environment (a chair, for example)". Dynamic systems do have the ability to act and relate to their environment. Within those, they distinguish between systems that react (they have a linear relation between system activation and response) and systems that interact. The categorization of interactive systems is further elaborated and, as said above, they present conversation as the most sophisticated form of interaction (Dubberly, Pangaro and Haque, 2009). A similar categorization of passive, reactive and interactive systems was presented in (Soler-Adillon, 2012).

Designing artistically behaving artifacts
When designing interactive artistic artifacts, the approach to interactivity differs from that of designing functional and efficient digital systems. In the latter, system's interfaces should be easy to use, robust, consistent with existing standards, etc. and users should know what to expect from the system and how to achieve their goals. The interactivity of interactive art, on the other hand, needs not to be constrained by these principles, as it fundamentally differs from functional interaction in its intentions. This dichotomy has been characterized, from the point of view of the creator, as the 'design for efficiency' and 'design for motivation' approaches to creating interactive media (Ribas, 2001). Along the same lines, CC E-JOURNAL ON ART, SCIENCE AND TECHNOLOGY http://artnodes.uoc.edu The intangible material of interactive art… artnodes CC Simon Penny has elaborated the notion of poetic interaction, which is at the heart of his theorization of the aesthetics of behavior (Penny, 2000;. The basic idea is that when designing interactive artistic artifacts, the type of interactivity that one is seeking to create is very different from the one in the case of functional interaction. Poetic interaction is the kind of interactivity that develops within a system formed by an interactor and a behaving system that has been designed with artistic purposes. The aesthetics of behavior refer to the analysis of such type of devices and the experiences they afford. In respect to functional systems, aesthetically behaving systems move towards unpredictability. Systems are neither necessarily easy to control in terms of how the interface is presented, nor are they necessarily predictable in their behavior. The idea is that they do so without falling into what is perceived by the interactor as complete random behaviors. Otherwise they become uninteresting in terms of interactivity, since when that is the case all sense of dialog or control is lost. Thus, artistic interaction is situated in the search for a point of equilibrium between what is predictable and what is not. According to Simon Penny it "should not be predictably instrumental, but should generate behavior which exists in the liminal territory between perceived predictability and perceived randomness, a zone of surprise, of poetry" (Penny, 2011). And it is in this liminal territory that it seeks to create conversation. Moving away from the linear responsive model of interaction, as described by Dubberly, Pangaro and Haque in (2009), conversation is here a more elaborated mode of relation to the artificial system: "artistic interaction [can be conceived] as an ongoing conversation between system and user rather than the conventional (Pavlovian) stimulus and response model" (Penny, 2000).
Poetic interaction lives in the realm of art and experimentation. In it, interactivity and behavior become aesthetic concerns and this has a series of implications in this type of art creation and on the discourses around it. In terms of understanding how interactivity is designed and understood, this approach is to be interpreted as differentiated from functional interaction. The objective of poetic interaction is to experiment with the possibilities of interactivity. As noted by Krueger when describing his early video tracking experiments, interactivity becomes here a central aesthetic concern (Krueger, 1989) and, as such, becomes a fundamental part of the artistic interest in these pieces, although not necessarily in an exclusive manner.
The methods of designing artifacts for poetic interaction may coincide with those of functional interaction, and some of the design principles that apply to the latter apply to the former. However, since the objectives are fundamentally different when designing such systems, their interpretation has to be made from a different point of view. These experiences and the processes that develop as the interactor and the system act and react to each other shall be read as any other aesthetic experience: a dance between expectations and surprise where the interactive artifact seeks to engage the user into exploring and experimenting with it; a dance of agency, to use Andrew Pickering's terms (1995). The artistic interactive system is not conceived as a tool. Instead of being a system to help a user perform a task efficiently, in this paradigm it exists on its own right, as an agent -a behaving entity -that will drive the experience of the visitor to the piece.
Thus, creating artistic interactive systems means designing performative artifacts that will relate in a meaningful way to an interactor. This means that there are mainly two focuses of interest in terms of design: interface and behavior. Within the system formed by interactor and performative artifact, the first is the point of connection between them. In accordance, its design is crucial in establishing this relation. The artist creating the artifact knows that the interactor relates to his or her environment by doing things with it. According to this approach, the idea is that he or she will acquire the necessary knowledge to relate to the interactive object through this embodied doing, not through an objective abstract examination of it. This means that, as opposed to the design of a website from the point of view of usability, for instance, the design approach must take into account the actions of the interactor at all levels. The analysis of the artifact's behavior comes through the doing things with it in relating to its interfaces (using the buttons, moving in front of it, etc.).
Behavior is the essential element in defining the artifact, and the interfaces are the means of establishing the relation to it. All other aspects of the piece are accessory to behavior. This doesn't mean they are not important, but external appearance, for instance, is here one step below in aesthetic relevance. Indeed, interactive art pieces can be analyzed aesthetically from the point of view of appearance or in relation to the technology of the time. But according to the approach proposed here, it is in terms of performativity, of behavior, that are most relevant. Since they aim at relating to their visitors in a meaningful way, and are designed to exploit such relations and to provoke exploration of behavior on the part of the human interactor, they are aesthetically relevant mostly in terms of interactive behavior.

Agency, behavior and performativity
An agent is a behaving entity. Andrew Pickering has described agency as a basic building block of the influence that an entity can have on its environment. Put simply, agency is about "doing things in the world" (Pickering, 2002), a notion that he elaborates in detail in his work (Pickering, 1995;. Other accounts take agency further into the ontological and metaphysical, and place it at the very center of the configuration of matter and reality. According to Barad, agency cannot be an attribute of things as it ontologically precedes the things themselves (Barrad, 2007, p. 178). In any case, arguably in terms of designing interactivity, it is sufficient to state that an agent is a non-passive entity. That is, one that either responds reactively or interactively to the actions that it is able to perceive from whomever is relating to it.
The creation of agents with artistic purposes (agents as artworks) has been studied by Simon Penny (2000). An agentic artwork, according to Penny, is not just one that does things in responding to its surroundings. It can be understood as a cultural actor in its own right. These agents are designed and built within a cultural tradition, and in a particular social context. And it is when they are understood and read as such, and not abstracted from their social and technological context, that they can be better understood.
Penny's own Petit Mal is a paradigmatic example of this approach. This robotic piece intentionally eschews all anthropomorphic or zoomorphic appearance, but its behavior is infallibly attributed to some sort of animal-like entity, as it hesitantly moves towards and away from the visitors in a constant loop of recognition, retreat and forgetfulness. The materialization of the piece is crucial, of course. Penny acknowledges that it did have the right size so as not to be scary, but at the same time not to be regarded as a sort of pet robot (Penny, 2015). However, what defines Petit Mal as much as its material instantiation is its behavior. Penny's naming it after the disease that informs how it responds to its environment is no coincidence here. The defining trait of the piece is its agency; one that performatively relates to the visitors in a neat example of embodied interaction.
With these type of pieces, then, the aesthetic focus on interactivity is a step away from the object and a step closer towards its relation to the interactor. The work is neither just the behaving system nor its behavior, but it too encompasses the situated actions of the visitor, the embodied 'contextualized doing' of whomever interacts with the system, much in accordance to the performative ontology that Andrew Pickering describes as the basis of Cybernetics. As Simon Penny puts it, in connection to ideas of embodied interaction and enaction, "the lesson of performativity is that the doing of the action by the subject in the context of the work is what constitutes the experience of the work. It is less the destination, or chain of destinations, and more the temporal process which constitutes the experience" (Penny, 2011).
As said, an important issue here is with what Pickering has identified as the performative ontology of Cybernetics, which is arguably extensible to interactive art practices in general. The basic idea is that, in opposition to the representational ontology of modern science, Cybernetics is based on a performative ontology: "a decentered perspective that is concerned with agency -doing things in the world -and with the emergent interplay of human and material agency" (Pickering, 2002). Within this context, Cybernetics appears as a paradigmatic discipline of the 'performative idiom'. The discipline as a whole "is all about this shift from epistemology to ontology, from representation to performativity, agency and emergence, not in the analysis of science but within the body of science itself" (Pickering, 1995).
Thus, the performative idiom, according to Pickering, is an ontology in its own right, and a nonmodern one in the sense that it differentiates itself from the reductionist approach of the scientific method (the modern ontology). In contrast to Bruno Latour's characterization of modernity as being determined by the dualism of people and things (Latour, 1993), the cybernetic approach, in which this frontier would be blurred, "stages for us a nonmodern ontology, in which people and things are not so different at all" (Pickering 2010, p. 18). As he has explained (Pickering, 2008;, this ontology allows Cybernetics to propose an image of the world that is performative rather than representational. He calls it the 'ontological theater', a nonmodern understanding of the world and our relationship with it that implies "a vision of knowledge as 'part of' performance rather than as an external controller of it" (Pickering 2010, p. 25).
Here the idea of the black box is useful to understand how this is brought into practice. According to Pickering, Cybernetics proposes a theory of knowledge that is largely built up through a performative relationship with what we can understand as black boxes. Rather than being about control in a classical sense, "the entire task of Cybernetics was to figure out how to get along in a world that was not enframable, that could not be subjugated to human designs -how to build machines and construct systems that could adapt performatively to whatever happened to come their way" (Pickering 2010, pp. 30-31).
In this context, he argues how Cybernetics assumes in fact an ontology that, contrary to the reductionist approach, involves a certain degree of unknowability, as it "tries to address the problematic of getting along performatively with systems that can always surprise us" (Pickering 2010, p. 23). These are what Stafford Beer labeled as exceedingly complex systems. That is, systems which are, unlike simple and merely complex systems, neither predictable nor susceptible to be treated by the methods of modern science and engineering. Exceedingly complex systems, like the interior of the black boxes, are unknowable. They are too complex to be grasped representationally, and they change over time, so that future behavior cannot be anticipated trough current knowledge.
So if Cybernetics was about that, as Pickering argues, it was not so much about control, as some literature has portrayed it, as it was about the study of the conditions under which the interaction among different parts of technology could result in these complex, unanticipated patterns of behavior. Some of the devices built by the cybernetic practitioners were clearly built in order to experiment in this direction, and were in this respect anticipative of interactive art. Ross Ashby's Homeostat was a machine whose sole purpose was to exist performatively. Once set in motion a homeostat unit would connect to others and seek its own equilibrium, with no other goal than that. Norbert Wiener defined it as the "brilliant idea of the unpurposeful random mechanism which seeks for its own purpose through a process of learning" and qualified it as "one of the great philosophical contributions of the present day" (Wiener, 1989, p. 38). Grey Walters tortoises, which behaved much similarly to the contemporary Roomba robots -certainly not by chance -, also represent an example of such practices. These avant-la-lettre bottom-up robotic devices were capable of surprising its own designer, Walter, who described them as having a 'remarkably unpredictable' behavior (Walter, 1950). As Owen Holland has noted, they were a clear anticipation of Artificial Life (Holland, 1997;. These two pieces, along with Gordon Pask's Musicolour and Colloquy of Mobiles, suffice to exemplify the early Cybernetic's remarkable anticipation of interactive art.

Emergence
The element of surprise, of impossibility of a full anticipation that Pickering attributes to Cybernetics, and that can be expanded to interactive art in general, clearly resonates with the idea of emergence, a concept that is often explained with the idea of a whole being 'more' than just the sum of its parts; of being irreducible to the mere addition of its constituting elements. These explanations are usually articulated in terms of different levels of complexity where parts or agents are the constituents of a larger systemic level (the whole) in which emergent phenomena appear. What is emergent are properties and behaviors that can be observed at the system level, and they have a degree of complexity that, as it is argued, cannot be accounted for by a simple addition of the properties and behaviors of the parts.
In the academic literature the concept of emergence appears as related to two different and not necessarily related phenomena: self-organization and the appearance of novelty (Soler-Adillon and Penny, 2014). Many authors present it as the result of multiple local interactions among agents within a system that produce observable patterns at the system level. These patterns are emergent phenomena in the sense that they could not be understood, nor anticipated, through the analysis of the elements and their behaviors in isolation (Langton, 1988;Holland, 1998;Bedau, 2008). For other authors, emergent phenomena are related to fundamental novelty and, thus, to creativity. For them, emergence is synonymous to the appearance of new functions or behaviors in a known system (Steels, 1995;Cariani, 2012).
The typical examples of emergence are systems that exhibit complex behaviors from a relatively small set of simple rules and behaviors. Ant or termite colonies and their social complexity, the human mind understood as a product of the interconnectivity of neurons in the brain, chemical clocks, traffic jams or cellular automata are some of the most cited examples. When related to novelty, emergence is often used when referring to learning systems or adaptive devices.
Prominent among the authors that are concerned with emergence as a generator of novelty and, thus, as creativity, Peter Cariani has elaborated the theory known as emergence-relative-to-a-model (eg Cariani 1992;. Cariani articulates a discourse that aims at identifying emergence as novelty in a given system in a way that can be scientifically communicated. His approach is concerned with how new functions can appear in systems or devices that perceive and act on their environment. The basic idea is that this newness can only be accounted for scientifically if, first, the observer of the system defines its states and state-transitions by creating a model of it. Once this is done, these observations are used to make predictions on the futures states of the system. In this context, emergence occurs whenever unanticipated behaviors, states or functions appear: "emergence is the appearance of novel entities that in one sense or another could not have been predicted from what came before" (Cariani, 2009). Cariani bases his modeling on three aspects: how the system reads and acts on its environment (semantics); how it decides how to act according to these readings (syntactics); and how it evaluates the actions performed according to its goals (pragmatics). All these actions are performed according to the basic building blocks of what the system can operate with: the primitives, in Cariani's terminology. Within this framework, he identifies two ways in which emergence can occur. The first is Combinatoric Emergence, which consists in the appearance of new system functions through new combinations of the primitives with which the system operates (eg, genetic algorithms). The second is Creative Emergence, which is the appearance of new functions trough the introduction of new primitives in the computations. This second type of emergence, equivalent to the introduction of a new sensory organ in an animal species through the course of evolution, is extremely rare in artificial systems, and in fact Cariani identifies in his literature only one case: Gordon Pask's electrochemical 'ear' (Cariani, 1993;. Despite these difficulties in the case of artificial systems, however, Cariani opens a door to mixed computer-human systems (ie, interactive systems) to be generators of this latter kind of emergence (Cariani, 2009;. The relationship between emergence and interactive art can be exemplified by the idea of the unpredictable black box (Soler-Adillon, 2011). In the prototypical engineering black box, the researcher can figure out the mappings between inputs and outputs; i.e. he or she must be able to develop a protocol to map what goes in and what comes out of the box (Ashby, 1957). In contrast, unpredictable black boxes would be devices that appear as black boxes even to their own creator; systems in which the relation of inputs and outputs is not fully foreseeable, and the inside of which is not only unknown but unknowable. Thus, the only possible relation with these unpredictable black boxes is the continuous performative act of interacting with them. Not even the eventual opening of the box (or zooming in into the system) would solve the problem of the unpredictability that they would have up to some degree.
Ideally, the result of the unpredictable black box is not mere randomness but something that, although coherent with the general behaviors of the system, was not explicitly built in it by its designersomething that lives in the liminal territory of poetic interaction and can be qualified as emergent. As often happens in interactive art (eg in generative art and especially in Artificial Live Art), it is sought by the artist to create systems or processes that exceed his or her own expectations. The idea is to do so not through some blind trial and error, but through the creation of the conditions where emergent phenomena can occur. This creative effort can be conceptualized with the idea of emergence as generation of novelty .
In terms of designing interactive artifacts potentially capable of exhibiting emergent behavior, Cariani's emergence-relative-toa-model is arguably the most solid framework to approach the challenging task. Designing such devices means facing the apparent paradox of designing something that is, in principle, impossible to be designed. However, designing the possibilities for emergence to appear is indeed possible, and it has been the goal of a series of practitioners, mostly in relation to Artificial Life Art.
A successful example of such pieces is Ruairi Glynn's Performative Ecologies (Glynn, 2008), a series of robots that learn how to best attract the gaze of the visitors to the installation space by 'dancing' in front of them. As they perform their moves, they calculate (through facial recognition) how much of the attention of a visitor these moves are capable of attracting. Then, using genetic algorithms, the individual robots will create new dance moves based on those that were successful within the group. With this, eventually the robots should be capable of attracting more and more of the users' attention as they learn over time to do so. The new behaviors that the robots learn and perform, and that were not pre-designed by the artist, can be regarded as a case of emergence -particularly, of combinatoric emergence according to Cariani's categorization (Soler-Adillon, 2015).

Conclusions
Interactive art is about behaving entities, agents, performatively relating to their environment, which includes the interactor (visitor, user, etc.). The theorization and practices of early Cybernetics, through what Pickering has labeled the performative idiom, can be a good theoretical framework to understand how these ideas relate to each other. The embodied doing of the interactor with the interactive systems, and the connection to ideas of unpredictability and emergence offer an interesting arena for the analysis and creation of interactive art pieces. According to this, the guidelines of design for emergent behavior are possible to produce. Cariani's emergence-relative-to-a-model and the theorization of emergence as self-organization and as generation of novelty constitute a theoretical framework within which this task can be undertaken.
However, any aesthetic discussion on this area requires a sound conceptual analysis; a clarification of terms that helps avoid the ambiguities of discussing interactivity and emergence without a clear statement of what is exactly meant by such multi-discursive concepts. In this respect, the proposed definition of interactivity, along with the theorization of emergence as self-organization and as generation of novelty  represents an effort to contribute to the elaboration of the aesthetics of artistically behaving systems.