Suspicious Behavior: A fictional annotation tutorial

Suspicious Behavior is a fictional annotation tutorial inviting readers to critically examine machine learning datasets assembled for anomaly detection in surveillance footage (Figure 1). Mimicking existing annotation interfaces [11] and practices [12] the tutorial, although fictive, provides insight into the hidden work of crowdsourced labor and how annotators engage in decision making (Figure 2). Readers in the role of annotator-trainees, advance through an introduction and three ‘advanced modules’ of the tutorial performing what is called ‘Human Intelligence Tasks.’ The assignment is to spot suspicious behavior in video segments. Throughout the interactive story the reader gets trained for an optimized annotation workflow, balancing between accuracy and efficiency.

Within the interactive story YouTube video montages are used to contextualize the hidden human labor of image annotators as fundamental for artificial intelligent. Research on dataset annotation on the other hand suggest that undesired bias gets embedded into datasets through the individual subjective annotator [3,7]. However, traversing through Suspicious Behavior it becomes gradually evident that annotation work is more about matching labels with images than making meaning out of them. Image annotation can be understood as a ‘sense-making practice’[10] in which visual images are translated into operational data. In the process of translating images into data they must be classified in some way [8]. Thus, classifications are embedded in machine learning datasets and become naturalized as part of AI infrastructures [1]. Therefore, understanding practices of image annotation and in which ways operative images as interfaces [6] facilitate human-image-machine interpretation is crucial. Inspired by artistic inquiries of exposing [5], excavating [2] and exhaustively watching [13] image datasets Suspicious Behavior expands on scrutinizing the origins, classification and categorization of image datasets by giving the audience the opportunity to explore and experience aspects of video dataset annotation. The artistic research for Suspicious Behavior involved exploring publicly available anomaly detection video datasets, documentation describing the assembly of those datasets as well as designing a video annotation interfaces. Insights of the artistic research are weaved into Suspicious Behavior. For example, by simulating existing image annotation interfaces Suspicious Behavior enforces optimized workflows which do not allow time for reflection. The reader experiences how the client oversees defining of classes, and control image interpretation through designed annotation interfaces and embedded instructions. Twelve posters instruct the annotator of Suspicious Behavior what to look for (Figure 3). Thus, gradually the reader realizes that definitions are imposed on annotators and passed onto datasets [10].

As a critical artifact Suspicious Behavior presents an approach to examine the power relationships between actors in the annotation apparatus. How is image interpretation distributed along the pipeline of assembling datasets? Who gets to decide what is suspicious or normal? Annotators along with data curators, project managers, engineers and authorities like law enforcement are part of the annotation apparatus shaping the dataset, which in turn frames what machines perceive as anomaly or normal. In addition, technical image interpretation in form of search queries and platform algorithms are enmeshed into practices of dataset assembly [9]. Nevertheless, as experienced through Suspicious Behavior agency is not equally distributed. In this manner the artwork criticizes the division of labor, in which the work of the annotator is devalued [4]. Hence, Suspicious Behavior also offers a starting point to discuss alternatives: What needs to change in order to achieve annotation workflows which put sustainable and ethical working conditions in focus?