Insight provenance for spatiotemporal visual analytics: Theory, review, and guidelines

: Research on provenance, which focuses on different ways to describe and record the history of changes and advances made throughout an analysis process, is an integral part of visual analytics. This paper focuses on providing the provenance of insight and rationale through visualizations while emphasizing, ﬁrst, that this entails a profound understanding of human cognition and reasoning and that, second, the special nature of spatiotemporal data needs to be acknowledged in this process. A recently proposed human reasoning framework for spatiotemporal analysis, and four guidelines for the creation of visualizations that provide the provenance of insight and rationale published in relation to that framework, work as a starting point for this paper. While these guidelines are quite abstract, this paper set out to create a set of more concrete guidelines. On the basis of a review of available provenance solutions, this paper identiﬁes a set of key features that are of relevance when providing the provenance of insight and rationale and, on the basis of these features, produces a new set of complementary guidelines that are more practically oriented than the original ones. Together, these two sets of guidelines provide both a theoretical and practical approach to the problem of providing the provenance of insight and rationale. Providing these kinds of guidelines represents a new approach in provenance research.


Introduction
Visual analytics (VA) seeks to combine the reasoning powers of humans and the automated analysis techniques of computers through interactive visual interfaces with the goal of c by the author(s) Licensed under Creative Commons Attribution 3.0 License CC achieving effective analysis and decision making [45].Research on provenance is an integral part of VA.It focuses on different ways to describe and record the history of changes and advances made throughout an analysis process [67].Provenance information is important both for the analyst or analysts, e.g., to keep track of their progress, and for persons external to the analysis that might need to trace and verify the process, accuracy, and sources of an analysis.For example, in analysis connected to serious real-life problems it is important to be able to repeat the analysis process as well as to know the argumentation of decisions made during the analysis.Provenance has traditionally been divided into analytical provenance, focusing on the analyst's reasoning processes [59,64,87], and data provenance, also known as data lineage, focusing on the computational workflow of the analysis [5,11,22,79].This research belongs to the former, focusing on ways to communicate, through visualizations, the reasoning and the knowledge used by an analyst to persons external to the analysis process.
A recent organizational framework of provenance in VA further differentiates between five different types and six different purposes of provenance [67].Of the types listed, this research deals with the provenance of insight, covering "cognitive outcomes and information derived from the analysis process," and the provenance of rationale, covering "the reasoning behind decisions, hypotheses, and interactions" [67].Of the purposes listed, this research relates to those of recall, replication, collaborative communication, and, most strongly, presentation, which is described as the communication of "how an analysis was conducted, how the findings were determined, or how the data justifies a conclusion" [67].
To provide the provenance of insight and rationale two things need to be recognized.First, a profound understanding of human cognition and reasoning is needed; studying technical sciences alone is not sufficient.This might seem obvious but the truth is that the progress of understanding human reasoning in VA has been slow and misunderstandings still exist.To explain this several authors point to the lack of an accepted theoretical framework through which to investigate and understand this kind of reasoning [1,52,53,60].Second, VA usually deals with data sets that have both spatial and temporal components, i.e., spatiotemporal data.These kinds of data provide both technically and cognitively unique challenges and possibilities which have been studied extensively within the field of geographical information science [16].This special nature of spatial and temporal data needs to be acknowledged.
In a recent dissertation Hall [31] proposed a human reasoning framework for spatiotemporal analysis and, on its basis, derived four guidelines for the creation of visualizations that communicate the reasoning of spatiotemporal analysis, i.e., visualizations that provide provenance.The framework is built on the prevailing theories on human cognition and reasoning in the domains of cognitive science and psychology.It was developed with the explicit goal of taking the special nature of spatial and temporal data into consideration.Hall [31] concluded that the guidelines are a good starting point for any project aiming at providing the provenance of insight or rationale in any domain addressing the visual analysis of spatiotemporal data but also points out that the guidelines are quite abstract and that further research is needed to make them more concrete.
This paper reviews available provenance solutions from the literature and discusses them in the light of the guidelines.The aim was to identify a set of key features that are of relevance when providing the provenance of insight and rationale and, on the basis of these features, to produce a new set of guidelines that are more technically oriented.The main research method is a literature review.The starting point was a set of projects that www.josis.org 1 The tool does not have a name in the original paper by Groth and Streefkerk (2006).To facilitate the readability of this paper the authors named it as they found best.
Table 1: The eighteen tools that made it to the paper, and their abbreviations when applicable, listed in alphabetical order.
emphasize the provenance of insight or rationale.These projects are listed by Ragan et al. in Table 2 on page 37 in [67].The mode of the review can be described as sprawling: when interesting references were found in a paper these were added to the review.The focus was on identifying realized tools and software.Table 1 provides a list of the tools that made it to this paper and their references.Henceforth in this paper the tools are only referenced by their names.Note that this paper does not assess these tools in any way and that in many cases the tools were developed with a completely different goal than providing the provenance of insight or rationale.Rather, this paper discusses how specific features of these tools fit into the provenance context studied in this paper.For the same reason, this paper does not go into the evaluations and user studies done on these tools as the findings would not necessarily be transferrable to the context of this paper.The rest of this paper is structured as follows.Section 2 gives a short theoretical foundation.The framework and guidelines in Hall [31] have not been previously published in the easily accessible format of a paper and are therefore presented here in Sections 3 and 4 respectively.A review of the available solutions can be found in Section 5.The paper ends by presenting a new and practical set of guidelines in Section 6.The new guidelines are discussed in Section 7, which also includes conclusions.

Theoretical foundations
This research takes the distributed cognition framework [1,75] as a theoretical framework for the study of interaction with visualizations and computers, as suggested by many re-searchers [14,37,52,53].Through reasoning we derive inferences or conclusions from a set of premises [74].There are two main types of reasoning, deduction and induction [40,42].Inductive reasoning, which uses knowledge not given in the premises to go beyond the given information, is the cornerstone of human reasoning [2].When reasoning inductively we use the meaning of premises and our knowledge to construct mental models that are iconic, i.e., based on the concept of resemblance [40][41][42].They are, however, not images [26,46], but rather abstract topological structures [26,46,68].
Spatial reasoning appears to have a primary role in our cognition [4,20].It is one of the most common and basic forms of human intelligence [17,49,65] and we use it to reason about other concepts, such as possession, circumstances, and time.Human reasoning about space and time is mainly qualitative in its nature [10,17,23,48,55,69].Qualitative categorical representations are also argued to be important when perceiving an object or scene visually [48].
The subject matter of spatiotemporal analysis can be conceptualized as being composed of spatial, temporal, and thematic components of data [56,61,88,89].Spatial and temporal data identify locations in space and time in relation to some frame of reference.Any other data are thematic and typically a property that can be sensed, measured, and assigned a qualitative or quantitative value [56].The task of the spatiotemporal analyst can be interpreted as finding the meaningful relations inside and between these three data components.

Human reasoning framework for spatiotemporal analysis
The human reasoning framework for spatiotemporal analysis identifies and relates the different cognitive mechanisms of human cognition to each other.The framework borrows its notation and builds to a large extent on the human cognition framework for information visualization by Patterson et al. [60].It incorporates the framework of spatiotemporal concepts (the STC framework [31,32]), which classifies spatial and temporal relations into three basic types, distance, direction, and relations of topology, and has a three-level hierarchical structure of spatiotemporal concepts consisting of basic, compound, and labeled concepts.The framework is thus tightly tied to spatiotemporal analysis and its unique nature.
The framework is depicted in Figure 1 and details the flow of information in human reasoning from stimulus to response.It has a vertical structure to reflect the fact that reasoning is a dynamic interplay of bottom-up and top-down processes.The framework only covers the analytical reasoning system as postulated by the dual-systems theory [19,44,81,82], not the intuitive system.
External information enters human reasoning through encoding, which converts the stimuli into a neural representation in human memory.In spatiotemporal analysis space and time and the relations therein are always present in the encoded stimulus.If the stimulus is a map, innumerable spatial relations are implicitly conveyed to the analyst, regardless of what aspect of the data is under analysis.The framework emphasizes this by explicitly listing basic spatial and temporal relations as a major part of the encoded information.Reference information and thematic information are also given special attention as they play a major role in spatiotemporal analysis.The neural representation that results from encoding thematic and reference information needs, however, to be interpreted using, amongst other things, knowledge from long-term memory, before thematic information or reference inforwww.josis.org E}OEu o (o}Á ~v}š š}‰r }Áv• mation can be extracted.In this way thematic information differs from spatial and temporal relations, which are more directly conveyed.On the basis of the goals and demands of the present analysis task, the analyst may direct his attention to certain features of the stimulus, thus exposing the encoding of the top-down influence of attention and working memory.It is also important to remember the influence of exogenous, stimulus-driven attention on the encoding process [18].These two kinds of attention are not explicitly depicted in Figure 1 but are included in the framework by Patterson et al. [60].
The actual analytical reasoning takes place in working memory.There, the encoded information is iteratively processed as mental models on several levels of abstraction, as illustrated by the STC framework.The basic concepts are distance, direction, and relations of topology and they are the direct equivalences to the basic types of spatiotemporal relations as defined by the STC framework.On the next level there are compound concepts.Contrary to the basic concepts, they need more than two points in space or time to apply and are in a way derived from and divisible into basic concepts.These are concepts such as intensity, size, shape, and distribution.The third level is called labeled concepts.These are basic and compound concepts enriched with thematic information or knowledge not inherent in the data.For example, the distribution of a set of points on a map is a compound concept, while the distribution of the subset of points that have a certain thematic value is a labeled concept.These concepts are processed as mental models or as parts of mental models in our working memory.See Hall [31] and Hall and Ahonen-Rainio [32] for a more detailed discussion of these concepts.
Spatiotemporal concepts are usually given a qualitative categorical representation when processed in our working memory, which might be complex, or as simple as short or long.This categorization is based on the perceived information and on previous knowledge about this kind of information, depicted through the arrows indicating top-down influence from long-term memory and normal processing flow from encoding in Figure 1.This processing might result in an insight, understood as a changed mental model, i.e., new knowledge, which is then stored in long-term memory and which might result in a decision.The causally structured narratives that we use to make sense of and organize new insights and information [15,21,80] are created in working memory on the basis of previous knowledge, a top-down process, and the perceived information, a bottom-up process.
Reasoning in spatiotemporal analysis is inductive and based on knowledge.This knowledge is situated in long-term memory and is depicted at the top of Figure 1.The knowledge is, at least in part, stored in the form of iconic mental models that have an abstract topological structure that corresponds to the structure of what it represents.Causality is often central in these models.Once one of these mental models is activated and accessible to working memory it influences our encoding processes by guiding our attention.These mental models can also be activated directly by an encoded stimulus, depicted by the arrow from encoding to long-term memory in Figure 1.
The final stage of reasoning is a decision, which might result in a response.Decisions can also guide working memory, e.g., by changing certain goals or eliminating certain choices under consideration.A response might be directed towards the representation of data, resulting in a change of stimulus.This then restarts the cognitive processing, perhaps resulting in a changed mental model in working memory or the activation of a new mental model from long-term memory.This is the iterative manipulation of physical and mental representations of information that leads to insight and new knowledge, as described in the sensemaking process by Pirolli and Card [66] and in the knowledge generation model by Sacha et al. [73].

The original guidelines for visualizing reasoning
The guidelines for the creation of visualizations that communicate the reasoning of spatiotemporal analysis are organized under four headings.It is central that the visualizations communicate not only the cognitive outcomes, such as insights and other forms of analytical findings, but the whole cognitive process that led to these outcomes.They are directly derived from the human reasoning framework for spatiotemporal analysis depicted in Figure 1 and each guideline relates to one of the four components in the framework, as indicated.

(1) Decisive tangible pieces of information-relates to encoding
The visualizations need to communicate the decisive tangible pieces of information that were identified in the stimulus and on which the reasoning was based.This is the encoded information that becomes part of our mental models.This communication entails the inclusion of the original physical representation or some simplified representation of it.In this, there needs to be a special focus on the decisive spatial and temporal relations identified www.josis.org in the data by the analyst.Important thematic information and the frame of reference also need to be taken into consideration at this stage.
(2) Decisive spatiotemporal concepts-relates to reasoning The visualizations need to communicate the decisive spatiotemporal concepts and their qualitative categorical representations, i.e., the mental models iteratively processed in our working memory on several levels of abstraction.The qualitative categorical representations are needed to communicate how the analyst's differentiation of spatiotemporal relations into normal behavior and anomalies affected the reasoning.If relevant, the origin of this classification should also be communicated, i.e., whether it is based on the perceived information or on previous knowledge about this kind of information. (

3) Cognitive outcomes-relates to decision
The visualizations need to communicate the cognitive outcomes, such as insights and other forms of analytical findings, as well as the narrative that guided the organization of these outcomes into existing knowledge structures.Regarding a specific insight, these visualizations will ideally communicate both the experience of having it as well as the product of that experience.Decisions made also need to be communicated if they are relevant.

(4) Knowledge-relates to knowledge
The visualizations need to communicate the knowledge used.Here knowledge does not only refer to representations of knowledge in long-term memory, partly in the form of mental models, but also to the mental models processed in working memory during the process of reasoning.As these mental models have an abstract topological structure, the preference, referring to the congruence principle of Tversky et al. [83], is that the visualizations should also have an abstract topological structure.When these abstract topological structures are being created, the possible role of causality in these mental models, or chains of mental models, needs to be considered.The visualizations will take both the bottom-up and top-down aspects of the reasoning into consideration if they are constructed according to these guidelines.In this way they will capture the dynamic interplay between bottom-up and top-down processing that constitutes human reasoning.Please refer to Hall [31] to get an idea of how these guidelines might be applied and in what kind of visualizations that they might result.

Review of available solutions
This review is divided into two parts.The first part looks at the solutions related most closely to Guideline 1.This entails ways of communicating the original physical representation used by the analyst in the analysis process, i.e., the stimuli, as well as ways of differentiating in more detail the information identified therein.There is a need for both an overview and detail, so to speak [77], with special attention being paid to decisive spatial and temporal relations.The second part deals with solutions more closely related to the ideas of Guidelines 2, 3, and 4, i.e., ways of communicating the reasoning and knowledge of the analyst as well as the cognitive outcomes of the analysis.This is more a question of explicating the bits and pieces of knowledge and information that were important for a certain analysis outcome and their relation to each other, with special attention being paid to spatiotemporal concepts.

How can the stimuli be captured?
To communicate the original physical representations and the decisive details therein, there needs to be a way to record or link back to the visualizations used.In general there are two ways of recording these kinds of data: designed systematic capture and manual user capture [35,87].The former usually aims at capturing all the visualization states, while the latter is about letting the user capture the relevant states.In some cases the delimitation between the visualization states to be automatically captured is clear-cut.Such an example is VisTrails, developed within the domain of scientific visualization, which was one of the first tools to formally propose a designed systematic capture.In other domains, such as VA, which include a lot of interaction with the visualizations, e.g., brushing, filtering, and zooming, the delimitation between different visualization states is fuzzier.In these cases designed systematic capture can be realized at different semantic levels.
To semantically model user analytical behavior many researchers apply the model by Gotz and Zhou [29], shown here in Figure 2.They characterized user behavior into a hierarchical system consisting of tasks, sub-tasks, actions, and events.These four categories describe levels of decreasing semantic richness where activities in one category are combined into sequences that form a single activity on the following level.Events such as mouse clicks and keystrokes and actions such as filtering and sorting are easy to capture but low in semantic content.Sub-tasks and tasks represent more concrete analytical goals.They have a higher semantic content but are difficult to capture without user input.One system that applies this model is the visual analytic system HARVEST.It performs systematic capture on the action level by capturing the user's actions, e.g., query, filter, and sort activities, by bundling events such as mouse clicks and drags into action-level expressions of user activity.Tools that apply designed systematic capture usually provide a visual overview of the stored states.The simplest structure is probably a linear trail, such as the "action trail" of HARVEST or the "breadcrumb track" of the Scalable Reasoning System (SRS).More common is a tree structure, where nodes usually represent visualization states and edges the changes applied to a certain state (the parent node) to achieve the new state (the child node).Typical examples of this are VisTrails, Aruvi, and the Provenance and Annotation Prototype (P&A).Figure 3 gives typical examples of a linear trail and a tree structure.In Aruvi [78] the history tree can be ordered by time so that it resembles a timeline and shows the temporal order of the visualization states.Some tools also use a pure timeline view inwww.josis.orgstead of a tree structure, e.g., SensePath.A variation on the tree structure is to use the actual visualization states as nodes.Examples of this can be found in ExPlatesJS and GraphTrail.
There is also the option of not applying any specific structure.In the Human Terrain Visual Analytics Prototype (HTVAP) snapshots representing visualization states are stored without any explicit structure.The user can browse and reorder them on the basis of a set of quantitative measures.To further help analysts handle the potentially large number of snapshots each of them is enriched with a graphical summary that succinctly summarizes the important aspects of the visualization state, including predefined data source uncertainty, the spatial and temporal extent of the data, and the visual variables used.
: Two illustrations of typical visual overviews of stored states.The linear structure in the upper row is inspired by the action trail of HARVEST and the tree structure in the bottom row is inspired by the history tree of Aruvi.These visual overviews of the stored states, especially those applying a tree or timeline structure, also often work as an overview of the interactions made with the system.A way to allow a more detailed understanding of the interaction with the visualization states is to record a screen capture video, a feature introduced by SensePath to support humancomputer interaction researchers in their analysis of subjects' browser-based sensemaking process.When a recorded action is selected on a timeline view, there is a browser view that displays the web page where the action was performed and in addition to this a video view that automatically jumps to the corresponding part of the screen capture video.The video can provide additional contextual information about subjects' interaction, such as scrolling and mouse movements.Lipford et al. [50] discuss the use of video for a similar purpose and they recommend that video should not be used without providing additional information such as recorded actions.
Many tools provide some kind of annotation functionality that enables users to record, organize, and communicate findings.These kinds of functionalities can be used to communicate the decisive details of the original physical representations.Maybe the easiest to implement is textual annotation of stored states.This kind of functionality is offered in many tools, such as P&A, Sense.us, and HTVAP.Although expressive, written text needs to be read and can easily remain disconnected from the details of the visualization.Overlaid graphical annotations, on the other hand, are implicitly conveyed and provide a way to easily explicate details in a visualization, such as pointing to or circling around certain details.They are also good for identifying spatial and temporal relations [33], as can be seen in Figure 4. Sense.us[36] allows traditional textual annotations to be enhanced with graphical annotations such as free-form ink, lines, arrows, shapes, and text that can be drawn over the visualization view.ExPlatesJS [38] provides a similar graphical annotation feature but instead of just single views being annotatable the whole "exploration canvas," where each visualization and data operation that is created is laid out in a node-link structure, is annotatable.Graphical annotations are of two general types: overlay and data-aware.They are discussed by Heer et al. [36] and Heer and Shneiderman [35].Overlay annotations are a direct and unconstrained way of interacting with a visualization but they lack an explicit tie to the underlying data.By making annotations data-aware they can persist over operations such as filtering, aggregation, and rotation and thus not be tied to only one particular view of the data.One way to make annotations data-aware is to realize them as selections [34].GeoTime Stories, a space-time cube tool for the analysis of events, allows data-aware graphical annotation of events by means of curves, arrows, callouts, outlines, and enlargements.These annotations can be re-rendered when the view is updated (e.g., when the space-time cube is rotated).
Textual annotations can also be made data-aware.The ManyInsights tool implements an approach called Click2Annotate that allows the user to annotate interesting aspects of a view by brushing the relevant data (e.g., a set of points), storing it as a finding, and choosing its type from a pre-defined set of types, such as cluster or outlier, corresponding to the sub-task level in the model by Gotz and Zhou [29].The system then automatically generates an annotation that describes the data and that the user can manually modify or add to.As these annotations are data-aware they can be flagged in all views where the relevant data items can be viewed.
Annotations can be marked with descriptive terms, called keywords or tags, to organize them for future navigation, filtering, or search.Tagging systems are non-hierarchical and inclusive and are often contrasted with hierarchical and exclusive taxonomies [25].Examples of tools that provide a tagging functionality are ManyInsights, which allows the user www.josis.org to create their own tags, and CommentSpace, which uses a small, fixed vocabulary of tags (question, hypothesis, to-do, evidence-for, and evidence-against).

How can the reasoning be captured?
Moving on to the subject matter of Guidelines 2-4, things get more abstract.There is no designed systematic capture that can record the reasoning of the analyst.Analysts need to record this information themselves.What is key here is to provide the possibility of linking the bits and pieces of information and knowledge that the reasoning was based upon together to show how they led the analyst to a specific insight or decision.A popular solution is to use a node-link structure for this purpose, often called reasoning graphs.Typical examples of this are CLIP and Jigsaw's Tablet.Figure 5 provides a typical example of a reasoning graph.These are described as visual thinking spaces that support sensemaking through allowing users to record, organize, and share their findings in the form of node-link graphs and notes.Interesting entities, such as people, locations, or events, are added as nodes and relationships between them as links.Aruvi [78] is another example.Compared to CLIP, it provides a larger set of different nodes and connectors that allow the user to create more expressive graphs.A third and more refined example is SRS [62,63], which provides the ability to record and link evidence, assumptions, hypotheses, and other artifacts using a node-link structure.There are also other approaches than the node-link structure.The analytical software environment nSpace provides the user with a flexible and expressive workspace called the Sandbox where users can "write anywhere, group anything, [and] place anything anywhere" [86].It supports node-link structures but can also handle text snippets and images and puts the focus on the ease of spatial arrangement of these.Another solution is to use a textual narrative, or story, to communicate the reasoning.This more formal and linear approach is used by GeoTime Stories.Instead of a graph or workspace, the main output is an authored interactive report.
Important distinctions for these kinds of systems are if they are stand-alone or integrated systems and if the evidence that the reasoning is based upon is left out of, linked to, or included in the recorded reasoning.Stand-alone systems are mainly for organizing thought and communicating findings, not for analyzing information.CLIP is a pure standalone tool, designed for recording findings made in other contexts.It does not support either the inclusion of evidence, or linking to it.Many tools, however, integrate functions for analyzing data and recording and organizing findings.This makes it easier to include or link to evidence, such as the visualization states used.In Aruvi the analysis of data and recording of reasoning literally take place side by side.There, a node in the reasoning graph can be linked to the original visualization state that the piece of information in the node is based upon.At the same time Aruvi also displays the location of this visualization state in a history tree of stored states.SRS also applies an integrated approach, with the additional option of including visualization states in the recorded reasoning.The idea is that any feature of a view, or an entire view, can be saved and added to the node-link structure as a node that contains a snapshot of that view.These nodes can be expanded to a full view that consists of an interactive visualization of what the analyst saw.
There are also tools that are not pure stand-alone or integrated systems.Sandbox, which is a part of nSpace, is integrated with a tool for information retrieval and analysis called TRIST [43].The analyst can add any relevant information from TRIST by dragging it into the Sandbox workspace and saving it as an object.These objects then work as links to the original interactive views in TRIST.It is also possible to include the original visualization states in the Sandbox workspace.There is also drag-and-drop support for information from other sources, such as text editors and web explorers, but these objects do not link back to the original sources.In relation to these sources, Sandbox is a standalone system.The future might be standalone systems that can be plugged into any VA system.In a recent publication Sacha et al. [71] present their current progress in developing a notetaking environment (NTE) that supports importing visualization states from, and recording interactions with, any VA system.
There is an evident difference when it comes to the structures for recording reasoning between tools that are mainly designed to support analysts in organizing their thoughts, and tools that are designed to support the analyst in presenting results.The former emphasize divergent and expanding structures, which are better suited for exploratory work, while the latter focus on more linear structures, which are natural to use when it is already known how things are connected and the causal relations between input and output.Typical examples of tools that are designed to support analysts in organizing their thought are CLIP, Jigsaw's Tablet, and Sandbox.
Two interesting and dissimilar examples of tools more focused on enabling the presentation of results are GeoTime Stories and HTVAP.As already mentioned, GeoTime Stories provides the analyst with the ability to author narrative reports, or stories, that link to decisive views of events in time and space.The story is a top-down approach that promotes the sharing of observations and understanding of complex phenomena.They can be presented to high-level decision makers without additional explanations.A narrative approach is also used in HTVAP.Its narrative is, instead of a written report, a graph-node structure consisting of a set of ordered and annotated snapshots of visualization states that communicate "facts, assumptions, assertions and interpretations" [84].Figure 6 illustrates the HTVAP narrative approach.This function was developed with a special focus on providing support for analysts giving short verbal briefings to superiors.In these tools the story is used as a vessel for communication and understanding in a similar way as in narrative visualization [24,47,76].Trust is an important aspect of analysis.Sacha et al. [72] describe analysts' trust as depending on the extent of their "awareness of the underlying uncertainties generated on the system side."Xu et al. [87] list three contexts of trust in analysis: "trust in the data, trust in the process, and trust in the result."Several tools provide support for analysts to record their trust in findings.In SRS analysts can describe their trust in evidence and the degree to which each node in the reasoning graph supports their hypotheses or conclusions.For this purpose each node and link in the graph is equipped with an adjustable bar graph that represents its current belief state.SRS propagates the trust and support scores across the graph and produces likelihood estimates for hypotheses and conclusions automatically.NTE provides similar functions by enabling the analyst to apply trust ratings to imported visualization states and notes.Figure 7 illustrates how ratings are aggregated into an evidence bar in NTE that describes if a hypothesis is rejected or accepted.Sandbox also provides these kinds of features.
Whether the tool supports designed systematic capture of the visualization states used or only relies on manual user capture has a considerable effect on the available design solutions when it comes to how visualization states are added to a reasoning graph.With tools that support designed systematic capture it is possible to let the user choose from the set of saved visualization states.This kind of design enables reasoning graphs to be created post-analysis because the decisive visualization states can still be accessed.That analysts explicitly need to save visualization states to use them in a reasoning graph might be problematic as it requires the analysts to be aware of their importance immediately, although this might only become clear later in the analysis.HTVAP presents a clever solution for how visualization states can be added to a reasoning graph from a set of stored states.There are two more design solutions related to the topic of Guideline 2-3 that deserve to be mentioned.First, many tools provide some kind of tagging system or taxonomy to describe the role of nodes in a reasoning graph.In SRS reasoning artifacts can be tagged as evidence, assumption, spatial or temporal pattern, causal relationship, or hypothesis.In NTE verifying, falsifying, and neutral tags for pieces of evidence are used.In Figure 7 these tags are represented by the happy, sad, and neutral faces.Second, some tools also provide timeline visualizations alongside the reasoning graphs.For example, CLIP allows nodes in the reasoning graph to be linked to and visualized on a timeline.Related to this, SchemaLine presents a solution for how entities on a timeline can be grouped together visually to facilitate analysis.

Synthesis: Key features
This section presents a synthesis of the review of available provenance solutions that was performed.A set of key features for an ideal provenance tool is identified.This is done on the basis of the guidelines and with the intention to provide the provenance of insight and rationale, with the main purpose being communication, according to the framework by Ragan et al. [67].
One of the first questions when designing a tool for recording and communicating reasoning is what kind of format to use.The recommendation of this paper is to use a graph with a node-link structure, i.e., a reasoning graph.This is an abstract topological structure as recommended in Guideline 4 and it is also one of the most popular solutions, on the evidence of the review.It furthermore enables the user to easily record causal links between pieces of information and knowledge.Optionally, support for other kinds of orwww.josis.orgganizational metaphors, such as those listed by Robinson [70], could be provided.What is of the uttermost importance is that these reasoning graphs include or link to the original physical representations that the reasoning was based upon, the bottom-up aspect of reasoning.This can, e.g., be achieved through including the actual visualization states in the graph (as in NTE, Figure 7), making them available in an adjacent view (as in Aruvi), or by including some simplified representations of them that link to the original visualization (as in HTVAP, Figure 6).If the original visualizations were interactive, then the visualizations accessed through the reasoning graph should also be interactive.
These reasoning graphs should be created post-analysis in order not to interrupt the actual analysis.This is not to say that graphing capabilities do not need to be provided to analysts during analysis, only that for reasoning graphs to be effective for presenting the results and outcomes of analysis they will necessarily different from graphs for organizing thought.In the kind of deterministic situation that prevails once the outcomes of analysis and the goal of a presentation are clear it is easier to create effective graphs that only include the relevant information.These graphs will thus be linear rather than divergent and clearly communicate the narrative of the analysis (see Guideline 3 and Figure 6).
A prerequisite for a smooth post-analysis creation of graphs is that there is a designed systematic capture of the visualization states used.Manual user capture at this stage would simply be too laborious and difficult.The saved visualization states need to be easily accessible and browsable by the analyst.This can be facilitated, e.g., by presenting the stored states as snapshots that include graphical summaries and allow the analyst to order them by qualitative measures (as in HTVAP) or by providing a tree-like visualization (illustrated in Figure 3) of the interaction process that created the states (as in VisTrails), possibly orderable by time (as in Aruvi).
Providing annotation abilities is of equal importance to including the original visualizations.Through textual annotation analysts are able to describe their reasoning and communicate their narrative.It also enables them to pinpoint the role of thematic information, the frame of reference, spatiotemporal concepts, their own knowledge, and other aspects mentioned in the guidelines.Providing graphical annotation is equally important as it enables the analysts to specify the decisive tangible pieces of information identified in visualizations, especially spatial and temporal relations (see Guideline 1 and Figure 4).Data-aware annotations have many strengths, especially in interactive interfaces where they can stay visible although the view changes, and are to be preferred if realizable.Overlay graphical annotations and data-loose textual annotations are, however, good alternatives in most situations.Annotation capabilities should be available both during analysis and post-analysis when creating reasoning graphs.Annotations created during analysis should be automatically captured and linked to the corresponding visualization view.Annotated visualization states should be easily discernible when browsing saved states to facilitate access to these supposedly important states.
Tagging systems or taxonomies can be used to enhance the information value of both annotated visualization states (as in CommentSpace) and entities in reasoning graphs (as in SRS).As for stored states and annotations, tags can also facilitate browsing and reasoning (the faces in Figure 7 are one example of the use of tags).There are many taxonomies and similar systems available that, if not usable as such, could work as a starting point.Some examples are the fact taxonomy by Chen et al. [8], the classification of visualization insights by Choe et al. [9], the classification of movement patterns by Dodge et al. [12], and the STC framework [31,32].A tagging system based on the STC framework could be useful to conform to Guideline 2. In a similar fashion entities in reasoning graphs and annotations can be tagged with the degree of trust that the analyst has in them.
A big challenge for analysts creating reasoning graphs post-analysis is how to remember how the reasoning progressed and which visualization states were actually decisive.A solution is to use cued recall, which can enable analysts to "remember more of their rationale and decision points" [50].Cues can be provided to the analyst in the form of interaction logs [39], screen capture video (as in SensePath), eye-tracking recordings [30,50,67], and think-aloud recordings (used for annotating visualizations in P&A).

New practical guidelines
As a summary four new guidelines are provided.

(1) Use node-link reasoning graphs as a communication format
The graphs should be created post-analysis so as not to infer with the analysis.They should furthermore have a clear communication objective, i.e., a certain decision or outcome that they aim at communicating, and a linear structure that clearly communicates the narrative and important causal links.They can be enhanced with tags indicating the level of trust and other relevant information.
(2) Include the original physical representations The reasoning graph needs to include or provide access to the decisive visualization states and the other physical representations that the reasoning was based upon.The smoother the access, the better.This requires there to be a designed systematic capture during analysis and the captured states to be easily browsable and accessible post-analysis.In this way the spatial and temporal relations identified in the data by the analyst are also included.This is especially important when analyses have included maps, timelines and similar spatiotemporal representations that include substantial amounts of possibly important relations.
(3) Provide annotation capabilities Textual and graphical annotation capabilities should be provided both during analysis and post-analysis.Annotations should be automatically captured and linked to the corresponding visualization states.Tags, possibly identifying spatiotemporal concepts and relations, can be used as one type of annotation.Graphical annotations are, especially in maps, timelines and similar spatiotemporal representations, an effective and important way to pinpoint the decisive spatial and temporal relations identified in the data by the analyst.
(4) Offer cued recall When reasoning graphs are created post-analysis analysts need help to remember their rationale and decision points.This kind of help can be achieved through cued recall.

Discussion and conclusions
While the original four guidelines were built around theories from cognitive science and psychology, and as such are quite abstract, this study created a set of complementary guidelines that are instead centered on tool features.The new set of guidelines is thus more concrete, practical, and comprehensible than the original ones.They do not, however, act as www.josis.orgsubstitutes for the original guidelines but work as a good complement to them.Together they provide both a theoretical and practical approach to the problem of providing the provenance of insight and rationale.
The idea of providing these kinds of guidelines represents a new approach in provenance research.Up to now, research on provenance has mostly been practically oriented and focused around a certain analysis tool or problem.These two sets of guidelines combine theoretically oriented findings from cognitive science with technical solutions from VA and look at provenance on a general level.These guidelines can work as a starting point for any project aiming at providing the provenance of insight or rationale in any domain addressing the visual analysis of spatiotemporal data.
Although these guidelines have been developed for a spatiotemporal application domain they are probably transferable to other domains with no, or only minor, changes.The main contribution of the spatiotemporal focus is the advice to include the original physical representation and the advice to enable graphical annotations.The original four guidelines are more centered around the spatiotemporal application domain and thus complement these new guidelines from that point of view as well.
Hall [31] did not address how the actual recording of provenance should happen, only what aspects of the reasoning need to be communicated.This study addresses this deficit by explicitly discussing what kinds of features are needed in a tool that implements the original four guidelines.By recommending that the graphs be created post-analysis this paper suggests that the recording of analysts' reasoning cannot be achieved during analysis without fragmenting the actual analysis process too much.To make up for the inconvenience and impracticality of creating reasoning graphs post-analysis this paper suggests the use of cued recall, perhaps in the form of interaction logs and screen capture video.
The next research step would be to create a provenance tool that implements these guidelines and perform user testing with it to study how it works in practice.Testing and refining the ease of use of the tool for recording reasoning would be as important as testing if it provides provenance.Here a successful implementation of the post-analysis graph creation and browsing of visualization states is key.User testing is also needed to figure out how to best provide cued recall.Implementing a tagging system based on the STC framework is also a direction for future research.
For a discussion on the human reasoning framework for spatiotemporal analysis and the original four guidelines, see Hall [31].

Figure 1 :
Figure 1: The human reasoning framework for spatiotemporal analysis.The flow of information starts with a stimulus and ends in a decision.Normal processing flow refers here to the traditional view of a one-directional flow of information in human perception.The components depicted in the diagram are explained in the text.The notation is inspired by Patterson et al. [60].

Figure 2 :
Figure 2: An illustration of the hierarchical model of user analytical behavior by Gotz and Zhou [29].

Figure 4 :
Figure 4: Overlaid graphical annotations (here in red) are good for identifying spatial (and temporal) relations.Map from MapAnt (www.mapant.fi),data from National Land Survey of Finland (www.nls.fi),both provided under the Creative Commons license.

Figure 5 :
Figure 5: A typical and fictional example of a reasoning graph inspired by the one found in Jigsaw's Tablet.

Figure 6 :
Figure 6: An illustration of the graph-node structured narrative of HTVAP.It consists of a set of ordered and annotated snapshots of visualization states.This fictional example uses lorem ipsum instead of real text.

Figure 7 :
Figure 7: Simplified illustration that describes how ratings are aggregated into an evidence bar in NTE that describes if a hypothesis is rejected or accepted.This fictional example uses lorem ipsum instead of real text.