Explainable Activity Recognition in Videos using Deep Learning and Tractable Probabilistic Models

2.1 Traditional Space-Time Approaches

Traditional activity recognition has been typically concerned with isolating segments of video and mapping them to particular actions. These traditional space-time approaches treat each video as a 3D volume along the X, Y (spatial) and T (temporal) axes. This 3D volume is then either (a) matched against smaller template volumes for each action [9, 42, 95, 106]; or (b) used to track the trajectories of the agent inside the volume in order to detect specific actions [13, 92, 107, 122]; or (c) used to extract spatio-temporal features or points of interest (changes in directions, collisions, etc.) and use these features to assign actions [8, 15, 19, 50, 51, 77, 105, 118, 124]. A detailed description of these methods can be found in the survey article by Aggarwal and Ryoo [2]. Unlike previous work, our model encodes additional context about each activity by also modeling the object and location associated with each action.

2.2 State-Based Sequential Approaches

These approaches use Hidden Markov Models (HMMs) and Dynamic Bayesian Networks (DBNs) to model the problem as a sequence of states and then use these sequences to determine the action in question. There exists a large body of work that uses these approaches [10, 12, 29, 35, 63, 73, 78, 81, 110, 121]; however, similar to the traditional space-time approaches, most of these approaches are concerned with single actions only and although they can be used to generate the explanations we use in our system, HMMs suffer from a lack of expressivity while DBNs are impractical either because exact inference over them is intractable or approximate inference algorithms such as particle filtering are inaccurate in practice. The Dynamic Conditional Cutset Network (DCCN) model we use in this article circumvents both of these issues and also allows for multiple action/object/location labels per time slice by framing the problem as a temporal multi-label classification (TMLC) problem.

2.3 Deep Learning Approaches

More recently, there have been a number of works that use Convolutional Neural Networks (CNNs) [53], which have already found extraordinary success in large-scale image classification and object detection, e.g., [48, 111]. Most of these works use CNNs as spatial feature extractors and then train a recurrent model such as an HMM or an RNN/LSTM on top of these spatial feature vectors to model the sequential nature of activities [20, 26, 27, 31, 37, 52, 59, 61, 67, 75, 79, 108, 119, 120, 123]. Unfortunately, as is typical of deep-learning approaches, these systems are not explainable. The approach used in this article aims to rectify this problem by training our tractable, dynamic, and probabilistic framework to create a transparent and robust decision-making process that can be used to answer any probabilistic query accurately and efficiently as well as to generate explanations to justify its decisions.

There has been another line of work from the variational autoencoder (VAE) community that composes graphical models with neural networks and performs joint learning and inference [38, 40, 47]. The key difference between the formulation used by these models and our formulation is the fact that the graphical models in the former are defined over latent variables while in our model, they are defined over the explanatory variables themselves. This makes our model better suited towards computing the answers to probabilistic queries posed over the explanatory variables that can be presented to the end-user as explanations.

2.4 Event Recognition Models That Use Black-box Models as Sensors

In addition to training HMMs and DBNs on top of low-level feature extractors, there have also been a number of grammar-based approaches [36, 39, 104, 125] that use stochastic context-free grammars to model the hierarchical nature of activities as well as the temporal relations between them. The work by Pei et al. [83], for instance, uses a black-box random forest model to predict noisy activities which, in turn, are fed into an AND/OR graph that represents the hierarchical semantic structure of each video. While such approaches are globally interpretable (since they provide a good, hierarchical overview of the event structure in each video), they are unable to efficiently compute conditional probabilities of actions over time e.g., the probability of a slicing activity taking place over an orange in the seventh interval given that the orange was taken out of the fridge in the second interval.

The second class of approaches [11, 68] involve generating candidate interval hypotheses using black-box models and then using a probabilistic relational model to refine these probabilities. Although probabilistic relational models work very well in situations where prior knowledge about the domain is known a priori, learning these relations from data is typically NP-hard [24]. In fact, inference is also intractable in these models unless some very restrictive assumptions are made. For example, the work by Morariu et al [68] restricts the treewidth of the underlying Markov Logic Network (MLN) in order to guarantee tractability of exact inference. Similarly, the work by Brendel et al. [11] defines their own probabilistic event logic; however, computation of the maximum a-posteriori (MAP) is required for inference which is intractable in their model and is approximated using a local search-based algorithm. Our model addresses all these problems since it treats each action, object, and location as individual labels and compactly encodes a joint distribution over them. Solving this temporal multi-label classification (TMLC) problem using our dynamic and tractable framework allows for both (a) refining the results of the black-box outputs and (b) generating efficient and accurate posterior probabilities as explanations using particle filtering.

2.5 Semantic Representation of Videos

This effort is also closely related to the work of Rohrbach et al. [96] and Donahue et al. [20] on generating a semantic representation from videos at an activity level using deep learning architectures. Instead of generating sentences in natural language, we assign a number of pre-defined labels divided into categories. Related efforts have considered the task of dense captioning [46], i.e., generating summaries of texts from particular segments. Song et al. [109] attempted to create captioning methods that require minimum supervision on the TaCOS dataset. Duan et al. [22] attempted to combine caption generation and sentence localization to feed off of each other to create a weakly supervised training model. These works focus on creating text summaries for video segments, and as is typical of deep learning approaches, they are essentially black-boxes. Our approach, on the other hand, aims to create a semantic representation for activities in each frame that can be used to both answer queries easily as well as generate explanations (via probabilistic inference) that justify these answers.

2.6 Explainable AI and Machine Learning

Most modern machine learning models use deep neural networks or ensembles as key components to achieve state-of-the-art results. However, due to the opaque nature of these black-box models, decision makers have been reluctant to adopt them in mission-critical applications where the cost of failure is high and where it is imperative to have at least some understanding of the underlying logic used by these models to arrive at their decisions (cf. Veale et al. [116]). There are many types of explanation techniques that are currently in use. The survey by Adadi and Berrada [1] lists some of these including visualization techniques such as surrogate models and partial dependency plots (PDPs); knowledge-extraction techniques such as rule extraction and model distillation; influence methods such as feature importance; as well as example-based techniques such as counterfactuals. While these techniques are both powerful and versatile, they are not specifically tailored to the task of activity recognition. There has been some work in this area regarding the usage of association rules [3] and computing feature relevance scores using interpretable models [7] as explanations; however, these systems primarily use sensor data as input (as opposed to video data that is used in this article). In addition, there have also been some successful attempts in the visual question-answering (VQA) community to generate meaningful visual explanations [57, 80]; however these use still images instead of videos. Very recently, Chen et al. [14] proposed a method that extended the VQA task to videos. However, the aim of most of these VQA methods is to generate natural-language answers to questions posed in natural-language which is significantly different from our aim of building a context-rich semantic representation that can be easily queried. Further, these methods do not incorporate uncertainty very well (for example, they cannot tell the user how confident the model is about certain components of the answer).

2.7 XAR in Videos

Technically, we can use image attribution approaches (see, for example, Grad-CAM [53] and LIME [94]) to perform XAR because each video is a sequence of images. However, these approaches do not model or reason about spatiotemporal relationships and often yield coarse-grained, unsatisfactory explanations [56]. To circumvent this issue, recently, researchers have begun investigating video attribution approaches [5, 56] which output important regions in each relevant frame of a video as explanations. The issue with the above approaches is that they use a separate explainable model or an external process (e.g., loss functions or gradient computations) to explain the original model’s decision. These external methods are unable to guarantee that the provided explanation will be close to the optimal explanation. In contrast, the method described in this article derives optimal or close to optimal explanations by reasoning over a tractable, interpretable probabilistic model defined over the explanatory and output variables.

Our work is also closely related to the work of Zhuo et al. [126] which uses logical reasoning over scene graphs to derive explanations. Our work is different from Zhou et al.’s approach in that we use tractable probabilistic models and reasoning.

2.8 AI and Trust

There have been a number of studies on how trust influences interactions between humans and automated systems, e.g., [32, 54, 69, 70]. These studies examine factors that might affect the trust of the user in the system, such as the past performance of the system and how understandable the system is to the user [54]. Hoffman [33] provides a more detailed taxonomy of such factors and explains how trust is context-specific and dynamic. In other words, trust might vary with respect to specific contexts of automation and must also be maintained over time. Our aim is to compare the demonstrated trust [66] between two systems (models), one that uses explanations and another that does not by conducting human subjects studies. More specifically, we first divide the participants into two groups and allow participants in the first and second groups to observe and interact with a model that uses explanations and a model that does not (use explanations), respectively. Then, we have participants in each group use the system they interacted with to answer yes/no queries about activities performed in the video. Finally, we assume that the participant has deemed a system more trustworthy than the other if they are able to answer the query in less time, more accurately, and agree with the system’s answer in more cases than the other system. An interesting future work is to measure participants’ self-reported perception of their trust (via questionnaires and surveys) using the trust scale for explainable AI proposed by Hoffmann et al. [34] and compare the results with the three metrics (accuracy, speed, and agreement) used in this article.

3.1 Desiderata and Assumptions for the Activity Recognition System

We assume that we are given a collection of third-person videos where each video contains exactly one actor performing simple activities such as “open door”, “take bottle out of fridge”, and so on but not complicated ones such as “cut a banana into ten 1-inch pieces with a knife on a cutting board”. Under these assumptions, we define an activity (performed by an actor) as a triple (action, object, location) where object and location can be “None” but action cannot be “None”. We assume that the domains of (namely, the values that can be assigned to) action, object, and location are provided to the system as input.

The action component forms the core part of the activity. These are usually verbs such as wash, cut, slice, open, and so on. The object component denotes the entities over which the activity is performed. These are generally nouns such as apples, refrigerator, cutting board, knife, and so on. Finally, the location component tells us where the activity is taking place. These are generally location nouns such as kitchen, bathroom, counter top, sink, and so on but can also overlap with the nouns we use as objects. For example, when a person “kicks open a door,” the activity is “kick” and the object is “door,” but the same entity might play a different semantic role in a different activity such as if a person “paints a picture on the door.” Here “paint” is the activity, “picture” is the object, and “door” is the location. For reflexive actions, such as “walking,” the object and location are “None.”

Users interact with our system by posing so-called selection questions: “Did a particular (simple) activity defined by the triple (action, object, location) happen in the video?” Examples of selection questions include: (1) “Did the person slice an orange on the counter?” where slice, orange, and counter denote the action, object, and location respectively; (2) “Did the person take out grapes from the refrigerator?” where take out, grapes, and refrigerator denote the action, object, and location, respectively; and (3) “Did the person open the refrigerator?” where open and refrigerator denote action and object, respectively, and location is None.

3.2 Desiderata for Explanations Provided by the Activity Recognition System

In addition to answering queries posed by the user, we also want the system to provide explanations to justify its answers. We seek three different types of explanations:

4.1 Video Classification Layer

The input to this layer is a set of frames (i.e., a video), and the output is a set of predicted activity labels, one for each frame. Each frame is passed through a deep neural network and the latter generates the output activity labels for the frame.

In our study, we used GoogleNet [111], a 22-layer CNN (see, for example, Dumoulin and Visin [23]) that is pre-trained on the ImageNet dataset [103]. Note that any deep neural network can be used instead of GoogleNet; the accuracy of our system will only (likely) improve with more advanced deep learning algorithms and architectures.

We modified the GoogleNet architecture slightly to accommodate our problem. The original GoogleNet, which is trained on the ImageNet dataset, solves a multi-class classification task in that it takes an image as input and classifies it into one of the roughly 1,000 object categories in ImageNet. In this work, we are interested in solving a multi-label classification task where each image can be associated with multiple labels. Specifically, the TACoS and wet lab datasets used in our experiments have 28 and 57 labels, respectively. To this end, we replaced the softmax classification layer of GoogLeNet which yields a probability distribution over roughly 1,000 object categories with a fully connected sigmoid layer having m nodes, one for each class label (where m equals 28 and 57 for the TACoS and wet lab datasets, respectively). Figure 2 depicts the difference between the original GoogleNet and our modification.

Training: As mentioned earlier, we used a feature extractor that was pre-trained on the ImageNet dataset [103]. Then, we retrained our modified architecture (see Figure 2(b)) for a fixed number of iterations using the ADAM gradient optimization technique [44] on the TACoS and wet lab datasets, respectively.

4.2 Explanation Layer

In this section, we present DCCNs, our new tractable, temporal probabilistic representation. We use DCCNs in the explanation layer to (a) correct errors in the labels predicted by GoogLeNet at each frame; (b) model the dynamics as well as persistence (activities do not change rapidly between frames) in the video; and (c) provide explanations via abductive poly-time probabilistic inference.

4.2.1 Cutset Networks.

Probabilistic Graphical Models (PGMs) such as Bayesian and Markov networks (see, for example, Koller and Friedman [45]) are widely used in practice to represent and reason about uncertainty. At a high level, they are a compact representation of the joint probability distribution over a large number of random variables. Once learned from data, they can be used to answer any query posed over the joint distribution via probabilistic inference. The two main types of inference (queries) tasks are posterior marginal (MAR) and MAP inference. In MAR inference, we are interested in computing the marginal probability distribution over each query variable given evidence where evidence (or observation) is an assignment of values to a subset of random variables. In MAP inference, we are interested in computing the most likely assignment to all query variables given evidence. Both tasks are notoriously difficult to solve in many practical networks, and theoretically, they are NP-hard in general [98]. As a result, in practice, one has to often use approximate inference algorithms to solve these problems (approximately). Unfortunately, these algorithms are unreliable and often yield inaccurate query answers [89].

Tractable probabilistic models (TPMs) [4, 17, 62] are special types of probabilistic models, which admit poly-time MAR and MAP inference and thus circumvent the problem of unreliability of approximate inference in Bayesian and Markov networks. Although TPMs are less expressive than intractable (latent) probabilistic models such as Bayesian and Markov networks, their prediction accuracy (at test time) is often much higher than intractable models. This is because tractable models use exact inference at prediction time while one has to use inaccurate approximate inference algorithms in Bayesian and Markov networks. Examples of popular TPMs include cutset networks [86, 87, 90], arithmetic circuits [17, 62], sum-product networks [84], and probabilistic sentential decision diagrams [6].

Cutset networks [90] are a class of TPMs that use recursive cutset conditioning [64, 82] to build a rooted OR tree where each non-leaf node corresponds to a conditioned variable and each leaf node corresponds to a tree-structured Bayesian Network defined over all variables not appearing on the path from the root to the leaf. Formally, given a set of variables \(\mathbf {X}=\lbrace X_1,\ldots ,X_n\rbrace\), a cutset network C is a pair \((O,T)\) where O represents an OR tree and T represents a set of tree-structured Bayesian Networks, one for each leaf node in O (see Figure 3(a) for an example). Assuming that all the variables in \(\mathbf {X}\) are binary, each non-leaf node in O will have two branches. We will assume that the left and right branches of an OR node labeled by \(X_i\) in O correspond to the values \(\overline{x}_i\) and \(x_i\) respectively where \(\overline{x}_i\) (similarly \(x_i\)) denotes an assignment of value 0 to \(X_i\) (similarly 1). Each directed edge between an OR node labeled by \(X_i\) and its child node in O is labeled with the conditional probability of the variable \(X_i\) taking the corresponding value given the assignment on the path from the root to \(X_i\). For example, in Figure 3(a), 0.92 equals the conditional probability \(P(\overline{y}_4|y_1)\). Every non-leaf node partitions the probability space into data points that agree with \(\overline{x}_i\) in the left sub-tree and those that agree with \(x_i\) in the right sub-tree. The probability of a full assignment x w.r.t. the cutset network C is given by (1) \(\begin{equation} P_C(\mathbf {x}) = T_{l(\mathbf {x})}\left(\mathbf {x}_{V\left(T_{l(\mathbf {x})}\right)}\right) \cdot \prod _{p \in O(\mathbf {x})}p , \end{equation}\)where \(l(\mathbf {x})\) denotes the leaf node in C corresponding to the assignment \(\mathbf {x}\), \(T_{l(\mathbf {x})}\) denotes the tree Bayesian network at \(l(\mathbf {x})\), \(V(T_{l(\mathbf {x})})\) denotes the set of variables over which \(T_{l(\mathbf {x})}\) is defined, \(x_{V\left(T_{l(\mathbf {x})}\right)}\) denotes the projection of the assignment \(\mathbf {x}\) on the variables \(V(T_{l(\mathbf {x})})\) (where \(V(T_{l(\mathbf {x})})\subseteq \mathbf {X}\)) and \(O(\mathbf {x})\) is the set of conditional probabilities on the path from root to the leaf node \(l(\mathbf {x})\) in the OR tree O.

The time complexity of posterior marginal estimation (MAR) and full MAP estimation is linear in the size of the cutset network and can be solved by making two passes over the network [90]. The fact that most prediction tasks can be reduced to these two types of inference queries makes these models an attractive choice for applications that rely heavily on exact inference at test time.

The structure and parameters of cutset networks can be learned from data using the top-down, recursive induction approach described in Algorithm 1. The algorithm has two main steps: base case and conditioning step. In the base case, the algorithm returns a tree Bayesian network if a pre-defined termination condition (a popular condition is described below) is satisfied. The tree Bayesian network is learned from data using the Chow-Liu algorithm [16]. This algorithm first constructs an undirected weighted complete graph in which each node corresponds to a variable \(X_i\) in X and each edge \((X_i,X_j)\) is weighed using the mutual information score (MIScore) between \(X_i\) and \(X_j\): \(\begin{equation*} \text{MIScore}(X_i,X_j) = \sum _{i=0}^{1} \sum _{j=0}^{1} P_{\mathbf {D}}(X_i=i,X_j=j) \log \frac{P_{\mathbf {D}}(X_i=i,X_j=j)}{P_{\mathbf {D}}(X_i=i)P_{\mathbf {D}}(X_j=j)}, \end{equation*}\)where \(P_{\mathbf {D}}(X_i=i,X_j=j)\) is estimated from the dataset \(\mathbf {D}\); the estimate equals the number of times the partial assignment \((X_i=i,X_j=j)\) appears in the data divided by the number of examples in \(\mathbf {D}\), and \(P_{\mathbf {D}}(X_i=i) = \sum _{j=0}^{1}P_{\mathbf {D}}(X_i=i,X_j=j)\) (similarly, \(P_{\mathbf {D}}(X_j=j)=\sum _{i=0}^{1} P_{\mathbf {D}}(X_i=i,X_j=j)\)). Then, the Chow-Liu algorithm finds a maximum spanning tree from the weighted complete graph and converts the tree to a directed tree K using depth-first search. The latter yields a tree Bayesian network which represents the following distribution: \(\begin{equation*} T(\mathbf {x}) = \prod _{i=1}^{n} P_{\mathbf {D}}(x_{\lbrace X_i\rbrace }|x_{pa_K(X_i)}), \end{equation*}\)where \(pa_K(X_i)\) is the set of parents of \(X_i\) in K. Note that \(pa_K(X_i)\le 1\) for all i.

The following termination condition is often used in practice [86, 87, 90]: stop growing the OR tree if any of the following conditions are satisfied

(1)	The number of examples is smaller than k;
(2)	The depth of the OR tree is larger than d (d is bounded by n).

Hyperparameters d and k are tuned using the validation set; namely, we search over possible choices of d and k and choose the combination that gives the highest log-likelihood score on the validation set.

In the conditioning step, the algorithm heuristically selects a variable \(X_i\) to condition on. The following heuristic is often used in practice [86]. We select a variable having the following sum mutual information score with ties broken randomly: \(\begin{equation*} \text{SumMIScore}(X_i) = \sum _{j: j\ne i} \text{MIScore}(X_i,X_j). \end{equation*}\)Once the variable (\(X_i\)) is selected, the algorithm induces an OR node o labeled by \(X_i\) (line 4). Then, the algorithm partitions the dataset \(\mathbf {D}\) into two datasets, \(\mathbf {D}_{\overline{x}_i}\) and \(\mathbf {D}_{x_i}\) where the former contains only those examples in \(\mathbf {D}\) which \(X_i\) is assigned the value 0 while the latter contains only those examples in \(\mathbf {D}\) in which \(X_i\) is assigned the value 1. It then creates two child nodes l and r and recursively constructs a CN on l and r using \(\mathbf {D}_{\overline{x}_i}\) and \(\mathbf {D}_{x_i}\), respectively. Finally, the algorithm estimates the conditional probability on the edges between l and o and between r and o (lines 11 and 12) and returns the OR node o.

4.2.2 Conditional Cutset Networks (CCNs).

CCNs are a new framework that was recently proposed by Rahman et al. [88]. As the name suggests, they generalize the cutset networks framework to compactly represent conditional distributions of the form \(P(\mathbf {Y}|\mathbf {X})\) where \(\mathbf {X}\) and \(\mathbf {Y}\) are sets of variables. In CCNs, the OR tree and each tree Bayesian network is defined over variables in \(\mathbf {Y}\). The conditional probabilities in the OR tree and tree Bayesian networks are given by a calibrated probabilistic classifier [76]. These classifiers take as input an assignment \(\mathbf {x}\) to a set of variables \(\mathbf {X}\) and output a probability distribution over the class label \(Y_i \in \mathbf {Y}\). Tractability over each individual distribution is still maintained since the number of parameters for most calibrated classifiers scales polynomially with the number of input variables \(\mathbf {X}\). For example, if we use the logistic regression classifier, \(P(\mathbf {Y}_i=1|\mathbf {X}=\mathbf {x})\) equals \(\sigma (w_0+\sum _{X_i \in \mathbf {X}} w_i\mathbf {x}_{\lbrace X_i\rbrace })\) where \(w_i\)’s are the weights (parameters) and \(\sigma\) denotes the sigmoid function.

Given an assignment \(\mathbf {x}\) to all variables in \(\mathbf {X}\), a CCN yields a cutset network because each calibrated classifier yields a marginal probability distribution over the class variable. Thus, given \(\mathbf {x}\), CCNs yield a tractable probabilistic model over \(\mathbf {X}\). Figure 3(b) shows an example of a conditional cutset network.

Structure and parameters of a CCN can be learned (see Algorithm 2) using the same top-down induction approach used for cutset networks. The differences between the two algorithms are

(1)	In LearnCCN, we learn the parameters on the edges of the OR tree and the conditional distributions at each node in each tree Bayesian network using a calibrated classifier \(\sigma (\mathbf {X})\) (logistic regression, neural networks, random forests, etc.). The best classifier is chosen using cross-validation.
(2)	In LearnCCN, we learn the tree Bayesian networks at each leaf node of the OR tree using conditional mutual information scores. Similarly, the splitting heuristic in the LearnCCN algorithm uses sum conditional mutual information scores as compared with sum mutual information scores in the LearnCNet algorithm. See Rahman et al. [88] for details.

4.2.4 DCCNs.

An issue with CCNs is that they are static and do not explicitly model temporal aspects of video. For instance, we can use persistence, namely, objects do not change their position rapidly between subsequent frames to correct prediction errors at a frame by using data from neighboring frames. To address this issue, we propose a novel framework called DCCNs. Formally, let a video consist of n frames, let \(\mathbf {Y}^t=\lbrace Y_1^t,\ldots ,Y_m^t\rbrace\) and \(\mathbf {X}^t=\lbrace X_1^t,\ldots ,X_m^t\rbrace\) be the set of true labels and predicted labels (evidence), respectively, at frame t. Then, the DCCN represents the following probability distribution: (2) \(\begin{align} P(\mathbf {y}^{1:T}|\mathbf {x}^{1:T})= P(\mathbf {y}^1|\mathbf {x}^1)\prod _{t=2}^{T} P(\mathbf {y}^t|\mathbf {x}^{1:t},\mathbf {y}^{1:t-1}), \end{align}\)where the notation \(\mathbf {y}^{1:T}\) (similarly \(\mathbf {x}^{1:T}\)) denotes an assignment of values to all true (predicted) labels in frames 1 to T. We will use the notation \(\mathbf {Y}^{1:T}\) to denote the set \(\bigcup _{t=1}^{T} \mathbf {Y}^t\).

The representation given in Equation (2) is not compact as the number of frames in a video (n) increases. To circumvent this issue, we adopt two standard assumptions widely used in temporal or dynamic probabilistic models—the 1-Markov and stationarity assumptions [85]. Specifically, we assume that each frame is conditionally independent of all frames before it given the previous frame (1-Markov) and all conditional distributions are identical (stationarity). With these assumptions, we can represent \(P(\mathbf {y}^{1:T}|\mathbf {x}^{1:T})\) using (3) \(\begin{align} P(\mathbf {y}^{1:T}|\mathbf {x}^{1:T})= P(\mathbf {y}^1|\mathbf {x}^1)\prod _{t=2}^{T} P(\mathbf {y}^t|\mathbf {x}^{t},\mathbf {y}^{t-1}), \end{align}\)where \(P(\mathbf {y}^1|\mathbf {x}^1)\) and \(P(\mathbf {y}^t|\mathbf {x}^t,\mathbf {y}^{t-1})\) are conditional cutset networks and \(P(\mathbf {y}^t|\mathbf {x}^t,\mathbf {y}^{t-1})\) has the same parameters and structure for \(t=2,\ldots ,T\) (see Figure 3(c)).

We learn DCCNs using the following approach. The prior model \(P(\mathbf {y}^1|\mathbf {x}^1)\) is the same as the CCN described in the previous section. To learn the structure and parameters of \(P(\mathbf {y}^t|\mathbf {x}^t,\mathbf {y}^{t-1})\), we construct the dataset as follows. Each frame in each video is a training example and is composed of (1) true labels at frame t, namely, \(\mathbf {Y}^t\)); (2) true labels at frame \(t-1\), namely, \(\mathbf {Y}^{t-1}\); and (3) labels predicted by GoogLeNet at frame t, namely, \(\mathbf {X}^t\) using the pixels in the frame as input.

Inference over DCCNs is intractable in general and can be approximately solved using the particle filtering algorithm [21, 60], a sequential importance sampling algorithm that generates samples from a proposal distribution and estimates the posterior distribution using a weighted average over the generated samples. The performance of the particle filtering algorithm is highly dependent on the quality of the proposal distribution; higher the quality better the estimate. It is known that the ideal proposal distribution is the posterior distribution \(P(\mathbf {y}^{1:T}|\mathbf {x}^{1:T})\). Unfortunately, it is NP-hard to compute in DCCNs [71]. Therefore, we propose to approximate the ideal proposal using the following: (4) \(\begin{align} P(\mathbf {y}^{1:T}|\mathbf {x}^{1:T}) \approx P(\mathbf {y}^1|\mathbf {x}^1) \prod _{t=2}^{T} P(\mathbf {y}^t|\mathbf {y}^{1:t-1},\mathbf {x}^{1:t}). \end{align}\)Thus (see Equation (4)), we propose to use evidence up to the current time slice to approximate the ideal proposal, namely, we use the filtering distribution \(P(\mathbf {y}^t|\mathbf {y}^{1:t-1},\mathbf {x}^{1:t})\) [71] to approximate the posterior distribution \(P(\mathbf {y}^t|\mathbf {y}^{1:t-1},\mathbf {x}^{1:T})\). It is easy to see that this approximation, because it uses evidence up to the current time slice, is likely to yield higher quality estimates as compared to the likelihood weighting approach which uses the prior distribution (namely a distribution that is not conditioned on evidence) as the proposal. Note that \(P(\mathbf {y}^t|\mathbf {y}^{1:t-1},\mathbf {x}^{1:t})\) can be computed in linear time in DCCNs because they use CCNs, which are conditionally tractable models to represent the transition distribution. Algorithm 3 uses this process to generate K particles that we will later use to generate the explanations.

4.3 User Interface

Our prototype system uses an interactive visual interface that allows users to load videos, ask queries, and review the model output along with explanations. The goal of the interface design was to limit the amount of model information presented to the users in order to avoid overwhelming them with information. For this reason, the system uses simple visual representations in the form of graphical annotations, textual component lists, and simple bar charts. Figure 5 shows the interface.

The interface allows the users to select a video and a relevant query based on that video. These would serve as inputs for the model. The interface would then provide two types of output: (1) the model’s answer to that query and (2) the explanations for why the answer is provided. For this purpose, the interface includes a video player with the selected video that would allow the users to go over the video if they wanted to review and analyze the system’s answer to the query or try to come up with their own answer for that query. The queries used in this system are in the form of yes/no questions, and hence, the answers are either yes or no.

The system also provides information to justify its answer to the selected query. For this purpose, it would first highlight the most relevant segments (a sequence of relevant and consecutive frames) in the video through visual annotations added under the video progress bar. These annotations are in the form of square-shaped blue buttons, as seen under the video in Figure 3. These annotated segments are buttons, and when clicked, the video jumps to the start of that segment, and new information is loaded that explains why this segment is relevant to the query answer. By default, the first identified segment is selected.

Each segment includes detailed explanations as to why it is related to the system answer. We display this information on the right-side of the interface, as seen in Figure 3. On the top right, the summary of detected video components (action, objects, and locations) for the given query is shown, which represents the highest-ranking combinations of components the model detected in this segment. On the bottom right, the interface shows the component scores that summarize the marginal probabilities of single components in the selected segment. To help users to quickly judge component scores, graphical bars are shown underneath detected components to visually represent the values of the component scores. Users can select different video segments to view the corresponding component scores and combinations from different portions of the video.

6.1 Goals and Hypotheses

VAR systems are valuable and have many real-world applications, from fire detection [49] and airport security [113], to elderly care [43] and autonomous vehicles [91]. As alluded to in the introduction, many state-of-the-art models exist that yield high accuracy on the VAR task. However, no matter how accurate the model is, these kinds of models typically suffer from false positives, which may be highly undesirable in mission-critical applications. At the very least, as users of these systems, we would like to predict the circumstances under which the system would be likely to generate erroneous results and if it does, what these results might look like. Thus, human-AI collaboration plays an important role in identifying the weaknesses of such systems. For this purpose, it is crucial that human users maintain a proper understanding of the systems and how they work in order to understand when to rely on them. This is the problem we expect to be addressed by the XAR framework that we defined earlier.

The goal of this study was to measure the degree to which the explanations generated by our system would benefit non-expert end users with little to no understanding of how such systems work. We hypothesized that given a set of videos and several yes/no queries where each query asks whether an activity was performed in a video or not, the time required by the user to answer the query as well as the error rate over all queries would decrease significantly with the introduction of explanations. We also hypothesized that the user’s level of agreement with the system’s answers would vary significantly if explanations were shown. A user’s answer is said to “agree” with the system’s when they are the same.

6.2 Experimental Design

The task used for evaluating user performance involved answering a series of questions (or queries) over a set of videos with the help of the system. The participants were divided into two groups: one with and the other without explanations. Participants from both groups had access to the video player and the system’s answer to each question. Participants in the with explanations category used the interface with all the explanation components available (i.e., the video segments, the detected combination of components, and the component scores), while participants in the without explanations category did not have these components shown (i.e., they were only able to view the system’s answers but no explanations). The experiment was conducted between subjects in order to allow the same set of questions and videos to be used for both groups and also to avoid learning effects about knowledge of the model performance.

The TACoS dataset that was used for training and evaluating our system was used since cooking videos typically do not require any domain-specific knowledge. Each user was presented with 20 queries spread out over four videos (i.e., five queries per video). Each query was a simple yes/no question about the video and had a single, unique answer (e.g., “Does the person cut a carrot?”). Participants of each group were presented with the system’s answer to each of these queries and their task was to determine the true answer by watching the video and using the system’s answers as point of reference.

Since the goal of the study was to evaluate whether the explanations helped the participants perform better than the model alone, it was necessary that the system made enough errors so that the explanations would actually be useful to determine the actual answer and that this improvement could be measured. It was, therefore, imperative that the task include multiple queries where the system provided incorrect answers so that participants would have opportunities to recognize system errors. However, since the actual model had a high accuracy score (The accuracy of our system was \(\gt 92\%\), also refer to Table 1 which shows two other measures related to accuracy, the Jaccard Index, and Hamming Loss), using a set of sample queries representative of the actual model accuracy would not have provided enough opportunities to view system errors since the system would have been too accurate for participants to see enough errors in the limited time of the study. To address this problem, we constrained the system accuracy to a constant 80% for this phase of the evaluation. This was done by controlling the composition of trials so that all participants experienced the system answering 80% of the queries correctly.

To assess task performance, we used the following three metrics: (1) error, (2) time taken for task completion, and (3) agreement. Error was calculated as the percentage of queries where the participant’s answer was incorrect with respect to the (known) true answer. Agreement was measured as the percentage of queries where the participant’s answer matched that of the system.

6.3 Procedure

The interface was a web-based application and the evaluation was conducted as an online study. We ran the experiment through the Amazon Mechanical Turk (AMT) crowdsourcing platform. The study was approved by our organization’s Institutional Review Board (IRB), and participants were compensated at a fixed rate per hour. The experiment consisted of a single session with the completion time of the participants varying from 25 minutes to 55 minutes (Note that the total session time includes the time required to complete the questionnaires and demographic information).

The study opened with a consent form followed by a background questionnaire asking for general information about participant demographics, education, and occupation. The participants were also provided with instructions on how to complete the task as well as a small tutorial prior to the main query review trials.

We slightly modified the interface described in Section 4.3 to (1) control the sequence of viewed videos and queries, and (2) include buttons for participants to provide their own answer for each query. Thus, instead of allowing participants to choose from the set of existing videos and queries, the interface only showed one video and one corresponding query at a time. For each trial, the system’s answers were available, but the participants were asked to answer each query themselves. After providing their answer, the system showed participants the correct answer (which was sometimes different from the system’s answer to simulate 80% accuracy as mentioned earlier) as feedback to help them estimate the system’s simulated accuracy as well as build an understanding of how the system made its decisions which, in turn, would help them decide whether or not to rely on its answers. Participants were not allowed to change their responses after submission.

6.4 Participants

The experiment was completed online by 80 AMT workers. Of these participants, 40 of them were shown explanations while the other 40 were not. With the exception of a single participant who reported himself to be a programmer, all the others had occupations that were not related to data science, machine learning, and statistics; hence, the participants were non-experts with regard to machine learning knowledge. After pre-processing the data and removing outliers that did not fall within 1.5 \(\times\) IQR, we analyzed results from 38 participants for the with explanations category and 40 for the without explanations category.

6.5 Results

Because the collected data is not normally distributed—verified using the Shapiro–Wilk test and graphically via a histogram—we analyzed the results using the Kruskal–Wallis non-parametric test to measure the difference between the two groups. The plots are shown in Figure 6. We observed a significant difference in error per trial \(\big (\chi ^2 (1, 76) = 5.63, p \lt 0.05 \big)\), showing that the participants with explanations had significantly less error than those without explanations. Our experiment also detected a significant difference in average time per trial \((\chi ^2 (1, 76) = 28.1, p \lt 0.001)\). Participants with explanations were significantly faster. Together, these results support our hypothesis that the addition of our explanations significantly improves user task-performance in our system.

Fig. 6.

View Figure

We also observed that participants with explanations agreed with the system significantly more than their counterparts \((\chi ^2 (1, 76) = 8.00, p \lt 0.01)\). This observation is aside from the fact that both groups showed high agreement levels with the system given its high accuracy (80%), as the mean and standard deviation are shown in Figure 6(d).

We can further analyze user agreement by using the contingency table (confusion matrix) presented in Table 2. In cases where the system is correct, we observe that the users in the with explanations group agreed with the system significantly (\((\chi ^2 (1, 76) = 12.01, p \lt 0.001 )\)) more than those in the no explanations group. This implies that providing more information helped participants understand the system and judge when it was correct; namely, users appropriately placed their trust in the with explanations system, in contrast to the no explanations system and the explanations encouraged participants to correctly rely on its output. Conversely, when the system was incorrect, on average, users in the with explanations category tend to agree with the system more than users in the no explanations category. However, there was no significant difference (\((\chi ^2 (1, 76) = 0.61, p \gt 0.43 )\)) in the user agreement when the system was incorrect, suggesting that no significant unwarranted (or inappropriate) trust is developed. Based on this analysis, it can be inferred that users in the with explanations category frequently develop appropriate trust with the system compared to users in the no explanations category.

Finally, since the with explanations category also had significantly better performance results, this suggests that the higher rate of agreement was not simply blind trust or automation bias [28], where humans tend to trust an intelligent system by virtue of its “intelligence” alone. Rather, the results of this study suggest that agreement was appropriately aligned with the queries where the system provided the correct answer. However, it is to be noted that our study was not designed to specifically focus on the potential effects of explanations on automation bias, and, therefore, this still remains an open area for further research.

7.1 Future Work

Although our new DCCN framework generates high-quality samples from the posterior distribution, both MAR and MAP inference over it are intractable (or NP-hard) in general. To this end, one line of future work is to investigate temporal models on which both MAP and MAR inference tasks can be solved in polynomial time. We are currently investigating specific structural constraints for achieving this objective building on our prior work [100] on this topic. A second avenue for future work is to combine DCCNs with more sophisticated deep learning architectures such as 3D CNNs [27] and transformers [115] and evaluate whether the combined approach improves (or at least does not degrade) performance or not.

In this article, we only considered simple selection queries with yes/no answers. An interesting direction to expand upon would be to use more complex kinds of queries such as counting queries (e.g., how many carrots are there in the video?) and queries where a given task can be accomplished in multiple ways (e.g., using a spoon instead of a spatula to stir soup) which would require the use of common-sense reasoning. In order to achieve this objective, we will have to develop a novel representation of activities and build an ontology to represent hierarchies of activities. This would allow for other interesting queries involving super-activities and sub-activities. A simple query of this type might be something like “Does the person in the video cook a potato?” which might consist of the sub-activities (cut, potato, \(\ast\)), (move, potato, pot), (move, pot, stove), (turn on, stove, \(\ast\)) and (turn off, stove, \(\ast\)). Since these are cooking videos, we might even ask the system if the person in the video follows the recipe correctly. Once our system is able to answer these kinds of complex queries, we could then think about more varieties of explanations that might be tailored to specific kinds of queries. Finally, we would re-design our human studies to accommodate these new queries and explanations and evaluate their usefulness to different categories of end users.