Integrating Click-Through and Eye-Tracking Logs for Decision-Making Process Mining

In current software every click of the users is logged, therefore a wealth of click-through information exists. Besides, recent technologies have made eye-tracking affordable and an alternative to other human-computer interaction means (e.g. mouse, touchscreens). A big challenge is to make sense of all this data and convert it into useful information. This paper introduces a possible solution placed in the context of decision-making processes. We show how the decision maker’s activity can be traced using two means: mouse tracing (i.e. clicks) and eye-tracking (i.e. eye fixations). Then, we discuss a mining approach, based on the log, which extracts a Decision Data Model (DDM). We use the DDM to determine, post-hoc, which decision strategy was employed. The paper concludes with a validation based on a controlled experiment.


Introduction
This paper aims to show how decisionmaking process models can be automatically extracted from human-computer interaction logs.Since our research is placed in IT, most of the interaction with a software is performed my mouse and by looking at a computer screen.Therefore, the traces left by the user interacting with the software are mouse clicks (an additional data like values typed from the keyboard) as well as screen objects (e.g.menus, textboxes) at which the user stared.Aggregating those two data sources as well as making sense of this new data is the main focus of this paper.Modern software stores more and more data about everything connected to it.For example, Google sores information about searches of each user, about pages visited, about links opened, etc.Also, any web-shop stores data on products viewed as well as any click performed during each visit (e.g.Amazon sends personalized e-mails with price updates if a user just looked at the details of certain products, let alone added anything in the shopping cart and then discarded it).ERP systems also log the activities of users, and even go as far as storing information on changes made to the tables storing data.Therefore, we think that there is enough support to claim that clickthrough data may be, or is, stored by any software.Eye-tracking is a technique used to output the point of a stimulus where a subject looks at.For our research, the stimulus is the interface of software displayed on a computer screen.Eye-tracking hardware follows the physical movements of the eye (most common technique is to film the pupils with video cameras).Eye-tracking software converts the physical movements into (computer screen) coordinates and matches them to object in the interface.Therefore, a log of objects (e.g.buttons, menu items, textboxes, etc.) that capture user's attention is available.Eye tracking becomes cheaper and more and more implementations are available.The most common example is the use of eyetracking in current smartphones.For example, Samsung Galaxy S4 uses the front camera and dedicated software to determine if the user is looking at the bottom of the page so it will automatically scroll documents or if the user is looking away from the screen so it will pause running videos.Also, eye-tracking systems are integrated into laptops on the market DOI: 10.12948/issn14531305/18.1.2014.05(Lenovo laptop presented at CeBit 2011) or soon to be put on the market (Tobii ultrabook presented at CeBit 2013).Those implementations seek to replace 'classical' interaction methods like mouse, touchpad or touchscreens with eye-gaze interaction (e.g.user clicks a button just by looking at it).Having established there is a wealth of logged data on the behavior of users interacting with software, we wish to argue that it can be mined for building models.This is the drive behind process mining research area.Activity logs are mined in order to extract the control-flow perspective of business processes.We got inspiration to apply the same approach to the decision making research.Basically, we see decision making as a process composed of distinct activities.The challenge is that those activities are mostly mental.Therefore, there is a need to extract or elicit them.Our approach assumes that modern decision making is supported by various systems that are used by the decision makers to get data, but also to create new information.By eyetracking and by logging clicks and other interaction data we get a footprint of what the decision making activities were.The difficult part is to convert those logs into explicit models.The novel thing in this paper is the integration of click-through logs with eyetracking outputs.On this basis, we apply the Decision Data Models (DDM) framework.This framework is a complete approach to modeling, mining and enacting data-centric business decision making processes.In our framework, the user is a decision maker who interacts with a decision support system in order to make a decision.Meanwhile, we log his mouse clicks and his eye movements.Then, we integrate the two logs and feed them into our decision-process mining tool.This outputs a DDM that explicitly and graphically depicts the data processing performed by the decision maker.This is an executable model that mirrors the mental activities performed by the decision maker in his quest to choose one of the available alternatives.
This paper is organized as follows.In the next section, Then, we introduce the DDM framework and stress on the two data sources that are the goal of this paper.The fourth section formalizes the researched problem and its solution.The fifth section seeks to support our claims by a controlled experiment.The paper is closed with a discussion and our concluding remarks.

Related Work
This section discusses similar approaches to the problem tacked and, while doing so, we point out open research questions that might be solved by employing our results.We first give a brief review of decision making research, which is the context of our research.Then, we review some research in two fields that approached similar problems: process mining and decision-making process tracing techniques.The common property of decision strategies is that there is a decomposition of the decision into sub-tasks.The decision process means that sub-tasks are solved first and their results will be aggregated in order to make the final decision.However, a person's decomposition ability varies greatly with decision complexity as well as knowledge and experience about the given decision [1].Our decision making-process model emphasizes exactly on this aspect.Since better knowledge leads to better decisions through decomposition, we aim to graphically depict the decomposition steps.Also, expertise allows the decision maker to better define his strategy and better or quicker identify the relevant factors for the decision [Browne et al., 2007].In this case, we argue that the decision making-process model needs to focus first on showing those factors as discrete elements.There are a lot of strategies that can be employed by a decision maker during his decision-making process.Sometimes just one strategy is employed, but it could also be that strategies are swapped during the process [2], [3].A short walk-through the most important decision-making strategies and a concise comparison is available in [4].DOI: 10.12948/issn14531305/18.1.2014.05 A lot of research effort focused on the issue of determining which decision strategy is/was used by some decision maker.The basic idea is that, given some process-tracing data, it is possible to identify which decision strategy was actually employed by the decision maker.In [5] there is an algorithm that relies on four metrics in order to identify a decision trace as one of the decision strategies.The process data used as input is extracted using choice matrixes (one of the techniques in the information display boards category -see the next section).The algorithm is implemented as the Decision Tracer tool.Metrics used for identifying decision strategies: a) Search index is the ratio of within alternative transitions divided to the between alternative transitions was introduced by Payne in the 70's [6].b) Ratio of time spent reviewing different attributes [5] c) Statistic approach [7].This is demonstrated to work well with eyetracking data.d) Process visualization [8] There is an area of decision making research, called dual-process models, dedicated to identifying whether decisions are intuitive or deliberate.An intuitive decision making process is described as unconscious, automatic, effortless, and fast [9].A deliberate decision making process is driven by conscious awareness therefore is limited by the decision maker's mental abilities and experience, sequential, effortful, rulegoverned, and slow [9].There is a classification of decision making based on the influence of each mode on the decision process [10], [11]: pre-emptive, parallelcompetitive, default-interventionist and integrative theories.In [12] it was investigated if instructioninduced decisions will result in processes in which intuition and deliberation are distinct or integrate.It gave evidence that intuitive and deliberate decisions are not clearly distinct.That is, if the decision maker is given instructions to deliberate he will not use a completely different rule-based decision making process.This impacts on our research because it shows that the mined instance decision-process model can be different from the 'ideal' process because the decision maker used a larger part of intuition in his process mix.In [13] it was argued that even a more complex decision-making strategy as WADD does not necessary require conscious calculation steps (i.e.calculation of weighted sums).A decision maker may approximate by an automatic/intuitive process, in a short amount of time, the weighted additive attribute value of the alternatives.The issue whether intuitive and deliberate decisions lead to 'good' or 'bad' decisions has not been settled [14].However, the debate on decision quality is important for us because we also face the problem whether following the decision model will eventually lead to a better decision.The process tracing techniques used by a study need to be closely connected with the tested hypothesis.There are a large number of decision process-tracing techniques that have been used and perfected in the last decades [15], [5].This section is a quick review of the available techniques, with a focus on the ones that best fit our aim of creating reference decision-process models.Process tracing techniques were classified in methods for investigating [15]: i) Information acquisition ii) Information integration and evaluation iii) The cognitive process (psychological, neurological and other Given that we are probing the cognitive process of the decision maker, just the techniques that can be applied for information acquisition are reviewed.The first three methods are of interest to us because they are most common for tracing external information acquisitions.Internal acquisitions (i.e.retrieval of information from the memory of the decision maker) cannot be traced by those techniques and need to be inferred and/or interpreted by the researcher.DOI: 10.12948/issn14531305/18.1.2014.05 An extensive literature review of process tracing studies, covering refereed journals in the period 1970 to 2008, arrived to some interesting conclusions [15].First, it shows that there was a steady increase of interest on process tracing, proven by the growing number of published papers on the topic (by a factor of 5).Secondly, it shows that reaction time studies took the lion's share and that bio-neural studies rocketed in the last decade.This happened, while only 10% of the studies in the researched timeframe focused on information acquisition and another 10% focused on information integration.Therefore, we think there is enough room for our research which is about decision information acquisition and integration.As with any method or technique, there are drawbacks of the process-tracing methods [15].On one side, process-tracing data can be unreliable due to the nature of the collection process or tools.On the other side, there are critical choices that need to be made related both to recording as well as data analysis.Third, collecting unambiguous data usually requires an artificial and static environment and stimuli.Relevant results useful for our research:  Looking patterns were used to differentiate between intuitive and deliberative thinking.It was proved that deliberation leads to a higher number of fixations and to a more complete information acquisition as a result of repeated information acquisitions [12]. MouseLab is a technique that provides researchers with data about: content and sequence of information acquired by the decision makers.Our proof-of-concept tool mimics MouseLab in our search for the same decision-making process data. MouseTracker complements our process-tracing tools  Eye-tracking based study showed that intuitive and deliberate decision making share the same process, with some additional operations in deliberate mode [12] It was found that deliberation seems to be associated with a thorough and extensive information search expressed as more fixated elements and more repeated information inspections.Therefore, we think we could cluster the decision makers based on the number of operations performed during the decision-making process. In Mouselab-based studies, it was concluded that the way information is displayed may have a huge impact on the information acquisition and integration and ultimately on the chosen alternative.For example, people follow the 'natural' reading order and will review first the data elements in the top-left corner of the screen.To mitigate this, we need to counterbalance the position/order of data elements. Mapping data acquisition to the decision making process is suggested to have two properties: occurrence (information not examined cannot be used by a hypothesized strategy) and adjacency (information used in temporal proximity by a hypothesized strategy should be acquired in close proximity).A risk is that adjacency will not hold if data can be memorized and, therefore, not needed to be re-acquired.To mitigate this risk, we use in our experiments values that are more difficult to remember (e.g.instead of 1000 we use 1273);  The decision making process was divided into reading, and choice phases [8].In Mouselab experiments, the reading phase was delimited from the choice phase by the trace element when all outcomes/data elements have been examined once. Coalescing is the assumption that branches of a decision model can be combined by adding their probabilities.Cancellation is the reverse operation, in which ranches common to both alternatives can be removed.Those operations can be defined on the DDM.

Preliminaries
This section introduced the user to the methods that are applied as a part of our DOI: 10.12948/issn14531305/18.1.2014.05solution.We give a very brief walk-through our DDM framework that explicitly depicts and employs models of decision-making processes.Then, we look at tools available for decision process tracing (namely information boards, Mouse Lab, eye-tracking and our proof-of-concept implementation).

The Decision Data Model Framework
We look at business decision making processes as at a data-centric environment.
Most of the decision making in this environment evolves on using and interpreting data and building information based on that.But, creating information from data is not a trivial task while it is essential for the outcome of a business decision.Therefore, the entire framework aims to extract and make explicit the activities related to data and information manipulation.
To gain insights into how a decision maker manipulated data, the obvious approach is to perform an interview.This leads to knowledge representation under different formalisms (rules, decision trees, influence diagrams, etc.).If, for various reasons, one expert is not enough, more interviews need to be carried out and the output models need to be updated to show the aggregate view.The problem with this approach is that is quite difficult and expensive to perform interviews with many individuals and troublesome to update models with extra behavior, so that a broader and more reliable model is created.
Our approach takes a different path.We argue that a model, comparable with the one extracted from several experts, can be created if we mine the behavior of a large number of decision makers.We created a framework (see Fig. Once the decision logs are available, we need some algorithm to mine the model.In [16] we showed that current process mining algorithms do not perform well on such decision logs.Therefore, in [17] we proposed a model (DDM -Decision Data Model), derived from the Product Data Model [16], that is better suited for such a field.In [16] we showed how multiple individual DDMs can be aggregated.A DDM can be easily DOI: 10.12948/issn14531305/18.1.2014.05read by humans, but it could also be used as input in a recommendation engine.In [18] we introduced two approaches to recommendations: one based on a Greedy approach and one derived from the A* pathfinding algorithm.In [19] we showed the third algorithm that uses an aggregated DDM as input.It is mapped to a Markov Decision Process and, unlike the other two algorithms that recommended the local next best action, provides recommendations for the optimal decision strategy that may be used.

Decision Process-tracing
In this sub-section we introduce the techniques and tools that inspired us in the decision-making data collection.The research on process tracing methods looks at decision making as at a process that can be observed using different means.The goal of this research is to open new windows into the intimate internal processing that drives an individual towards making a decision.The techniques reviewed in this sub-section are 'pure'.In our approach we took advantage of all of them and used them concurrently.The measured decision data promoted by prominent decision process-tracing techniques, which we implemented up to a certain degree, are: a) Active information search This method aims to remove the need for prestructured information.Therefore, subjects get a basic description of the decision that needs to be made and need to acquire further information by various means (e.g. by asking questions, or by searching for data on-line).Metrics associated with this method are type, sequence, and acquisition time of information [15].

b) Information display boards This method
investigates what information is acquired by the decision maker.Research data relates to the content and sequence of the information explicitly acquired by the subject.The method evolved from envelopes with text (pioneered by Payne in 1976) to computerized information retrieval systems (which fall in the next category).The critiques of this method argue that there is a need of prestructured information which needs to be formalized as written data.c) Computer-based process tracing 1) Mouselab is a well-known technique [4] that builds on the information display boards principle.Besides information content and sequence it allows the researcher to measure the time needed by the subject to acquire some piece of information.2) MouseTrace is a technique that relies on logging mouse movements.The software package MouseTracker (http://mousetracker.com/) allows experimental designs to be created, data to be collected and then analyzed or exported.The key feature of mouse tracking is the possibility to directly observe the timeline of key cognitive processes.d) Eye-tracking.Eye movements are obviously directly linked to information acquisition.Also, at some level, they are closely connected to the cognitive processes of the decision makers [20].
There are several metrics that can be calculated based on eye-tracking, the most important are linked to saccades and fixations [21].Connected to saccades, decision-making researchers found of interest: latency, duration, and amplitude [15].About fixations are of interest the duration, frequency and scanning path [15].In the remainder of the section, we discuss the different metrics that can be recorded using some process-tracing tools.Some of these metrics are of capital importance to us.

Click-through Data
Click-through is basically a record of the sequence of mouse clicks.The click-through data was found to be correlated to decision making in online searches and information retrieval by humans [22], [23], [24].The metrics that was used in these studies are: were used by the user.For our problem, this is a subset of the set of interface objects.

Eye-tracking Data
Eye-tracking equipment helps to create a record of the elements the subject's eyes fixated upon.Other interesting data extracted using this method is the fixation sequence and fixation times of the different elements.Fixation means that a person's eyes are aimed at some object, therefore he investigates it.Fixation sequence is the order of the items a person looks at.Fixation time is the period of time over which the subject's eyes are directed at the object.In our eyetracking experiment, subjects look at process models.Therefore, a fixation occurs when the subject looks at the model node for a period of time over a certain threshold that will allow his brain to capture the meaning of the visual stimulus [25], [26], [27].
The eye-tracking software calculates fixation time as the length of time the eye velocity was below both the saccade velocity criterion and the drift distance criterion.Saccades are fast rotations of the eyes that occur several times each second and are commanded automatically by the brain (without getting awareness) [27].Saccades show up when the subject's attention shifts from one point on the screen to another.

The Formal Approach
This section introduces a formalization of our problem and the proposed output of an integrated click-through and eye fixations log.
Problem statement: Given a usage session of a system, output an event log that traces the user interaction with the software.Definition 1 (Usage Session): Let S be any software system.A usage session US is a set of all the activities performed by the user within the software from the moment he logs-in (or launches the program if it is a desktop-based software) until he logs-out (or closes the software).Definition 2 (Usage Trace): A usage trace, UT = (T, A), is a tuple where:  T is a set of timestamps recorded by the software such that t 0 is the recorded time of the first activity and t f is the timestamp of the last activity and any t i ∈ [t0, tf];  A is a set of activities performed by a user of software towards an individual, identifiable goal.During a usage session the user might aim for different goals, sequentially or concurrently.Therefore, A ⊆ US.Definition 3 (Mouse Click-through Trace): A mouse click-through trace, MT = (TC, OC), is a tuple where:  TC is a set of timestamps recorded by the software such that any t ∈ T and t c ∈ [t0, tf];  OC is a set of interface objects that were clicked.In other words, the SP is the order in which the decision maker fixates the elements of the decision-aware system interface.Definition 4 (Scan Path): Using eye-tracking recorded fixation sequences, we define Scan Path, SP = (TF, OF), as a tuple where:  T is a set of timestamps recorded by the eye-tracker such that any t ∈ T and t f ∈ [t0, tf];  OF is a set of interface objects that a subject has fixated (looked at).In other words, the SP is the order in which the decision maker fixates the elements of the decision-aware system interface.Definition 5 (Integrated log): An integrated log, IL = (T, O) is a tuple where:  T is a set of timestamps such that T = TC + TF;  O is a set of interface objects such that O = OC + OF.

Validation
In order to validate our claims, we conducted a controlled experiment with 12 master student at our faculty.The hypothesis that we DOI: 10.12948/issn14531305/18.1.2014.05aim to support is "an individual decision making strategy can be identified using an integrated log".The prediction based on this hypothesis is that a tool that mines DDM from integrated logs will be able to correctly classify (when it comes to the most common choice strategies) over 50% of the instances.
The experiment was performed in six steps as follows: a) Hardware set-up.For experimenting we used a eye-tracking system produced by Mirametrix (http://mirametrix.com/products/eye-tracker/).b) Calibration.This is an essential step that influences data accuracy.Calibration means mapping eye vectors (left and right) to a position on the screen.For the experiments, we calibrated a number of 9 points.c) Calibration confirmation.This step give assurance over the calibration quality; d) Show decision-aware software interface.
Recording eye movements starts when the log-in window is displayed on screen and ends when the user clicks the Log-out button.
According to the formal approach introduced in the previous section, this is an Usage Session that overlaps with a Usage Trace because the user solves only one and nothing else but one decision problem; e) Record clicks.The decision-aware system logs all activities of the user.The main events in the log are clicks on objects and calculations performed with the dedicated tool (see for details).Also, the eye-tracking system records automatically a mouse click layer.This layer is superimposed on the screen video capture and indicates the objects that were clicked by the decision maker; f) Record eye coordinates.The eye-tracking system records the coordinates of the eye gaze at 60Hz.There are two settings that allow the calculation of fixations: drift distance and fixation time.Drift is due to the fact that there is always an intra-ocular blood pressure that makes each person's eyes to twitch and therefore move slightly.This doesn't mean that the user is willingly moving his eyes.We used a threshold of 40 pixels, meaning that a saccade is recorded (and implicitly the fixation is broken) if two consecutive coordinates are farther apart than 40 pixels.Fixation time is the minimum interval for which gaze coordinates should indicate the user looks at the same point on the screen in order to classify it as a fixation.We used 4 microseconds as a threshold.
The participants in the experiment were required to choose between 3 banks to open an account with.There were 6 criterions for this decision: the credit facilities offered by each bank, the perceived customer friendliness, the deposit interest rate, the distance from home, the familiarity with the bank, and the number of ATMs.Each participant (decision maker) was asked to perform the following activities:  Select one choice strategy from: lexicographic, disjunctive, elimination by aspects, additive differences, majority, and equal weights;  Use the decision-aware software to investigate the available data while employing the selected choice strategy;  Input the selected alternative (one of the three banks to open an account with) and log-out.After performing the experiment, some datapreprocessing was needed.The main activity was to define Regions of Interest (ROI) for the eye-tracking analysis software.A ROI is an area of the screen that belongs together (e.g. the area of a button from the interface).A ROI was defined for every interface object.That allowed us to work with aggregated data rather than raw screen coordinates.Screen fixations were filtered so that only the ones on ROIs were kept for further processing.Another activity was to filter the click logs.Only clicks on textboxes, menu items and the '=' button that performs the calculation were kept.By default the values for all textboxes are hidden.Therefore clicks on textboxes are required to reveal the value of a certain criterion.The value is shown for 5 seconds then is hidden again, requiring the user to click the textbox again if he doesn't remember the value that was shown.This  The eye-tracking log shows the fixations of the decision maker while interacting with the interface of the decision-aware system in order to gather information on the decision at hand.A partial eye-tracking log, of the same user as in Fig. 3, is shown in Fig. 4. Please note that setting ROIs is similar to delimiting areas such the ones depicted in Fig. 2.

Fig. 4. Partial eye-tracking log
One can note that at various timestamps the eye-tracking system logs the coordinates of the two eyes (A = left eye and B = right eye) in columns ALX and ALY.Also, the software aggregates the times spent in one ROI once the gaze leaves the delimited area.Hence, the recording of event with codes 711 in the log (e.g.first record of an event code 711 that shows up in the log said that the user spent 0.15 sec looking at the interface object in ROI31).The partial integrated log obtained by merging the click-through (Fig. 3) and eyetracking (Fig. 4) partial logs is introduced in Fig. 5.

Fig. 2 . 3 .
Fig. 2. Interface of the decision-aware system for one user