Optimizing Scatterplot-Matrices for Decision-Support:

Abstract

The scatterplot matrix is defined to be a standard method for multivariate data visualization; nonetheless, their use for decision-support in a corporate environment is scarce. Amongst others, longstanding criticism lies in the lack of empirical testing to investigate optimal design specifications as well as areas of application from a business related perspective. Thus, on the basis of an innovative approach to assess a visualization’s fitness for efficient and effective decision-making given a user’s situational cognitive load, this study investigates the usability of a scatterplot matrix while performing typical tasks associated with multidimensional datasets (correlation and distribution assessment). A laboratory experiment recording eye-tracking data investigates the design of the matrix and its influence on the decision-maker’s ability to process the presented information. Especially, the information content presented in the diagonal as well as the size of the matrix are tested and linked to the user’s individual processing capabilities. Results show that the design of the scatterplot as well as the size of the matrix influenced the decision-making greatly.

Appendix

1.1 Appendix A: Pre-study 1 – Scatterplot Design

Study Design:

Participants had to first assess the type of correlation (positive, negative of no correlation) and then the correlation’s magnitude on a scale from 0 to 100 for positive correlations and a scale from -100 to 0 for negative correlations. Participants were recruited on Amazon Mechanical Turk (50 per design). For each design question, a different experiment was created keeping the experiment short and simple, as recommended for online experiments. Each experiment included scatterplots showing two strong positive (r > 0,8 and r > 0,6) as well as two strong negative correlations (r < -0,6, r > −0,8), two weak positive (r > 0,2) and two weak negative correlations (r < −0,2), as well as four scatterplots showing no correlation (−0,2 < r > 0,2).

Procedure:

Participants were asked to perform an online experiment using Lime Survey. The experiment started by explaining how to assess correlations by reading scatterplots. To check for the participant’s attention, they had to answer six questions on the presented content. Experimental tasks were presented in randomized order. Assessment was based on TA (task accuracy) which is measured by using the deviation of the participant’s estimate to the true score of the presented correlation. Lastly, a self-assessment on previous experience and question on the demography of the participants were presented.

Design Question 1 and Stimuli Material:

Should a regression-line be used to indicate the direction and the strength of the correlation coefficient? (Fig. 3).

Fig. 3.

Stimuli material design question 1 (left: default; right: trendline)

Design Question 2 and Stimuli Material:

Should a monochrome or a multi-color scheme be used to differentiate identified data clusters? (Fig. 4).

Fig. 4.

Stimuli material design question 2 (left: default, right color scheme mono-color)

Results:

Table 6 demonstrates, that assessing positive correlations is easier than assessing negative ones. Further, it is clearly indicated that a regression-line stressing the type (positive or negative) and the magnitude of the correlation increases its assessment. With respect to the color scheme used, no clear evidence in terms of TA could be found, nonetheless, participants preferred the multi-color scheme.

Table 6. Overview results pre-study 1

The results of this study are applied to design the scatterplots used in the scatterplot matrix in pre-study 2 as well as the main experiment presented in this paper.

1.2 Appendix B: Pre-study 2 – Scatterplot Matrix Design

Study Design:

The same five questions and the same data set were used in the pre-study as in the main experiment. Participants (N = 20), recruited at the University of Applied Sciences Upper Austria, educated in business administration were recruited but for this study a different group was used compared to the main experiment. Stimuli material was similar, but targeted towards answering two particular design question: (1) the design of the diagonal (a scatterplot vs. using a histogram) and (2) the design of the bottom left corner (scatterplots vs. the corresponding correlation coefficient).

Procedure:

Participants were asked to perform an online experiment, which was sent out with a link and designed using Lime Survey. The experiment started by explaining how to assess correlations and how to derive information on the distribution of a variable by reading scatterplots and histograms. To check for the participant’s attention, they had to answer questions on the presented content (six concerning a single scatterplot and how to assess correlations; four concerning the distribution of a variable; and four concerning the scatterplot matrix). After informing participants on the content of the study, experimental tasks were presented in randomized order. Assessment is based on TA (task accuracy) and TT (task time). Lastly, a self-assessment on previous experience and question on the demography of the participants were presented.

As experience is of utmost importance when analyzing information in visualized form and self-assessment has proven to be inefficient in the past [24], the experiment was repeated one week after the first trial to check for improvement through learning and repetition. One student did not participate in the second trial and one student did not pass the validity questions at the beginning of the experiment and thus, these two were excluded from analysis.

Design Question 1 and Stimuli Material:

Should the area beneath the diagonal (bottom left) also include scatterplots or should the respective correlation coefficient be presented instead? In addition to the correlation coefficient, also a heatmap representing the strength of the correlation was included as it has proven to increase efficiency and effectiveness in tabular representations (Fig. 5).

Fig. 5.

Stimuli material design question 1

Results:

Tasks targeted towards assessing one specific correlation as well as counting the amount of positive or negative correlations presented within the matrix (tasks 1–3) were significantly better supported by using the second design (correlation coefficient in the bottom left corner). TA was significantly higher when correlation coefficients were presented. Additionally, the effect size (eta²) is stronger for the tasks targeted towards counting the amount of positive and negative correlations. Supporting previous research, showing correlation coefficients is more important for counting negative correlations, as those are harder to assess (see task 3).

Further, TA was higher in the second trial indicating participants did increase their effectiveness with repetition. As expected, no effects were visible for the tasks targeted towards assessing the variable’s distribution (tasks 4–5). TT shows significance with respect to the repetition of the trial for all posed tasks, but no direct effect can be detected concerning the design of the area in the bottom left corner. However, when analyzing the significant interaction effects, design 2 increased the participant’s efficiency in the second trial indicating better decision-support after getting used to its design (Table 7).

Table 7. Overview on repeated measures ANOVA on matrix design (bottom left)

Design Question 2 and Stimuli Material:

The second design questions was targeted towards identifying the best information to be presented in the diagonal. Identified options form previous literature as well as published options on visualization platforms highlight three possibilities (Fig. 6):

Fig. 6.

Stimuli material design question 2 (left: scatterplot; middle: histogram including normal distribution curve; right: histogram on identified clusters within the dataset)

Results:

For the tasks targeted towards assessing a variable’s distribution, no direct effect on TA could be found. Only when examining interaction effects, differences on the improvement based on repetition can be found: performance using the first histogram design (hole dataset and normal distribution curve) improved, while performance using the second design decreased. Using a correlation coefficient shows the lowest TA. With respect to TT, the histogram representing cluster information significantly increased efficiency for task 4, which asked for an assessment of cluster information (Table 8).

Table 8. Overview on repeated measures ANOVA on design of the diagonal

1.3 Appendix C: Second-Order Formative Construct – Situational CL

For analysis of the second order formative construct we used SmartPLS. We applied the repeated measures approach to analyze the latent variable of situational cognitive load, while we consider balanced indicators for the first order constructs as suggested by Perkhofer and Lehner [13] (Fig. 7).

Fig. 7.

Second-order formative construct to measure situational CL

Situational cognitive load is calculated during information processing, using an adaptive baseline to focus not only on the user and his/her capabilities but also on the presented stimulus. The variable is supposed to indicate a mismatch between the presented visualization and the needs of the decision maker, as it indicates increased cognitive resources by focusing on the duration of events [25]. Saccade and blink duration are suspected to be decelerated as the cognitive burden increases, and the size of pupil diameter is said to increase [13, 14, 26,27,28]. Further, by considering measures on blink and saccade count, also the need to search through bad design (not instantly locating the information) is accounted for [26, 29, 30]. Both, information on count measures (highly correlated with time) and information on the duration of events (correlated with the complexity of the material and task accuracy), explain situational cognitive load and thus increase understanding of good or bad design depending on the decision-maker and his/her capabilities.