1932
0
  1. Home
  2. A-Z Publications
  3. Annual Review of Statistics and Its Application
  4. Volume 7, 2020
  5. Article

Annual Review of Statistics and Its Application

Volume 7, 2020

Testing Statistical Charts: What Makes a Good Graph?

Abstract

It has been approximately 100 years since the very first formal experimental evaluations of statistical charts were conducted. In that time, technological changes have impacted both our charts and our testing methods, resulting in a dizzying array of charts, many different taxonomies to classify graphics, and several different philosophical approaches to testing the efficacy of charts and graphs experimentally. Once rare, charts and graphical displays are now everywhere—but do they help us understand? In this article we review the history of graphical testing across disciplines, discuss different direct approaches to testing graphics, and contrast direct tests with visual inference, which requires that the viewer determine both the question and the answer. Examining the past 100 years of graphical testing, we summarize best practices for creating effective graphics and discuss what the future holds for graphics and empirical testing of interactive statistical visualizations.

Keyword(s): chart design, graphics, perception, user testing, visual inference, visualization
Loading

Article metrics loading...

/content/journals/10.1146/annurev-statistics-031219-041252
2020-03-07
2024-04-16
Loading full text...

Full text loading...

/deliver/fulltext/statistics/7/1/annurev-statistics-031219-041252.html?itemId=/content/journals/10.1146/annurev-statistics-031219-041252&mimeType=html&fmt=ahah

Literature Cited

  1. Amer T. 2005. Bias due to visual illusion in the graphical presentation of accounting information. J. Inf. Syst. 19:1–18
     [Google Scholar]
  2. Baird JC, Lewis C, Romer D. 1970. Relative frequencies of numerical responses in ratio estimation. Percept. Psychophys. 8:358–62
     [Google Scholar]
  3. Becker G, Moore SE, Lawrence M. 2019. trackr: a framework for enhancing discoverability and reproducibility of data visualizations and other artifacts in R. J. Comput. Graph. Stat. 28:644–58
     [Google Scholar]
  4. Becker RA, Cleveland WS. 1987. Brushing scatterplots. Technometrics 29:127–42
     [Google Scholar]
  5. Becker RA, Cleveland WS, Wilks AR. 1987. Dynamic graphics for data analysis. Stat. Sci. 2:355–83
     [Google Scholar]
  6. Beecham R, Dykes J, Meulemans W, Slingsby A, Turkay C, Wood J. 2017. Map LineUps: effects of spatial structure on graphical inference. IEEE Trans. Vis. Comput. Graph. 23:391–400
     [Google Scholar]
  7. Beniger JR, Robyn DL. 1978. Quantitative graphics in statistics: a brief history. Am. Stat. 32:1–11
     [Google Scholar]
  8. Bernstein EM, Cowden DJ. 1937. Graphic presentation of trend data. South. Econ. J. 3:443–51
     [Google Scholar]
  9. Bertin J, Berg WJ 1983. Semiology of Graphics: Diagrams, Networks, MapsVol 1 Madison, WI: Univ. Wis. Press
  10. Borland D, Ii RMT. 2007. Rainbow color map (still) considered harmful. IEEE Comput. Graph. Appl. 27:14–17
     [Google Scholar]
  11. Br. Mus 1882. The map of the world. British Museum Collection Online https://www.britishmuseum.org/research/collection_online/collection_object_details.aspx?assetId=404485001&objectId=362000&partId=1
     [Google Scholar]
  12. Brandes D. 1976. The present state of perceptual research in cartography. Cartogr. J. 13:172–76
     [Google Scholar]
  13. Brewer CA 2019. ColorBrewer2.0: color advice for cartography. Software diagnostic tool for maps http://colorbrewer2.org
     [Google Scholar]
  14. Brinton WC 1917. Graphic Methods for Presenting Facts New York: Eng. Mag. Co.
  15. Broersma H, Molenaar IW. 1985. Graphical perception of distributional aspects of data. Comput. Stat. Q. 2:53–72
     [Google Scholar]
  16. Buja A, Asimov D, Hurley C, McDonald JA. 1988. Statistical inference for exploratory data analysis and model diagnostics. In Dynamic Graphics for Statistics WS Cleveland, ME McGill277–308 Belmont, CA: Wadsworth
     [Google Scholar]
  17. Buja A, Cook D, Hofmann H, Lawrence M, Lee EK, et al. 2009. Statistical inference for exploratory data analysis and model diagnostics. Philos. Trans. R. Soc. Lond. A 367:4361–83
     [Google Scholar]
  18. Callaghan TC. 1984. Dimensional interaction of hue and brightness in preattentive field segregation. Percept. Psychophys. 36:25–34
     [Google Scholar]
  19. Carr DB. 1994. Using gray in plots. Stat. Comput. Graph. Newsl. 5:19–23
     [Google Scholar]
  20. Carswell CM, Wickens CD. 1987. Information integration and the object display: an interaction of task demands and display superiority. Ergonomics 30:511–27
     [Google Scholar]
  21. Chandar N, Collier D, Miranti P. 2012. Graph standardization and management accounting at AT&T during the 1920s. Account. Hist. 17:35–62
     [Google Scholar]
  22. Chang W, Cheng J, Allaire J, Xie Y, McPherson J 2019. shiny: web application framework for R. R package, version 1.3.2. https://shiny.rstudio.com
     [Google Scholar]
  23. Chang W, Wickham H 2016. ggvis: interactive grammar of graphics. R package, version 0.4.3. https://ggvis.rstudio.com/
     [Google Scholar]
  24. Cleveland WS, McGill R. 1984. Graphical perception: theory, experimentation, and application to the development of graphical methods. J. Am. Stat. Assoc. 79:531–54
     [Google Scholar]
  25. Cleveland WS, McGill R. 1985. Graphical perception and graphical methods for analyzing scientific data. Science 229:828–33
     [Google Scholar]
  26. Cleveland WS, McGill R. 1987. Graphical perception: the visual decoding of quantitative information on graphical displays of data. J. R. Stat. Soc. A 150:192–229
     [Google Scholar]
  27. Cooke L. 2010. Assessing concurrent think-aloud protocol as a usability test method: a technical communication approach. IEEE Trans. Prof. Commun. 53:202–15
     [Google Scholar]
  28. Croxton FE. 1932. Graphic comparisons by bars, squares, circles, and cubes. J. Am. Stat. Assoc. 27:54–60
     [Google Scholar]
  29. Croxton FE, Stryker RE. 1927. Bar charts versus circle diagrams. J. Am. Stat. Assoc. 22:473–82
     [Google Scholar]
  30. Desnoyers L. 2011. Toward a taxonomy of visuals in science communication. Tech. Commun. 58:119–34
     [Google Scholar]
  31. Dunbar K. 1995. How scientists really reason: scientific reasoning in real-world laboratories. Nat. Insight 18:365–95
     [Google Scholar]
  32. Dunn R. 1988. Framed rectangle charts or statistical maps with shading: an experiment in graphical perception. Am. Stat. 42:123
     [Google Scholar]
  33. Eells WC. 1926. The relative merits of circles and bars for representing component parts. J. Am. Stat. Assoc. 21:119–32
     [Google Scholar]
  34. Eick SG, Wills GJ. 1995. High interaction graphics. Eur. J. Oper. Res. 81:445–59
     [Google Scholar]
  35. Fabrikant SI, Hespanha SR, Hegarty M. 2010. Cognitively inspired and perceptually salient graphic displays for efficient spatial inference making. Ann. Assoc. Am. Geogr. 100:13–29
     [Google Scholar]
  36. Fienberg SE. 1979. Graphical methods in statistics. Am. Stat. 33:165–78
     [Google Scholar]
  37. Flury B, Riedwyl H 1988. Multivariate Statistics: A Practical Approach London: Chapman & Hall
  38. Funkhouser HG. 1937. Historical development of the graphical representation of statistical data. Osiris 3:269–404
     [Google Scholar]
  39. Gelman A, Unwin A. 2013. Infovis and statistical graphics: different goals, different looks. J. Comput. Graph. Stat. 22:2–28
     [Google Scholar]
  40. Goldberg JH, Helfman JI. 2010. Comparing information graphics: a critical look at eye tracking. In Proceedings of the Third BELIV’10 Workshop on Beyond Time and Errors: Novel Evaluation Methods for Information Visualization71–78 New York: ACM
     [Google Scholar]
  41. Green TM, Fisher B. 2011. The personal equation of complex individual cognition during visual interface interaction. In Human Aspects of Visualization A Ebert, A Dix, ND Gershon, M Pohl38–57 Berlin: Springer
     [Google Scholar]
  42. Guan Z, Lee S, Cuddihy E, Ramey J. 2006. The validity of the stimulated retrospective think-aloud method as measured by eye tracking. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems—CHI ’06 R Grinter, T Rodden, P Aoki, E Cutrell, R Jeffries, G Olson1253–62 New York: ACM
     [Google Scholar]
  43. Harms H. 1991. August Friedrich Wilhelm Crome (1753-1833). Cartogr. Helv. 33:33–38
     [Google Scholar]
  44. Hasanhodzic J, Lo AW, Viola E 2010. Is it real, or is it randomized?: A financial Turing test. arXiv:1002.4592 [q-fin.GN]
    [Google Scholar]
  45. Healey CG, Booth KS, Enns JT. 1996. High-speed visual estimation using preattentive processing. ACM Trans. Comput. Hum. Interact. 3:107–35
     [Google Scholar]
  46. Healey CG, Enns JT. 1999. Large datasets at a glance: combining textures and colors in scientific visualization. IEEE Trans. Vis. Comput. Graph. 5:145–67
     [Google Scholar]
  47. Heer J, Bostock M. 2010. Crowdsourcing graphical perception: using Mechanical Turk to assess visualization design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems203–12 New York: ACM
     [Google Scholar]
  48. Hofmann H, Follett L, Majumder M, Cook D. 2012. Graphical tests for power comparison of competing designs. IEEE Trans. Vis. Comput. Graph. 18:2441–48
     [Google Scholar]
  49. Hughes BM. 2001. Just noticeable differences in 2D and 3D bar charts: a psychophysical analysis of chart readability. Percept. Motor Skills 92:495–503
     [Google Scholar]
  50. Javed W, McDonnel B, Elmqvist N. 2010. Graphical perception of multiple time series. IEEE Trans. Visualization Comput. Graph. 16:927–34
     [Google Scholar]
  51. Karsten K 1923.Charts and Graphs: An Introduction to Graphic Methods in the Control and Analysis of Statistics. Upper Saddle River, NJ: Prentice-Hall
  52. Kibirige H 2017. plotnine: a grammar of graphics for Python. Graphical software https://plotnine.readthedocs.io/en/stable/
     [Google Scholar]
  53. Kim S, Lombardino LJ, Cowles W, Altmann LJ. 2014. Investigating graph comprehension in students with dyslexia: an eye tracking study. Res. Dev. Disabil. 35:1609–22
     [Google Scholar]
  54. Kirschenbaum SS. 2003. Comparative cognitive task analysis: the cognition of weather forecasting. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 47:473–77
     [Google Scholar]
  55. Kostelnick C. 2016. The re-emergence of emotional appeals in interactive data visualization. Tech. Commun. 63:116–35
     [Google Scholar]
  56. Kruskal W. 1977. Visions of maps and graphs. In Proceedings of the International Symposium on Computer-Assisted Cartography (Auto-Carto II) J Kavaliunas27–36 Washington, DC: US Dep. Commer.
     [Google Scholar]
  57. Lawrence M, Wickham H, Cook D, Hofmann H, Swayne DF. 2009. Extending the GGobi pipeline from R: rapid prototyping of interactive visualizations. Comput. Stat. 24:195–205
     [Google Scholar]
  58. Legge GE, Gu Y, Luebker A. 1989. Efficiency of graphical perception. Percept. Psychophys. 46:365–74
     [Google Scholar]
  59. Lewandowsky S, Spence I. 1989. The perception of statistical graphs. Sociol. Methods Res. 18:200–42
     [Google Scholar]
  60. Light A, Bartlein PJ. 2004. The end of the rainbow? Color schemes for improved data graphics. Eos Trans. Am. Geophys. Union 85:385–91
     [Google Scholar]
  61. Loy A, Follett L, Hofmann H 2016. Variations of Q-Q plots: the power of our eyes!. Am. Stat. 70:202–14
     [Google Scholar]
  62. Loy A, Hofmann H. 2015. Are you normal? The problem of confounded residual structures in hierarchical linear models. J. Comput. Graph. Stat. 24:1191–209
     [Google Scholar]
  63. Lumley T 2013. dichromat: color schemes for dichromats. R package, version 2.0-0. https://cran.r-project.org/web/packages/dichromat/index.html
     [Google Scholar]
  64. MacDonald-Ross M. 1977. How numbers are shown: a review of research on the presentation of quantitative data in texts. AV Commun. Rev. 25:359–409
     [Google Scholar]
  65. Majumder M, Hofmann H, Cook D. 2013. Validation of visual statistical inference, applied to linear models. J. Am. Stat. Assoc. 108:942–56
     [Google Scholar]
  66. McDonald L. 2014. Florence Nightingale, statistics and the Crimean War. J. R. Stat. Soc. A 177:569–86
     [Google Scholar]
  67. Morel P. 2018. Gramm: grammar of graphics plotting in Matlab. J. Open Source Softw. 3:568
     [Google Scholar]
  68. Netzel R, Vuong J, Engelke U, O'Donoghue S, Weiskopf D, Heinrich J. 2017. Comparative eye-tracking evaluation of scatterplots and parallel coordinates. Vis. Inform. 1:118–31
     [Google Scholar]
  69. Normand MP, Bailey JS. 2006. The effects of celeration lines on visual data analysis. Behav. Modif. 30:295–314
     [Google Scholar]
  70. North C. 2006. Toward measuring visualization insight. IEEE Comput. Graph. Appl. 26:6–9
     [Google Scholar]
  71. Okan Y, Galesic M, Garcia-Retamero R. 2016. How people with low and high graph literacy process health graphs: evidence from eye-tracking. J. Behav. Decis. Making 29:271–94
     [Google Scholar]
  72. Peterson LV, Schramm W 1954. How accurately are different kinds of graphs read?. Audio Vis. Commun. Rev. 2:178–89
     [Google Scholar]
  73. Peuchet J, Gilbert C 1805. Statistique élémentaire de la France Paris: Chez Gilbert Cie.
  74. Playfair W 1801. The Statistical Breviary: Shewing the Resources of Every State and Kingdom in Europe London: J. Wallis
  75. Roy Chowdhury N, Cook D, Hofmann H, Majumder M, Lee EK, Toth AL. 2015. Using visual statistical inference to better understand random class separations in high dimension, low sample size data. Comput. Stat. 30:293–316
     [Google Scholar]
  76. Ryu YS, Yost B, Convertino G, Chen J, North C. 2003. Exploring cognitive strategies for integrating multiple-view visualizations. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 47:591–95
     [Google Scholar]
  77. Satyanarayan A, Moritz D, Wongsuphasawat K, Heer J. 2017. Vega-Lite: a grammar of interactive graphics. IEEE Trans. Vis. Comput. Graph. 23:341–50
     [Google Scholar]
  78. Shah P, Carpenter PA. 1995. Conceptual limitations in comprehending line graphs. J. Exp. Psychol. Gen. 124:43–61
     [Google Scholar]
  79. Shah P, Miyake A 2005. The Cambridge Handbook of Visuospatial Thinking Cambridge, UK: Cambridge Univ. Press
  80. Sievert C 2018. plotly: create interactive web graphics via ‘plotly.js’. R package, version 4.9.0. https://cran.r-project.org/web/packages/plotly/index.html
     [Google Scholar]
  81. Simons DJ, Levin DT. 1997. Change blindness. Trends Cogn. Sci. 1:261–67
     [Google Scholar]
  82. Smith CD. 1996. Imago Mundi's logo the Babylonian map of the world. Imago Mundi 48:209–11
     [Google Scholar]
  83. Spence I. 1990. Visual psychophysics of simple graphical elements. J. Exp. Psychol. Hum. Percept. Perform. 16:683–92
     [Google Scholar]
  84. Swayne D, Klinke S 1999. Introduction to the special issue on interactive graphical data analysis: What is interaction?. Comput. Stat. 14:1–6
     [Google Scholar]
  85. Tan JK. 1994. Human processing of two-dimensional graphics: information-volume concepts and effects in graph-task fit anchoring frameworks. Int. J. Hum. Comput. Interact. 6:414–56
     [Google Scholar]
  86. Teghtsoonian M. 1965. The judgment of size. Am. J. Psychol. 78:392–402
     [Google Scholar]
  87. Tomasetti N 2015.Comparing the power of plot designs to reveal correlation. Honors Thesis, Fac. Bus. Econ., Monash Univ., Melbourne, Aust. https://github.com/dicook/lineplots_v_scatterplot
  88. Trafton GJ, Kirschenbaum SS, Tsui TL, Miyamoto RT, Ballas JA, Raymond PD. 2000. Turning pictures into numbers: extracting and generating information from complex visualizations. Int. J. Hum. Comput. Stud. 53:827–50
     [Google Scholar]
  89. Treisman AM. 1980. A feature-integration theory of attention. Cogn. Psychol. 12:97–136
     [Google Scholar]
  90. Trickett SB, Fu WT, Schunn CD, Trafton JG 2000a. From dipsy-doodles to streaming motions: changes in representation in the analysis of visual scientific data. . In Proceedings of the Annual Meeting of the Cognitive Science Society, Vol. 22:, ed. LR Gleitman, AK Joshi, pp. 959–64 Ann Arbor, MI: Cogn. Sci. Soc.
     [Google Scholar]
  91. Trickett SB, Trafton JG, Schunn CD 2000b. Blobs, dipsy-doodles and other funky things: Frame-work anomalies in exploratory data analysis. . In Proceedings of the Annual Meeting of the Cognitive Science Society, Vol. 22:, ed. LR Gleitman, AK Joshi, pp. 965–70 Ann Arbor, MI: Cogn. Sci. Soc.
     [Google Scholar]
  92. Tufte E 1991. The Visual Display of Quantitative Information Cheshire, CT: Graphics Press. 2nd ed.
  93. Tukey JW. 1965. The technical tools of statistics. Am. Stat. 19:23–28
     [Google Scholar]
  94. Tukey JW. 1972. Some graphic and semigraphic displays. In Statistical Papers in Honor of George W. Snedecor T Bancroft293–316 Ames, IA: Iowa State Univ. Press
     [Google Scholar]
  95. Ulery BT, Hicklin RA, Buscaglia J, Roberts MA. 2011. Accuracy and reliability of forensic latent fingerprint decisions. PNAS 108:7733–38
     [Google Scholar]
  96. Unwin A. 1999. Requirements for interactive graphics software for exploratory data analysis. Comput. Stat. 14:7–22
     [Google Scholar]
  97. Unwin A 2019. Why is data visualization important? What is important in data visualization?. Harv. Data Sci. Rev. In press
    [Google Scholar]
  98. Vanderplas S, Goluch R, Hofmann H. 2019. Framed! Reproducing and revisiting 150-year-old charts. J. Comput. Graph. Stat. 28:620–34
     [Google Scholar]
  99. Vanderplas S, Hofmann H. 2015. Signs of the sine illusion–why we need to care. J. Comput. Graph. Stat. 24:1170–90
     [Google Scholar]
  100. Vanderplas S, Hofmann H. 2017. Clusters beat trend!? Testing feature hierarchy in statistical graphics. J. Comput. Graph. Stat. 26:231–42
     [Google Scholar]
  101. von Huhn R. 1927. Further studies in the graphic use of circles and bars. I. A discussion of the Eells’ experiment. J. Am. Stat. Assoc. 22:31–39
     [Google Scholar]
  102. Wagemans J, Elder JH, Kubovy M, Palmer SE, Peterson MA, et al. 2012a. A century of gestalt psychology in visual perception: I. Perceptual grouping and figure-ground organization. Psychol. Bull. 138:1172–217
     [Google Scholar]
  103. Wagemans J, Feldman J, Gepshtein S, Kimchi R, Pomerantz JR, et al. 2012b. A century of gestalt psychology in visual perception: II. Conceptual and theoretical foundations. Psychol. Bull. 138:1218–52
     [Google Scholar]
  104. Wakita K, Shimamura K. 2005. SmartColor: disambiguation framework for the colorblind. In Proceedings of the 7th International ACM SIGACCESS Conference on Computers and Accessibility158–65 New York: ACM
     [Google Scholar]
  105. Walker FA 1874. Statistical Atlas of the United States, Based on the Results of the Ninth Census, 1870, with Contributions from Many Eminent Men of Science, and Several Departments of the Government New York: Bien
  106. Wickham H. 2010. A layered grammar of graphics. J. Comput. Graph. Stat. 19:3–28
     [Google Scholar]
  107. Wickham H. 2013. Graphical criticism: some historical notes. J. Comput. Graph. Stat. 22:38–44
     [Google Scholar]
  108. Wickham H 2016. ggplot2: Elegant Graphics for Data Analysis New York: Springer
  109. Wickham H, Chowdhury NR, Cook D, Hofmann H 2018. nullabor: tools for graphical inference. R package, version 0.3.5. https://cran.r-project.org/web/packages/nullabor/index.html
     [Google Scholar]
  110. Wickham H, Cook D, Hofmann H, Buja A. 2010. Graphical inference for infovis. IEEE Trans. Vis. Comput. Graph. 16:973–79
     [Google Scholar]
  111. Wickham H, Grolemund G 2017. R for Data Science: Import, Tidy, Transform, Visualize, and Model Data Sebastopol, CA: O'Reilly. 1st ed.
  112. Wickham H, Lawrence M, Cook D, Buja A, Hofmann H, Swayne DF. 2009. The plumbing of interactive graphics. Comput. Stat. 24:207–15
     [Google Scholar]
  113. Widen HM, Elsner JB, Pau S, Uejio CK. 2016. Graphical inference in geographical research. Geogr. Anal. 48:115–31
     [Google Scholar]
  114. Wilkinson L 1999. The Grammar of Graphics New York: Springer
  115. Woller-Carter MM, Okan Y, Cokely ET, Garcia-Retamero R. 2012. Communicating and distorting risks with graphs: an eye-tracking study. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 56:1723–27
     [Google Scholar]
  116. Xie Y, Hofmann H, Cheng X. 2014. Reactive programming for interactive graphics. Stat. Sci. 29:201–13
     [Google Scholar]
  117. Yang W, Jeppson H, Lyttle IJ 2019. ggvega: translator from ‘ggplot2’ to ‘Vega-Lite’. R package , version 0.0.0.9001 https://github.com/vegawidget/ggvega
     [Google Scholar]
  118. Yates J. 1985. Graphs as a managerial tool: a case study of Du Pont's use of graphs in the early twentieth century. J. Bus. Commun. 22:5–33
     [Google Scholar]
  119. Zeileis A, Hornik K, Murrell P. 2009. Escaping RGBland: selecting colors for statistical graphics. Comput. Stat. Data Anal. 53:3259–70
     [Google Scholar]
  120. Zgraggen E, Zhao Z, Zeleznik R, Kraska T 2018. Investigating the effect of the multiple comparisons problem in visual analysis. Proceedings of the 2018 CHI Conference on Human Factors in Computing New York: ACM
     [Google Scholar]
/content/journals/10.1146/annurev-statistics-031219-041252
Loading
Testing Statistical Charts: What Makes a Good Graph?
Annual Review of Statistics and Its Application 7, 61 (2020); https://doi.org/10.1146/annurev-statistics-031219-041252
/content/journals/10.1146/annurev-statistics-031219-041252
/content/journals/10.1146/annurev-statistics-031219-041252
Loading

Data & Media loading...

  • Article Type: Review Article

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error