Students’ misconceptions of statistical inference: A review of the empirical evidence from research on statistics education

https://doi.org/10.1016/j.edurev.2007.04.001Get rights and content

Abstract

A solid understanding of inferential statistics is of major importance for designing and interpreting empirical results in any scientific discipline. However, students are prone to many misconceptions regarding this topic. This article structurally summarizes and describes these misconceptions by presenting a systematic review of publications that provide empirical evidence of them. This group of publications was found to be dispersed over a wide range of specialized journals and proceedings, and the methodology used in the empirical studies was very diverse. Three research needs rise from this review: (1) further empirical studies that identify the sources and possible solutions for misconceptions in order to complement the abundant theoretical and statistical discussion about them; (2) new insights into effective research designs and methodologies to perform this type of research; and (3) structured and systematic summaries of findings like the one presented here, concerning misconceptions in other areas of statistics, that might be of interest both for educational researchers and teachers of statistics.

Introduction

Since the early 1970s, there has been an increasing interest in research about people's understanding and performance in probability and statistics. Researchers especially focused on misconceptions and heuristics regarding probability, chance, and randomness (e.g., Kahneman, Slovic, & Tversky, 1982; Konold, 1989, Konold, 1991; Nisbett & Ross, 1980; Shaughnessy, 2003). This article reports on a systematic review of the available empirical evidence of students’ misconceptions in statistical inference. It constitutes a starting point for educational researchers interested in the relation between misconceptions and the conceptual change theory (e.g., Finch & Cumming, 1998; Guzzetti, Snyder, Glass, & Gamas, 1993; Smith, diSessa, & Roschelle, 1993), or a helpful tool for teachers of statistical inference to become aware of the most common misconceptions that their students may hold (e.g., Batanero, Godino, Vallecillos, & Holmes, 1994; Brewer, 1985; Haller & Krauss, 2002).

In educational research, the term misconception is used to refer to several concepts. On the one hand, authors often consider a broad definition of the word, using it to label different concepts such as preconception, misunderstanding, misuse, or misinterpretation interchangeably (Smith et al., 1993). On the other hand, misconceptions are sometimes defined in a more restrictive way, as misunderstandings generated during instruction, emphasizing a distinction with alternative conceptions resulting from ordinary life and experience (Guzzetti et al., 1993). In this manuscript, a refinement of the first definition is applied, and the term refers to any sort of fallacies, misunderstandings, misuses, or misinterpretations of concepts, provided that they result in a documented systematic pattern of error (Cohen, Smith, Chechile, Burns, & Tsai, 1996).

The interest on statistical inference arises from three realities. First, this is a topic of main relevance for the development of research in all empirical sciences in general and psychology and education in particular (Belia, Fidler, Williams, & Cumming, 2005; Krauss & Wassner, 2002). Second, inference receives special attention in statistical courses from almost all scientific areas, where hypotheses tests and confidence intervals are taught to students as the methods for evaluating scientific hypotheses (Aberson, Berger, Healy, & Romero, 2003; APA, 2001). Finally, inferential ideas seem to be especially sensitive to be misunderstood and students are often prone to fall into deep misconceptions (Daniel, 1998, Kirk, 2001) because they require students to understand and connect many abstract concepts such as sampling distribution and significance level.

After presenting our methodology of search for this review (Section 2), we provide an overview of the misconceptions mentioned and exemplified in the literature, and describe to what extent and under which conditions they occur, discussing the methodology of the presented group of studies (Section 3). Finally, we conclude with some suggestions for further research (Section 4).

Section snippets

Method

We performed a thorough literature exploration in order to bring together publications that report on studies providing empirical evidence of university students’ misconceptions that have been published during the last 15 years (from 1990 to the beginning of 2006). Therefore, studies based on personal experience and anecdotes only or publications oriented to other groups (such as professionals or younger students) were excluded.

Other publications that did not match our inclusion criteria, for

Results

As our findings show, the literature on statistics education, and particularly publications providing empirical evidence of misconceptions in statistics, is sparse (see also Batanero, 2005; Ware & Chastain, 1991). The four searches defined above yielded more than 500 references that contained only 21 publications (cf. Table 1) reporting on 17 different studies (cf. Table A1) that provide evidence of misconceptions about topics related to statistical inference and that satisfied our selection

Conclusion

The group of publications analyzed in this review provides empirical evidence of deep and spread students’ misconceptions in statistical inference. This evidence demonstrates that, although they may be able to manipulate and carry out calculations with statistical data, students have severe misconceptions concerning the interpretation of results from inferential techniques. According to this literature, the following summary and classification of students’ misconceptions in statistical

References (75)

  • C. Batanero

    Statistics education as a field for research and practice

  • C. Batanero et al.

    Errors and difficulties in understanding elementary statistical concepts

    International Journal of Mathematics Education in Science and Technology

    (1994)
  • C. Batanero et al.

    Significado y comprensión de la distribución normal en un curso introductorio de análisis de datos [Meaning and understanding of the normal distribution in an introductory data analysis course]

    Quadrante

    (2001)
  • C. Batanero et al.

    Students’ reasoning about the normal distribution

  • S. Belia et al.

    Researchers misunderstand confidence intervals and standard error bars

    Psychological Methods

    (2005)
  • M. Borovcnik et al.

    Probability

  • Bower, K. M. (2003). Some misconceptions about the Normal distribution. Paper presented at the Six Sigma Forum....
  • J.K. Brewer

    Behavioral statistics textbooks: Source of myths and misconceptions?

    Journal of Educational Statistics

    (1985)
  • B. Chance et al.

    Reasoning about sampling distributions

  • S.L. Chow

    Statistical significance

    (1996)
  • J. Cohen

    Things I have learned (so far)

    American Psychologist

    (1990)
  • S. Cohen et al.

    Identifying impediments to learning probability and statistics from an assessment of instructional software

    Journal of Educational and Behavioral Statistics

    (1996)
  • G. Cumming

    Understanding replication: Confidence intervals, p-values, and what's likely to happen next time

  • G. Cumming et al.

    Confidence intervals and replication: Where will the next mean fall?

    Psychological Methods

    (2006)
  • G. Cumming et al.

    Replication, and researchers’ understanding of confidence intervals and standard error bars

    Understanding Statistics

    (2004)
  • L.G. Daniel

    Statistical significance testing: A historical overview of misuse and misinterpretation with implications for the editorial policies of educational journals

    Research in the Schools

    (1998)
  • R.C. delMas et al.

    Exploring students’ conceptions of the standard deviation

    Statistics Education Research Journal

    (2005)
  • R. Falk

    Misconceptions of statistical significance

    Journal of Structural Learning

    (1986)
  • R. Falk et al.

    Significance tests die hard

    Theory and Psychology

    (1995)
  • F. Fidler

    Should psychology abandon p-values and teach CIs instead? Evidence-based reforms in statistics education

  • F. Fidler et al.

    Editors can lead researchers to confidence intervals, but can’t make them think: Statistical reform lessons from medicine

    Psychological Science

    (2004)
  • S. Finch

    Explaining the law of large numbers

  • S. Finch et al.

    Assessing conceptual change in learning statistics

  • Garfield

    Assessing statistical reasoning

    Statistics Education Research Journal

    (2003)
  • G. Gigerenzer

    The superego, the ego, and the id in statistical reasoning

  • G. Gigerenzer

    On narrow norms and vague heuristics: A reply to Kahneman and Tversky

    Psychological Review

    (1996)
  • J.A. Gliner et al.

    Problems with null hypothesis significance testing (NHST): What do the textbooks say?

    The Journal of Experimental Education

    (2002)
  • Cited by (139)

    • Probabilistic prediction of engineering demand parameters using Bayesian inference approach

      2022, Soil Dynamics and Earthquake Engineering
      Citation Excerpt :

      When zero is excluded from the 95% Bayesian Credible Interval (BCI), the predictor's predictability power is somewhat confirmed. A BCI of 95% indicates that the model correctly predicts the unobserved parameter 95% of the time [44]. Fig. 2a depicts a schematic diagram of the proposed hierarchical models based on high shrinkage priors.

    View all citing articles on Scopus
    View full text