Students’ misconceptions of statistical inference: A review of the empirical evidence from research on statistics education
Introduction
Since the early 1970s, there has been an increasing interest in research about people's understanding and performance in probability and statistics. Researchers especially focused on misconceptions and heuristics regarding probability, chance, and randomness (e.g., Kahneman, Slovic, & Tversky, 1982; Konold, 1989, Konold, 1991; Nisbett & Ross, 1980; Shaughnessy, 2003). This article reports on a systematic review of the available empirical evidence of students’ misconceptions in statistical inference. It constitutes a starting point for educational researchers interested in the relation between misconceptions and the conceptual change theory (e.g., Finch & Cumming, 1998; Guzzetti, Snyder, Glass, & Gamas, 1993; Smith, diSessa, & Roschelle, 1993), or a helpful tool for teachers of statistical inference to become aware of the most common misconceptions that their students may hold (e.g., Batanero, Godino, Vallecillos, & Holmes, 1994; Brewer, 1985; Haller & Krauss, 2002).
In educational research, the term misconception is used to refer to several concepts. On the one hand, authors often consider a broad definition of the word, using it to label different concepts such as preconception, misunderstanding, misuse, or misinterpretation interchangeably (Smith et al., 1993). On the other hand, misconceptions are sometimes defined in a more restrictive way, as misunderstandings generated during instruction, emphasizing a distinction with alternative conceptions resulting from ordinary life and experience (Guzzetti et al., 1993). In this manuscript, a refinement of the first definition is applied, and the term refers to any sort of fallacies, misunderstandings, misuses, or misinterpretations of concepts, provided that they result in a documented systematic pattern of error (Cohen, Smith, Chechile, Burns, & Tsai, 1996).
The interest on statistical inference arises from three realities. First, this is a topic of main relevance for the development of research in all empirical sciences in general and psychology and education in particular (Belia, Fidler, Williams, & Cumming, 2005; Krauss & Wassner, 2002). Second, inference receives special attention in statistical courses from almost all scientific areas, where hypotheses tests and confidence intervals are taught to students as the methods for evaluating scientific hypotheses (Aberson, Berger, Healy, & Romero, 2003; APA, 2001). Finally, inferential ideas seem to be especially sensitive to be misunderstood and students are often prone to fall into deep misconceptions (Daniel, 1998, Kirk, 2001) because they require students to understand and connect many abstract concepts such as sampling distribution and significance level.
After presenting our methodology of search for this review (Section 2), we provide an overview of the misconceptions mentioned and exemplified in the literature, and describe to what extent and under which conditions they occur, discussing the methodology of the presented group of studies (Section 3). Finally, we conclude with some suggestions for further research (Section 4).
Section snippets
Method
We performed a thorough literature exploration in order to bring together publications that report on studies providing empirical evidence of university students’ misconceptions that have been published during the last 15 years (from 1990 to the beginning of 2006). Therefore, studies based on personal experience and anecdotes only or publications oriented to other groups (such as professionals or younger students) were excluded.
Other publications that did not match our inclusion criteria, for
Results
As our findings show, the literature on statistics education, and particularly publications providing empirical evidence of misconceptions in statistics, is sparse (see also Batanero, 2005; Ware & Chastain, 1991). The four searches defined above yielded more than 500 references that contained only 21 publications (cf. Table 1) reporting on 17 different studies (cf. Table A1) that provide evidence of misconceptions about topics related to statistical inference and that satisfied our selection
Conclusion
The group of publications analyzed in this review provides empirical evidence of deep and spread students’ misconceptions in statistical inference. This evidence demonstrates that, although they may be able to manipulate and carry out calculations with statistical data, students have severe misconceptions concerning the interpretation of results from inferential techniques. According to this literature, the following summary and classification of students’ misconceptions in statistical
References (75)
The role of sample size in sample evaluation
Organizational Behavior and Human Performance
(1979)- et al.
Subjective probability: A judgment of representativeness
Cognitive Psychology
(1972) Probabilistic logic
Artificial Intelligence
(1986)- et al.
Development of understanding of sampling for statistical literacy
Journal of Mathematical Behavior
(2000) - et al.
Understanding the effects of sample size on the variability of the mean
Organizational Behavior and Human Decision Processes
(1990) - et al.
Evaluation of an interactive tutorial for teaching hypothesis testing concepts
Teaching of Psychology
(2003) The language of conditional probability
Journal of Statistics Education
(2006)Publication manual of the American Psychological Association
(2001)PsycINFO
(2006)Controversies around the role of statistical tests in experimental research
Mathematical Thinking and Learning (An International Journal)
(2000)
Statistics education as a field for research and practice
Errors and difficulties in understanding elementary statistical concepts
International Journal of Mathematics Education in Science and Technology
Significado y comprensión de la distribución normal en un curso introductorio de análisis de datos [Meaning and understanding of the normal distribution in an introductory data analysis course]
Quadrante
Students’ reasoning about the normal distribution
Researchers misunderstand confidence intervals and standard error bars
Psychological Methods
Probability
Behavioral statistics textbooks: Source of myths and misconceptions?
Journal of Educational Statistics
Reasoning about sampling distributions
Statistical significance
Things I have learned (so far)
American Psychologist
Identifying impediments to learning probability and statistics from an assessment of instructional software
Journal of Educational and Behavioral Statistics
Understanding replication: Confidence intervals, p-values, and what's likely to happen next time
Confidence intervals and replication: Where will the next mean fall?
Psychological Methods
Replication, and researchers’ understanding of confidence intervals and standard error bars
Understanding Statistics
Statistical significance testing: A historical overview of misuse and misinterpretation with implications for the editorial policies of educational journals
Research in the Schools
Exploring students’ conceptions of the standard deviation
Statistics Education Research Journal
Misconceptions of statistical significance
Journal of Structural Learning
Significance tests die hard
Theory and Psychology
Should psychology abandon p-values and teach CIs instead? Evidence-based reforms in statistics education
Editors can lead researchers to confidence intervals, but can’t make them think: Statistical reform lessons from medicine
Psychological Science
Explaining the law of large numbers
Assessing conceptual change in learning statistics
Assessing statistical reasoning
Statistics Education Research Journal
The superego, the ego, and the id in statistical reasoning
On narrow norms and vague heuristics: A reply to Kahneman and Tversky
Psychological Review
Problems with null hypothesis significance testing (NHST): What do the textbooks say?
The Journal of Experimental Education
Cited by (139)
When perceived informativity is not enough. How teachers perceive and interpret statistical results of educational research
2023, Teaching and Teacher EducationProbabilistic prediction of engineering demand parameters using Bayesian inference approach
2022, Soil Dynamics and Earthquake EngineeringCitation Excerpt :When zero is excluded from the 95% Bayesian Credible Interval (BCI), the predictor's predictability power is somewhat confirmed. A BCI of 95% indicates that the model correctly predicts the unobserved parameter 95% of the time [44]. Fig. 2a depicts a schematic diagram of the proposed hierarchical models based on high shrinkage priors.
Effectiveness of Gamification in Enhancing Learning and Attitudes: A Study of Statistics Education for Health School Students
2023, Journal of Advances in Medical Education and ProfessionalismThe integration of undergraduate students’ informal and formal inferential reasoning
2023, Educational Studies in Mathematics