“Averages and relationships and trends and graphs are not always what they seem. There may be more in them than meets the eye, and there may be a good deal less.”
Darrell Huff, “How to Lie with Statistics”
Abstract
We comment on Sijtsma’s (2014) thought-provoking essay on how to minimize questionable research practices (QRPs) in psychology. We agree with Sijtsma that proactive measures to decrease the risk of QRPs will ultimately be more productive than efforts to target individual researchers and their work. In particular, we concur that encouraging researchers to make their data and research materials public is the best institutional antidote against QRPs, although we are concerned that Sijtsma’s proposal to delegate more responsibility to statistical and methodological consultants could inadvertently reinforce the dichotomy between the substantive and statistical aspects of research. We also discuss sources of false-positive findings and replication failures in psychological research, and outline potential remedies for these problems. We conclude that replicability is the best metric of the minimization of QRPs and their adverse effects on psychological research.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abelson, R. P. (1995). Statistics as principled argument. Hillsdale, NJ: Erlbaum.
Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J., Fiedler, K., et al. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108–119.
Bertamini, M., & Munafò, M. R. (2012). Bite-size science and its undesired side effects. Perspectives on Psychological Science, 7, 67–71.
Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., et al. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365–376.
Cohen, J. (1962). The statistical power of abnormal-social psycological research: A review. Journal of Abnormal and Social Psycology, 65, 145–153.
Cohen, J. (1969). Statistical Power Analysis for the Behavioral Sciences. Hillsdale, NJ: Erlbaum.
Cohen, J. (1994). The earth is round \((p<. 05)\). American Psychologist, 49, 997–1003.
Collins, A. L., & Sullivan, P. F. (2013). Genome-wide association studies in psychiatry: What have we learned. British Journal of Psychiatry, 202, 1–4.
Cumming, G. (2014). The new statistics why and how. Psychological Science, 25, 7–29.
Fidler, F., Thomason, N., Cumming, G., Finch, S., & Leeman, J. (2004). Editors can lead researchers to confidenceintervals, but can’t make them think: Statistical reform lessons from medicine. Psychological Science, 15, 119–126.
Fisher, R. A. (1925). Statistical methods for research workers. London: Oliver & Boyd.
Frank, M. C., & Saxe, R. (2012). Teaching replication. Perspectives on Psychological Science, 7, 600–604.
Gigerenzer, G. (1998). We need statistical thinking, not statistical rituals. Behavioral and Brain Sciences, 21, 199–200.
Giner-Sorolla, R. (2012). Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science, 7, 562–571.
Gray, K., & Wegner, D. M. (2013). Six guidelines for interesting research. Perspectives on Psychological Science, 8, 549–553.
Harlow, L. L., Mulaik, S. A., & Steiger, J. H. (Eds.). (1997). What if there were no significance tests? Hillsdale, NJ: Erlbaum.
Harzing, A. W. (2011). The publish or perish book: A guide to the software. London: Tarma Software Research.
Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. New York: Academic Press.
Hersh, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences, 16569–16572.
Huff, D. (1954). How to Lie with Statistics. New York: W.W. Norton & Company.
Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. doi:10.1371/journal.pmed.0020124.
Ioannidis, J., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014). Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention. Trends in Cognitive Sciences, 18, 235–241.
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524–532.
Kahneman, D. (2011). Thinking: Fast and slow. New York: Farrar, Straus and Giroux.
Kuhn, T. S. (1970). The Structure of Scientific Revolutions. Chicago, IL: University of Chicago Press.
Lehrer, J. (2010). The truth wears off. The New Yorker, 52–57.
Lykken, D. T. (1968). Statistical significance in psychological research. Psychological Bulletin, 70, 1151–1159.
Meehl, P. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806–834.
Neyman, J., & Pearson, E. S. (1928). On the use and interpretation of certain test criteria for purposes of statistical inference. Biomettika, 20A, 175–240, 263–294.
Nickerson, R. S. (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5, 241–301.
Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific Utopia II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631.
Nosek, B. A., & Lakens, D. (2014). Registered reports. Social Psychology, 45, 137–141.
Oakes, M. (1986). Statistical inference: A commentary for the social and behavioral sciences. New York: Wiley.
Pashler, H., & Wagenmakers, E. J. (2012). Editors’ Introduction to the Special Section on replicability in psychological science. A crisis of confidence? Perspectives on Psychological Science, 7, 528–530.
Ripke, S., et al. (2013). Genome-wide association analysis identifies 13 new risk loci for schizophrenia. Nature Genetics, 45, 150–159.
Rosenthal, R. (2001). Meta-analytic procedures for social research (2nd ed.). Beverly Hills, CA: Sage Publications.
Rosnow, R. L., & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276–1284.
Rozeboom, W. W. (1960). The fallacy of the null-hypothesis significance test. Psychological Bulletin, 57, 416–428.
Schmidt, F. L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47, 1173–1181.
Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies. Psychological Bulletin, 105(2), 309–316.
Simons, D. J. (2014). The value of direct replication. Perspectives on Psychological Science, 9, 76–80.
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2013). P-Curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143, 534–537.
Smith, M. L., Glass, G. V., & Miller, T. I. (1980). The benefits of psychotherapy. Baltimore, MD: Johns Hopkins University Press.
Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185, 1124–1131.
Vrij, A., Granhag, P. A., & Porter, S. (2010). Pitfalls and opportunities in nonverbal and verbal lie detection. Psychological Science in the Public Interest, 11, 89–121.
Wicherts, J.M. (2014, May 24). The power paradox and the myth of the failed study. Paper presented at the Annual Convention of the Association for Psychological Science, San Francisco, California.
Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PloS One, 6, e26828.
Author information
Authors and Affiliations
Corresponding authors
Rights and permissions
About this article
Cite this article
Waldman, I.D., Lilienfeld, S.O. Thinking About Data, Research Methods, and Statistical Analyses: Commentary on Sijtsma’s (2014) “Playing with Data”. Psychometrika 81, 16–26 (2016). https://doi.org/10.1007/s11336-015-9447-z
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11336-015-9447-z