Skip to main content
SpringerLink
Log in

Combatting carelessness: Can placement of quality check items help reduce careless responses?

  • Published:
Current Psychology Aims and scope Submit manuscript

Abstract

There is extensive literature on identifying careless responses in survey data and acknowledging their negative impact on accurate data analysis (Goodman, Cryder, & Cheema, 2013; Huang et al., 2015a, b; Meade and Craig, 2012). However, there are very limited findings on how researchers can help prevent participants from responding carelessly in the first place. The current study manipulated the placement of a quality check item (i.e., early placement versus late placement) and showed that participants were less likely to carelessly respond when the quality check items were placed later in the survey. Quality check items were also able to identify careless responses more than other approaches (i.e., LongString index, completion time index, and self-reported indicators). When using quality check items, it appears to be best when used toward the end of the survey. There are plenty of opportunities for future studies to expand on this research to uncover ways that can help deter careless responding.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. As a reviewer suggested, I also performed an independent samples t-test to determine if there was a difference between the two conditions regarding the raw completion time (in seconds). There was not a statistically significant difference between the early condition (M = 10,991.25, SD = 40,448.07) and the late condition (M = 8260.41, SD = 35,600.41) regarding the completion time index; t(283) = 0.61, p = .55. This result is very similar to the statistically insignificant result using the completion time index.

References

  • Adair, J. G. (1984). The Hawthorne effect: A reconsideration of the methodological artifact. Journal of Applied Psychology, 69(2), 334–345.

    Article  Google Scholar 

  • Baer, R. A., Ballenger, J., Berry, D. T., & Wetter, M. W. (1997). Detection of random responding on the MMPI–A. Journal of Personality Assessment, 68(1), 139–151.

    Article  Google Scholar 

  • Beach, D. A. (1989). Identifying the random responder. The Journal of Psychology, 123(1), 101–103.

    Article  Google Scholar 

  • Berinsky, A. J., Margolis, M. F., & Sances, M. W. (2016). Can we turn shirkers into workers? Journal of Experimental Social Psychology, 66, 20–28.

    Article  Google Scholar 

  • Berry, D. T., Wetter, M. W., Baer, R. A., Larsen, L., Clark, C., & Monroe, K. (1992). MMPI-2 random responding indices: Validation using a self-report methodology. Psychological Assessment, 4(3), 340–345.

    Article  Google Scholar 

  • Bowling, N. A., Huang, J. L., Bragg, C. B., Khazon, S., Liu, M., & Blackmore, C. E. (2016). Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. Journal of Personality and Social Psychology, 111(2), 218–229.

    Article  Google Scholar 

  • Chiaburu, D. S., Huang, J. L., Hutchins, H. M., & Gardner, R. G. (2014). Trainees’ perceived knowledge gain unrelated to the training domain: The joint action of impression management and motives. International Journal of Training and Development, 18(1), 37–52.

    Article  Google Scholar 

  • Galesic, M., & Bosnjak, M. (2009). Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly, 73(2), 349–360.

    Article  Google Scholar 

  • Goldberg, L. R. (1999). A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. Personality Psychology in Europe, 7(1), 7–28.

    Google Scholar 

  • Goldberg, L. R., & Kilkowski, J. M. (1985). The prediction of semantic consistency in self-descriptions: Characteristics of persons and of terms that affect the consistency of responses to synonym and antonym pairs. Journal of Personality and Social Psychology, 48(1), 82–98.

    Article  Google Scholar 

  • Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of mechanical Turk samples. Journal of Behavioral Decision Making, 26(3), 213–224.

    Article  Google Scholar 

  • Gordon, M. E., Slade, L. A., & Schmitt, N. (1986). The “science of the sophomore” revisited: From conjecture to empiricism. Academy of Management Review, 11(1), 191–207.

    Google Scholar 

  • Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27(1), 99–114.

    Article  Google Scholar 

  • Huang, J. L., Bowling, N. A., Liu, M., & Li, Y. (2015a). Detecting insufficient effort responding with an infrequency scale: Evaluating validity and participant reactions. Journal of Business and Psychology, 30(2), 299–311.

    Article  Google Scholar 

  • Huang, J. L., Liu, M., & Bowling, N. A. (2015b). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100(3), 828–845.

    Article  Google Scholar 

  • Johnson, J. A. (2005). Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39(1), 103–129.

    Article  Google Scholar 

  • Joinson, A. (1999). Social desirability, anonymity, and internet-based questionnaires. Behavior Research Methods, Instruments, & Computers, 31(3), 433–438.

    Article  Google Scholar 

  • Landsberger, H. (1958). Hawthorne revisited. Ithaca, New York: Cornell University.

    Google Scholar 

  • Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455.

    Article  Google Scholar 

  • Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867–872.

    Article  Google Scholar 

  • Spelke, E., Hirst, W., & Neisser, U. (1976). Skills of divided attention. Cognition, 4(3), 215–230.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychology, Rice University, 6100 Main Street, Houston, TX, 77005-1827, USA

    Denise L. Reyes

Authors
  1. Denise L. Reyes
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Denise L. Reyes.

Ethics declarations

Informed Consent

This study was approved by the Institutional Review Board (IRB#: IRB-FY2017–339).

Data Availability Statement

The dataset generated and analyzed during the current study are available from the author on reasonable request.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix 1

Table 1 Correlations Between All Variables
Full size table
Table 2 Effect of Quality Check Item Placement
Full size table

Appendix 2

Fig. 1
figure 1

Distribution of self-reported index for early condition

Full size image
Fig. 2
figure 2

Distribution of self-reported index for late condition

Full size image
Fig. 3
figure 3

Distribution of self-reported index for both conditions

Full size image
Fig. 4
figure 4

Distribution of LongString index for early condition

Full size image
Fig. 5
figure 5

Distribution of LongString index for late condition

Full size image
Fig. 6
figure 6

Distribution of LongString for both conditions

Full size image
Fig. 7
figure 7

Distribution of completion time index for early condition

Full size image
Fig. 8
figure 8

Distribution of completion time index for late condition

Full size image
Fig. 9
figure 9

Distribution of completion time index for both conditions

Full size image

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Reyes, D.L. Combatting carelessness: Can placement of quality check items help reduce careless responses?. Curr Psychol 41, 6858–6866 (2022). https://doi.org/10.1007/s12144-020-01183-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12144-020-01183-4

Keywords

Search

Navigation