Skip to main content

Advertisement

SpringerLink
Log in

Should We Be Concerned About Nonresponse Bias in College Student Surveys? Evidence of Bias from a Validation Study

  • Published:
Research in Higher Education Aims and scope Submit manuscript

Abstract

This study uses college student survey data and corresponding administrative data on campus recreation facility usage, academic performance, physical education class attendance, and co-curricular participation to examine nonresponse bias in college student surveys. Within the context of the Groves (Public Opin Q 70:646–675, 2006) Alternative Cause Model, we found compelling evidence of the presence of nonresponse error observed as student characteristics related to the survey topic that also explain their response propensity. An individual’s survey response propensity has a statistically significant relationship with their actual behavior for 2 of 3 survey topics. In 11 of the 13 survey questions used to measure the survey topic behaviors, we found statistically significant differences between the respondent and nonrespondent behavioral measures. These findings hold important implications for survey researchers and those using student surveys for high-stakes accountability measures because survey summary statistics may not be generalizable to the target population.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1

Similar content being viewed by others

References

  • Abraham, K. G., Helms, S., & Presser, S. (2009). How social processes distort measurement: The impact of survey nonresponse on estimates of volunteer work in the United States. American Journal of Sociology, 114(4), 1129–1165.

    Article  Google Scholar 

  • Adams, M. J. D., & Umbach, P. D. (2012). Nonresponse and online student evaluations of teaching: Understanding the influence of salience, fatigue, and academic environments. Research in Higher Education, 53(5), 576–591.

    Article  Google Scholar 

  • Aviv, A. L., Zelenski, J. M., Rallo, L., & Larsen, R. J. (2002). Who comes when: Personality differences in early and later participation in a university subject pool. Personality and Individual Differences, 33, 487–496.

    Article  Google Scholar 

  • Bowman, N. A. (2014). The meaning and interpretation of college student self-reporting gains. In S. Herzog & N. A. Bowman (Eds.), The validity of college student self-reported data. New Directions for Institutional Research. San Francisco: Jossey Bass.

  • Clarkberg, M., Robertson, D., & Einarson, M. (2008). Engagement and student surveys: Nonresponse and implications for reporting survey data. Paper presented at the Annual Forum of the Association for Institutional Research, Seattle, WA. Ithaca, NY: Office of Institutional Research and Planning at Cornell University.

  • Crawford, S. D., Couper, M. P., & Lamias, M. J. (2001). Web surveys: Perceptions of burden. Social Science Computer Review, 19(2), 146–162.

    Article  Google Scholar 

  • Curtin, R., Presser, S., & Singer, E. (2000). The effects of response rate changes on the index of consumer sentiment. Public Opinion Quarterly, 64(4), 413–428.

    Article  Google Scholar 

  • De Leeuw, E., & de Heer, W. (2002). Trends in household survey nonresponse: A longitudinal and international comparison. In R. M. Groves, D. A. Dillman, & J. L. Eltinge (Eds.), Survey nonresponse. New York: Wiley.

    Google Scholar 

  • Dey, E. L. (1997). Working with low survey response rates: The efficacy of weighting adjustments. Research in Higher Education, 44, 1–19.

    Google Scholar 

  • Groves, R. M. (2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 70, 646–675.

    Article  Google Scholar 

  • Groves, R. M., & Peytcheva, E. (2008). The impact of nonresponse rates on nonresponse bias: A meta-analysis. Public Opinion Quarterly, 72(2), 167–189.

    Article  Google Scholar 

  • Groves, R. M., Singer, E., & Corning, A. (2000). Leverage-saliency theory of survey participation. Public Opinion Quarterly, 64(3), 299–308.

    Article  Google Scholar 

  • Keeter, S., Miller, C., Kohut, A., Groves, R. M., & Presser, S. (2000). Consequences of reducing nonresponse in a national telephone survey. Public Opinion Quarterly, 64(2), 125–148.

    Article  Google Scholar 

  • Kuh, G. D. (2003). The national survey of student engagement: Conceptual framework and overview of psychometric properties. Bloomington, IN: Indiana University Center for Postsecondary Research.

    Google Scholar 

  • Merkle, D., & Edelman, M. (2002). Nonresponse in exit polls: A comprehensive analysis. In R. M. Groves, D. A. Dillman, & J. L. Eltinge (Eds.), Survey nonresponse. New York: Wiley.

    Google Scholar 

  • National Survey of Student Engagement. (2016). NSSE response rate FAQ. Retrieved from http://nsse.indiana.edu/pdf/Resp_Rate_FAQ.pdf.

  • Olson, K. (2006). Survey participation, nonresponse bias, measurement error bias, and total bias. Public Opinion Quarterly, 70(5), 737–758.

    Article  Google Scholar 

  • Olson, K., & Kennedy, C. (2006). Examination of the relationship between nonresponse and measurement error in a validation study of alumni. Sociology Department, Faculty Publications. Paper 146.

  • Pike, G. R. (2007). Adjusting for nonresponse in surveys. In John C. Smart (Ed.), Higher education: Handbook of theory and research (Vol. 22, pp. 411–450). Dordrecht: Springer.

    Google Scholar 

  • Porter, S. R. (2011). Do college student surveys have any validity? The Review of Higher Education, 35(1), 45–76.

    Article  Google Scholar 

  • Porter, S. R., & Umbach, P. D. (2006). Student survey response rates across institutions: Why do they vary? Research in Higher Education, 47(2), 229–247.

    Article  Google Scholar 

  • Porter, S. R., & Whitcomb, M. E. (2005). Non-response in student surveys: The role of demographics, engagement, and personality. Research in Higher Education, 46(2), 127–152.

    Article  Google Scholar 

  • Rubin, D. B. (1987). Multiple imputation for nonresponse in surveys. New York: John Wiley.

    Book  Google Scholar 

  • Sax, L. J., Gilmartin, S. K., & Bryant, A. N. (2003). Assessing response rates and nonresponse bias in web and paper surveys. Research in Higher Education, 44(4), 409–432.

    Article  Google Scholar 

  • Sax, L. J., Gilmartin, S. K., Lee, J. J., & Hagedorn, L. S. (2008). Using web surveys to reach community college students: An analysis of response rates and response bias. Community College Journal of Research and Practice, 32, 712–729.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. SAS Institute, Inc, Cary, NC, USA

    Trey Standish

  2. North Carolina State University, Raleigh, NC, USA

    Paul D. Umbach

Authors
  1. Trey Standish
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paul D. Umbach
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Trey Standish.

Appendices

Appendix 1

Survey Instrument Core Questions

  1. (1)

    Alternative versions of the CRF use questions randomly offered

    • In the past 7 days, how many times have you entered the Carmichael Recreation Complex for any reason? Specifically, how many times did you enter the indoor areas that require you to present your student identification card? Count each entry separately, even if you entered multiple times per day.—Drop down menu from 0 to 20

    • Approximately how many times this academic year—that is, since the start of fall 2015—have you entered the indoor parts of the Carmichael Recreation Complex for any reason? Specifically, how many times did you enter the indoor areas that require you to present your student identification card? Count each entry separately, even if you entered multiple times per day. – (A)Very often; (B)Often; (C)Sometimes; (D)Never

    • Approximately how many times this academic year—that is, since the start of fall 2015—have you entered the indoor parts of the Carmichael Recreation Complex for any reason? Specifically, how many times did you enter the indoor areas that require you to present your student identification card? Count each entry separately, even if you entered multiple times per day.—(A)I never entered Carmichael Recreation Complex; (B)Less than; (C)Once per week; (D)Twice per week; (E)Three or more times per week

  2. (2)

    Rounded to the nearest one hundredth (.01), what is your current cumulative grade point average (please include only for classes taken at NC State)? If you are unsure, please estimate.—(type here)

  3. (3)

    Rounded to the nearest one hundredth (.01), what was your semester grade point average in fall 2015 (please include only for classes taken at NC State)? If you are unsure, please estimate.—(type here)

  4. (4)

    In how many courses was your final grade an Unsatisfactory, F, D−, D, or D+ (please include only for classes taken at NC State)? If you are unsure, please estimate.—(A)0 (Never received F, D-, D, or D+); (B)1 course; (C)2 courses; (D)3 courses; (E)4 courses; (F)5 or more courses

  5. (5)

    In how many courses was your final grade an A+, A, or A− (please include only for classes taken at NC State)? If you are unsure, please estimate.—Drop down list 0 to 20 or more

  6. (6)

    How many for credit Health and Exercise Studies/Physical Education courses have you taken at NC State only? Include all courses at any level.?—Drop down list 0 to 4 or more

  7. (7)

    Including courses in which you are currently enrolled, select the most recent of the Health and Exercise Studies Fitness 100 level course you have taken. These are the HESF 100 level (previously called Physical Education) courses that are required to complete general education requirements. If you have not taken an HESF 100 level course, click on “I have not taken…” option; if you cannot remember the course please indicate “Not sure.” – (A)PE/HESF 100 (Cross Training); (B)PE/HESF 101 (Fitness and Wellness); (C)PE/HESF 102 (Fitness Walking); (D)PE/HESF 103 (Water Aerobics); (E)PE/HESF 104 (Swim Conditioning); (F)PE/HESF 105 (Aerobics and Body Conditioning); (G)PE/HESF 106 (Triathlon); (H)PE/HESF 107 (Run Conditioning); (I)PE/HESF 108 (Water Step Aerobics); (J)PE/HESF 109 (Step Aerobics); (K)PE/HESF 110 (Adapted Physical Education); (L)PE/HESF 111 (Indoor Group Cycling); (M)I have not taken a PE/HESF 100 level course (skip to question 12); (N)Not sure (skip to question 12)

  8. (8)

    What grade did you earn in [piped in name of question 9 course]? (select one)—(A)Currently enrolled (no grade); (B)Satisfactory; (C)A-/A/ A+; (D)B-/B/B+; (E)C-/C/C+; (F)D-/D/D+; (G)F; (H) Not sure

  9. (9)

    How many class meetings did you miss for any reason (e.g. you were sick, didn’t feel like going, had other obligations)? Only count class meetings you missed entirely; do not count class meetings you were late for or left early. – (A)0; (B)1; (C)2; (D)3; (E)4; (F)5; (G)6; (H)7 or more; (I)I took a distance education section

  10. (10)

    For each activity listed below, please identify if you have participated at any time while enrolled at NC State? Please include each event or competition separately; for example, count each game of an intramural basketball season.

    1. (10A)

      Intramural athletics, including only games, tournaments, and other competitions (excluding practice) – (A)I have never participated; (B)For one semester; (C)During multiple semesters; (D)In all semesters

    2. (10B)

      Club sports, including only games, tournaments, and other competitions (excluding practice)—(A)I have never participated; (B)For one semester; (C)During multiple semesters; (D)In all semesters

    3. (10C)

      Formal fitness classes, including only fitness meetings located inside NC State’s Carmichael Complex—(A)I have never participated; (B)For one semester; (C)During multiple semesters; (D)In all semesters

    4. (10D)

      Outdoor Adventures, counting each instructional activity, day-long trek, or weekend excursion as one event—(A)I have never participated; (B)For one semester; (C)During multiple semesters; (D)In all semesters.

Appendix 2

Email Communications

Invitation Email (March 14, 2016)

Subject: Gym Users Feedback

Hello,

My name is Trey Standish  and I am a PhD student working under the supervision of Dr. Paul Umbach in the Department of Educational Leadership, Policy and Human Development at NC State University. I am contacting you to ask you to take a 7 minute survey about Gym usage and academics. Your name was selected as a part of a random sample of all undergraduate students at NC State. I am a fellow student and you completing the survey is instrumental to my earning a degree.

Your answers to the survey questions are instrumental in understanding how your use of Carmichael Gymnasium and academics affects student outcomes. Participation in this study is voluntary. If you choose not to participate or to withdraw from the study at any time, there will be no penalty. The results of the research study may be published, but your name and responses will not be used. I would like to assure you that the study has been reviewed and received clearance through the NC State's Institutional Review Board.

Reminder Email 1 (March 21, 2016)

Subject: Class Skipping and Grades

Hello,

This is a friendly reminder to complete my brief survey about class skipping, gym usage and academics. I am a fellow student and you completing the survey is instrumental to me earning a degree. I very much appreciate your time and assure you it only involves completing a 7 minute survey about use of the Carmichael Gymnasium and academic history.

Reminder Email 2 (March 23, 2016)

Subject: Last Chance – Extracurricular Survey

Hello,

This is a friendly reminder to complete my brief survey about class skipping, gym usage and academics. I am a fellow student and you completing the survey is instrumental to me earning a degree. I very much appreciate your time and assure you it only involves completing a 7 minute survey about use of the Carmichael Gymnasium and academic history.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Standish, T., Umbach, P.D. Should We Be Concerned About Nonresponse Bias in College Student Surveys? Evidence of Bias from a Validation Study. Res High Educ 60, 338–357 (2019). https://doi.org/10.1007/s11162-018-9530-2

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11162-018-9530-2

Keywords

Search

Navigation