Disentangling Administration Errors From Scale Development Errors in Survey Research
DOI:
https://doi.org/10.33423/jmdc.v18i1.6820Keywords:
marketing development, survey design, information entropy, scale development, Tobit regressionAbstract
In a recent manuscript, Friesner, et al. (2023) used the concept of information entropy to assess the quantity of information in survey responses. They demonstrate how assessments of the quantity of information can be used to identify possible errors in a survey’s administration. A major limitation of their methodology is that it assumes the survey items used to elicit consumer preferences were created appropriately and contained a meaningful quantity of information. The current study addresses this limitation by incorporating a methodology developed by Friesner et al. (2021) into the Friesner et al. (2023) methodology. The combined methodology is applied to the same data studied in both Friesner et al. (2021) and Friesner et al. (2023), which allows for a direct comparison of the quantity of information gained/lost from survey administration versus scale development. The results indicate that the survey used in the empirical application exhibits flaws in both scale design and survey administration.
References
Abd Gani, N., Rathakrishnan, M., & Krishnasamy, H. (2020). A pilot test for establishing validity and reliability of qualitative interview in the blended learning English proficiency course. Journal of Critical Reviews, 7(5), 140–143.
Ballard, A. (2019). Framing bias in the interpretation of quality improvement data: Evidence from an experiment. International Journal of Health Policy and Management, 8(5), 307–314.
Batra, R., Homer, P., & Kahle, L. (2001). Values, susceptibility to normative influence, and attribute importance weights: A nomological analysis. Journal of Consumer Psychology, 11(2), 115–128.
Beatty, S., Kahle, L., Homer, P., & Misra, S. (1985). Alternative management approaches to consumer values: The list of values and the Rokeach value survey. Psychology and Marketing, 2(3), 181–200.
Bozman, C.S., Kurpis, L.V., & Frye, C. (2010). Hoopfest: Using longitudinal economic impact data to assess the success of a strategic reorientation. Sport Management Review, 13, 65–81.
Bradley, K., Peabody, M., & Sampson, S. (2015). Quality control in survey design: Evaluating a survey of educators’ attitudes concerning differentiated compensation. International Journal of Assessment Tools in Education, 2(1), 3–21.
Carré, A., Stefaniak, N., D’Ambrosio, F., Bensalah, L., & Besche-Richard, C. (2013). The Basic Empathy Scale in Adults (BES-A): Factor structure of a revised form. Psychological Assessment, 25(3), 679–691.
Dahl, F., & Osteras, N. (2010). Quantifying information content in survey data by entropy. Entropy, 12(2), 161–163.
Deming, W.E. (1944). On errors in surveys. American Sociological Review, 9(4), 359–369.
Dillman, D.A. (2000). Mail and internet surveys: The tailored design method. New York, NY: John Wiley and Sons.
Draugalis, J., Coons, S., & Plaza C. (2008). Best practices for survey research reports: A synopsis for authors and reviewers. American Journal of Pharmaceutical Education, 72(11), Article 11. Retrieved from https://www.ajpe.org/content/72/1/11
Entman, R. (2007). Framing bias: Media in the distribution of power. Journal of Communication, 57(1), 163–173.
Fenner, K., Hyde, M., Crean, A., & McGreevy, P. (2020). Identifying sources of potential bias when using online survey data to explore horse training, management, and behaviour: A systematic literature review. Veterinary Sciences, 7(3), Article 140. https://doi.org/10.3390/vetsci7030140
Friesner, D., Bozman, C., McPherson, M., Valente, F., & Zhang, A. (2021). Information entropy and scale development. Journal of Survey Statistics and Methodology, 9(5), 1183–1203.
Friesner, D., Bozman, C., McPherson, M., Valente, F., & Zhang, A. (2023). Information entropy as a quality control tool in survey research. Journal of Marketing Development and Competitiveness, 17(1), 46–61.
Friesner, D., Valente, F., & Bozman, C.S. (2016). Using entropy-based information theory to evaluate survey research. Journal of Marketing Development and Competitiveness, 10(3), 32–48.
Golan, A. (2006). Information and entropy econometrics – A review and Synthesis. Foundations and Trends in Econometrics, 2(1–2), 1–145.
Golan, A., Judge, G., & Miller, D. (1996). Maximum entropy econometrics: Robust estimation with limited data. New York, NY: John Wiley and Sons.
Golan, A., Judge, G., & Perloff, J. (1996). A maximum entropy approach to recovering information from multinomial response data. Journal of the American Statistical Association, 91(434), 841–853.
González-Cabrera, M., Ortega-Martínez, A., Martínez-Galiano, J., Hernández-Martínez, A., Parra-Anguita, L., & Frías-Osuna, A. (2020). Design and validation of a questionnaire on communicating bad news in nursing: A pilot study. International Journal of Environmental Research and Public Health, 17(2), Article 457. https://doi.org/10.3390/ijerph17020457
Hair, J., Black, W., Babin, B., Anderson, R., & Tatham, R. (2006). Multivariate data analysis (6th Ed.). Upper Saddle River, NJ: Pearson/Prentice Hall.
Hendra, R., & Hill, A. (2019). Rethinking response rates: New evidence of little relationship between survey response rates and nonresponse bias. Evaluation Review, 43(5), 307–330.
Imbens, G., & Lancaster, T. (1996). Efficient estimation and stratified sampling. Journal of Econometrics, 74(2), 289–318.
Jaynes, E. (1957). Information theory and statistical mechanics. Physics Review, 106(4), 620–630.
Jaynes, E. (1982). On the rationale of maximum-entropy methods. Proceedings of the IEEE, 70(9), 939–952.
Johnson, T., Cho, Y., Holbrook, A., O’Rourke, D., Warnecke, R., & Chavez, N. (2006). Cultural variability in the effects of question design features on respondent comprehension of health surveys. Annals of Epidemiology, 16(9), 661–668.
Jolliffe, D., & Farrington, D.P. (2006). Development and validation of the basic empathy scale. Journal of Adolescence, 29, 589–611.
Jordan, P., & Troth, A. (2020). Common method bias in applied settings: The dilemma of researching in organizations. Australian Journal of Management, 45(1), 3–14.
Kahle, L., Beatty, S., & Homer, P. (1986). Alternative measurement approaches to consumer values: The list of values (LOV) and values and life style (VALS). Journal of Consumer Research, 13(3), 405–409.
Kahle, L.R. (1983). Social Values and Social Change: Adaptation to Life in America. New York, NY: Praeger.
Kurpis, L.V., Bozman, C.S., & Kahle, L.R. (2010). Distinguishing between amateur sport participants and spectators: The list of values approach. International Journal of Sport Management and Marketing, 7(3/4), 190–201.
Marquis, K., Marquis, M., & Polich, J. (1986). Response bias and reliability in sensitive topic surveys. Journal of the American Statistical Association, 81(394), 381–389.
Schnell, L. (2014, June 27). Hoopfest, the world’s largest 3-3 tourney, turns 25 this weekend. Retrieved from https://www.si.com/college-basketball/2014/06/27/spokane-hoopfest-3-3-basketball-tournament-25-years
Shannon, C. (1948). A mathematical theory of communication. Bell System Technical Journal, 27, 37–423.
Suchman, E.A. (1962). An analysis of “bias” in survey research. The Public Opinion Quarterly, 26(1), 102–111.
Tripathi, G. (2011). Generalized method of moments (GMM) based inference with stratified samples when the aggregate shares are known. Journal of Econometrics, 165(2), 258–265.
van Teijlingen, E., & Hundley, V. (2001). The importance of pilot studies. Social Research Update, (35), 1–4. Retrieved from http://sru.soc.surrey.ac.uk/SRU35.pdf
van Teijlingen, E., Rennie, A.-M., Hundley, V., & Graham, W. (2001). The importance of conducting and reporting pilot studies: The example of the Scottish Births Survey. Journal of Advanced Nursing, 34(3), 289–295.
Wikman, A., & Wärneryd, B. (1990). Measurement errors in survey questions: Explaining response variability. Social Indicators Research, 22(2), 199–212.
Wing, C., Simon, K., & Bello-Gomez, R. (2018). Designing difference in difference studies: Best practices for public health policy research. Annual Review of Public Health, 39(1), 453–469.
Wiseman, F. (1972). Methodological bias in public opinion surveys. The Public Opinion Quarterly, 36(1), 105–108.
