Abstract
A Web-based coding application was designed to improve coding efficiency and to provide a systematic means of evaluating responses to open-ended assessments. The system was developed for use by multiple raters to assign open-ended responses to predetermined categories. The application provides a software environment for efficiently supervising the work of coders and evaluating the quality of the coding by (1) systematically presenting open-ended responses to coders, (2) tracking each coder’s categorized responses, and (3) assessing interrater consistency at any time in order to identify coders in need of further training. In addition, the application can be set to automatically assign repeated responses to categories previously identified as appropriate for those responses. To evaluate the efficacy of the coding application and to determine the statistical reliability of coding open-ended data within this application, we examined data from two empirical studies. The results demonstrated substantial interrater agreement on items assigned to various categories across free and controlled association tasks. Overall, this new coding application provides a feasible method of reliably coding open-ended data and makes the task of coding these data more manageable.
Article PDF
Similar content being viewed by others
References
Ames, S. L., &Stacy, A. W. (1998). Implicit cognition in the prediction of substance use among drug offenders.Psychology of Addictive Behaviors,12, 272–281.
Bishop, G. F., Tuchfarber, A. J., &Oldendick, R. W. (1986). Opinions on fictitious issues: The pressure to answer survey questions.Public Opinion Quarterly,50, 240–250.
Bornstein, R. F. (1999). Criterion validity of objective and projective dependency tests: A meta-analytic assessment of behavioral prediction.Psychology Assessment,11, 48–57.
Carey, J., Wenzel, P., Reilly, C., Sheridan, J., &Steinberg, J. (1998). CDC EZ-Text: Software for management and analysis of semistructured qualitative data sets.Cultural Anthropology Methods,10, 14–20.
Cramer, P. (1968).Word association. New York: Academic Press.
Crutcher, R. (2003). A computer-aided digital audio recording and encoding system for improving the encoding of verbal reports.Behavior Research Methods, Instruments, & Computers,35, 263–268.
Dosher, B. A., &Rosedale, G. S. (1997). Configural processing in memory retrieval: Multiple cues and ensemble representations.Cognitive Psychology,33, 209–265.
James, J., &Sanderson, P. (1991). Heuristic and statistical support for protocol analysis with SHAPA Version 2.01.Behavior Research Methods, Instruments, & Computers,23, 449–460.
Krosnick, J. A. (1999). Survey research.Annual Review of Psychology,50, 537–567.
Landis, J. R., &Koch, G. G. (1977). The measurement of observer agreement for categorical data.Biometrics,33, 159–174.
Murray, D. M., &Perry, C. L. (1987). The measurement of substance use among adolescents: When is the “bogus pipeline” method needed?Addictive Behaviors,12, 225–233.
Nelson, D. L., McEvoy, C. L., & Schreiber, T. A. (1998).The University of South Florida word association, rhyme, and word fragment norms. www.usf.edu/FreeAssociation/.
Stacy, A. W. (1997). Memory activation and expectancy as prospective predictors of alcohol and marijuana use.Journal of Abnormal Psychology,106, 61–73.
Stacy, A. W., Ames, S. L., Sussman, S., &Dent, C. W. (1996). Implicit cognition in adolescent drug use.Psychology of Addictive Behaviors,10, 190–203.
Stacy, A. W., Widaman, K. F., Hayes, R., &DiMatteo, M. R. (1985). Validity of self-reports of alcohol and other drug use: A multitraitmultimethod assessment.Journal of Personality & Social Psychology,49, 219–232.
Sussman, S., Dent, C. W., Simon, T. R., Stacy, A. W., Galaif, E. R., Moss, M. A., et al. (1995). Immediate impact of social influenceoriented substance abuse prevention curricula in traditional and continuation high schools.Drugs & Society,8, 65–81.
Sussman, S., Stacy, A. W., Ames, S. L., &Freedman, L. B. (1998). Self-reported high-risk locations of adolescent drug use.Addictive Behaviors,23, 405–411.
Sussman, S., Stacy, A. W., Dent, C. W., Simon, T. R., Galaif, E. R., Moss, M. A., et al. (1995). Continuation high schools: Youth at risk for drug abuse.Journal of Drug Education,25, 191–209.
Verbastat 7.O [Computer software]. (2004). Chicago: SPSS. www.spss.com/verbastat/.
Walonick, D. (2004). StatPack [Computer software]. Minneapolis: Author. www.statpac.com.
White, D., King, A., &Duncan, S. (2002). Voice recognition technology as a tool for behavioral research.Behavior Research Methods, Instruments, & Computers,34, 1–5.
Author information
Authors and Affiliations
Corresponding author
Additional information
This research was supported by National Institute on Drug Abuse Grants DA12101 and DA16094 and National Institute on Alcohol Abuse and Alcoholism Grant AA12128. The authors thank Wei Sun and Juan Huang for their contributions to the development of the Web-based coding application. The Web-based coding application described in this manuscript is available to researchers free of charge. If you are interested in using the application, please contact the first author.
Note—This article was accepted by the previous editor, Jonathan Vaughan.
Rights and permissions
About this article
Cite this article
Ames, S.L., Gallaher, P.E., Sun, P. et al. A Web-based program for coding open-ended response protocols. Behavior Research Methods 37, 470–479 (2005). https://doi.org/10.3758/BF03192716
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/BF03192716