Abstract
Cognitive diagnosis models (CDMs) continue to generate interest among researchers and practitioners because they can provide diagnostic information relevant to classroom instruction and student learning. However, its modeling component has outpaced its complementary component—test construction. Thus, most applications of cognitive diagnosis modeling involve retrofitting of CDMs to assessments constructed using classical test theory (CTT) or item response theory (IRT). This study explores the relationship between item statistics used in the CTT, IRT, and CDM frameworks using such an assessment, specifically a large-scale mathematics assessment. Furthermore, by highlighting differences between tests with varying levels of diagnosticity using a measure of item discrimination from a CDM approach, this study empirically uncovers some important CTT and IRT item characteristics. These results can be used to formulate practical guidelines in using IRT- or CTT-constructed assessments for cognitive diagnosis purposes.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46, 443–459.
Burke, M. J., & Henson, R. (2008). LCDM user’s manual. Greensboro: University of North Carolina at Greensboro.
de la Torre, J. (2007, April). Evaluation of model fit in a large-scale assessment application of cognitive diagnosis. Presentation at the annual meeting of the national council on measurement in education, Chicago, IL.
de la Torre, J. (2008). An empirically-based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45, 343–362.
de la Torre, J. (2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34, 115–130.
de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76, 179–199.
de la Torre, J., & Douglas, J. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333–353.
de la Torre, J., & Douglas, J. (2005, April). Modeling multiple strategies in cognitive diagnosis. Presentation at the annual meeting of the national council on measurement in education, Montreal, Canada.
de la Torre, J., & Douglas, J. (2008). Model evaluation and multiple strategies in cognitive diagnosis: An analysis of fraction subtraction data. Psychometrika, 73, 595–624.
de la Torre, J., & Karelitz, T. (2009). Impact of diagnosticity on the adequacy of models for cognitive diagnosis under a linear attribute structure. Journal of Educational Measurement, 46, 450–469.
Doignon, J. P., & Falmagne, J. C. (1999). Knowledge spaces. New York: Springer.
Doornik, J. A. (2002). Object-oriented matrix programming using Ox (Version 3.1). [Computer software]. London: Timberlake Consultants Press.
Florida Department of Education. (2003a). Florida comprehensive assessment test. Tallahassee, FL: Author.
Florida Department of Education. (2003b). Florida comprehensive assessment test for reading and mathematics: Technical report for test administrations of FCAT 2003. San Antonio, TX: Harcourt Educational Measurement.
Fu, J., & Li, Y. (2007, April). An integrative review of cognitively diagnostic psychometric models. Presentation at the annual meeting of the national council on measurement in education, Chicago, IL.
Haertel, E. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 333–352.
Henson, R., Templin, J., & Willse, J. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74, 191–210.
Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272.
Lee, Y.-S., Choi, K.-M., & Park, Y. S. (2009, April). A comparison between the U.S. and Korea in the TIMSS 8th grade mathematics assessment: An application of cognitive diagnostic modeling. Presentation at the annual meeting of the american education research association, San Diego, CA.
Lee, Y.-S., Park, Y. S., & Taylan, D. (2011). An analysis of attribute mastery via cognitive diagnostic modeling: A comparison of Massachusetts, Minnesota, and the U.S. national average via TIMSS 2007 4th grade mathematics. International Journal of Testing, 11(2), 144–177.
Macready, G. B., & Dayton, C. M. (1977). The use of probabilistic models in the assessment of mastery. Journal of Educational Statistics, 33, 379–416.
Maris, E. (1999). Estimating multiple classification latent class models. Psychometrika, 64, 187–212.
Mislevy, R. (1994). Evidence and inference in educational assessment. Psychometrika, 59, 439–483.
Mislevy, R. (1995). Probability-based inference in cognitive diagnosis. In P. D. Nichols, S. F. Chipman, & R. L. Brennan (Eds.), Cognitively diagnostic assessment (pp. 43–71). Hillsdale: Erlbaum.
National Council of Teachers of Mathematics (NCTM). (2000). Principles and standards for school mathematics. Reston: NCTM.
National Research Council. (2001). Knowing what students know: The science and design of educational assessment. Washington: National Academies Press.
Rupp, A. A., & Templin, J. L. (2008). Unique characteristics of diagnostic classification models: A comprehensive review of the current state-of-the-art. Measurement, 6, 219–262.
Tatsuoka, K. K. (1983). Rule-space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345–354.
Tatsuoka, K. K. (1990). Toward an integration of item–response theory and cognitive error diagnosis. In N. Frederiksen, R. Glaser, A. Lesgold, & M. Safto (Eds.), Monitoring skills and knowledge acquisition (pp. 453–488). Hillsdale: Erlbaum.
Tatsuoka, C. (2002). Data-analytic methods for latent partially ordered classification models. Journal of the Royal Statistical Society Series C (Applied Statistics), 51, 337–350.
Turhan, A. (2006). Multilevel 2PL item response model vertical equating with the presence of differential item functioning. Unpublished doctoral dissertation, The Florida State University, Tallahassee, FL.
von Davier, M. (2005). A general diagnostic model applied to language testing data (RR-05–16). Princeton: Educational Testing Service.
Zimowski, M. F., Muraki, E., Mislevy, R. J., & Bock, R. D. (1996). BILOG-MG: Multiple group IRT analysis and test maintenance for binary items. Chicago: Scientific Software International.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Lee, YS., de la Torre, J. & Park, Y.S. Relationships between cognitive diagnosis, CTT, and IRT indices: an empirical investigation. Asia Pacific Educ. Rev. 13, 333–345 (2012). https://doi.org/10.1007/s12564-011-9196-3
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12564-011-9196-3