Abstract
Cognitive diagnostic models can be classified into two categories based on the type of interaction between attributes: disjunctive and conjunctive. A representative example of the former is the “Deterministic Input Noisy-Or gate” (DINO) model, and of the latter is the “Deterministic Input Noisy-And gate” (DINA) model. However, fixing the interaction form to be either disjunctive or conjunctive may be based on a strong assumption. Therefore, we developed a new hybrid cognitive diagnostic model in which the item response function is represented as a weighted mixture of disjunctive and conjunctive item response functions. This made it possible to estimate the quantitative degree of each interaction type for each item, while keeping the parameters within reasonable limits. The proposed model was formalized as a Bayesian model and estimated using the Hamiltonian Monte Carlo algorithm. A Monte Carlo simulation confirmed adequate parameter recovery of the proposed method. In an empirical application to actual mathematics test data, the proposed model achieved better predictive performance than the DINA and DINO models. The obtained posteriors of the mixture weights were found to be heterogeneous among items, indicating key advantages of the proposed approach.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
The dataset is available at the TIMSS website (https://timssandpirls.bc.edu/TIMSS2007/idb_ug.html) together with the item information (https://timssandpirls.bc.edu/TIMSS2007/PDF/T07_Items.zip).
References
Bernardo JM, Smith AFM (2000) Bayesian theory. Wiley, Amsterdam
Carpenter B, Gelman A, Hoffman MD, Lee D, Goodrich B, Betancourt M, Riddell A (2017) Stan: a probabilistic programming language. J Stat Softw 76:1–32. https://doi.org/10.18637/jss.v076.i01
Chen Y, Liu J, Xu G, Ying Z (2015) Statistical analysis of Q-matrix based diagnostic classification models. J Am Stat Assoc 110:850–866. https://doi.org/10.1080/01621459.2014.934827
Culpepper SA (2015) Bayesian estimation of the DINA model with Gibbs sampling. J Educ Behav Stat 40:454–476. https://doi.org/10.3102/1076998615595403
de la Torre J (2009) A cognitive diagnosis model for cognitively based multiple-choice options. Appl Psychol Meas 33(3):163–183. https://doi.org/10.1177/0146621608320523
de la Torre J (2011) The generalized DINA model framework. Psychometrika 76:179–199. https://doi.org/10.1007/S11336-011-9207-7
de la Torre J, Lee Y-S (2010) A note on the invariance of the DINA model parameters. J Educ Meas 47:115–127. https://doi.org/10.1111/j.1745-3984.2009.00102.x
de la Torre J, Minchen N (2014) Cognitively diagnostic assessments and the cognitive diagnosis model framework. Psicologia Educativa 20:89–97. https://doi.org/10.1016/j.pse.2014.11.001
DiBello LV, Roussos LA, Stout W (2006) Review of cognitively diagnostic assessment and a summary of psychometric models. In: Rao CR, Sinharay S (eds) Handbook of statistics. Elsevier, Amsterdam. https://doi.org/10.1016/S0169-7161(06)26031-0
Foy P, Olson JF (2009) TIMSS 2007 user guide for the international database. Chestnut Hill, MA: TIMSS and PIRLS International Study Center, Boston College.
Gelman A, Rubin DB (1992) Inference from iterative simulation using multiple sequences. Stat Sci 7:457–472. https://doi.org/10.1214/ss/1177011136
Gelman A, Carlin JB, Stern HS, Dunson DB, Vehtari A, Rubin DB (2013) Bayesian data analysis, 3rd edn. Chapman and Hall/CRC, Boca Raton
Hartz S, Roussos L (2008) The fusion model for skills diagnosis: Blending theory with practice. ETS Res Rep Ser. https://scholar.google.com/scholar?hl=en&q=roussos+diagnostic+fusion&btnG=&as_sdt=1,30&as_sdtp=#5
Henson RA, Templin JL, Willse JT (2009) Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika 74:191–210. https://doi.org/10.1007/S11336-008
Hoffman MD, Gelman A (2014) The no-U-turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo. J Mach Learn Res 15: 1593–1623. https://jmlr.org/papers/volume15/hoffman14a/hoffman14a.pdf
Huo Y, de la Torre J (2014) Estimating a cognitive diagnostic model for multiple strategies via the EM algorithm. Appl Psychol Meas 38:464–485. https://doi.org/10.1177/0146621614533986
Jang EE (2009) Demystifying a Q-matrix for making diagnostic inferences about L2 reading skills. Lang Assess Q 6:210–238. https://doi.org/10.1080/15434300903071817
Jasra A, Holmes CC, Stephens DA (2005) Markov chain Monte Carlo methods and the label switching problem in Bayesian mixture modeling. Stat Sci 20:50–67. https://doi.org/10.1214/088342305000000016
Junker BW, Sijtsma K (2001) Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Appl Psychol Meas 25:258–272. https://doi.org/10.1177/01466210122032064
Lee MD, Wagenmakers E-J (2013) Bayesian cognitive modeling: a practical course. Cambridge University Press, New York
Lee Y-S, Park YS, Taylan D (2011) A cognitive diagnostic modeling of attribute mastery in Massachusetts, Minnesota, and the U.S. national sample using the TIMSS 2007. Int J Test 11:144–177. https://doi.org/10.1080/15305058.2010.534571
Li H, Hunter CV, Lei P-W (2016) The selection of cognitive diagnostic models for a reading comprehension test. Lang Test 33:391–409. https://doi.org/10.1177/0265532215590848
Liu Y, Douglas JA, Henson RA (2009) Testing person fit in cognitive diagnosis. Appl Psychol Meas 33:579–598. https://doi.org/10.1177/0146621609331960
Ma W, Guo W (2019) Cognitive diagnosis models for multiple strategies. Br J Math Stat Psychol 72:370–392. https://doi.org/10.1111/bmsp.12155
McLachlan G, Peel D (2000) Finite mixture models. Wiley, New York
McLachlan GJ, Lee SX, Rathnayake SI (2019) Finite mixture models. Ann Rev Stat Appl 6:355–375. https://doi.org/10.1146/annurev-statistics-031017-100325
Okada K, Lee MD (2016) A Bayesian approach to modeling group and individual differences in multidimensional scaling. J Math Psychol 70:35–44. https://doi.org/10.1016/j.jmp.2015.12.005
Park C, Cho S (2017) An exploratory analysis of compensatory cognitive diagnosis in EFL reading comprehension. Korea J Engl Lang Linguist 17: 85–104. https://www.riss.kr/link?id=A103044266
Park YS, Lee Y-S (2014) An extension of the DINA model using covariates: examining factors affecting response probability and latent classification. Appl Psychol Meas 38:376–390. https://doi.org/10.1177/0146621614523830
Ravand H (2016) Application of a cognitive diagnostic model to a high-stakes reading comprehension test. J Psychoeduc Assess 34:782–799. https://doi.org/10.1177/0734282915623053
Richardson S, Green PJ (1997) On Bayesian analysis of mixtures with an unknown number of components. J R Stat Soc B 59:731–792
Rojas G, de la Torre J, Olea J (2012) Choosing between general and specific cognitive diagnosis models when the sample size is small. Paper presented at the annual meeting of the National Council on Measurement in Education, Vancouver, British Columbia, Canada.
Rupp AA, Templin J, Henson RA (2010) Diagnostic measurement: theory, methods, and applications. The Guilford Press, New York, NY
Schwarz G (1978) Estimating the dimension of a model. Ann Stat 6:461–464. https://doi.org/10.1214/aos/1176344136
Stan Development Team (2018) RStan: the R interface to Stan [Computer software]. https://mc-stan.org/
Tatsuoka KK (1983) Rule space: an approach for dealing with misconceptions based on item response theory. J Educ Meas 20:345–354. https://doi.org/10.1111/j.1745-3984.1983.tb00212.x
Templin JL, Henson RA (2006) Measurement of psychological disorders using cognitive diagnosis models. Psychol Methods 11:287–305. https://doi.org/10.1037/1082-989X.11.3.287
Tijmstra J, Bolsinova M, Jeon M (2018) General mixture item response models with different item response structures: Exposition with an application to Likert scales. Behav Res Methods 50:2325–2344. https://doi.org/10.3758/s13428-017-0997-0
Tjoe H, de la Torre J (2014) The identification and validation process of proportional reasoning attributes: an application of a cognitive diagnosis modeling framework. Math Educ Res J 26:237–255. https://doi.org/10.1007/s13394-013-0090-7
Vehtari A, Gelman A, Gabry J (2017) Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Stat Comput 27(1413–1432):1–20. https://doi.org/10.1007/s11222-016-9696-4
Vehtari A, Gelman A, Gabry J (2018) loo: efficient leave-one-out cross-validation and WAIC for Bayesian models [Computer software]. https://cran.r-project.org/package=loo.
von Davier M (2008) A general diagnostic model applied to language testing data. Br J Math Stat Psychol 61:287–307. https://doi.org/10.1348/000711007X193957
Watanabe S (2010) Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory. J Mach Learn Res 11: 3571–3594. https://www.jmlr.org/papers/volume11/watanabe10a/watanabe10a.pdf
Yamaguchi K, Okada K (2018) Comparison among cognitive diagnostic models for the TIMSS 2007 fourth grade mathematics assessment. PLoS ONE 13:e0188691. https://doi.org/10.1371/journal.pone.0188691
Zhan P, Wang WC, Jiao H, Bian Y (2018) Probabilistic-input, noisy conjunctive models for cognitive diagnosis. Front Psychol 9:1–11. https://doi.org/10.3389/fpsyg.2018.00997
Zhan P, Jiao H, Liao M, Bian Y (2019a) Bayesian DINA modeling incorporating within-item characteristic dependency. Appl Psychol Meas 43:143–158. https://doi.org/10.1177/0146621618781594
Zhan P, Jiao H, Man K, Wang L (2019b) Using JAGS for Bayesian cognitive diagnosis modeling: a tutorial. J Educ Behav Stat 44:473–503. https://doi.org/10.3102/1076998619826040
Funding
This work was supported by JSPS Grant-in-Aid for JSPS Research Fellow 18J01312 and JSPS KAKANHI 17H04787, 19H00616.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
We have no conflicts of interest to declare.
Additional information
Communicated by Wim J. van der Linden.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix
Appendix
Data analysis code is available in Supplementary Material (https://osf.io/z53mw/).
About this article
Cite this article
Yamaguchi, K., Okada, K. Hybrid cognitive diagnostic model. Behaviormetrika 47, 497–518 (2020). https://doi.org/10.1007/s41237-020-00111-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41237-020-00111-x