Skip to main content
SpringerLink
Log in

Online Cognitive Diagnostic Assessment with Ordered Multiple-Choice Items for Word Problems involving ‘Time’

  • Published:
Education and Information Technologies Aims and scope Submit manuscript

Abstract

Solving word problems involving 'Time' is an important skill but poor mastery of the skill among elementary students has often been reported in the literature. In addition, the available diagnostic tools in the literature might be less efficient for identifying the various errors made by many students in solving word problems. Thus, an online problem solving Cognitive Diagnostic Assessment (CDA) with Ordered Multiple-Choice (OMC) items was developed as a web application with automated scoring features to increase the efficiency in assessing Grade Five students’ mastery level of word problem-solving attributes involving the topic of Time. The online problem-solving CDA with OMC items was developed based on the principled assessment design which comprised four building blocks: (i) construct map, (ii) item design, (iii) outcome space, and (iv) measurement model. Newman's Error Analysis was adapted to specify the construct map required for the development of OMC items. In this article, we documented the empirical evidence regarding item quality, the validity of the cognitive models, the reliability of the instrument, and the diagnostic analysis of students' responses. This study involved 128 Grade Five students in three elementary schools. The results indicated that the instrument showed appropriate difficulty level and discrimination power, satisfactory model-data fit, and high reliability, but most of the students were not at the highest mastery level of the word problem-solving attributes. The findings of the study suggested that the web application could be a valid and reliable diagnostic tool for pinpointing students' errors made in solving word problems. Besides, the findings also highlight the need for intervention to enhance the students’ mastery of word problem-solving attributes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  • Alonzo, A. C., & Steedle, J. T. (2009). Developing and assessing a force and motion learning progression. Science Education, 93(3), 389–421.

    Article  Google Scholar 

  • Alves, C. B. (2012). Making diagnostic inferences about student performance on the Alberta education diagnostic mathematics project: An application of the Attribute Hierarchy Method. (Publication No. 919011661) [Doctoral Thesis, University of Alberta, Ann Arbor, Canada]. ProQuest Dissertations and Theses database.

  • Authors. (2019). International Journal of Science and Mathematics Education.

  • Authors. (2021). Journal of Computers in Education.

  • Bardhoshi, G., & Erford, B. T. (2017). Processes and procedures for estimating score reliability and precision. Measurement and Evaluation in Counseling and Development, 50(4), 256–263.

    Article  Google Scholar 

  • Boonen, A. J., de Koning, B. B., Jolles, J., & van der Schoot, M. (2016). Word problem solving in contemporary math education: A plea for reading comprehension skills training. Frontiers in Psychology, 7(191), 1–10.

    Google Scholar 

  • Boora, S., Pasiphol, S., & Tangdhanakanond, K. (2015). Development of cognitive diagnostic testing on basic arithmetic operation. Procedia-Social and Behavioral Sciences, 191, 769–772.

    Article  Google Scholar 

  • Bradshaw, L. (2017). Diagnostic classification models. In A. A. Rupp & J. P. Leighton (Eds.), the handbook of cognition and assessment: Frameworks, methodologies, and applications (1st ed., pp. 297–327). Wiley Blackwell.

  • Briggs, D. C., & Alonzo, A. (2012). The psychometric modeling of ordered multiple-choice item responses for diagnostic assessment with a learning progression. In A. C. Alonzo & A. W. Gotwals (Eds.), Learning progressions in science: Current challenges and future directions (pp. 293–316). Sense Publishers.

    Chapter  Google Scholar 

  • Briggs, D. C., Alonzo, A., Schwab, C., & Wilson, M. (2006). Diagnostic assessment with ordered multiple-choice items. Educational Assessment, 11(1), 33–63.

    Article  Google Scholar 

  • Broaddus, A. E. (2011). An investigation into foundational concepts related to slope: An application of the Attribute Hierarchy Method (UMI No: 3487353) [Doctoral Thesis, University of Kansas, Kansas, United State]. ProQuest Dissertations and Theses database.

  • Cai, J., Morris, A., Hohensee, C., Hwang, S., Robison, V., & Hiebert, J. (2018). Using data to understand and improve students' learning: Empowering teachers and researchers through building and using a knowledge base. Journal for Research in Mathematics Education, 49(4), 362–372.

    Article  Google Scholar 

  • Clarkson, P. C. (1991). Language comprehension errors: A further investigation. Mathematics Education Research Journal, 3(2), 24–33.

    Article  Google Scholar 

  • Cui, Y., & Leighton, J. P. (2009). The Hierarchy Consistency Index: Evaluating person fit for cognitive diagnostic assessment. Journal of Educational Measurement, 46(4), 429–449.

    Article  Google Scholar 

  • De Champlain, A. F. (2010). A primer on classical test theory and item response theory for assessments in medical education. Medical Education, 44(1), 109–117.

    Article  Google Scholar 

  • Downing, S. M. (2004). Reliability: on the reproducibility of assessment data. Medical education, 38(9), 1006–1012.

    Article  Google Scholar 

  • Ebel, R. L. & Frisbie, D.A. (1991). Essentials of educational measurement (5th ed.). Prentice-Hall.

  • Fuchs, L. S., Fuchs, D., Seethaler, P. M., & Craddock, C. (2020a). Improving language comprehension to enhance word-problem solving. Reading & Writing Quarterly, 36(2), 142–156.

    Article  Google Scholar 

  • Fuchs, L., Fuchs, D., Seethaler, P. M., & Barnes, M. A. (2020b). Addressing the role of working memory in mathematical word-problem solving when designing intervention for struggling learners. ZDM, 52(1), 87–96.

    Article  Google Scholar 

  • Fulmer, G. W. (2015). Validating proposed learning progressions on force and motion using the force concept inventory: Findings from Singapore secondary schools. International Journal of Science and Mathematics Education, 13(6), 1235–1254.

    Article  Google Scholar 

  • Gay, L. R., Mills, G. E., & Airasian, P. W. (2012). Educational research: Competencies for analysis and applications (10th ed.). Merrill.

  • George, D., & Mallery, P. (2019). IBM SPSS statistics 26 step by step: A simple guide and reference. Routledge.

  • Gierl, M. J., & Zhou, J. (2008). Computer adaptive-attribute testing: A new approach to cognitive diagnostic assessment. Zeitschrift für Psychologie/Journal of Psychology, 216(1), 29–39.

    Article  Google Scholar 

  • Gierl, M. J., Alves, C., & Taylor-Majeau, R. (2010). Using the Attribute Hierarchy Method to make diagnostic inferences about examinees’ knowledge and skills in mathematics: An operational implementation of cognitive diagnostic assessment. International Journal of Testing, 10(4), 318–341.

    Article  Google Scholar 

  • Gierl, M. J., Cui, Y., & Zhou, J. (2009a). Reliability and attribute-based scoring in cognitive diagnostic assessment. Journal of Educational Measurement, 46(3), 293–313.

    Article  Google Scholar 

  • Gierl, M. J., Roberts, M. P. R., Alves, C., & Gotzmann, A. (2009b). Using judgments from content specialists to develop cognitive models for diagnostic assessments. Paper presented at the Annual Meeting of National Council on Measurement in Education, San Diego, CA.

  • Hadenfeldt, J. C., Bernholt, S., Liu, X., Neumann, K., & Parchmann, I. (2013). Using ordered multiple-choice items to assess students’ understanding of the structure and composition of matter. Journal of Chemical Education, 90(12), 1602–1608.

    Article  Google Scholar 

  • Hadenfeldt, J. C., Neumann, K., Bernholt, S., Liu, X., & Parchmann, I. (2016). Students’ progression in understanding the matter concept. Journal of Research in Science Teaching, 53(5), 683–708.

    Article  Google Scholar 

  • Haladyna, T. M. (1992). Context-dependent item sets. Educational Measurement: Issues and Practice, 11(1), 21–25.

    Article  Google Scholar 

  • Haladyna, T. M. (2004). Developing and validating multiple-choice test items (3rd ed.). Routledge.

    Book  Google Scholar 

  • Hassan, F. N. A., Mohamad, Q. A., & Azuddin, S. K. Y. (2020). The implementation of higher order thinking skills (HOTS) in Malaysia secondary school: Post PISA 2009. International Journal of Psychosocial Rehabilitation, 24(5), 5510–5517.

    Article  Google Scholar 

  • Ihudiebube-Splendor, C. N., & Chikeme, P. C. (2020). A descriptive cross-sectional study: Practical and feasible design in investigating health care–seeking behaviors of undergraduates.In SAGE Research Methods Cases. https://doi.org/10.4135/9781529742862

  • Keehner, M., Gorin, J. S., Feng, G., & Katz, I. R. (2017). Developing and validating cognitive models in assessment. In A. A. Rupp & J. P. Leighton (Eds.), The handbook of cognition and assessment: Frameworks, methodologies, and applications (1st ed., pp. 75-101). John Wiley & Sons.

  • LaDuca, A., Downing, S. M., & Henzel, T. R. (1995). Systematic item writing and test construction. In J. C. Impara (Ed.), Licensure testing: Purposes, procedures, and practices (pp. 117-149). Buros.

  • Lai, H., Gierl, M. J., & Babenko, O. (2015). Application of conditional means for diagnostic scoring. International Journal of Learning, Teaching and Educational Research, 12(3), 61–79.

    Google Scholar 

  • Leighton, J. P., & Gierl, M. J. (2011). The learning sciences in educational assessment: The role of cognitive models. Cambridge University Press.

    Book  Google Scholar 

  • Leighton, J. P., Gierl, M. J., & Hunka, S. M. (2004). The Attribute Hierarchy Method for cognitive assessment: A variation on Tatsuoka's Rule-Space Approach. Journal of Educational Measurement, 41(3), 205–237.

    Article  Google Scholar 

  • Leong, Y. H., Tay, E. G., Toh, T. L., Quek, K. S., Toh, P. C., & Dindyal, J. (2016). Infusing mathematical problem solving in the mathematics curriculum: Replacement Units. In P. Felmer, E. Pehkonen, & J. Kilpatrick (Eds.), Posing and solving mathematical problems: Advances and new perspective (pp. 309–325). Springer.

    Chapter  Google Scholar 

  • Liljedahl, P., Santos-Trigo, M., Malaspina, U., & Bruder, R. (2016). Problem solving in mathematics education. Springer.

    Book  Google Scholar 

  • Lithner, J. (2008). A research framework for creative and imitative reasoning. Educational Studies in Mathematics, 67(3), 255–276.

    Article  Google Scholar 

  • Ministry of Education Malaysia [MOE] (2016). Kurikulum standard sekolah rendah: Standard kurikulum dan pentaksiran Matematik Tahun 4 [Primary school standard curriculum: Year 4 Mathematics curriculum and assessment standard document]. Ministry of Education Malaysia.

  • Newman, M. A. (1977). An analysis of sixth-grade pupil's error on written mathematical tasks. Victorian Institute for Educational Research Bulletin, 39, 31–43.

    Google Scholar 

  • Nichols, P. D., Kobrin, J. L., Lai, E., & Koepfler, J. D. (2017). The role of theories of learning and cognition in assessment design and development. In A. A. Rupp & J. P. Leighton (Eds.), The handbook of cognition and assessment: Frameworks, methodologies, and applications (1st ed., pp. 41–74). Wiley Blackwell.

  • O’Connor, B. R., & Norton, S. (2020). Supporting indigenous primary students’ success in problem-solving: Learning from Newman interviews. Mathematics Education Research Journal, 1-24.

  • Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what's being reported? Critique and recommendations. Research in Nursing & Health, 29(5), 489–497.

    Article  Google Scholar 

  • Quaigrain, K., & Arhin, A. K. (2017). Using reliability and item analysis to evaluate a teacher-developed test in educational measurement and evaluation. Cogent Education, 4(1), 1–11.

    Article  Google Scholar 

  • Roberts, M. R., Alves, C. B., Chu, M. W., Thompson, M., Bahry, L. M., & Gotzmann, A. (2014). Testing expert based versus student based cognitive models for a Grade 3 diagnostic mathematics assessment. Applied Measurement in Education, 27(3), 173–195.

    Article  Google Scholar 

  • Rohmah, M., & Sutiarso, S. (2018). Analysis problem solving in mathematical using theory Newman. Eurasia Journal of Mathematics, Science and Technology Education, 14(2), 671–681.

    Google Scholar 

  • Salkind, N. (2010) Convenience sampling. In N. Salkind (Ed.), Encyclopedia of research design (p. 254). Sage.

  • Scheibling-Seve, C., Pasquinelli, E., & Sander, E. (2020). Assessing conceptual knowledge through solving arithmetic word problems. Educational Studies in Mathematics, 1-19.

  • Schultz, M., Lawrie, G. A., Bailey, C. H., Bedford, S. B., Dargaville, T. R., O'Brien, G., .Tasker, R., Thompson, C. D., Williams, M. & Wright, A. H. (2017). Evaluation of diagnostic tools that tertiary teachers can apply to profile their students’ conceptions. International Journal of Science Education, 39(5), 565–586.

    Article  Google Scholar 

  • Scully, D. (2017). Constructing multiple-choice items to measure higher-order thinking. Practical Assessment, Research, and Evaluation, 22(1), 1–4.

    MathSciNet  Google Scholar 

  • Sia, C. J. L. (2017). Development and validation of Cognitive Diagnostic Assessment (CDA) for primary mathematics learning of time [Unpublished master's thesis]. Universiti Sains Malaysia, Malaysia.

  • Sia, C. J. L., & Lim, C. S. (2018). Cognitive diagnostic assessment: An alternative mode of assessment for learning. In D. R. Thompson, M. Burton, A. Cusi, & D. Wright (Eds.), Classroom assessment in mathematics (pp. 123–137). Springer.

    Google Scholar 

  • Sibanda, L. (2017). Grade 4 learners’ linguistic difficulties in solving mathematical assessments. African Journal of Research in Mathematics, Science and Technology Education, 21(1), 86–96.

    Article  MathSciNet  Google Scholar 

  • Tan, P. L., Lim, C. S., & Kor, L. K. (2017). Diagnosing primary pupils' learning of the concept of" after" in the topic "Time" through knowledge states by using cognitive diagnostic assessment. Malaysian Journal of Learning and Instruction, 14(2), 145–175.

    Article  Google Scholar 

  • Tavakol, M., & Dennick, R. (2011). Post-examination analysis of objective tests. Medical Teacher, 33(6), 447–458.

    Article  Google Scholar 

  • Thompson, N. A. (2010). KR-20. In N. Salkind (Ed.), Encyclopedia of research design (pp. 667–668). Sage.

  • Verschaffel, L., Van Dooren, W., & De Smedt, B. (2012). Mathematical Learning. In N. M. Seel (Eds.), Encyclopedia of the sciences of learning (pp. 2107–2110). Springer.

  • White, A. L. (2005). Active mathematics in classrooms: Finding out why children make mistakes-and then doing something to help them. Square One, 15(4), 15–19.

    Google Scholar 

  • White, A. L. (2009). A revaluation of Newman's error analysis. In D. Martin, T. Fitzpatrick, R. Hunting, D. Itter, C. Lenard, T. Mills, L. Milne (Eds), Mathematics of prime importance, Proceedings of 46th Annual Conference of the Mathematics Association of Victoria (pp. 249-257). M.A.V.

  • White, A. L. (2018). Australian Mathematics Literacy Perspectives: Assessment. Southeast Asian Mathematics Education Journal, 8(1), 69–81.

    Article  Google Scholar 

  • Wijaya, A., van den Heuvel-Panhuizen, M., Doorman, M., & Robitzsch, A. (2014). Difficulties in solving context-based PISA mathematics tasks: An analysis of students' errors. The Mathematics Enthusiast, 11(3), 555–584.

    Article  Google Scholar 

  • Wilson, M. (2009). Measuring progressions: Assessment structures underlying a learning progression. Journal of Research in Science Teaching, 46(6), 716–730.

    Article  Google Scholar 

Download references

Acknowledgements

This study is made possible with funding from the Research University Grant (RUI) Scheme 1001/PGURU/8011027 of Universiti Sains Malaysia. The authors would like to thank the computer science experts, the web developer, and the technician from Centre for Knowledge, Communication and Technology, USM who involved in the Online CDA web application development process. The authors would also like to thank all the teachers and students who voluntarily participated in this study.

Availability of data and material

Not applicable.

Code availability

Not applicable.

Funding

This study is made possible with funding from the Research University Grant (RUI) Scheme 1001/PGURU/8011027 of Universiti Sains Malaysia, Penang, Malaysia.

Author information

Authors and Affiliations

  1. School of Educational Studies, Universiti Sains Malaysia, 11800 USM, Penang, Malaysia

    Chin Huan & Chew Cheng Meng

Authors
  1. Chin Huan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chew Cheng Meng
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by Chin Huan and Chew Cheng Meng. The first draft of the manuscript was written by Chin Huan and Chew Cheng Meng commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Chew Cheng Meng.

Ethics declarations

Conflicts of interest/Competing interests

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Word Problem Solving Construct Map

Level

Descriptor

0

Does not master any attribute.

1

Master attribute:

RU: Read and understand the word problem

2

Master attributes:

RU: Read and understand the word problem, and

T: Transform the word problem into correct mathematical sentences

3

Master attributes:

RU: Read and understand the word problem, and

T: Transform the word problem into correct mathematical sentences, and

P: Perform mathematical operations necessary to solve the word problem after the transformation process

4

Master attributes:

RU: Read and understand the word problem, and

T: Transform the word problem into correct mathematical sentences, and

P: Perform mathematical operations necessary to solve the word problem after the transformation process, and

E: Encode the answer in an acceptable form

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chin, H., Chew, C.M. Online Cognitive Diagnostic Assessment with Ordered Multiple-Choice Items for Word Problems involving ‘Time’. Educ Inf Technol 27, 7721–7748 (2022). https://doi.org/10.1007/s10639-022-10956-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10639-022-10956-2

Keywords

Search

Navigation