Abstract
The use of e-assessments has gained significant prominence in educational settings as they offer a range of benefits such as improved scalability, accessibility, and cost-effectiveness. This chapter discusses the possibilities of addressing diversity in educational e-assessments. We propose a workaround using assessment analytics to identify test items prone to diversity-related performance differences. Based on this, we offer theory-guided suggestions on how diversity-appropriate assessments could be realized. In a proof of concept using data collected in a mock e-assessment, we demonstrate a workaround for identifying potential diversity biases on the item-level with small sample sizes. To put this into practice, a technical infrastructure using Experience API is proposed. Based on established cognitive theories focusing on learning processes, it is outlined how the flexibility of e-assessments regarding item presentation can be used to address the needs and preferences of diverse test takers. The chapter concludes by highlighting the potential of e-assessments to address diversity and proposes perspectives for research and practical application.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
In this chapter item is used synonymously for a question or task in a test.
- 2.
For data and analyses see https://osf.io/f85e9/?view_only=6dd08dabf2214124b014fc3683c3baf5.
- 3.
Students consented to the use of their data for the stated study purpose and were able to withdraw from the survey at any time without giving a reason. Book vouchers were raffled off to increase the incentive to participate.
- 4.
xAPI-Spec, https://github.com/adlnet/xAPI-Spec, last accessed: 13.07.2023.
- 5.
- 6.
Translation programs like Google Translate© or DeepL© could be incorporated into examination systems.
- 7.
The test material can be tested for these properties on the internet page: https://contrastchecker.com/.
References
Adenuga, K. I., Mbarika, V. W., & Omogbadegun, Z. O. (2019). Technical support: Towards mitigating effects of computer anxiety on acceptance of e-assessment amongst university students in Sub Saharan African countries. In Y. Dwivedi, E. Ayaburi, R. Boateng, & J. Effah (Eds.), ICT unbounded, social impact of bright ICT adoption (pp. 48–72). Springer. https://doi.org/10.1007/978-3-030-20671-0_5
Al-Azawei, A., Baiee, W. R., & Mohammed, M. A. (2019). Learners’ experience towards e-assessment tools: A comparative study on virtual reality and moodle quiz. International Journal of Emerging Technologies in Learning (IJET), 14(05), Article 05. https://doi.org/10.3991/ijet.v14i05.9998
Alruwais, N. (2018). Advantages and challenges of using e-assessment. International Journal of Information and Education Technology, 8, 34–37. https://doi.org/10.18178/ijiet.2018.8.1.1008
Appiah, M., & Tonder, F. (2018). E-assessment in higher education: A review. International Journal of Business Management and Economic Research, 9, 1454–1460.
Aristarkhova, M. (2023). Developement of foreign language competence through multimedia technologies. Collection of scientific papers “SCIENTIA”, (May 19, 2023; Berlin, Germany), pp. 99–100. https://previous.scientia.report/index.php/archive/article/view/976
Berking, P. (2015). Choosing a learning record store (LRS). Advanced Distributed Learning (ADL) Initiative. https://doi.org/10.13140/RG.2.1.1652.7763
Blau, I., Weiser, O., & Eshet-Alkalai, Y. (2017). How do medium naturalness and Personality traits shape academic achievement and perceived learning? An experimental study of face-to-face and synchronous E-Learning. Research in Learning Technology, 25. https://doi.org/10.25304/rlt.v25.1974
Bottiroli, S., & Cavallini, E. (2009). Can computer familiarity regulate the benefits of computer-based memory training in normal aging? A study with an Italian sample of older adults. Neuropsychology, Development, and Cognition. Section B, Aging, Neuropsychology and Cognition, 16(4), 401–418. https://doi.org/10.1080/13825580802691763
Breuer, M., Brocker, A., Persike, M., & Schroeder, U. (2023). AxEL—Eine modulare Softwarekomponente für ein dediziertes E-Prüfungssystem zur Generierung von xAPI-Statements für Assessment Analytics (AxEL—A modular software component for a dedicated e-assessment system to generate xAPI statements for assessment analytics). In R. Röpke & U. Schroeder (Eds.), 21. Fachtagung Bildungstechnologien (DELFI) (pp. 91–96). Gesellschaft für Informatik.
Brinckerhoff, L. C., & Banerjee, M. (2007). Misconceptions regarding accommodations on high-stakes tests: Recommendations for preparing disability documentation for test takers with learning disabilities. Learning Disabilities Research & Practice, 22(4), 246–255. https://doi.org/10.1111/j.1540-5826.2007.00253.x
Brom, C., Stárková, T., & D’Mello, S. K. (2018). How effective is emotional design? A meta-analysis on facial anthropomorphisms and pleasant colors during multimedia learning. Educational Research Review, 25, 100–119. https://doi.org/10.1016/j.edurev.2018.09.004
Brown, G., & Harris, L. (2016). Handbook of human and social conditions in assessment. Routledge.
Camilli, G. (2013). Ongoing issues in test fairness. Educational Research and Evaluation, 19(2–3), 104–120. https://doi.org/10.1080/13803611.2013.767602
Chamorro-Premuzic, T., & Furnham, A. (2002). Neuroticism and “special treatment” in university examinations. Social Behavior and Personality: An International Journal, 30(8), 807–811. https://doi.org/10.2224/sbp.2002.30.8.807
Cizek, G. J. (Ed.). (2011). Setting performance standards: Foundations, methods, and innovations. Routledge. https://doi.org/10.4324/9780203848203
Collis, B., & Moonen, J. (2002). Flexible learning in a digital world. Open Learning: The Journal of Open, Distance and E-Learning, 17(3), 217–230. https://doi.org/10.1080/0268051022000048228
De Jong, T., Sotiriou, S., & Gillet, D. (2014). Innovations in STEM education: The Go-Lab federation of online labs. Smart Learning Environments, 1(1), 3. https://doi.org/10.1186/s40561-014-0003-6
De Soete, B., Lievens, F., & Druart, C. (2012). An update on the diversity-validity dilemma in personnel selection: A review. Psihologijske Teme, 21(3), 399–424.
De Soete, B. D., Lievens, F., & Druart, C. (2013). Strategies for dealing with the diversity-validity dilemma in personnel selection: Where are we and where should we go? Revista de Psicología Del Trabajo y de Las Organizaciones, 29(1), 3–12. https://doi.org/10.5093/tr2013a2
Dirkx, K. J. H., Skuballa, I., Manastirean-Zijlstra, C. S., & Jarodzka, H. (2021). Designing computer-based tests: Design guidelines from multimedia learning studied with eye tracking. Instructional Science, 49(5), 589–605. https://doi.org/10.1007/s11251-021-09542-9
Dittrich, S., Weiss, Z., Schröter, H., & Meurers, D. (2019). Integrating large-scale web data and curated corpus data in a search engine supporting German literacy education. In Proceedings of the 8th workshop on NLP for computer assisted language learning, pp. 41–56. https://aclanthology.org/W19-6305
Dronkers, J., & van der Velden, R. (2012). Positive but also negative effects of ethnic diversity in schools on educational performance? An empirical test using PISA data. CReAM Discussion Paper Series, Article 1211. https://ideas.repec.org//p/crm/wpaper/1211.html
Dube, T., & Ma, M. (2010). A flexible e-assessment system inspired by design methodology management to accommodate diverse learning styles. International Journal for Digital Society, 1(1), 44–52. https://doi.org/10.20533/ijds.2040.2570.2010.0008
Ehlenz, M., Heinemann, B., & Schroeder, U. (2022). Information sources and their potential for multimodal learning analytics in laboratory-based learning. In D. Uckelmann, G. Romagnoli, J. Baalsrud Hauge, & V. Kammerlohr (Eds.), Online-labs in education (pp. 139–152). Nomos Verlagsgesellschaft mbH & Co. KG. https://doi.org/10.5771/9783957104106-139
Eng, T. S. (2005). The impact of ICT on learning: A review of research. International Education Journal, 6(5), 635–650.
Farzin, S., & Mohamed Dahlan, H. (2018). An empirical study of the behavioral factors affecting students’ perception on adopting electronic assessment system. Journal of Theoretical and Applied Information Technology, 96(1), Article 1.
Froehlich, L., Sassenberg, K., Jonkmann, K., Scheiter, K., & Stürmer, S. (2023). Student diversity and e-exam acceptance in higher education. Journal of Computer Assisted Learning, 39(4), 1196–1210. https://doi.org/10.1111/jcal.12794
Gewin, V. (2014). Diversity: Equal access. Nature, 511, 499–500. https://doi.org/10.1038/nj7510-499a
Glaser, M., & Schwan, S. (2015). Explaining pictures: How verbal cues influence processing of pictorial learning material. Journal of Educational Psychology, 107(4), 1006–1018. https://doi.org/10.1037/edu0000044
Goldman, N. (2013). Rewordify.com | Understand what you read. Retrieved July 13, 2023, from https://rewordify.com/
Grassi, A., Gaggioli, A., & Riva, G. (2011). New technologies to manage exam anxiety. In Annual review of cybertherapy and telemedicine 2011 (pp. 57–62). IOS Press. https://doi.org/10.3233/978-1-60750-766-6-57
Green, A., & O’Sullivan, B. (2011). Test taker characteristics. Cambridge University Press. https://uobrep.openrepository.com/handle/10547/238220
Hammer, M., Scheiter, K., & Stürmer, K. (2021). New technology, new role of parents: How parents’ beliefs and behavior affect students’ digital media self-efficacy. Computers in Human Behavior, 116, 106642. https://doi.org/10.1016/j.chb.2020.106642
Hoebel, J., Müters, S., Kuntz, B., Lange, C., & Lampert, T. (2015). Messung des subjektiven sozialen Status in der Gesundheitsforschung mit einer deutschen Version der MacArthur Scale (Measurement of subjective social status in health research using a German version of the MacArthur Scale). Bundesgesundheitsblatt—Gesundheitsforschung—Gesundheitsschutz, 58(7), 749–757. https://doi.org/10.1007/s00103-015-2166-x
Hu, L., Chen, G., Li, P., & Huang, J. (2021). Multimedia effect in problem solving: A meta-analysis. Educational Psychology Review, 33(4), 1717–1747. https://doi.org/10.1007/s10648-021-09610-z
Ifenthaler, D., & Greiff, S. (2021). Leveraging learning analytics for assessment and feedback. In J. Liebowitz (Ed.), Online learning analytics (pp. 1–18). Auerbach Publications. https://doi.org/10.1201/9781003194620
Imus, A., Schmitt, N., Kim, B., Oswald, F. L., Merritt, S., & Wrestring, A. F. (2011). Differential item functioning in biodata: Opportunity access as an explanation of gender- and race-related DIF. Applied Measurement in Education, 24(1), 71–94. https://doi.org/10.1080/08957347.2011.532412
Jaap, A., Dewar, A., Duncan, C., Fairhurst, K., Hope, D., & Kluth, D. (2021). Effect of remote online exam delivery on student experience and performance in applied knowledge tests. BMC Medical Education, 21(1), 86. https://doi.org/10.1186/s12909-021-02521-1
Jansen, A. E., & Searle, B. J. (2021). Diverse effects of team diversity: A review and framework of surface and deep-level diversity. Personnel Review, 50(9), 1838–1853. https://doi.org/10.1108/PR-12-2019-0664
Judel, S., & Schroeder, U. (2022). EXCALIBUR LA—An extendable and scalable infrastructure build for learning analytics. In 2022 International conference on advanced learning technologies (ICALT), pp. 155–157. https://doi.org/10.1109/ICALT55010.2022.00053
Kaczorowski, T., McMahon, D., Gardiner-Walsh, S., & Hollingshead, A. (2023). Designing an inclusive future: Including diversity and equity with innovations in special education technology. Teaching Exceptional Children, 55(5), 376–383. https://doi.org/10.1177/00400599221090506
Karami, H., & Mok, M. M. C. (2016). Introduction: Fairness issues in educational assessment. In H. Karami (Ed.), Fairness issues in educational assessment. Routledge.
Kasim, N., & Khalid, F. (2016). Choosing the right learning management system (LMS) for the higher education institution context: A systematic review. International Journal of Emerging Technologies in Learning (IJET), 11, 55. https://doi.org/10.3991/ijet.v11i06.5644
King-Sears, M. (2009). Universal design for learning: Technology and pedagogy. Learning Disability Quarterly, 32(4), 199–201. https://doi.org/10.2307/27740372
Kirschner, P. A., Park, B., Malone, S., & Jarodzka, H. (2017). Toward a cognitive theory of multimedia assessment (CTMMA). In M. J. Spector, B. B. Lockee, & M. D. Childress (Eds.), Learning, design, and technology: An international compendium of theory, research, practice, and policy (pp. 1–23). Springer. https://doi.org/10.1007/978-3-319-17727-4_53-1
Klašnja-Milićević, A., Vesin, B., Ivanović, M., Budimac, Z., & Jain, L. C. (2017). Introduction to e-learning systems. In A. Klašnja-Milićević, B. Vesin, M. Ivanović, Z. Budimac, & L. C. Jain (Eds.), E-learning systems: Intelligent techniques for personalization (pp. 3–17). Springer. https://doi.org/10.1007/978-3-319-41163-7_1
Kunnan, A. J. (2004). Test fairness. European Language Testing in a Global Context, 18, 27–48.
Küppers, B., Kerber, F., Meyer, U., & Schroeder, U. (2017). Beyond lockdown: Towards reliable e-assessment. Gesellschaft für Informatik. https://dl.gi.de/handle/20.500.12116/4841
Ladonlahti, T., Laamanen, M., & Uotinen, S. (2020). Ensuring diverse user experiences and accessibility while developing the TeSLA e-assessment system. In D. Baneres, M. E. Rodríguez, & A. E. Guerrero-Roldán (Eds.), Engineering data-driven adaptive trust-based e-assessment systems: Challenges and infrastructure solutions (pp. 213–238). Springer. https://doi.org/10.1007/978-3-030-29326-0_10
Lauring, J., & Selmer, J. (2010). Is university internationalization bad for performance? Examining two different types of diversity. International Journal of Educational Research, 49(4), 161–171. https://doi.org/10.1016/j.ijer.2011.02.002
Lee, C. D. (2003). Toward a framework for culturally responsive design in multimedia computer environments: Cultural modeling as a case. Mind, Culture, and Activity, 10(1), 42–61. https://doi.org/10.1207/S15327884MCA1001_05
Lei, P.-W., & Li, H. (2013). Small-sample DIF estimation using SIBTEST, cochran’s Z, and log-linear smoothing. Applied Psychological Measurement, 37(5), 397–416. https://doi.org/10.1177/0146621613478150
Lenhard, W., & Lenhard, A. (2011). Berechnung des Lesbarkeitsindex LIX nach Björnson. Psychometrica. https://doi.org/10.13140/RG.2.1.1512.3447
Lievens, F., Sackett, P. R., Dahlke, J. A., Oostrom, J. K., & Soete, B. D. (2019). Constructed response formats and their effects on minority-majority differences and validity. Journal of Applied Psychology, 104(5), 715–726. https://doi.org/10.1037/apl0000367
Lindner, M. A. (2020). Representational and decorative pictures in science and mathematics tests: Do they make a difference? Learning and Instruction, 68, 101345. https://doi.org/10.1016/j.learninstruc.2020.101345
Lindner, M. A. (2021). Principles for educational assessment with multimedia. In R. E. Mayer & L. Fiorella (Eds.), The Cambridge handbook of multimedia learning (3rd ed., pp. 552–565). Cambridge University Press. https://doi.org/10.1017/9781108894333.055
Lindner, M. A., Lüdtke, O., Grund, S., & Köller, O. (2017). The merits of representational pictures in educational assessment: Evidence for cognitive and motivational effects in a time-on-task analysis. Contemporary Educational Psychology, 51, 482–492. https://doi.org/10.1016/j.cedpsych.2017.09.009
Liu, H.-J. (2012). Understanding EFL undergraduate anxiety in relation to motivation, autonomy, and language proficiency. Electronic Journal of Foreign Language Teaching, 9, 123–139.
Mather, N., & Abu-Hamour, B. (2013). Individual assessment of academic achievement. In K. F. Geisinger, B. A. Bracken, J. F. Carlson, J.-I. C. Hansen, N. R. Kuncel, S. P. Reise, & M. C. Rodriguez (Eds.), APA handbook of testing and assessment in psychology, Vol. 3: Testing and assessment in school psychology and education (pp. 101–128). American Psychological Association. https://doi.org/10.1037/14049-005
Matthíasdóttir, Á., & Arnalds, H. (2016). E-assessment: Students’ point of view. In Proceedings of the 17th international conference on computer systems and technologies 2016, pp. 369–374. https://doi.org/10.1145/2983468.2983497
Mayer, R. E. (2002). Multimedia learning. In B. H. Ross (Ed.), Psychology of learning and motivation (Vol. 41, pp. 85–139). Academic. https://doi.org/10.1016/S0079-7421(02)80005-6
Mayer, R. E. (2008). Applying the science of learning: Evidence-based principles for the design of multimedia instruction. The American Psychologist, 63(8), 760–769. https://doi.org/10.1037/0003-066X.63.8.760
Mayer, R. E. (Ed.). (2014). The Cambridge handbook of multimedia learning (2nd ed.). Cambridge University Press. https://doi.org/10.1017/CBO9781139547369
Mayer, R. E., & Moreno, R. (2002). Animation as an aid to multimedia learning. Educational Psychology Review, 14(1), 87–99. https://doi.org/10.1023/A:1013184611077
Miller, C. (2011). Aesthetics and e-assessment: The interplay of emotional design and learner performance. Distance Education, 32(3), 307–337. https://doi.org/10.1080/01587919.2011.610291
Mislevy, R. J., Haertel, G., Cheng, B. H., Ructtinger, L., DeBarger, A., Murray, E., ... & Vendlinski, T. (2016). A “conditional” sense of fairness in assessment. In Fairness issues in educational assessment (pp. 33–52). Routledge.
Moreno, R., & Mayer, R. (2007). Interactive multimodal learning environments. Educational Psychology Review, 19(3), 309–326. https://doi.org/10.1007/s10648-007-9047-2
OECD. (2018). Equity in education: Breaking down barriers to social mobility. Organisation for Economic Cooperation and Development. https://doi.org/10.1787/9789264073234-en
Park, S., Ryu, J., & McChesney, K. (2019). Collaborative studio experiences between South Korean and American pre-service teachers: A case study of designing culturally-responsive virtual classroom simulation. TechTrends, 63(3), 271–283. https://doi.org/10.1007/s11528-019-00392-4
Pellegrino, J. (2005). The challenge of knowing what students know. Measurement: Interdisciplinary Research and Perspectives, 1(2), 7–11.
Penfield, R. D., & Camilli, G. (2006). Differential item functioning and item bias. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics (Vol. 26, pp. 125–167). Elsevier. https://doi.org/10.1016/S0169-7161(06)26005-X
Plass, J. L., Heidig, S., Hayward, E. O., Homer, B. D., & Um, E. (2014). Emotional design in multimedia learning: Effects of shape and color on affect and learning. Learning and Instruction, 29, 128–140. https://doi.org/10.1016/j.learninstruc.2013.02.006
Rose, D. (2000). Universal design for learning. Journal of Special Education Technology, 15(4), 47–51. https://doi.org/10.1177/016264340001500407
Sackett, P. R., Kuncel, N. R., Arneson, J. J., Cooper, S. R., & Waters, S. D. (2009). Does socioeconomic status explain the relationship between admissions tests and post-secondary academic performance? Psychological Bulletin, 135(1), 1–22. https://doi.org/10.1037/a0013978
Sanadgol, F., & Abdolmanafi-Rokni, S. J. (2015). Anxiety, motivation and autonomy in Iranian high school students: A quantitative study. Advances in Language and Literary Studies, 6(1), 143–149.
Scheidig, F., & Schweinberger, K. (2022). Assessment Analytics—Daten digitaler Prüfungen auswerten (Assessment analytics—Evaluate data from digital exams) (H 3.9, pp. 2–23).
Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis: Modeling change and event occurrence. Oxford University Press. https://doi.org/10.1093/acprof:oso/9780195152968.001.0001
Supsakova, B. (2016). Visual literacy for the 21st century. IJAEDU- International E-Journal of Advances in Education, 2(5), 202. https://doi.org/10.18768/ijaedu.07737
Sweller, J. (2011). Cognitive load theory. In J. P. Mestre & B. H. Ross (Eds.), The psychology of learning and motivation: Cognition in education (Vol. 55, pp. 37–76). Elsevier Academic Press. https://doi.org/10.1016/B978-0-12-387691-1.00002-8
Teresi, J. A., Ramirez, M., Jones, R. N., Choi, S., & Crane, P. K. (2012). Modifying measures based on differential item functioning (DIF) impact analyses. Journal of Aging and Health, 24(6), 1044–1076. https://doi.org/10.1177/0898264312436877
Tsai, F.-H., Tsai, C.-C., & Lin, K.-Y. (2015). The evaluation of different gaming modes and feedback types on game-based formative assessment in an online learning environment. Computers & Education, 81, 259–269. https://doi.org/10.1016/j.compedu.2014.10.013
Um, E., Plass, J. L., Hayward, E. O., & Homer, B. D. (2012). Emotional design in multimedia learning. Journal of Educational Psychology, 104(2), 485–498. https://doi.org/10.1037/a0026609
Viberg, O., & Grönlund, Å. (2012). Mobile assisted language learning: A literature review. In CEUR workshop proceedings (Vol. 955).
Zeidner, M. (2014). Anxiety in education. In International handbook of emotions in education (pp. 265–288). Routledge/Taylor & Francis Group.
Zhao, W., Wang, B., Coniam, D., & Xie, B. (2017). Calibrating the CEFR against the China standards of English for college English vocabulary education in China. Language Testing in Asia, 7(1), 5. https://doi.org/10.1186/s40468-017-0036-1
Funding
This chapter was written by cooperation partners of the project NOVA:ea, which is supported by the Stiftung Innovation in der Hochschullehre (Foundation for Innovation in Higher Education; FBM2020-VA-294-3-97008).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Zinke, N. et al. (2024). Towards Fair and Diversity-Appropriate E-assessments. In: Sahin, M., Ifenthaler, D. (eds) Assessment Analytics in Education. Advances in Analytics for Learning and Teaching. Springer, Cham. https://doi.org/10.1007/978-3-031-56365-2_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-56365-2_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-56364-5
Online ISBN: 978-3-031-56365-2
eBook Packages: EducationEducation (R0)