Abstract
The OpenAnswer system has the goal of exploiting teacher mediated peer-assessment for the evaluation of answers to open ended questions. The system models both the learning state of each student and their choices during peer-assessment. In OpenAnswer, each student is represented as a Bayesian network made of a triple of finite-domain variables: K for student’s Knowledge about a topic, J for the estimated ability to evaluate (”Judge”) the answer of another peer, C for Correctness of the answer to a given question. The student’s individual sub-networks are connected through further Bayesian variables which model each peer-assessment choice, depending on the type of peer-assessment performed: (G for grading, B for choosing the best, W for choosing the worst). During an assessment session, each student grades a fixed number of peers’ answers. The final result for a given session is a full set of grades for all students’ answers, although the teacher had actually graded only a part of them. The student’s assessments are instantiated in the network as evidence, together with the teacher’s (perhaps partially complete) grades, so that OpenAnswer deduces the remaining grades. In the former OpenAnswer implementation, all variables were represented through a probability distribution over three values (Good/Fair/Bad for K and J, correct/fair/wrong for C). We present experiments and simulations showing that, by increasing the domain granularity for all variables from 3 to 6 values (A to F), the information obtained from the Bayesian network achieves higher reliability.
Chapter PDF
Similar content being viewed by others
References
Limongelli, C., Sciarrone, F., Temperini, M., Vaste, G.: Lecomps5: A web-based learning system for course personalization and adaptation. In: Nunes, M.B., McPherson, M. (eds.) IADIS International Conference e-Learning 2008, Amsterdam, The Netherlands, July 22-25, vol. 1, pp. 325–332 (2008)
Popescu, E.: Adaptation Provisioning with respect to Learning Styles in a Web-Based Educational System: An Experimental Study. Journal of Computer Assisted Learning 26(4), 243–257 (2010)
Sterbini, A., Temperini, M.: Selection and sequencing constraints for personalized courses. In: Proc. 40th IEEE Frontiers in Education (FIE) Conference, Washington, DC, USA, October 27-30, pp. T2C1–T2C6 (2010), doi:10.1109/FIE.2010.5673146
Essalmi, F., Jemni Ben Ayed, L., Jemni, M., Kinshuk, Graf, S.: A fully Personalization Strategy of E-Learning Scenarios. Computers in Human Behavior 26(4), 581–591 (2010)
Limongelli, C., Sciarrone, F., Temperini, M., Vaste, G.: The Lecomps5 Framework for Personalized Web-Based Learning: a Teacher’s Satisfaction Perspective. Computers in Human Behavior 27(4), 1310–1320 (2011) ISSN 0747-5632
Kirschner, P.A.: Using integrated electronic environments for collaborative teaching/learning. Learning and Instruction 10, 1–9 (2001)
Sterbini, A., Temperini, M.: Learning from Peers: Motivating Students through Reputation Systems. In: Proc. IEEE/IPSJ International Symposium on Applications and the Internet, July 28-August 01, pp. 305–308 (2008), doi:ieeecomputersociety.org/10.1109/SAINT.2008.107
Kreijns, K., Kirschner, P.A., Jochems, W.: Identifying the pitfalls for social interaction in computer supported collaborative learning environments: a review of the research. Computers in Human Behavior 19, 335–353 (2003)
Cheng, Y., Ku, H.: An investigation of the effects of reciprocal peer tutoring. Computers in Human Behavior 25, 40–49 (2009)
Popescu, E.: Providing Collaborative Learning Support with Social Media in an Integrated Environment. World Wide Web (2012), doi:10.1007/s11280-012-0172-6, ISSN: 1386-145X
De Marsico, M., Sterbini, A., Temperini, M.: The Definition of a Tunneling Strategy between Adaptive Learning and Reputation-based Group Activities. In: Proc. 11th IEEE International Conference on Advanced Learning Technologies, ICALT 2011, Athens, GA, USA, July 6-8, pp. 498–500 (2011), doi:10.1109/ICALT.2011.155
De Marsico, M., Sterbini, A., Temperini, M.: A strategy to join adaptive and reputation-based social-collaborative e-learning, through the Zone of Proximal Development. Int. Journal of Distance Education Technology, IJDET 11(3), 12–31 (2013)
De Marsico, M., Sterbini, A., Temperini, M.: A Framework to Support Social-Collaborative Personalized e-Learning. In: Kurosu, M. (ed.) Human-Computer Interaction, HCII 2013, Part II. LNCS, vol. 8005, pp. 351–360. Springer, Heidelberg (2013)
Palmer, R.: On-line assessment and free-response input – a pedagogic and technical model for squaring the circle. In: Proceedings of the 7th CAA Conference, Loughborough University (2003)
Sterbini, A., Temperini, M.: Correcting open-answer questionnaires through a Bayesian-network model of peer-based assessment. In: Proc. Int. Conference on Information Technology Based Higher Education and Training, ITHET 2012, pp. 1–6 (2012)
Sterbini, A., Temperini, M.: OpenAnswer, a framework to support teacher’s management of open answers through peer assessment. In: Proc. 43th ASEE/IEEE Frontiers in Education, FIE, Oklahoma City, OK, October 23-26. IEEE Computer Society (2013)
Sterbini, A., Temperini, M.: Analysis of OpenAnswers via mediated peer-assessment. In: Proc. 17th IEEE Int Conf. on System Theory, Control and Computing, ICSTCC 2013 (2013)
Yamanishi, K., Li, H.: Mining Open Answers in Questionnaire Data. IEEE Intelligent Systems, 58–63 (2002)
Morinaga, S., Yamanishi, K., Tateishi, K., Fukushima, T.: Mining product reputations on the Web. In: Proc. 8th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, KDD 2002, pp. 341–349 (2002)
Jackson, K., Trochim, W.: Concept mapping as an alternative approach for the analysis of open-ended survey responses. Organizational Research Methods 5, 307–336 (2002)
Romero, C., Ventura, S.: Educational Data Mining: A Review of the State of the Art. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews 40(6), 601–618 (2010)
Castellanos-Nieves, D., Fernández-Breis, J., Valencia-García, R., Martínez-Béjar, R., Iniesta-Moreno, M.: Semantic Web Technologies for supporting learning assessment. Information Sciences 181(9), 1517–1537 (2011)
El-Kechaï, N., Delozanne, É., Prévit, D., Grugeon, B., Chenevotot, F.: Evaluating the Performance of a Diagnosis System in School Algebra. In: Leung, H., Popescu, E., Cao, Y., Lau, R.W.H., Nejdl, W. (eds.) ICWL 2011. LNCS, vol. 7048, pp. 263–272. Springer, Heidelberg (2011)
Formisano, A., Omodeo, E.G., Temperini, M.: Layered map reasoning: An experimental approach put to trial on sets. Electronic Notes in Theoretical Computer Science 48, 1–28 (2001)
Chung, H., Graf, S., Robert Lai, K., Kinshuk: Enrichment of Peer Assessment with Agent Negotiation. IEEE Trans. on Learning Technologies (TLT) 4(1), 35–46 (2011)
Sterbini, A., Temperini, M.: Supporting Assessment of Open Answers in a Didactic Setting. In: Proc. 12th IEEE Int. Conference on Advanced Learning Technologies (ICALT 2012), Rome, Italy, July 4-6 (2012)
Sterbini, A., Temperini, M.: Dealing with open-answer questions in a peer-assessment environment. In: Popescu, E., Li, Q., Klamma, R., Leung, H., Specht, M. (eds.) ICWL 2012. LNCS, vol. 7558, pp. 240–248. Springer, Heidelberg (2012)
Costa Santos, V.: CLP (BN): Constraint logic programming for probabilistic knowledge. In: Proc. 19th Conf. on Uncertainty in Artificial Intelligence. Morgan Kaufmann (2002)
Moij, J.M.: liDAI: a free and open source C++ library for Discrete Approximate Inference in graphical models. J. of Machine Learning Research 11, 2169–2173 (2010)
Kschischang, F.R., Frey, B.J., Loeliger, H.-A.: Factor graphs and the sum-product algorithm. IEEE Trans. on Information Theory 47(2), 498–519 (2001)
Bloom, B.S., Mesia, B.B., Krathwohl, D.R.: Taxonomy of Educational Objectives (two vols: The Affective Domain & The Cognitive Domain), NY. David McKay (1964)
Chatti, M.A., Sodhi, T., Specht, M., Klamma, R., Klemke, R.: u-Annotate: An application for user-driven freeform digital ink annotation of e-Learning content. In: Proc. Sixth International Conference on Advanced Learning Technologies, ICALT 2006, pp. 1039–1043 (2006)
Limongelli, C., Sciarrone, F., Starace, P., Temperini, M.: An Ontology-driven OLAP System to Help Teachers in the Analysis of Web Learning Object Repositories. Information Systems Management 27(3), 198–206, doi:10.1080/10580530.2010.493810
Dyckhoff, A.L., Zielke, D., Bültmann, M., Chatti, M.A., Schroeder, U.: Design and implementation of a learning analytics toolkit for teachers. Educational Technology and Society 15(3), 58–76 (2012)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
De Marsico, M., Sterbini, A., Temperini, M. (2014). How the Granularity of Evaluation Affects Reliability of Peer-Assessment Modelization in the OpenAnswer System. In: Schmorrow, D.D., Fidopiastis, C.M. (eds) Foundations of Augmented Cognition. Advancing Human Performance and Decision-Making through Adaptive Systems. AC 2014. Lecture Notes in Computer Science(), vol 8534. Springer, Cham. https://doi.org/10.1007/978-3-319-07527-3_20
Download citation
DOI: https://doi.org/10.1007/978-3-319-07527-3_20
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-07526-6
Online ISBN: 978-3-319-07527-3
eBook Packages: Computer ScienceComputer Science (R0)