Skip to main content

Advertisement

SpringerLink
Log in

Cloud and edge based data analytics for privacy-preserving multi-modal engagement monitoring in the classroom

  • Published:
Information Systems Frontiers Aims and scope Submit manuscript

Abstract

Learning management systems are service platforms that support the administration and delivery of training programs and educational courses. Prerecorded, real-time or interactive lectures can be offered in blended, flipped or fully online classrooms. A key challenge with such service platforms is the adequate monitoring of engagement, as it is an early indicator for a student’s learning achievements. Indeed, observing the behavior of the audience and keeping the participants engaged is not only a challenge in a face-to-face setting where students and teachers share the same physical learning environment, but definitely when students participate remotely. In this work, we present a hybrid cloud and edge-based service orchestration framework for multi-modal engagement analysis. We implemented and evaluated an edge-based browser solution for the analysis of different behavior modalities with cross-user aggregation through secure multiparty computation. Compared to contemporary online learning systems, the advantages of our hybrid cloud-edge based solution are twofold. It scales up with a growing number of students, and also mitigates privacy concerns in an era where the rise of analytics in online learning raises questions about the responsible use of data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. https://campustechnology.com/articles/2018/05/02/when-learning-analytics-violate-student-privacy.aspx

  2. https://www.imec-int.com/en/what-we-offer/research-portfolio/lecture

  3. https://js.tensorflow.org/

  4. https://multiparty.org/, https://github.com/multiparty/jiff

  5. The data collection was approved on April 2018 by the Social and Societal Ethics Committee of the university with case number G-2018 041206

References

  • Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D.G., Steiner, B., Tucker, P., Vasudevan, V., Warden, P., Wicke, M., Yu, Y., Zheng, X. (2016a) Tensorow: A system for large-scale machine learning. In: Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation. pp. 265–283. Berkeley: OSDI'16, USENIX Association. http://dl.acm.org/citation.cfm?id=3026877.3026899

  • Abadi, M., Chu, A., Goodfellow, I., McMahan, H.B., Mironov, I., Talwar, K., Zhang, L. (2016b) Deep learning with differential privacy. In: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. pp. 308–318. ACM.

  • Abbas, N., Zhang, Y., Taherkordi, A., & Skeie, T. (2018). Mobile edge computing: A survey. IEEE Internet of Things Journal, 5(1), 450–465.

    Article  Google Scholar 

  • Acar, A., Aksu, H., Uluagac, A. S., & Conti, M. (2018). A survey on homomorphic encryption schemes: Theory and implementation. ACM Computing Surveys, 51(4), 79:1–79:35. https://doi.org/10.1145/3214303.

    Article  Google Scholar 

  • de Assuncao, M. D., da Silva Veith, A., & Buyya, R. (2018). Distributed data stream processing and edge computing: A survey on resource elasticity and future directions. Journal of Network and Computer Applications, 103, 1–17.

    Article  Google Scholar 

  • Atherton, M., Shah, M., Vazquez, J., Griffths, Z., Jackson, B., & Burgess, C. (2017). Using learning analytics to assess student engagement and academic outcomes in open access enabling programmes. Open Learning: The Journal of Open, Distance and e-Learning, 32(2), 119–136. https://doi.org/10.1080/02680513.2017.1309646.

    Article  Google Scholar 

  • Baker, R.S. (2007) Modeling and understanding students off-task behavior in intelligent tutoring systems. In: Proceedings of the SIGCHI Conference on Hu- man Factors in Computing Systems. pp. 1059–1068. New York: CHI '07, ACM. https://doi.org/10.1145/1240624.1240785.

  • Beimel, A. (2011) Secret-sharing schemes: a survey. In: International Conference on Coding and Cryptology. pp. 11–46. Springer.

  • Ben-David, A., Nisan, N., Pinkas, B.(2008) Fairplaymp: A system for secure multi-party computation. In: Proceedings of the 15th ACM Conference on Computer and Communications Security. pp. 257–266. New York: CCS '08, ACM https://doi.org/10.1145/1455770.1455804.

  • Bonomi, F., Milito, R., Natarajan, P., Zhu, J. (2014) Fog computing: A platform for internet of things and analytics. In: Big data and internet of things: A roadmap for smart environments, pp. 169–186. Springer.

  • Bonomi, F., Milito, R., Zhu, J., MAddepalliilito, S. (2012) Fog computing and its role in the internet of things. In: Proceedings of the First Edition of the MCC Workshop on Mobile Cloud Computing. pp. 13–16. New York: MCC '12, ACM. https://doi.org/10.1145/2342509.2342513.

  • Cao, H., Wachowicz, M., Cha, S. (2017) Developing an edge computing platform for real-time descriptive analytics. In: 2017 IEEE International Conference on Big Data (Big Data). pp. 4546–4554. https://doi.org/10.1109/BigData.2017.8258497.

  • Cetintas, S., Si, L., Xin, Y.P., Hord, C., Zhang, D. (2009) Learning to identify students' off-task behavior in intelligent tutoring systems. In: Proceedings of the 2009 Conference on Artificial Intelligence in Education: Building Learning Systems That Care: From Knowledge Representation to Affective Modelling. pp. 701–703. Amsterdam: IOS Press, http://dl.acm.org/citation.cfm?id=1659450.1659578

  • Cramer, R., Damgrd, I. B., & Nielsen, J. B. (2015). Secure Multiparty Computation and Secret Sharing (1st ed.). New York, NY, USA: Cambridge University Press.

    Book  Google Scholar 

  • Danezis, G., Domingo-Ferrer, J., Hansen, M., Hoepman, J., Métayer, D.L., Tirtea, R., Schiffner, S. (2015) Privacy and data protection by design - from policy to engineering. CoRR abs/1501.03726, http://arxiv.org/abs/1501.03726

  • Daniel, B.K. (2016) Big data and learning analytics in higher education. Springer.

  • Dantcheva, A., Elia, P., & Ross, A. (2016). What else does your biometric data reveal? a survey on soft biometrics. IEEE Transactions on Information Forensics and Security, 11(3), 441–467. https://doi.org/10.1109/TIFS.2015.2480381.

    Article  Google Scholar 

  • van Dijk, M., Gentry, C., Halevi, S., & Vaikuntanathan, V. (2010). Fully homomorphic encryption over the integers. In H. Gilbert (Ed.), Advances in Cryptology – EUROCRYPT 2010 (pp. 24–43). Berlin Heidelberg, Berlin, Heidelberg: Springer.

    Chapter  Google Scholar 

  • Dwork, C. (2011) Differential privacy. Encyclopedia of Cryptography and Security pp. 338–340.

  • El-Yahyaoui, A., El Kettani, M.D.E.C. (2017) Fully homomorphic encryption: Searching over encrypted cloud data. In: Proceedings of the 2Nd International Conference on Big Data, Cloud and Applications. pp. 10:1–10:5. New York: BDCA'17, ACM. https://doi.org/10.1145/3090354.3090364

  • Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59–109. https://doi.org/10.3102/00346543074001059.

    Article  Google Scholar 

  • Gong, L., Liu, Y., Zhao, W. (2018) Using learning analytics to promote student engagement and achievement in blended learning: An empirical study. In: Proceedings of the 2Nd International Conference on E-Education, E-Business and E-Technology. pp. 19–24. New York: ICEBT 2018, ACM. https://doi.org/10.1145/3241748.3241760.

  • Gray, C. C., & Perkins, D. (2019). Utilizing early engagement and machine learning to predict student outcomes. Computers & Education, 131, 22–32. https://doi.org/10.1016/j.compedu.2018.12.006http://www.sciencedirect.com/science/article/pii/S0360131518303191.

    Article  Google Scholar 

  • He, J., Wei, J., Chen, K., Tang, Z., Zhou, Y., & Zhang, Y. (2018). Multitier fog computing with large-scale iot data analytics for smart cities. IEEE Internet of Things Journal, 5(2), 677–686.

    Article  Google Scholar 

  • Henrie, C. R., Halverson, L. R., & Graham, C. R. (2015). Measuring student engagement in technology-mediated learning: A review. Computers & Education, 90, 36–53. https://doi.org/10.1016/j.compedu.2015.09.005http://www.sciencedirect.com/science/article/pii/S0360131515300427.

    Article  Google Scholar 

  • Holzer, A., Franz, M., Katzenbeisser, S., Veith, H. (2012) Secure two-party computations in ansi c. In: Proceedings of the 2012 ACM Conference on Computer and Communications Security. pp. 772–783. New York: CCS '12, ACM . https://doi.org/10.1145/2382196.2382278

  • Hukkelås, H., Mester, R., Lindseth, F.(2019) DeepPrivacy: A Generative Adversarial Network for Face Anonymization. arXiv e-prints arXiv:1909.04538.

  • Kahu, E. R. (2013). Framing student engagement in higher education. Studies in Higher Education, 38(5), 758–773. https://doi.org/10.1080/03075079.2011.598505.

    Article  Google Scholar 

  • Kahu, E. R., & Nelson, K. (2018). Student engagement in the educational interface: understanding the mechanisms of student success. Higher Education Research & Development, 37(1), 58–71. https://doi.org/10.1080/07294360.2017.1344197.

    Article  Google Scholar 

  • Kawamura, R., Toyoda, Y., Niinuma, K. (2019) Engagement estimation based on synchrony of head movements: Application to actual e-learning scenarios. In: Proceedings of the 24th International Conference on Intelligent User Interfaces: Companion. pp. 25–26. New York: IUI '19, ACM. https://doi.org/10.1145/3308557.3308660.

  • Konecný, J., McMahan, H.B., Yu, F.X., Richárik, P., Suresh, A.T., Bacon, D. (2016) Federated learning: Strategies for improving communication effciency. CoRRabs/1610.05492. http://arxiv.org/abs/1610.05492

  • Krawiecka, K., Kurnikov, A., Paverd, A., Mannan, M., Asokan, N. (2018) Safekeeper: Protecting web passwords using trusted execution environments. In: Proceedings of the 2018 World Wide Web Conference. pp. 349–358. WWW '18, International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland., https://doi.org/10.1145/3178876.3186101.

  • Li, N., Li, T., Venkatasubramanian, S. (2007) t-closeness: Privacy beyond k-anonymity and l-diversity. In: 2007 IEEE 23rd International Conference on Data Engineering. pp. 106–115. IEEE.

  • Lorenz, B., Sousa, S., & Tomberg, V. (2013). Privacy awareness of students and its impact on online learning participation – a case study. In T. Ley, M. Ruohonen, M. Laanpere, & A. Tatnall (Eds.), Open and Social Technologies for Networked Learning. pp. 189–192. Berlin Heidelberg, Berlin, Heidelberg: Springer.

    Google Scholar 

  • Ma, X., Zhang, F., Chen, X., & Shen, J. (2018). Privacy preserving multi-party computation delegation for deep learning in cloud computing. Information Sciences, 459, 103–116.

    Article  Google Scholar 

  • May, M., George, S.: Using students' tracking data in e-learning: Are we always aware of security and privacy concerns? In: 2011 IEEE 3rd International Conference on Communication Software and Networks. pp. 10{14 (May 2011). https://doi.org/10.1109/ICCSN.2011.6013764.

  • Melis, L., Song, C., Cristofaro, E.D., Shmatikov, V.(2018) Inference attacks against collaborative learning. CoRR abs/1805.04049 http://arxiv.org/abs/1805.04049

  • Mohan, P., Thakurta, A., Shi, E., Song, D., Culler, D. (2012) Gupt: Privacy preserving data analysis made easy. In: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data. pp. 349–360. New York: SIGMOD '12, ACM https://doi.org/10.1145/2213836.2213876.

  • Monkaresi, H., Bosch, N., Calvo, R.A., D'Mello, S.K.: Automated detection of engagement using video-based estimation of facial expressions and heart rate. IEEE Transactions on Affective Computing 8(1), 15–28 (Jan 2017). https://doi.org/10.1109/TAFFC.2016.2515084

  • Prabhakaran, M., & Sahai, A. (2013). Secure Multi-Party Computation. Amsterdam, The Netherlands, The Netherlands: IOS Press.

    Google Scholar 

  • Preuveneers, D., Joosen, W. (2019) Edge-based and privacy-preserving multi-modal monitoring of student engagement in online learning environments. In: Proceedings of the IEEE International Conference on Edge Computing (IEEE EDGE 2019). pp. 1–3. IEEE.

  • Raes, A., Vanneste, P., Pieters, M., Windey, I., Noortgate, W. V. D., & Depaepe, F. (2020). Learning and instruction in the hybrid virtual classroom: An investigation of students' engagement and the effect of quizzes. Computers & Education, 143, 103682. https://doi.org/10.1016/j.compedu.2019.103682http://www.sciencedirect.com/science/article/pii/S0360131519302350.

  • Roman, R., Lopez, J., & Mambo, M. (2018). Mobile edge computing, fog et al.: A survey and analysis of security threats and challenges. Future Generation Computer Systems, 78, 680–698.

    Article  Google Scholar 

  • Satyanarayanan, M. (2017). The emergence of edge computing. Computer, 50(1), 30–39.

    Article  Google Scholar 

  • Shan, Z., Ren, K., Blanton, M., & Wang, C. (2018). Practical secure computation outsourcing: a survey. ACM Computing Surveys (CSUR), 51(2), 31.

    Article  Google Scholar 

  • Shi, W., Cao, J., Zhang, Q., Li, Y., & Xu, L. (2016). Edge computing: Vision and challenges. IEEE Internet of Things Journal, 3(5), 637–646.

    Article  Google Scholar 

  • Strauss, V. (2018) Why parents and students are protesting an online learning program backed by mark zuckerberg and facebook. https://www.washingtonpost.com/education/2018/12/20/why-parents-students-are-protesting-an-online-learning-program-backed-by-mark-zuckerberg-facebook/

  • Thomas, C., Jayagopi, D.B. (2017) Predicting student engagement in classrooms using facial behavioral cues. In: Proceedings of the 1st ACM SIGCHI International Workshop on Multimodal Interaction for Education. pp. 33–40. New York: MIE 2017, ACM https://doi.org/10.1145/3139513.3139514.

  • Toshev, A., Szegedy, C. (2014) Deeppose: Human pose estimation via deep neural networks. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition. pp. 1653–1660. https://doi.org/10.1109/CVPR.2014.214.

  • Whitehill, J., Serpell, Z., Lin, Y., Foster, A., & Movellan, J. R. (2014). The faces of engagement: Automatic recognition of student engagementfrom facial expressions. IEEE Transactions on Affective Computing, 5(1), 86–98. https://doi.org/10.1109/TAFFC.2014.2316163.

    Article  Google Scholar 

  • Yang, B., Gu, F., Niu, X. (2006) Block mean value based image perceptual hashing. In: Proceedings of the 2006 International Conference on Intelligent Information Hiding and Multimedia. pp. 167–172.Washington: IIH-MSP '06, IEEE Computer Society. https://doi.org/10.1109/IIH-MSP.2006.66.

  • Yi, S., Hao, Z., Qin, Z., Li, Q. (2015) Fog computing: Platform and applications. In: 2015 Third IEEE Workshop on Hot Topics in Web Systems and Technologies (HotWeb). pp. 73–78. IEEE.

  • Zhang, X., Meng, Y., de Pablos, P. O., & Sun, Y. (2017). Learning analytics in collaborative learning supported by slack: From the perspective of engagement. Computers in Human Behavior. https://doi.org/10.1016/j.chb.2017.08.012http://www.sciencedirect.com/science/article/pii/S0747563217304788.

Download references

Acknowledgments

This research is partially funded by the Research Fund KU Leuven, and by imec through ICON LECTURE+. LECTURE+ is a project realized in collaboration with imec, with Barco, Televic Education and Limecraft as project partners and with project support from VLAIO.

We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan V GPU used for this research.

Author information

Authors and Affiliations

  1. imec-DistriNet, KU Leuven, Celestijnenlaan 200A, B-3001, Heverlee, Belgium

    Davy Preuveneers, Giuseppe Garofalo & Wouter Joosen

Authors
  1. Davy Preuveneers
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Giuseppe Garofalo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wouter Joosen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Davy Preuveneers.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Preuveneers, D., Garofalo, G. & Joosen, W. Cloud and edge based data analytics for privacy-preserving multi-modal engagement monitoring in the classroom. Inf Syst Front 23, 151–164 (2021). https://doi.org/10.1007/s10796-020-09993-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10796-020-09993-4

Keywords

Search

Navigation