Anne Lamb
Justin Reich
ABSTRACT
Massive open online course (MOOC) platforms increasingly allow easily implemented randomized experiments. The heterogeneity of MOOC students, however, leads to two methodological obstacles in analyzing interventions to increase engagement. (1) Many MOOC participation metrics have distributions with substantial positive skew from highly active users as well as zero-inflation from high attrition. (2) High attrition means that in some experimental designs, most users assigned to the treatment never receive it; analyses that do not consider attrition result in "intent-to-treat" (ITT) estimates that underestimate the true effects of interventions. We address these challenges in analyzing an intervention to improve forum participation in the 2014 JusticeX course offered on the edX MOOC platform. We compare the results of four ITT models (OLS, logistic, quantile, and zero-inflated negative binomial regressions) and three "treatment-on-treated" (TOT) models (Wald estimator, 2SLS with a second stage logistic model, and instrumental variables quantile regression). A combination of logistic, quantile, and zero-inflated negative binomial regressions provide the most comprehensive description of the ITT effects. TOT methods then adjust the ITT underestimates. Substantively, we demonstrate that self-assessment questions about forum participation encourage more students to engage in forums and increases the participation of already active students.
- Abadie, A., Angrist, J. and Imbens, G. Instrumental variables estimates of the effect of subsidized training on the quantiles of trainee earnings. Econometrica, 70, 1 ( 2002), 91--117. DOI=10.1111/1468-0262.00270.Google Scholar
Cross Ref
- Anderson, A., Huttenlocher, D., Kleinberg, J. and Leskovec, J. Engaging with Massive Online Courses. in Proceedings of the 2014 International World Wide Web Conference, (2014), 687--698. Google ScholarDigital Library
- Angrist, J. D. Mostly harmless econometrics: An empiricist's companion. Princeton: Princeton University Press, Princeton, 2009.Google ScholarCross Ref
- Bakshy, E., Eckles, D. and Bernstein, M. S. Designing and deploying online field experiments. in Proceedings of the 23rd international conference on World wide web. (2014), 283--292. Google ScholarDigital Library
- Breslow, L., Pritchard, D. E., DeBoer, J., Stump, G. S., Ho, A. D. and Seaton, D. Studying learning in the worldwide classroom: Research into edX's first MOOC. Research & Practice in Assessment, 8( 2013), 13--25.Google Scholar
- Guo, P. J. and Reinecke, K. Demographic differences in how students navigate through MOOCs. in Proceedings of the First ACM Conference on Learning@Scale Conference. (Atlanta, USA, 2014). ACM, 21--30. Google ScholarDigital Library
- Ho, A. D., Reich, J., Nesterko, S. O., Seaton, D. T., Mullaney, T., Waldo, J. and Chuang, I. HarvardX and MITx: The First Year of Open Online Courses, Fall 2012-Summer 2013. HarvardX & MITx Working Paper No. 1. , 2014.Google Scholar
- Huang, J., Dasgupta, A., Ghosh, A., Manning, J. and Sanders, M. Superposter behavior in MOOC forums. in Proceedings of the First ACM Conference on Learning@Scale Conference. (Atlanta, USA, 2014). ACM, 117--126. Google ScholarDigital Library
- Kizilcec, R. F., Schneider, E., Cohen, G. and McFarland, D. Encouraging forum participation in online courses with collectivist, individualist, and neutral motivational framings. eLearning Papers, 37( 2014), 13--22.Google Scholar
- Manning, J. and Sanders, M. How widely used are MOOC forums? A first look. Signals: Thoughts on Online Learning, ( 2013).Google Scholar
- Nichols, A. IVPOIS: Stata module to estimate an instrumental variables Poisson regression via GMM. Statistical Software Components, ( 2008).Google Scholar
- O'Hara, R. B. and Kotze, D. J. Do not log-transform count data. Methods in Ecology and Evolution, 1, 2 ( 2010), 118--122. DOI=10.1111/j.2041--210X.2010.00021.x.Google ScholarCross Ref
- Rabe-Hesketh, S., Skrondal, A. and Stata Corporation. Multilevel and longitudinal modeling using Stata. Stata Press Publication, College Station, Tex., 2008. Google ScholarDigital Library
- Reich, J. MOOC Completion and retention in the context of student intent. EDUCAUSE Review Online, (2014).Google Scholar
- Reich, J., Nesterko, S. O., Seaton, D. T., Mullaney, T., Waldo, J., Chuang, I. and Ho, A. D. JusticeX: Spring 2013 Course Report. HarvardX Working Paper No. 4., 2014.Google Scholar
Index Terms
- Addressing Common Analytic Challenges to Randomized Experiments in MOOCs: Attrition and Zero-Inflation
Recommendations
A Literature Review on MOOCs Integrated With Learning Analytics
Although there have been numerous studies committed to MOOCs integrated with learning analytics, fewer of them have systematically reviewed the related literature. Using clustering techniques, bibliographic network visualization, content analysis, and ...
From the learner's perspective: A systematic review of MOOC learner experiences (2008–2021)
AbstractWe report the results of a systematic review of learners' experiences and perspectives in massive open online courses (MOOCs). This systematic review includes 51 articles published between 2008 and 2021 that appeared in top educational ...
Highlights- MOOC learners' diverse intentions and motivations for enrollment influence their experiences within the course.
Using learning analytics to explore help-seeking learner profiles in MOOCs
LAK '17: Proceedings of the Seventh International Learning Analytics & Knowledge ConferenceIn online learning environments, learners are often required to be more autonomous in their approach to learning. In scaled online learning environments, like Massive Open Online Courses (MOOCs), there are differences in the ability of learners to ...
Comments