Understanding Hybrid-MOOC Effectiveness with a Collective Socio-Behavioral Model



Published Dec 28, 2019
Sabina Tomkins Lise Getoor


Online courses for high school students promise the opportunity to bring critical education to youth most at need, bridging gaps which may exist in brick-and-mortar institutions. In this work, we investigate a hybrid Massive Open Online Course for high schoolers which includes an in-person coaching component. We address the efficacy of these courses and the contribution of in-person coaching. We first analyze features of student behavior and their effect on post-test performance and then propose a novel probabilistic model for inferring student success on an AP exam post-test. Our proposed model exploits relationships between students to collectively infer student success. When these relationships are not directly observed, we formulate latent constructs to capture social dynamics of learning. By collectively inferring student success as a function of both unobserved individual characteristics and relational dynamics, we improve predictive performance by up to 6.8% over an SVM model with only observable features. We propose this general socio-behavioral modeling framework as a flexible approach for including unobserved aspects of learning in meaningful ways, in order to better understand and infer student success.

How to Cite

Tomkins, S., & Getoor, L. (2019). Understanding Hybrid-MOOC Effectiveness with a Collective Socio-Behavioral Model. Journal of Educational Data Mining, 11(3), 42–77. https://doi.org/10.5281/zenodo.3594773
Abstract 857 | PDF Downloads 591



latent-variable modeling, collective inference, probabilistic modeling, socio-behavioral modeling, high school, MOOC

ANDERSON, A., HUTTENLOCHER, D., KLEINBERG, J., AND LESKOVEC, J. 2014. Engaging with massive online courses. In International Conference on World Wide Web (WWW). 687–698.

ANDERSON, T. 2003. Getting the mix right again: An updated and theoretical rationale for interaction. The International Review of Research in Open and Distributed Learning 4, 2. Retrieved from: http://www.irrodl.org/index.php/irrodl/article/view/149/230.

ANDREWS-TODD, J., FORSYTH, C., STEINBERG, J., AND RUPP, A. 2018. Identifying profiles of collaborative problem solvers in an online electronics environment. In International Conference on Educational Data Mining (EDM), K. E. Boyer and M. Yudelson, Eds. 239–245.

AVIV, R., ERLICH, Z., RAVID, G., AND GEVA, A. 2003. Network analysis of knowledge construction in asynchronous learning networks. Journal of Asynchronous Learning Networks 7, 3, 1–23.

BACH, S. H., BROECHELER, M., HUANG, B., AND GETOOR, L. 2017. Hinge-loss Markov random fields and probabilistic soft logic. Journal of Machine Learning Research 18, 109, 3846–3912.

BAKER, R., CORBETT, A., KOEDINGER, K., AND WAGNER, A. 2004. Off-task behavior in the cognitive tutor classroom: When students "game the system". In SIGCHI Conference on Human Factors in Computing Systems. 383–390.

BLEI, D. M., NG, A. Y., AND JORDAN, M. I. 2003. Latent Dirichlet Allocation. Journal of Machine Learning Research 3, 993–1022.

BLOM, J., VERMA, H., LI, N., SKEVI, A., AND DILLENBOURG, P. 2013. MOOCs are more social than you believe. 33, 1–3.

BOCK, M. AND O'DEA, V. 2013. Virtual educators critique value of MOOCs for K-12. https://www. edweek.org/ew/articles/2013/02/06/20moocs.h32.html.

CHEN, G., DAVIS, D., HAUFF, C., AND HOUBEN, G.-J. 2016. On the impact of personality in massive open online learning. In User Modeling Adaptation and Personalization. 121–130.

CORBETT, A. T. AND ANDERSON, J. R. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction 4, 4, 253–278.

ER, E., GÓMEZ-SÁNCHEZ, E., BOTE-LORENZO, M. L., ASENSIO-PÉREZ, J. I., AND DIMITRIADIS, Y. 2019. Informing the design of collaborative activities in MOOCs using actionable predictions. In ACM Conference on Learning@ Scale (L@S). 27:1–27:4.

FRIEDMAN, N., GETOOR, L., KOLLER, D., AND PFEFFER, A. 1999. Learning probabilistic relational models. In IJCAI. Vol. 99. 1300–1309.

GARDNER, J. AND BROOKS, C. 2018. Student success prediction in MOOCs. User Modeling and UserAdapted Interaction 28, 2, 127–203.

GITINABARD, N., KHOSHNEVISAN, F., LYNCH, C., AND WANG, E. Y. 2018. Your actions or your associates? Predicting certification and dropout in MOOCs with behavioral and social features. In International Conference on Educational Data Mining (EDM), K. E. Boyer and M. Yudelson, Eds. 404–410.

HANZAKI, M. R. AND EPP, C. D. 2018. The effect of personality and course attributes on academic performance in MOOCs. In European conference on technology enhanced learning. 497–509.

HUANG, J., DASGUPTA, A., GHOSH, A., MANNING, J., AND SANDERS, M. 2014. Superposter behavior in MOOC forums. In ACM Conference on Learning @ Scale Conference (L@S). 117–126.

JURAFSKY, D. AND MARTIN, J. H. 2000. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Prentice Hall.

KELLOGG, S., BOOTH, S., AND OLIVER, K. 2014. Asocial network perspective on peer supported learning in MOOCs for educators. The International Review of Research in Open and Distributed Learning 15, 5, 263–289.

KENNEDY, G., COFFRIN, C., DE BARBA, P., AND CORRIN, L. 2015. Predicting success: How learners' prior knowledge, skills and activities predict MOOC performance. In International Conference on Learning Analytics And Knowledge (LAK). 136–140.

KIZILCEC, R. F., PIECH, C., AND SCHNEIDER, E. 2013. Deconstructing disengagement: Analyzing learner subpopulations in massive open online courses. In International Conference on Learning Analytics and Knowledge (LAK). 170–179.

KLÜSENER, M. AND FORTENBACHER, A. 2015. Predicting students' success based on forum activities in MOOCs. In International Conference on Intelligent Data Acquisition and Advanced Computing Systems (IDAACS). Vol. 2. 925–928.

KUMAR, R. AND ROSE, C. P. 2011. Architecture for building conversational agents that support collaborative learning. IEEE Transactions on Learning Technologies 4, 1, 21–34.

KURHILA, J. AND VIHAVAINEN, A. 2015. Apurposeful MOOC to alleviate insufficient CS education in Finnish schools. Transactions of Computing Education 15, 2, 1–18.

LI, N., VERMA, H., SKEVI, A., ZUFFEREY, G., BLOM, J., AND DILLENBOURG, P. 2014. Watching MOOCs together: investigating co-located MOOC study groups. Distance Education 35, 2, 217–233.

LI, X., WANG, T., AND WANG, H. 2017. Exploring n-gram features in clickstream data for MOOC learning achievement prediction. In International Conference on Database Systems for Advanced Applications. 328–339.

LONGSTAFF, E. 2017. Ritual in online communities: Astudy of post-voting in MOOC discussion forums. International Journal of Human Computer Interaction 33, 8, 655–663.

LOYA, A., GOPAL, A., SHUKLA, I., JERMANN, P., AND TORMEY, R. 2015. Conscientious behaviour, flexibility and learning in massive open on-line courses. Procedia-Social and Behavioral Sciences 191, 519–525.

MCCALLUM, A. 2002. MALLET: Amachine learning for language toolkit. http://mallet.cs.umass.edu/.

MEGA, C., RONCONI, L., AND DE BENI, R. 2014. What makes agood student? How emotions, selfregulated learning, and motivation contribute to academic achievement. Journal of Educational Psychology 106, 1, 121–131.

MOTZ, B., BUSEY, T., RICKERT, M., AND LANDY, D. 2018. Finding topics in enrollment data. In International Conference on Educational Data Mining (EDM), K. E. Boyer and M. Yudelson, Eds. 424–430.

NAJAFI, H., EVANS, R., AND FEDERICO, C. 2014. MOOC integration into secondary school courses. The International Review of Research in Open and Distributed Learning 15, 5, 306–322.

PEDREGOSA ET AL., F. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, Oct, 2825–2830.

RAMESH, A., GOLDWASSER, D., HUANG, B., DAUME III, H., AND GETOOR, L. 2014. Learning latent engagement patterns of students in online courses. In AAAI Conference on Artificial Intelligence. 1272–1278.

RICHARDSON, M. AND DOMINGOS, P. 2006. Markov logic networks. Machine learning 62, 1-2, 107– 136.

ROSÉ, C., WANG, Y.-C., CUI, Y., ARGUELLO, J., STEGMANN, K., WEINBERGER, A., AND FISCHER, F. 2008. Analyzing collaborative learning processes automatically: Exploiting the advances of computational linguistics in computer-supported collaborative learning. International Journal of Computer-Supported Collaborative Learning 3, 3, 237–271.

ROSÉ, C. P., CARLSON, R., YANG, D., WEN, M., RESNICK, L., GOLDMAN, P., AND SHERER, J. 2014. Social factors that contribute to attrition in MOOCs. In ACM Conference on Learning @ Scale Conference (L@S). 197–198.

ROSÉ, C. P., GOLDMAN, P., ZOLTNERS SHERER, J., AND RESNICK, L. 2015. Supportive technologies for group discussion in MOOCs. Current Issues in Emerging eLearning 2, 1, 5. Available at: https://scholarworks.umb.edu/ciee/vol2/iss1/5.

SIMON, B., PARRIS, J., AND SPACCO, J. 2013. How we teach impacts student learning: Peer instruction vs. lecture in CS0. In ACM Technical Symposium on Computer Science Education (SIGCSE). 41–46.

STAUBITZ, T. AND MEINEL, C. 2019. Graded team assignments in MOOCs: Effects of team composition and further factors on team dropout rates and performance. In ACM Conference on Learning@ Scale (L@S). 5:1–5:10.

SU, Y.-S., HUANG, C. S.-J., AND DING, T.-J. 2016. Examining the effects of MOOCs learning social search results on learning behaviors and learning outcomes. Eurasia Journal of Mathematics, Science and Technology Education 1, 9, 2517–2529.

SUN, Y., NI, L., ZHAO, Y., SHEN, X.-L., AND WANG, N. 2018. Understanding students engagement in MOOCs: An integration of self-determination theory and theory of relationship quality. British Journal of Educational Technology, 1–19.

TOMKINS, S., RAMESH, A., AND GETOOR, L. 2016. Predicting post-test performance from online student behavior: Ahigh school MOOC case study. In International Conference on Educational Data Mining (EDM), T. Barnes, M. Chi, and M. Feng, Eds. 239–246.

VELETSIANOS, G., COLLIER, A., AND SCHNEIDER, E. 2015. Digging deeper into learners' experiences in MOOCs: Participation in social networks outside of MOOCs, notetaking and contexts surrounding content consumption. British Journal of Educational Technology 46, 3, 570–587.

WANG, X., YANG, D., WEN, M., KOEDINGER, K., AND ROSÉ, C. P. 2015. Investigating how student's cognitive behavior in MOOC discussion forums affect learning gains. In International Conference on Educational Data Mining (EDM), O. C. Santos, J. G. Boticario, C. Romero, M. Pechenizkiy, A. Merceron, P. Mitros, J. M. Luna, C. Mihaescu, P. Moreno, A. Hershkovitz, S. Ventura, and M. Desmarais, Eds. 226–233.

WEN, M., YANG, D., AND ROSÉ, C. P. 2015. Virtual teams in massive open online courses. In International Conference on Artificial Intelligence in Education, C. Conati, N. Heffernan, A. Mitrovic, and F. M. Verdejo, Eds. 820–824.