Understanding Hybrid-MOOC Effectiveness with a Collective Socio-Behavioral Model
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
Dec 28, 2019
Sabina Tomkins
Lise Getoor
https://orcid.org/0000-0002-2632-8173
Abstract
Online courses for high school students promise the opportunity to bring critical education to youth most at need, bridging gaps which may exist in brick-and-mortar institutions. In this work, we investigate a hybrid Massive Open Online Course for high schoolers which includes an in-person coaching component. We address the efficacy of these courses and the contribution of in-person coaching. We first analyze features of student behavior and their effect on post-test performance and then propose a novel probabilistic model for inferring student success on an AP exam post-test. Our proposed model exploits relationships between students to collectively infer student success. When these relationships are not directly observed, we formulate latent constructs to capture social dynamics of learning. By collectively inferring student success as a function of both unobserved individual characteristics and relational dynamics, we improve predictive performance by up to 6.8% over an SVM model with only observable features. We propose this general socio-behavioral modeling framework as a flexible approach for including unobserved aspects of learning in meaningful ways, in order to better understand and infer student success.
How to Cite
Tomkins, S., & Getoor, L. (2019). Understanding Hybrid-MOOC Effectiveness with a Collective Socio-Behavioral Model. Journal of Educational Data Mining, 11(3), 42–77. https://doi.org/10.5281/zenodo.3594773
##plugins.themes.bootstrap3.article.details##
Keywords
latent-variable modeling, collective inference, probabilistic modeling, socio-behavioral modeling, high school, MOOC
References
ANDERSON, A., HUTTENLOCHER, D., KLEINBERG, J., AND LESKOVEC, J. 2014. Engaging with massive online courses. In International Conference on World Wide Web (WWW). 687–698.
ANDERSON, T. 2003. Getting the mix right again: An updated and theoretical rationale for interaction. The International Review of Research in Open and Distributed Learning 4, 2. Retrieved from: http://www.irrodl.org/index.php/irrodl/article/view/149/230.
ANDREWS-TODD, J., FORSYTH, C., STEINBERG, J., AND RUPP, A. 2018. Identifying profiles of collaborative problem solvers in an online electronics environment. In International Conference on Educational Data Mining (EDM), K. E. Boyer and M. Yudelson, Eds. 239–245.
AVIV, R., ERLICH, Z., RAVID, G., AND GEVA, A. 2003. Network analysis of knowledge construction in asynchronous learning networks. Journal of Asynchronous Learning Networks 7, 3, 1–23.
BACH, S. H., BROECHELER, M., HUANG, B., AND GETOOR, L. 2017. Hinge-loss Markov random fields and probabilistic soft logic. Journal of Machine Learning Research 18, 109, 3846–3912.
BAKER, R., CORBETT, A., KOEDINGER, K., AND WAGNER, A. 2004. Off-task behavior in the cognitive tutor classroom: When students "game the system". In SIGCHI Conference on Human Factors in Computing Systems. 383–390.
BLEI, D. M., NG, A. Y., AND JORDAN, M. I. 2003. Latent Dirichlet Allocation. Journal of Machine Learning Research 3, 993–1022.
BLOM, J., VERMA, H., LI, N., SKEVI, A., AND DILLENBOURG, P. 2013. MOOCs are more social than you believe. 33, 1–3.
BOCK, M. AND O'DEA, V. 2013. Virtual educators critique value of MOOCs for K-12. https://www. edweek.org/ew/articles/2013/02/06/20moocs.h32.html.
CHEN, G., DAVIS, D., HAUFF, C., AND HOUBEN, G.-J. 2016. On the impact of personality in massive open online learning. In User Modeling Adaptation and Personalization. 121–130.
CORBETT, A. T. AND ANDERSON, J. R. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction 4, 4, 253–278.
ER, E., GÓMEZ-SÁNCHEZ, E., BOTE-LORENZO, M. L., ASENSIO-PÉREZ, J. I., AND DIMITRIADIS, Y. 2019. Informing the design of collaborative activities in MOOCs using actionable predictions. In ACM Conference on Learning@ Scale (L@S). 27:1–27:4.
FRIEDMAN, N., GETOOR, L., KOLLER, D., AND PFEFFER, A. 1999. Learning probabilistic relational models. In IJCAI. Vol. 99. 1300–1309.
GARDNER, J. AND BROOKS, C. 2018. Student success prediction in MOOCs. User Modeling and UserAdapted Interaction 28, 2, 127–203.
GITINABARD, N., KHOSHNEVISAN, F., LYNCH, C., AND WANG, E. Y. 2018. Your actions or your associates? Predicting certification and dropout in MOOCs with behavioral and social features. In International Conference on Educational Data Mining (EDM), K. E. Boyer and M. Yudelson, Eds. 404–410.
HANZAKI, M. R. AND EPP, C. D. 2018. The effect of personality and course attributes on academic performance in MOOCs. In European conference on technology enhanced learning. 497–509.
HUANG, J., DASGUPTA, A., GHOSH, A., MANNING, J., AND SANDERS, M. 2014. Superposter behavior in MOOC forums. In ACM Conference on Learning @ Scale Conference (L@S). 117–126.
JURAFSKY, D. AND MARTIN, J. H. 2000. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Prentice Hall.
KELLOGG, S., BOOTH, S., AND OLIVER, K. 2014. Asocial network perspective on peer supported learning in MOOCs for educators. The International Review of Research in Open and Distributed Learning 15, 5, 263–289.
KENNEDY, G., COFFRIN, C., DE BARBA, P., AND CORRIN, L. 2015. Predicting success: How learners' prior knowledge, skills and activities predict MOOC performance. In International Conference on Learning Analytics And Knowledge (LAK). 136–140.
KIZILCEC, R. F., PIECH, C., AND SCHNEIDER, E. 2013. Deconstructing disengagement: Analyzing learner subpopulations in massive open online courses. In International Conference on Learning Analytics and Knowledge (LAK). 170–179.
KLÜSENER, M. AND FORTENBACHER, A. 2015. Predicting students' success based on forum activities in MOOCs. In International Conference on Intelligent Data Acquisition and Advanced Computing Systems (IDAACS). Vol. 2. 925–928.
KUMAR, R. AND ROSE, C. P. 2011. Architecture for building conversational agents that support collaborative learning. IEEE Transactions on Learning Technologies 4, 1, 21–34.
KURHILA, J. AND VIHAVAINEN, A. 2015. Apurposeful MOOC to alleviate insufficient CS education in Finnish schools. Transactions of Computing Education 15, 2, 1–18.
LI, N., VERMA, H., SKEVI, A., ZUFFEREY, G., BLOM, J., AND DILLENBOURG, P. 2014. Watching MOOCs together: investigating co-located MOOC study groups. Distance Education 35, 2, 217–233.
LI, X., WANG, T., AND WANG, H. 2017. Exploring n-gram features in clickstream data for MOOC learning achievement prediction. In International Conference on Database Systems for Advanced Applications. 328–339.
LONGSTAFF, E. 2017. Ritual in online communities: Astudy of post-voting in MOOC discussion forums. International Journal of Human Computer Interaction 33, 8, 655–663.
LOYA, A., GOPAL, A., SHUKLA, I., JERMANN, P., AND TORMEY, R. 2015. Conscientious behaviour, flexibility and learning in massive open on-line courses. Procedia-Social and Behavioral Sciences 191, 519–525.
MCCALLUM, A. 2002. MALLET: Amachine learning for language toolkit. http://mallet.cs.umass.edu/.
MEGA, C., RONCONI, L., AND DE BENI, R. 2014. What makes agood student? How emotions, selfregulated learning, and motivation contribute to academic achievement. Journal of Educational Psychology 106, 1, 121–131.
MOTZ, B., BUSEY, T., RICKERT, M., AND LANDY, D. 2018. Finding topics in enrollment data. In International Conference on Educational Data Mining (EDM), K. E. Boyer and M. Yudelson, Eds. 424–430.
NAJAFI, H., EVANS, R., AND FEDERICO, C. 2014. MOOC integration into secondary school courses. The International Review of Research in Open and Distributed Learning 15, 5, 306–322.
PEDREGOSA ET AL., F. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, Oct, 2825–2830.
RAMESH, A., GOLDWASSER, D., HUANG, B., DAUME III, H., AND GETOOR, L. 2014. Learning latent engagement patterns of students in online courses. In AAAI Conference on Artificial Intelligence. 1272–1278.
RICHARDSON, M. AND DOMINGOS, P. 2006. Markov logic networks. Machine learning 62, 1-2, 107– 136.
ROSÉ, C., WANG, Y.-C., CUI, Y., ARGUELLO, J., STEGMANN, K., WEINBERGER, A., AND FISCHER, F. 2008. Analyzing collaborative learning processes automatically: Exploiting the advances of computational linguistics in computer-supported collaborative learning. International Journal of Computer-Supported Collaborative Learning 3, 3, 237–271.
ROSÉ, C. P., CARLSON, R., YANG, D., WEN, M., RESNICK, L., GOLDMAN, P., AND SHERER, J. 2014. Social factors that contribute to attrition in MOOCs. In ACM Conference on Learning @ Scale Conference (L@S). 197–198.
ROSÉ, C. P., GOLDMAN, P., ZOLTNERS SHERER, J., AND RESNICK, L. 2015. Supportive technologies for group discussion in MOOCs. Current Issues in Emerging eLearning 2, 1, 5. Available at: https://scholarworks.umb.edu/ciee/vol2/iss1/5.
SIMON, B., PARRIS, J., AND SPACCO, J. 2013. How we teach impacts student learning: Peer instruction vs. lecture in CS0. In ACM Technical Symposium on Computer Science Education (SIGCSE). 41–46.
STAUBITZ, T. AND MEINEL, C. 2019. Graded team assignments in MOOCs: Effects of team composition and further factors on team dropout rates and performance. In ACM Conference on Learning@ Scale (L@S). 5:1–5:10.
SU, Y.-S., HUANG, C. S.-J., AND DING, T.-J. 2016. Examining the effects of MOOCs learning social search results on learning behaviors and learning outcomes. Eurasia Journal of Mathematics, Science and Technology Education 1, 9, 2517–2529.
SUN, Y., NI, L., ZHAO, Y., SHEN, X.-L., AND WANG, N. 2018. Understanding students engagement in MOOCs: An integration of self-determination theory and theory of relationship quality. British Journal of Educational Technology, 1–19.
TOMKINS, S., RAMESH, A., AND GETOOR, L. 2016. Predicting post-test performance from online student behavior: Ahigh school MOOC case study. In International Conference on Educational Data Mining (EDM), T. Barnes, M. Chi, and M. Feng, Eds. 239–246.
VELETSIANOS, G., COLLIER, A., AND SCHNEIDER, E. 2015. Digging deeper into learners' experiences in MOOCs: Participation in social networks outside of MOOCs, notetaking and contexts surrounding content consumption. British Journal of Educational Technology 46, 3, 570–587.
WANG, X., YANG, D., WEN, M., KOEDINGER, K., AND ROSÉ, C. P. 2015. Investigating how student's cognitive behavior in MOOC discussion forums affect learning gains. In International Conference on Educational Data Mining (EDM), O. C. Santos, J. G. Boticario, C. Romero, M. Pechenizkiy, A. Merceron, P. Mitros, J. M. Luna, C. Mihaescu, P. Moreno, A. Hershkovitz, S. Ventura, and M. Desmarais, Eds. 226–233.
WEN, M., YANG, D., AND ROSÉ, C. P. 2015. Virtual teams in massive open online courses. In International Conference on Artificial Intelligence in Education, C. Conati, N. Heffernan, A. Mitrovic, and F. M. Verdejo, Eds. 820–824.
ANDERSON, T. 2003. Getting the mix right again: An updated and theoretical rationale for interaction. The International Review of Research in Open and Distributed Learning 4, 2. Retrieved from: http://www.irrodl.org/index.php/irrodl/article/view/149/230.
ANDREWS-TODD, J., FORSYTH, C., STEINBERG, J., AND RUPP, A. 2018. Identifying profiles of collaborative problem solvers in an online electronics environment. In International Conference on Educational Data Mining (EDM), K. E. Boyer and M. Yudelson, Eds. 239–245.
AVIV, R., ERLICH, Z., RAVID, G., AND GEVA, A. 2003. Network analysis of knowledge construction in asynchronous learning networks. Journal of Asynchronous Learning Networks 7, 3, 1–23.
BACH, S. H., BROECHELER, M., HUANG, B., AND GETOOR, L. 2017. Hinge-loss Markov random fields and probabilistic soft logic. Journal of Machine Learning Research 18, 109, 3846–3912.
BAKER, R., CORBETT, A., KOEDINGER, K., AND WAGNER, A. 2004. Off-task behavior in the cognitive tutor classroom: When students "game the system". In SIGCHI Conference on Human Factors in Computing Systems. 383–390.
BLEI, D. M., NG, A. Y., AND JORDAN, M. I. 2003. Latent Dirichlet Allocation. Journal of Machine Learning Research 3, 993–1022.
BLOM, J., VERMA, H., LI, N., SKEVI, A., AND DILLENBOURG, P. 2013. MOOCs are more social than you believe. 33, 1–3.
BOCK, M. AND O'DEA, V. 2013. Virtual educators critique value of MOOCs for K-12. https://www. edweek.org/ew/articles/2013/02/06/20moocs.h32.html.
CHEN, G., DAVIS, D., HAUFF, C., AND HOUBEN, G.-J. 2016. On the impact of personality in massive open online learning. In User Modeling Adaptation and Personalization. 121–130.
CORBETT, A. T. AND ANDERSON, J. R. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction 4, 4, 253–278.
ER, E., GÓMEZ-SÁNCHEZ, E., BOTE-LORENZO, M. L., ASENSIO-PÉREZ, J. I., AND DIMITRIADIS, Y. 2019. Informing the design of collaborative activities in MOOCs using actionable predictions. In ACM Conference on Learning@ Scale (L@S). 27:1–27:4.
FRIEDMAN, N., GETOOR, L., KOLLER, D., AND PFEFFER, A. 1999. Learning probabilistic relational models. In IJCAI. Vol. 99. 1300–1309.
GARDNER, J. AND BROOKS, C. 2018. Student success prediction in MOOCs. User Modeling and UserAdapted Interaction 28, 2, 127–203.
GITINABARD, N., KHOSHNEVISAN, F., LYNCH, C., AND WANG, E. Y. 2018. Your actions or your associates? Predicting certification and dropout in MOOCs with behavioral and social features. In International Conference on Educational Data Mining (EDM), K. E. Boyer and M. Yudelson, Eds. 404–410.
HANZAKI, M. R. AND EPP, C. D. 2018. The effect of personality and course attributes on academic performance in MOOCs. In European conference on technology enhanced learning. 497–509.
HUANG, J., DASGUPTA, A., GHOSH, A., MANNING, J., AND SANDERS, M. 2014. Superposter behavior in MOOC forums. In ACM Conference on Learning @ Scale Conference (L@S). 117–126.
JURAFSKY, D. AND MARTIN, J. H. 2000. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Prentice Hall.
KELLOGG, S., BOOTH, S., AND OLIVER, K. 2014. Asocial network perspective on peer supported learning in MOOCs for educators. The International Review of Research in Open and Distributed Learning 15, 5, 263–289.
KENNEDY, G., COFFRIN, C., DE BARBA, P., AND CORRIN, L. 2015. Predicting success: How learners' prior knowledge, skills and activities predict MOOC performance. In International Conference on Learning Analytics And Knowledge (LAK). 136–140.
KIZILCEC, R. F., PIECH, C., AND SCHNEIDER, E. 2013. Deconstructing disengagement: Analyzing learner subpopulations in massive open online courses. In International Conference on Learning Analytics and Knowledge (LAK). 170–179.
KLÜSENER, M. AND FORTENBACHER, A. 2015. Predicting students' success based on forum activities in MOOCs. In International Conference on Intelligent Data Acquisition and Advanced Computing Systems (IDAACS). Vol. 2. 925–928.
KUMAR, R. AND ROSE, C. P. 2011. Architecture for building conversational agents that support collaborative learning. IEEE Transactions on Learning Technologies 4, 1, 21–34.
KURHILA, J. AND VIHAVAINEN, A. 2015. Apurposeful MOOC to alleviate insufficient CS education in Finnish schools. Transactions of Computing Education 15, 2, 1–18.
LI, N., VERMA, H., SKEVI, A., ZUFFEREY, G., BLOM, J., AND DILLENBOURG, P. 2014. Watching MOOCs together: investigating co-located MOOC study groups. Distance Education 35, 2, 217–233.
LI, X., WANG, T., AND WANG, H. 2017. Exploring n-gram features in clickstream data for MOOC learning achievement prediction. In International Conference on Database Systems for Advanced Applications. 328–339.
LONGSTAFF, E. 2017. Ritual in online communities: Astudy of post-voting in MOOC discussion forums. International Journal of Human Computer Interaction 33, 8, 655–663.
LOYA, A., GOPAL, A., SHUKLA, I., JERMANN, P., AND TORMEY, R. 2015. Conscientious behaviour, flexibility and learning in massive open on-line courses. Procedia-Social and Behavioral Sciences 191, 519–525.
MCCALLUM, A. 2002. MALLET: Amachine learning for language toolkit. http://mallet.cs.umass.edu/.
MEGA, C., RONCONI, L., AND DE BENI, R. 2014. What makes agood student? How emotions, selfregulated learning, and motivation contribute to academic achievement. Journal of Educational Psychology 106, 1, 121–131.
MOTZ, B., BUSEY, T., RICKERT, M., AND LANDY, D. 2018. Finding topics in enrollment data. In International Conference on Educational Data Mining (EDM), K. E. Boyer and M. Yudelson, Eds. 424–430.
NAJAFI, H., EVANS, R., AND FEDERICO, C. 2014. MOOC integration into secondary school courses. The International Review of Research in Open and Distributed Learning 15, 5, 306–322.
PEDREGOSA ET AL., F. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, Oct, 2825–2830.
RAMESH, A., GOLDWASSER, D., HUANG, B., DAUME III, H., AND GETOOR, L. 2014. Learning latent engagement patterns of students in online courses. In AAAI Conference on Artificial Intelligence. 1272–1278.
RICHARDSON, M. AND DOMINGOS, P. 2006. Markov logic networks. Machine learning 62, 1-2, 107– 136.
ROSÉ, C., WANG, Y.-C., CUI, Y., ARGUELLO, J., STEGMANN, K., WEINBERGER, A., AND FISCHER, F. 2008. Analyzing collaborative learning processes automatically: Exploiting the advances of computational linguistics in computer-supported collaborative learning. International Journal of Computer-Supported Collaborative Learning 3, 3, 237–271.
ROSÉ, C. P., CARLSON, R., YANG, D., WEN, M., RESNICK, L., GOLDMAN, P., AND SHERER, J. 2014. Social factors that contribute to attrition in MOOCs. In ACM Conference on Learning @ Scale Conference (L@S). 197–198.
ROSÉ, C. P., GOLDMAN, P., ZOLTNERS SHERER, J., AND RESNICK, L. 2015. Supportive technologies for group discussion in MOOCs. Current Issues in Emerging eLearning 2, 1, 5. Available at: https://scholarworks.umb.edu/ciee/vol2/iss1/5.
SIMON, B., PARRIS, J., AND SPACCO, J. 2013. How we teach impacts student learning: Peer instruction vs. lecture in CS0. In ACM Technical Symposium on Computer Science Education (SIGCSE). 41–46.
STAUBITZ, T. AND MEINEL, C. 2019. Graded team assignments in MOOCs: Effects of team composition and further factors on team dropout rates and performance. In ACM Conference on Learning@ Scale (L@S). 5:1–5:10.
SU, Y.-S., HUANG, C. S.-J., AND DING, T.-J. 2016. Examining the effects of MOOCs learning social search results on learning behaviors and learning outcomes. Eurasia Journal of Mathematics, Science and Technology Education 1, 9, 2517–2529.
SUN, Y., NI, L., ZHAO, Y., SHEN, X.-L., AND WANG, N. 2018. Understanding students engagement in MOOCs: An integration of self-determination theory and theory of relationship quality. British Journal of Educational Technology, 1–19.
TOMKINS, S., RAMESH, A., AND GETOOR, L. 2016. Predicting post-test performance from online student behavior: Ahigh school MOOC case study. In International Conference on Educational Data Mining (EDM), T. Barnes, M. Chi, and M. Feng, Eds. 239–246.
VELETSIANOS, G., COLLIER, A., AND SCHNEIDER, E. 2015. Digging deeper into learners' experiences in MOOCs: Participation in social networks outside of MOOCs, notetaking and contexts surrounding content consumption. British Journal of Educational Technology 46, 3, 570–587.
WANG, X., YANG, D., WEN, M., KOEDINGER, K., AND ROSÉ, C. P. 2015. Investigating how student's cognitive behavior in MOOC discussion forums affect learning gains. In International Conference on Educational Data Mining (EDM), O. C. Santos, J. G. Boticario, C. Romero, M. Pechenizkiy, A. Merceron, P. Mitros, J. M. Luna, C. Mihaescu, P. Moreno, A. Hershkovitz, S. Ventura, and M. Desmarais, Eds. 226–233.
WEN, M., YANG, D., AND ROSÉ, C. P. 2015. Virtual teams in massive open online courses. In International Conference on Artificial Intelligence in Education, C. Conati, N. Heffernan, A. Mitrovic, and F. M. Verdejo, Eds. 820–824.
Issue
Section
Articles
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors who publish with this journal agree to the following terms:
- The Author retains copyright in the Work, where the term “Work” shall include all digital objects that may result in subsequent electronic publication or distribution.
- Upon acceptance of the Work, the author shall grant to the Publisher the right of first publication of the Work.
- The Author shall grant to the Publisher and its agents the nonexclusive perpetual right and license to publish, archive, and make accessible the Work in whole or in part in all forms of media now or hereafter known under a Creative Commons 4.0 License (Attribution-Noncommercial-No Derivatives 4.0 International), or its equivalent, which, for the avoidance of doubt, allows others to copy, distribute, and transmit the Work under the following conditions:
- Attribution—other users must attribute the Work in the manner specified by the author as indicated on the journal Web site;
- Noncommercial—other users (including Publisher) may not use this Work for commercial purposes;
- No Derivative Works—other users (including Publisher) may not alter, transform, or build upon this Work,with the understanding that any of the above conditions can be waived with permission from the Author and that where the Work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.
- The Author is able to enter into separate, additional contractual arrangements for the nonexclusive distribution of the journal's published version of the Work (e.g., post it to an institutional repository or publish it in a book), as long as there is provided in the document an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post online a pre-publication manuscript (but not the Publisher’s final formatted PDF version of the Work) in institutional repositories or on their Websites prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (see The Effect of Open Access). Any such posting made before acceptance and publication of the Work shall be updated upon publication to include a reference to the Publisher-assigned DOI (Digital Object Identifier) and a link to the online abstract for the final published Work in the Journal.
- Upon Publisher’s request, the Author agrees to furnish promptly to Publisher, at the Author’s own expense, written evidence of the permissions, licenses, and consents for use of third-party material included within the Work, except as determined by Publisher to be covered by the principles of Fair Use.
- The Author represents and warrants that:
- the Work is the Author’s original work;
- the Author has not transferred, and will not transfer, exclusive rights in the Work to any third party;
- the Work is not pending review or under consideration by another publisher;
- the Work has not previously been published;
- the Work contains no misrepresentation or infringement of the Work or property of other authors or third parties; and
- the Work contains no libel, invasion of privacy, or other unlawful matter.
- The Author agrees to indemnify and hold Publisher harmless from Author’s breach of the representations and warranties contained in Paragraph 6 above, as well as any claim or proceeding relating to Publisher’s use and publication of any content contained in the Work, including third-party content.