Exploring the Effect of Student Confusion in Massive Open Online Courses



Published Nov 3, 2016
Diyi Yang Robert Kraut Carolyn Rose


Although thousands of students enroll in Massive Open Online Courses (MOOCs) for learning and self-improvement, many get confused, harming learning and increasing dropout rates. In this paper, we quantify these effects in two large MOOCs. We first describe how we automatically estimate students' confusion by looking at their clicking behavior on course content and participation in the course discussion forums. We then apply survival analysis to quantify the impact of confusion on students' dropout. The results demonstrate that the more confusion students express themselves and the more they are exposed to other students' confusion, the sooner they drop out of the course. We also explore the effects of confusion expressed in different contexts and related to different aspects of courses. We conclude with implications for the design of interventions to improve student retention in MOOCs.

How to Cite

Yang, D., Kraut, R., & Rose, C. (2016). Exploring the Effect of Student Confusion in Massive Open Online Courses. Journal of Educational Data Mining, 8(1), 52–83. https://doi.org/10.5281/zenodo.3554605
Abstract 747 | PDF Downloads 558



MOOCs, drop out, affective states, confusion, discussion forums, clickstream data

AGRAWAL, A., VENKATRAMAN, J., LEONARD, S., AND PAEPCKE, A. 2015. Youedu: Addressing confusion in MOOC discussion forums by recommending instructional video clips. In National Science Foundation. Stanford InfoLab.

ALARIO-HOYOS, C., PEREZ-SANAGUSTIN, M., DELGADO-KLOOS, C., MUNOZ-ORGANERO, M., RODRIGUEZ-DE-LAS HERAS, A., ET AL. 2013. Analysing the impact of built-in and external social tools in a MOOC on educational technologies. In Scaling up learning for sustained impact. Springer, 5–18.

ANDERSON, A., HUTTENLOCHER, D., KLEINBERG, J., AND LESKOVEC, J. 2014. Engaging with massive online courses. In Proceedings of the 23rd International Conference on World Wide Web. WWW '14. ACM, New York, NY, USA, 687–698.

ARROYO, I., COOPER, D. G., BURLESON, W., WOOLF, B. P., MULDNER, K., AND CHRISTOPHERSON, R. 2009. Emotion sensors go to school. In AIED. Vol. 200. 17–24.

BAKER, R. S., GOWDA, S. M., WIXON, M., KALKA, J., WAGNER, A. Z., SALVI, A., ALEVEN, V., KUSBIT, G. W., OCUMPAUGH, J., AND ROSSI, L. 2012. Towards sensor-free affect detection in cognitive tutor algebra. International Educational Data Mining Society.

BOSCH, N., CHEN, Y., AND DMELLO, S. 2014. Its written on your face: detecting affective states from facial expressions while learning computer programming. In Intelligent Tutoring Systems. Springer, 39–44.

BOYER, S. AND VEERAMACHANENI, K. 2015. Transfer learning for predictive models in massive open online courses. In International Conference on Artificial Intelligence in Education. Springer, 54–63.

BUHRMESTER, M., KWANG, T., AND GOSLING, S. D. 2011. Amazon's mechanical turk a new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science 6, 1, 3–5.

CALVO, R. A. AND D'MELLO, S. 2010. Affect detection: An interdisciplinary review of models, methods, and their applications. Affective Computing, IEEE Transactions on 1, 1, 18–37.

CAPRARA, G. V., FIDA, R., VECCHIONE, M., DEL BOVE, G., VECCHIO, G. M., BARBARANELLI, C., AND BANDURA, A. 2008. Longitudinal analysis of the role of perceived self-efficacy for self-regulated learning in academic continuance and achievement. Journal of Educational Psychology 100, 3, 525.

CHATURVEDI, S., GOLDWASSER, D., AND DAUME III, H. 2014. Predicting instructor's intervention in MOOC forums. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Baltimore, Maryland, 1501–1511.

CONATI, C. AND MACLAREN, H. 2009. Empirically building and evaluating a probabilistic model of user affect. User Modeling and User-Adapted Interaction 19, 3, 267–303.

CONOLE, G. 2013. MOOCs as disruptive technologies: strategies for enhancing the learner experience and quality of MOOCs. Revista de Educacion a Distancia 39 , 1–17.

D'MELLO, S. AND GRAESSER, A. 2007. Mind and body: Dialogue and posture for affect detection in learning environments. Frontiers in Artificial Intelligence and Applications 158, 161.

D'MELLO, S., OLNEY, A., AND PERSON, N. 2010. Mining collaborative patterns in tutorial dialogues. JEDM-Journal of Educational Data Mining 2, 1, 2–37.

DMELLO, S. AND GRAESSER, A. 2012. Dynamics of affective states during complex learning. Learning and Instruction 22, 2, 145–157.

DMELLO, S., LEHMAN, B., PEKRUN, R., AND GRAESSER, A. 2014. Confusion can be beneficial for learning. Learning and Instruction 29, 153–170.

DMELLO, S. K., CRAIG, S. D., WITHERSPOON, A., MCDANIEL, B., AND GRAESSER, A. 2008. Automatic detection of learners affect from conversational cues. User modeling and user-adapted interaction 18, 1-2, 45–80.

EZEN-CAN, A. AND BOYER, K. E. 2015. Understanding student language: An unsupervised dialogue act classification approach. JEDM-Journal of Educational Data Mining 7, 1, 51–78.

GUPTA, N. K. AND ROSE, C. P. 2010. Understanding instructional support needs of emerging internet users for web-based information seeking. JEDM-Journal of Educational Data Mining 2, 1, 38–82.

HAN, J., CHENG, H., XIN, D., AND YAN, X. 2007. Frequent pattern mining: current status and future directions. Data Mining and Knowledge Discovery 15, 1, 55–86.

HOWLEY, I. 2015. Leveraging educational technology to overcome social obstacles to help seeking. Ph.D. thesis, Carnegie Mellon University.

HOWLEY, I., TOMAR, G., YANG, D., FERSCHKE, O., AND ROSE, C. P. 2015. Alleviating the negative effect of up and downvoting on help seeking in MOOC discussion forums. In AIED.

HUSSAIN, M. S., MONKARESI, H., AND CALVO, R. A. 2012. Categorical vs. dimensional representations in multimodal affect detection during learning. In Proceedings of the 11th International Conference on Intelligent Tutoring Systems. ITS'12. Springer-Verlag, Berlin, Heidelberg, 78–83.

JORDAN, K. 2014. Initial trends in enrolment and completion of massive open online courses. The International Review Of Research In Open And Distributed Learning 15, 1.

KIM, J., GUO, P. J., SEATON, D. T., MITROS, P., GAJOS, K. Z., AND MILLER, R. C. 2014. Understanding in-video dropouts and interaction peaks inonline lecture videos. In Proceedings of the First ACM Conference on Learning @ Scale Conference. L@S '14. ACM, New York, NY, USA, 31–40.

KIZILCEC, R. F. AND SCHNEIDER, E. 2015. Motivation as a lens to understand online learners: Toward data-driven design with the olei scale. ACM Trans. Comput.-Hum. Interact. 22, 2 (Mar.), 6:1–6:24.

KOEDINGER, K. R., KIM, J., JIA, J. Z., MCLAUGHLIN, E. A., AND BIER, N. L. 2015. Learning is not a spectator sport: Doing is better than watching for learning from a MOOC. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale. L@S '15. ACM, New York, NY, USA, 111–120.

KOLLER, D., NG, A., DO, C., AND CHEN, Z. 2013. Retention and intention in massive open online courses: In depth. Educause Review 48, 3, 62–63.

LARSON, R. W. AND RICHARDS, M. H. 1991. Boredom in the middle school years: Blaming schools versus blaming students. American Journal of Education, 418–443.

LEE, D. M. C., RODRIGO, M. M. T., D BAKER, R. S., SUGAY, J. O., AND CORONEL, A. 2011. Exploring the relationship between novice programmer confusion and achievement. In Affective Computing and Intelligent Interaction. Springer, 175–184.

LEHMAN, B., DMELLO, S., AND GRAESSER, A. 2012. Interventions to regulate confusion during learning. In Intelligent Tutoring Systems. Springer, 576–578.

LEHMAN, B. AND GRAESSER, A. 2015. To resolve or not to resolve? that is the big question about confusion. In Artificial Intelligence in Education. Springer, 216–225.

LEHMAN, B., MATTHEWS, M., DMELLO, S., AND PERSON, N. 2008. What are you feeling? investigating student affective states during expert human tutoring sessions. In Intelligent Tutoring Systems. Springer, 50–59.

LIU, C., RANI, P., AND SARKAR, N. 2005. An empirical study of machine learning techniques for affect recognition in human-robot interaction. In Intelligent Robots and Systems, 2005.(IROS 2005). 2005 IEEE/RSJ International Conference on. IEEE, 2662–2667.

NICOLAOU, M. A., GUNES, H., AND PANTIC, M. 2011. Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space. Affective Computing, IEEE Transactions on 2, 2, 92–105.

OCUMPAUGH, J., BAKER, R., GOWDA, S., HEFFERNAN, N., AND HEFFERNAN, C. 2014. Population validity for educational data mining models: A case study in affect detection. British Journal of Educational Technology 45, 3, 487–501.

PARDOS, Z. A., BAKER, R. S., SAN PEDRO, M. O., GOWDA, S. M., AND GOWDA, S. M. 2013. Affective states and state tests: Investigating how affect throughout the school year predicts end of year learning outcomes. In Proceedings of the Third International Conference on Learning Analytics and Knowledge. ACM, 117–124.

PEKRUN, R., GOETZ, T., TITZ, W., AND PERRY, R. P. 2002. Academic emotions in students' selfregulated learning and achievement: A program of qualitative and quantitative research. Educational psychologist 37, 2, 91–105.

PENNEBAKER, J. W., FRANCIS, M. E., AND BOOTH, R. J. 2001. Linguistic inquiry and word count: Liwc 2001. Mahway: Lawrence Erlbaum Associates 71, 2001.

RAMESH, A., GOLDWASSER, D., HUANG, B., DAUME III, H., AND GETOOR, L. 2013. Modeling learner engagement in MOOCs using probabilistic soft logic. In NIPS Workshop on Data Driven Education.

ROSE, C. P., CARLSON, R., YANG, D., WEN, M., RESNICK, L., GOLDMAN, P., AND SHERER, J. 2014. Social factors that contribute to attrition in MOOCs. In Proceedings of the first ACM conference on Learning@ scale conference. ACM, 197–198.

SAARELA, M. AND KARKKAINEN, T. 2015. Analysing student performance using sparse data of core bachelor courses. JEDM-Journal of Educational Data Mining 7, 1, 3–32.

SHUTE, V. J. 2008. Focus on formative feedback. Review of educational research 78, 1, 153–189.

SINHA, T., JERMANN, P., LI, N., AND DILLENBOURG, P. 2014. Your click decides your fate: Leveraging clickstream patterns in MOOC videos to infer students' information processing and attrition behavior. CoRR abs/1407.7131.

SNOW, R., O'CONNOR, B., JURAFSKY, D., AND NG, A. Y. 2008. Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. In EMNLP. Association for Computational Linguistics, 254–263.

STATA, C. 2011. Stata statistical software release. In 7.0: Programming. Stata Corporation. 418–443.

VANLEHN, K., SILER, S., MURRAY, C., YAMAUCHI, T., AND BAGGETT, W. B. 2003. Why do only some events cause learning during human tutoring? Cognition and Instruction 21, 3, 209–249.

WANG, Y.-C., KRAUT, R., AND LEVINE, J. M. 2012. To stay or leave?: The relationship of emotional and informational support to commitment in online health support groups. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work. CSCW '12. ACM, New York, NY, USA, 833–842.

WEN, M., YANG, D., AND ROSE, C. P. 2014a. Linguistic reflections of student engagement in massive open online courses. In Proceedings of the International Conference on Weblogs and Social Media.

WEN, M., YANG, D., AND ROSE, C. P. 2014b. Sentiment analysis in MOOC discussion forums: What does it tell us? In Proceedings of Educational Data Mining.

WERNER, L., MCDOWELL, C., AND DENNER, J. 2013. A first step in learning analytics: preprocessing low-level alice logging data of middle school students. JEDM-Journal of Educational Data Mining 5, 2, 11–37.

WILSON, N. 1989. Learning from confusion: Questions and change in reading logs. English Journal, 62–69.

WIXON, M., ARROYO, I., MULDNER, K., BURLESON, W., RAI, D., AND WOOLF, B. 2014. The opportunities and limitations of scaling up sensor-free affect detection. In Educational Data Mining 2014.

YANG, D., ADAMSON, D., AND ROSE, C. P. 2014. Question recommendation with constraints for massive open online courses. In Proceedings of the 8th ACM Conference on Recommender Systems. RecSys '14. ACM, New York, NY, USA, 49–56.

YANG, D., SINHA, T., ADAMSON, D., AND ROSE, C. P. 2013. Turn on, tune in, drop out: Anticipating student dropouts in massive open online courses. In Proceedings of the 2013 NIPS Data-Driven Education Workshop. Vol. 10. 13.

YANG, D., WEN, M., HOWLEY, I., KRAUT, R., AND ROSE, C. 2015. Exploring the effect of confusion in discussion forums of massive open online courses. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale. L@S '15. ACM, New York, NY, USA, 121–130.

YANG, D., WEN, M., AND ROSE, C. 2014. Peer influence on attrition in massive open online courses. Proceedings of Educational Data Mining.
EDM 2016 Journal Track