Considering Alternate Futures to Classify Off-Task Behavior as Emotion Self-Regulation: A Supervised Learning Approach

##plugins.themes.bootstrap3.article.main##

##plugins.themes.bootstrap3.article.sidebar##

Published May 1, 2013
Jennifer L. Sabourin Jonathan P. Rowe Bradford W. Mott James C. Lester

Abstract

Over the past decade, there has been growing interest in real-time assessment of student engagement and motivation during interactions with educational software. Detecting symptoms of disengagement, such as offtask behavior, has shown considerable promise for understanding students' motivational characteristics during learning. In this paper, we investigate the affective role of off-task behavior by analyzing data from student interactions with CRYSTAL ISLAND, a narrative-centered learning environment for middle school microbiology. We observe that off-task behavior is associated with reduced student learning, but preliminary analyses of students' affective transitions suggest that off-task behavior may also serve a productive role for some students coping with negative affective states such as frustration. Empirical findings imply that some students may use off-task behavior as a strategy for self-regulating negative emotional states during learning. Based on these observations, we introduce a supervised machine learning procedure for detecting whether students' off-task behaviors are cases of emotion self-regulation. The method proceeds in three stages. During the first stage, a dynamic Bayesian network (DBN) is trained to model the valence of students' emotion self-reports using collected data from interactions with the learning environment. In the second stage, a novel simulation process uses the DBN to generate alternate futures by modeling students' affective trajectories as if they had engaged in fewer off-task behaviors than they did during their actual learning interactions. The alternate futures are compared to students' actual traces to produce labels denoting whether students' off-task behaviors are cases of emotion self-regulation. In the final stage, the generated emotion self-regulation labels are predicted using off-the-shelf classifiers and features that can be computed in run-time settings. Results suggest that this approach shows promise for identifying cases of off-task behavior that are emotion self-regulation. Analyses of the first two phases suggest that trained DBN models are capable of accurately modeling relationships between students' off-task behaviors and self-reported emotional valence in CRYSTAL ISLAND. Additionally, the proposed simulation process produces emotion self-regulation labels with high levels of reliability. Preliminary analyses indicate that support vector machines, bagged trees, and random forests show promise for predicting the generated emotion self-regulation labels, but room for improvement remains. The findings underscore the methodological potential of considering alternate futures when modeling students' emotion self-regulation processes in narrative-centered learning environments.

How to Cite

Sabourin, J. L., Rowe, J. P., Mott, B. W., & Lester, J. C. (2013). Considering Alternate Futures to Classify Off-Task Behavior as Emotion Self-Regulation: A Supervised Learning Approach. JEDM | Journal of Educational Data Mining, 5(1), 9-38. https://doi.org/10.5281/zenodo.3554607
Abstract 409 | PDF Downloads 293

##plugins.themes.bootstrap3.article.details##

Keywords

affective states, assessment, dynamic Bayesian network, off-task behavior, self-regulation, simulation

References
ALEVEN, V., MCLAREN, B.M., ROLL, I., and KOEDINGER, K.R. 2006. Toward meta-cognitive tutoring: A model of help-seeking with a cognitive tutor. Internaitonal Journal of Artificial Intelligence in Education, 16, 101-128.

ALPAYDIN, E. 2004. Introduction to Machine Learning, MIT Press.

ARROYO, I., FERGUSON, K., JOHNS, J., DRAGON, T., MEHERANIAN, H., FISHER, D., BARTO, A., MAHADEVAN, S., and WOOLF, B.P. 2007. Repairing disengagement with non-invasive interventions. Proceedings of the 13th International Conference on Artificial Intelligence in Education, 195-202.

BAKER, R.S. 2007. Modeling and understanding students' off-task behavior in intelligent tutoring systems, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1059-1068.

BAKER, R.S., CORBETT, A.T., and ALEVEN, V. 2008. More accurate student modeling through contextual estimation of slip and guess probabilities in Bayesian knowledge tracing. Proceedings of the Ninth International Conference on Intelligent Tutoring Systems, 406-415.

BAKER, R.S., CORBETT, K.R., KOEDINGER, K.R., EVENSON, S., ROLL, I., WAGNER, A., NAIM, M., ... BECK, J. 2006. Adapting to when students game an intelligent tutoring system, Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 392-401.

BAKER, R.S., CORBETT, A.T., KOEDINGER, K.R., and WAGNER, A. 2004. Off-task behavior in the cognitive tutor classroom: When students game the system. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 383-390.

BAKER, R.S., CORBETT, A.T., ROLL, I., and KOEDINGER, K.R. 2008. Developing a generalizable detector of when students game the system. User Modeling and UserAdapted Interaction, 18, 287-314.

BAKER, R.S., D'MELLO, S.K., RODRIGO, S.K., and GRAESSER, A.C. 2010. Better to be frustrated than bored: The incidence, persistence, and impact of learners' cognitive-affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68, 223-241.

BAKER, R.S., MOORE, G., WAGNER, A., KALKA, J., SALVI, A., KARABINOS, M., ASHE, C., and YARON, D. 2011. The dynamics between student affect and behavior occurring outside of educational software. Proceedings of the 4th International Conference on Affective Computing and Intelligent Interaction, 14-24.

BEAL, C.R., MITRA, S., and COHEN, P.R. 2007. Modeling learning patterns of students with a tutoring system using Hidden Markov Models. In Proceedings of the 13th International Conference of Artificial Intelligence in Education, 238-245.

BEAL, C.R., QU, L., and LEE, H. 2006. Classifying learner engagement through integration of multiple data sources. Proceedings of the National Conference on Artificial Intelligence, 2-8.

BLOOM, B.S. 1984. The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13, 4-16.

BRUNER, J.S., 1990. Acts of Meaning, Cambridge, MA: Harvard University Press.

BUNT, A. and CONATI, C. 2003. Probabilistic student modeling to improve exploratory behavior. User Modeling and User-Adapted Interaction, 13, 269-309.

CETINTAS, S., LUO, S., XIN, Y., HORD, C., and ZHANG, D. 2009. Learning to identify students off-task behavior in intelligent tutoring systems, Proceedings of the 14th International Conference on Artificial Intelligence in Education, 701-703.

COCEA, M., HERSHKOVITZ, A., and BAKER, R.S. 2009. The impact of off-task and gaming behaviors on learning: Immediate or aggregate. Proceedings of the 14th International Conference on Artificial Intelligence in Education, 507-514.

CONATI, C. and MACLAREN, H. 2009. Empirically building and evaluating a probabilistic model of user affect. User Modeling and User-Adapted Interaction, 19, 267- 303.

CORBETT, A.T. AND ANDERSON, J.R. 1994. Knowledge tracing: Modeling the Acquisition of Procedural Knowledge. User Modeling and User-Adapted Interaction, 4, 253-278.

D'MELLO, S., TAYLOR, R.S., and GRAESSER, A.C. 2007. Monitoring affective trajectories during complex learning. Proceedings of the 29th Annual Meeting of the Cognitive Science Society, 203-208.

DWECK, C.S. and LEGGET, E.L. A social-cognitive approach to motivation and personality. Psychological Review, 95, 256-273.

ELLIOT, A. and MCGREGOR, H.A. 2001. A 2 x 2 achievement goal framework, Journal of Personality and Social Psychology, 80, 501-519.

ELLIOT, A. and PEKRUN, R. 2007. Emotion in the hierarchical model of approachavoidance achievement motivation. Emotion in Education, P. SCHUTZ and R. PEKRUN, eds., London: Elsevier, 57-74.

GERNEFSKI, N. and KRAATI, V. 2006. Cognitive Emotion Regulation Questionaire: Development of a short 18-item version. Personality and Individual Differences, 41, 1045-1053.

GERTNER, A., CONATI, C., and VANLEHN, K. 1998. Procedural help in Andes: Generating hints using a Bayesian network student model. Proceedings of the 15th National Conference on Artificial Intelligence.

GONG, Y., BECK, J., HEFFERNAN, N., and FORBES-SUMMERS, E. 2010. The finegrained impact of gaming on learning. Proceedings of the 10th International Conference on Intelligent Tutoring Systems, 194-203.

GRAESSER, A.C., PERSON, N.K., MAGLIANO, J.P. Collaborative dialogue patterns in naturalistic one-to-one tutoring, Applied Cognitive Psychology, 9, 495-522.

HARP, S. and MAYER, R.E. 1998. How seductive details do their damage: A theory of cognitive interest in science learning. Journal of Educational Psychology, 90, 414-434.

HALL, M., FRANK, E., HOLMES, G., PFAHRINGER, B., REUTEMANN, P., and WITTEN, I. The WEKA data mining software: An update. SIGKDD Explorations, 11, 2009.

HICKEY, D.T., INGRAM-GOBLE, A.A., and JAMESON, E.M. 2009. Designing assessments and assessing designs in virtual educational environments. Journal of Science Education and Technology, 18, 187-208.

KETELHUT, D.J. The impact of student self-efficacy on scientific inquiry skills: An exploratory investigation in River City, a multi-user virtual environment. Journal of Science Education and Technology, 6, 99-111.

MANDLER, J. and JOHNSON, N. 1988. Remeberance of things parsed: Story structure and recall. Journal of Cognitive Psychology, 9, 111-151.

MARSELLA, S.C., JOHNSON, W.L., and LABORE, C.M. 2000. Interactive pedagogical drama. Proceedings of the 5th International Conference on Intelligent Virtual Agents, 305-315.

MCCRAE, R. and COSTA, P. 1993. Personality in Adulthood: A Five-Factor Theory Perspective, New York: Guilford Press.

MEYER, D. and TURNER, J. 2006. Reconceptualizing emotion and motivation to learn in classroom contexts, Educational Psychology Review, 18, 377-390.

MULDNER, K., BURLESON, B., VAN DE SAND, B., and VANLEHN, K. 2010. An analysis of gaming behaviors in an intelligent tutoring system. Proceedings of the 10th International Conference on Intelligent Tutoring Systems, 184-193.

MURRAY, R.C. and VANLEHN, K. Effects of dissuading unnecessary help requests while providing proactive help, Proceedings of the 12th International Conference on Artificial Intelligence in Education, 887-889.

O'NEILL, B. and RIEDL, M.O. 2011. Toward a computational framework of suspense and dramatic arc. Proceedings of the 4th International Conference on Affective Computing and Intelligent Interaction, 256-263.

RODRIGO, M., BAKER, R.S., LAGUD, M., LIM, S., MACAPANPAN, A., PASCUA, S., SANTILLANO, J., ...VIEHLAND, N. Affect and usage choices in simulation problem-solving environments, Proceedings of the 13th International Conference on Artificial Intelligence in Education, 145-152.

ROWE, J.P., MCQUIGGAN, S.W., ROBISON, J.L., and LESTER, J.C. 2009. Off-task behavior in narrative-centered learning environments. Proceedings of the 14th International Conference on Artificial Intelligence and Education, 99-106.

ROWE, J.P., SHORES, L.R., MOTT, B.W., and LESTER, J.C. 2011. Integrating learning, problem solving, and engagement in narrative-centered learning environments, International Journal of Artificial Intelligence in Education, 21, 115-133.

SABOURIN, J.L., MOTT, B.W., and LESTER, J.C. 2011. Modeling learner affect with theoretically grounded dynamic Bayesian networks. Proceedings of the 4th International Conference on Affective Computing and Intelligent Interaction, 286-295.

SABOURIN, J.L., ROWE, J.P., MOTT, B.W., and LESTER, J.C. 2011. When off-task in on-task: The affective role of off-task behavior in narrative-centered learning environments. Proceedings of the 15th International Conference on Artificial Intelligence and Education, 534-536.

VANLEHN, K. 2011. The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46, 197-221.

VANLEHN, K. 2006. The behaviour of tutoring systems, International Journal of Artificial Intelligence in Education, 16, 227-265.
Section
Articles