Over the past decade, there has been growing interest in real-time assessment of student engagement and motivation during interactions with educational software. Detecting symptoms of disengagement, such as offtask behavior, has shown considerable promise for understanding students' motivational characteristics during learning. In this paper, we investigate the affective role of off-task behavior by analyzing data from student interactions with CRYSTAL ISLAND, a narrative-centered learning environment for middle school microbiology. We observe that off-task behavior is associated with reduced student learning, but preliminary analyses of students' affective transitions suggest that off-task behavior may also serve a productive role for some students coping with negative affective states such as frustration. Empirical findings imply that some students may use off-task behavior as a strategy for self-regulating negative emotional states during learning. Based on these observations, we introduce a supervised machine learning procedure for detecting whether students' off-task behaviors are cases of emotion self-regulation. The method proceeds in three stages. During the first stage, a dynamic Bayesian network (DBN) is trained to model the valence of students' emotion self-reports using collected data from interactions with the learning environment. In the second stage, a novel simulation process uses the DBN to generate alternate futures by modeling students' affective trajectories as if they had engaged in fewer off-task behaviors than they did during their actual learning interactions. The alternate futures are compared to students' actual traces to produce labels denoting whether students' off-task behaviors are cases of emotion self-regulation. In the final stage, the generated emotion self-regulation labels are predicted using off-the-shelf classifiers and features that can be computed in run-time settings. Results suggest that this approach shows promise for identifying cases of off-task behavior that are emotion self-regulation. Analyses of the first two phases suggest that trained DBN models are capable of accurately modeling relationships between students' off-task behaviors and self-reported emotional valence in CRYSTAL ISLAND. Additionally, the proposed simulation process produces emotion self-regulation labels with high levels of reliability. Preliminary analyses indicate that support vector machines, bagged trees, and random forests show promise for predicting the generated emotion self-regulation labels, but room for improvement remains. The findings underscore the methodological potential of considering alternate futures when modeling students' emotion self-regulation processes in narrative-centered learning environments.
How to Cite
affective states, assessment, dynamic Bayesian network, off-task behavior, self-regulation, simulation
ALPAYDIN, E. 2004. Introduction to Machine Learning, MIT Press.
ARROYO, I., FERGUSON, K., JOHNS, J., DRAGON, T., MEHERANIAN, H., FISHER, D., BARTO, A., MAHADEVAN, S., and WOOLF, B.P. 2007. Repairing disengagement with non-invasive interventions. Proceedings of the 13th International Conference on Artificial Intelligence in Education, 195-202.
BAKER, R.S. 2007. Modeling and understanding students' off-task behavior in intelligent tutoring systems, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1059-1068.
BAKER, R.S., CORBETT, A.T., and ALEVEN, V. 2008. More accurate student modeling through contextual estimation of slip and guess probabilities in Bayesian knowledge tracing. Proceedings of the Ninth International Conference on Intelligent Tutoring Systems, 406-415.
BAKER, R.S., CORBETT, K.R., KOEDINGER, K.R., EVENSON, S., ROLL, I., WAGNER, A., NAIM, M., ... BECK, J. 2006. Adapting to when students game an intelligent tutoring system, Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 392-401.
BAKER, R.S., CORBETT, A.T., KOEDINGER, K.R., and WAGNER, A. 2004. Off-task behavior in the cognitive tutor classroom: When students game the system. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 383-390.
BAKER, R.S., CORBETT, A.T., ROLL, I., and KOEDINGER, K.R. 2008. Developing a generalizable detector of when students game the system. User Modeling and UserAdapted Interaction, 18, 287-314.
BAKER, R.S., D'MELLO, S.K., RODRIGO, S.K., and GRAESSER, A.C. 2010. Better to be frustrated than bored: The incidence, persistence, and impact of learners' cognitive-affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68, 223-241.
BAKER, R.S., MOORE, G., WAGNER, A., KALKA, J., SALVI, A., KARABINOS, M., ASHE, C., and YARON, D. 2011. The dynamics between student affect and behavior occurring outside of educational software. Proceedings of the 4th International Conference on Affective Computing and Intelligent Interaction, 14-24.
BEAL, C.R., MITRA, S., and COHEN, P.R. 2007. Modeling learning patterns of students with a tutoring system using Hidden Markov Models. In Proceedings of the 13th International Conference of Artificial Intelligence in Education, 238-245.
BEAL, C.R., QU, L., and LEE, H. 2006. Classifying learner engagement through integration of multiple data sources. Proceedings of the National Conference on Artificial Intelligence, 2-8.
BLOOM, B.S. 1984. The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13, 4-16.
BRUNER, J.S., 1990. Acts of Meaning, Cambridge, MA: Harvard University Press.
BUNT, A. and CONATI, C. 2003. Probabilistic student modeling to improve exploratory behavior. User Modeling and User-Adapted Interaction, 13, 269-309.
CETINTAS, S., LUO, S., XIN, Y., HORD, C., and ZHANG, D. 2009. Learning to identify students off-task behavior in intelligent tutoring systems, Proceedings of the 14th International Conference on Artificial Intelligence in Education, 701-703.
COCEA, M., HERSHKOVITZ, A., and BAKER, R.S. 2009. The impact of off-task and gaming behaviors on learning: Immediate or aggregate. Proceedings of the 14th International Conference on Artificial Intelligence in Education, 507-514.
CONATI, C. and MACLAREN, H. 2009. Empirically building and evaluating a probabilistic model of user affect. User Modeling and User-Adapted Interaction, 19, 267- 303.
CORBETT, A.T. AND ANDERSON, J.R. 1994. Knowledge tracing: Modeling the Acquisition of Procedural Knowledge. User Modeling and User-Adapted Interaction, 4, 253-278.
D'MELLO, S., TAYLOR, R.S., and GRAESSER, A.C. 2007. Monitoring affective trajectories during complex learning. Proceedings of the 29th Annual Meeting of the Cognitive Science Society, 203-208.
DWECK, C.S. and LEGGET, E.L. A social-cognitive approach to motivation and personality. Psychological Review, 95, 256-273.
ELLIOT, A. and MCGREGOR, H.A. 2001. A 2 x 2 achievement goal framework, Journal of Personality and Social Psychology, 80, 501-519.
ELLIOT, A. and PEKRUN, R. 2007. Emotion in the hierarchical model of approachavoidance achievement motivation. Emotion in Education, P. SCHUTZ and R. PEKRUN, eds., London: Elsevier, 57-74.
GERNEFSKI, N. and KRAATI, V. 2006. Cognitive Emotion Regulation Questionaire: Development of a short 18-item version. Personality and Individual Differences, 41, 1045-1053.
GERTNER, A., CONATI, C., and VANLEHN, K. 1998. Procedural help in Andes: Generating hints using a Bayesian network student model. Proceedings of the 15th National Conference on Artificial Intelligence.
GONG, Y., BECK, J., HEFFERNAN, N., and FORBES-SUMMERS, E. 2010. The finegrained impact of gaming on learning. Proceedings of the 10th International Conference on Intelligent Tutoring Systems, 194-203.
GRAESSER, A.C., PERSON, N.K., MAGLIANO, J.P. Collaborative dialogue patterns in naturalistic one-to-one tutoring, Applied Cognitive Psychology, 9, 495-522.
HARP, S. and MAYER, R.E. 1998. How seductive details do their damage: A theory of cognitive interest in science learning. Journal of Educational Psychology, 90, 414-434.
HALL, M., FRANK, E., HOLMES, G., PFAHRINGER, B., REUTEMANN, P., and WITTEN, I. The WEKA data mining software: An update. SIGKDD Explorations, 11, 2009.
HICKEY, D.T., INGRAM-GOBLE, A.A., and JAMESON, E.M. 2009. Designing assessments and assessing designs in virtual educational environments. Journal of Science Education and Technology, 18, 187-208.
KETELHUT, D.J. The impact of student self-efficacy on scientific inquiry skills: An exploratory investigation in River City, a multi-user virtual environment. Journal of Science Education and Technology, 6, 99-111.
MANDLER, J. and JOHNSON, N. 1988. Remeberance of things parsed: Story structure and recall. Journal of Cognitive Psychology, 9, 111-151.
MARSELLA, S.C., JOHNSON, W.L., and LABORE, C.M. 2000. Interactive pedagogical drama. Proceedings of the 5th International Conference on Intelligent Virtual Agents, 305-315.
MCCRAE, R. and COSTA, P. 1993. Personality in Adulthood: A Five-Factor Theory Perspective, New York: Guilford Press.
MEYER, D. and TURNER, J. 2006. Reconceptualizing emotion and motivation to learn in classroom contexts, Educational Psychology Review, 18, 377-390.
MULDNER, K., BURLESON, B., VAN DE SAND, B., and VANLEHN, K. 2010. An analysis of gaming behaviors in an intelligent tutoring system. Proceedings of the 10th International Conference on Intelligent Tutoring Systems, 184-193.
MURRAY, R.C. and VANLEHN, K. Effects of dissuading unnecessary help requests while providing proactive help, Proceedings of the 12th International Conference on Artificial Intelligence in Education, 887-889.
O'NEILL, B. and RIEDL, M.O. 2011. Toward a computational framework of suspense and dramatic arc. Proceedings of the 4th International Conference on Affective Computing and Intelligent Interaction, 256-263.
RODRIGO, M., BAKER, R.S., LAGUD, M., LIM, S., MACAPANPAN, A., PASCUA, S., SANTILLANO, J., ...VIEHLAND, N. Affect and usage choices in simulation problem-solving environments, Proceedings of the 13th International Conference on Artificial Intelligence in Education, 145-152.
ROWE, J.P., MCQUIGGAN, S.W., ROBISON, J.L., and LESTER, J.C. 2009. Off-task behavior in narrative-centered learning environments. Proceedings of the 14th International Conference on Artificial Intelligence and Education, 99-106.
ROWE, J.P., SHORES, L.R., MOTT, B.W., and LESTER, J.C. 2011. Integrating learning, problem solving, and engagement in narrative-centered learning environments, International Journal of Artificial Intelligence in Education, 21, 115-133.
SABOURIN, J.L., MOTT, B.W., and LESTER, J.C. 2011. Modeling learner affect with theoretically grounded dynamic Bayesian networks. Proceedings of the 4th International Conference on Affective Computing and Intelligent Interaction, 286-295.
SABOURIN, J.L., ROWE, J.P., MOTT, B.W., and LESTER, J.C. 2011. When off-task in on-task: The affective role of off-task behavior in narrative-centered learning environments. Proceedings of the 15th International Conference on Artificial Intelligence and Education, 534-536.
VANLEHN, K. 2011. The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46, 197-221.
VANLEHN, K. 2006. The behaviour of tutoring systems, International Journal of Artificial Intelligence in Education, 16, 227-265.
Authors who publish with this journal agree to the following terms:
- The Author retains copyright in the Work, where the term “Work” shall include all digital objects that may result in subsequent electronic publication or distribution.
- Upon acceptance of the Work, the author shall grant to the Publisher the right of first publication of the Work.
- The Author shall grant to the Publisher and its agents the nonexclusive perpetual right and license to publish, archive, and make accessible the Work in whole or in part in all forms of media now or hereafter known under a Creative Commons 4.0 License (Attribution-Noncommercial-No Derivatives 4.0 International), or its equivalent, which, for the avoidance of doubt, allows others to copy, distribute, and transmit the Work under the following conditions:
- Attribution—other users must attribute the Work in the manner specified by the author as indicated on the journal Web site;
- Noncommercial—other users (including Publisher) may not use this Work for commercial purposes;
- No Derivative Works—other users (including Publisher) may not alter, transform, or build upon this Work,with the understanding that any of the above conditions can be waived with permission from the Author and that where the Work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.
- The Author is able to enter into separate, additional contractual arrangements for the nonexclusive distribution of the journal's published version of the Work (e.g., post it to an institutional repository or publish it in a book), as long as there is provided in the document an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post online a pre-publication manuscript (but not the Publisher’s final formatted PDF version of the Work) in institutional repositories or on their Websites prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (see The Effect of Open Access). Any such posting made before acceptance and publication of the Work shall be updated upon publication to include a reference to the Publisher-assigned DOI (Digital Object Identifier) and a link to the online abstract for the final published Work in the Journal.
- Upon Publisher’s request, the Author agrees to furnish promptly to Publisher, at the Author’s own expense, written evidence of the permissions, licenses, and consents for use of third-party material included within the Work, except as determined by Publisher to be covered by the principles of Fair Use.
- The Author represents and warrants that:
- the Work is the Author’s original work;
- the Author has not transferred, and will not transfer, exclusive rights in the Work to any third party;
- the Work is not pending review or under consideration by another publisher;
- the Work has not previously been published;
- the Work contains no misrepresentation or infringement of the Work or property of other authors or third parties; and
- the Work contains no libel, invasion of privacy, or other unlawful matter.
- The Author agrees to indemnify and hold Publisher harmless from Author’s breach of the representations and warranties contained in Paragraph 6 above, as well as any claim or proceeding relating to Publisher’s use and publication of any content contained in the Work, including third-party content.