Considering Alternate Futures to Classify Off-Task Behavior as Emotion Self-Regulation: A Supervised Learning Approach

Abstract

Over the past decade, there has been growing interest in real-time assessment of student engagement and motivation during interactions with educational software. Detecting symptoms of disengagement, such as offtask behavior, has shown considerable promise for understanding students' motivational characteristics during learning. In this paper, we investigate the affective role of off-task behavior by analyzing data from student interactions with CRYSTAL ISLAND, a narrative-centered learning environment for middle school microbiology. We observe that off-task behavior is associated with reduced student learning, but preliminary analyses of students' affective transitions suggest that off-task behavior may also serve a productive role for some students coping with negative affective states such as frustration. Empirical findings imply that some students may use off-task behavior as a strategy for self-regulating negative emotional states during learning. Based on these observations, we introduce a supervised machine learning procedure for detecting whether students' off-task behaviors are cases of emotion self-regulation. The method proceeds in three stages. During the first stage, a dynamic Bayesian network (DBN) is trained to model the valence of students' emotion self-reports using collected data from interactions with the learning environment. In the second stage, a novel simulation process uses the DBN to generate alternate futures by modeling students' affective trajectories as if they had engaged in fewer off-task behaviors than they did during their actual learning interactions. The alternate futures are compared to students' actual traces to produce labels denoting whether students' off-task behaviors are cases of emotion self-regulation. In the final stage, the generated emotion self-regulation labels are predicted using off-the-shelf classifiers and features that can be computed in run-time settings. Results suggest that this approach shows promise for identifying cases of off-task behavior that are emotion self-regulation. Analyses of the first two phases suggest that trained DBN models are capable of accurately modeling relationships between students' off-task behaviors and self-reported emotional valence in CRYSTAL ISLAND. Additionally, the proposed simulation process produces emotion self-regulation labels with high levels of reliability. Preliminary analyses indicate that support vector machines, bagged trees, and random forests show promise for predicting the generated emotion self-regulation labels, but room for improvement remains. The findings underscore the methodological potential of considering alternate futures when modeling students' emotion self-regulation processes in narrative-centered learning environments.