Adjusting the L Statistic when Self-Transitions are Excluded in Affect Dynamics

##plugins.themes.bootstrap3.article.main##

##plugins.themes.bootstrap3.article.sidebar##

Published Dec 29, 2020
Jeffrey Matayoshi Shamya Karumbaiah

Abstract

Affect dynamics, the investigation of how student affect transitions from one state to another, is a popular area of research in adaptive learning environments. Recently, the commonly used transition metric L has come under critical examination when applied to data that exclude self-transitions (i.e., transitions where a student remains in the same affective state on consecutive observations); in this situation, recent work has shown that L potentially overestimates the significance of certain transitions. In order to deal with the unintended side effects of removing self-transitions, a solution was proposed that shifts the chance value of L from zero to a positive value that varies based on the number of affective states in the study. Although this treatment compensates for the aforementioned issues, it could lead to misinterpretation of the results due to the counterintuitive nonzero chance value. Motivated by these issues, in this work we study a modified version of the L statistic, which we refer to as L*, with an aim to shift the chance value back to zero when self-transitions are removed. We begin by studying the mathematical and statistical properties of L*, and we also compare these properties to those of the L statistic. After analyzing the theoretical attributes of L*, we then evaluate its performance when applied to data. In our first evaluation, we compute L* values on simulated sequences of affective states, where the base rates of the states vary from being uniform to highly non-uniform. The results provide evidence that the L* values at chance are not sensitive to unequal base rates, as the computed values from our experiments are centered closely around zero. Our second evaluation applies L* to actual student data generated from students working in a digital learning environment; in this case, the use of L* seemingly gives a more coherent picture in comparison to the values returned by the L statistic. Finally, along with these analyses, we also outline a recommended procedure for applying L* to sequences of affective states.

How to Cite

Matayoshi, J., & Karumbaiah, S. (2020). Adjusting the L Statistic when Self-Transitions are Excluded in Affect Dynamics. Journal of Educational Data Mining, 12(4), 1-23. https://doi.org/10.5281/zenodo.4399681
Abstract 242 | PDF Downloads 227

##plugins.themes.bootstrap3.article.details##

Keywords

affect dynamics, L statistic, self-transitions, transition metric

References
ANDRES, J. M. L., RODRIGO, M. M. T., SUGAY, J. O., BANAWAN, M. P., PAREDES, Y. V. M., CRUZ, J. S. D., AND PALAOAG, T. D. 2015. More fun in the Philippines? Factors affecting transfer of western field methods to one developing world context. In Proceedings of the Sixth International Workshop on Culturally-Aware Tutoring Systems at the 17th International Conference on Artificial Intelligence in Education, J. Boticario and K. Muldner, Eds. CEUR Workshop Proceedings, vol. 1432. 31–40.

BAKER, R. S., D'MELLO, S. K., RODRIGO, M. T., AND GRAESSER, A. C. 2010. Better to be frustrated than bored: The incidence, persistence, and impact of learners' cognitive-affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies 68, 4, 223–241.

BENJAMINI, Y. AND HOCHBERG, Y. 1995. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological) 57, 1, 289–300.

BENJAMINI, Y. AND YEKUTIELI, D. 2001. The control of the false discovery rate in multiple testing under dependency. Annals of Statistics 29, 4, 1165–1188.

BISWAS, G., JEONG, H., KINNEBREW, J., SULCER, B., AND ROSCOE, R. D. 2010. Measuring selfregulated learning skills through social interactions in a teachable agent environment. Research and Practice in Technology Enhanced Learning 5, 2, 123–152.

BOSCH, N. AND D'MELLO, S. 2017. The affective experience of novice computer programmers. International Journal of Artificial Intelligence in Education 27, 1, 181–206.

BOSCH, N. AND D'MELLO, S. K. 2013. Sequential patterns of affective states of novice programmers. In Proceedings of the First Workshop on AI-supported Education for Computer Science at the 16th International Conference on Artificial Intelligence in Education, E. Walker and C. Looi, Eds. CEUR Workshop Proceedings, vol. 1009. 1–10.

BOSCH, N. AND PAQUETTE, L. 2020. What's next? Edge cases in measuring transitions between sequential states. Submitted for publication.

BOTELHO, A. F., BAKER, R. S., AND HEFFERNAN, N. T. 2017. Improving sensor-free affect detection using deep learning. In Proceedings of the 18th International Conference on Artificial Intelligence in Education, E. André, R. Baker, X. Hu, M. M. T. Rodrigo, and B. du Boulay, Eds. Springer International Publishing, Cham, 40–51.

BOTELHO, A. F., BAKER, R. S., OCUMPAUGH, J., AND HEFFERNAN, N. T. 2018. Studying affect dynamics and chronometry using sensor-free detectors. In Proceedings of the 11th International Conference on Educational Data Mining, K. E. Boyer and M. Yudelson, Eds. International Educational Data Mining Society, 157–166.

DE FALCO, J., ROWE, J., PAQUETTE, L., GEORGOULAS -SHERRY, V., BRAWNER, K., MOTT, B., BAKER, R., AND LESTER, J. 2018. Detecting and addressing frustration in a serious game for military training. International Journal of Artificial Intelligence in Education 28, 152–193.

D'MELLO, S. AND GRAESSER, A. 2010. Modeling cognitive-affective dynamics with hidden Markov models. In Proceedings of the 32nd Annual Meeting of the Cognitive Science Society, S. Ohlsson and R. Catrambone, Eds. Vol. 32. Cognitive Science Society, Austin, TX, 2721–2726.

D'MELLO, S. AND GRAESSER, A. 2012. Dynamics of affective states during complex learning. Learning and Instruction 22, 2, 145–157.D'MELLO, S., PERSON, N., AND LEHMAN, B. 2009. Antecedent-consequent relationships and cyclical patterns between affective states and problem solving outcomes. In Proceedings of the 14th International Conference on on Artificial Intelligence in Education. IOS Press, NLD, 57–64.

D'MELLO, S., TAYLOR, R. S., AND GRAESSER, A. 2007. Monitoring affective trajectories during complex learning. In Proceedings of the 29th Annual Cognitive Science Society, D. S. McNamara and J. G. Trafton, Eds. Cognitive Science Society, Austin, TX, 203–208.

KARUMBAIAH, S., ANDRES, J., BOTELHO, A. F., BAKER, R. S., AND OCUMPAUGH, J. S. 2018. The implications of a subtle difference in the calculation of affect dynamics. In Proceedings of the 26th International Conference on Computers in Education, J. C. Yang, M. Chang, L.-H. Wong, and M. M. T. Rodrigo, Eds. 29–38.

KARUMBAIAH, S., BAKER, R. S., AND OCUMPAUGH, J. 2019. The case of self-transitions in affective dynamics. In Proceedings of the 20th International Conference on Artificial Intelligence in Education, S. Isotani, E. Millán, A. Ogan, P. Hastings, B. McLaren, and R. Luckin, Eds. Springer International Publishing, Cham, 172–181.

KARUMBAIAH, S., LIZARRALDE, R., ALLESSIO, D., WOOLF, B. P., AND ARROYO, I. 2017. Addressing student behavior and affect with empathy and growth mindset. In Proceedings of the 10th International Conference on Educational Data Mining, X. Hu, T. Barnes, A. Hershkovitz, and L. Paquette, Eds. International Educational Data Mining Society, 96–103.

KUPPENS, P. 2015. It's about time: A special section on affect dynamics. Emotion Review 7, 4, 297–300.

MATAYOSHI, J. 2020. jmatayoshi/affect-transitions: Release 1.0.0. M C QUIGGAN, S. W. AND LESTER, J. C. 2009. Modelling affect expression and recognition in an interactive learning environment. International Journal of Learning Technology 4, 3-4, 216–233.

NYE, B. D., KARUMBAIAH, S., TOKEL, S. T., CORE, M. G., STRATOU, G., AUERBACH, D., AND GEORGILA, K. 2018. Engaging with the scenario: Affect and facial patterns from a scenariobased intelligent tutoring system. In Proceedings of the 19th International Conference on Artificial Intelligence in Education, C. Penstein Rosé, R. Martı́nez-Maldonado, H. U. Hoppe, R. Luckin, M. Mavrikis, K. Porayska-Pomsta, B. McLaren, and B. du Boulay, Eds. Springer International Publishing, Cham, 352–366.

OCUMPAUGH, J., ANDRES, J. M., BAKER, R., D E FALCO, J., PAQUETTE, L., ROWE, J., MOTT, B., LESTER, J., GEORGOULAS, V., BRAWNER, K., AND SOTTILARE, R. 2017. Affect dynamics in military trainees using vMedic: From engaged concentration to boredom to confusion. In Proceedings of the 18th International Conference on Artificial Intelligence in Education, E. André, R. Baker, X. Hu, M. M. T. Rodrigo, and B. du Boulay, Eds. Springer International Publishing, Cham, 238–249.

OCUMPAUGH, J., BAKER, R. S., AND RODRIGO, M. M. T. 2015. Baker Rodrigo Ocumpaugh Monitoring Protocol (BROMP) 2.0 Technical and Training Manual.

RODRIGO, M., ANGLO, E., SUGAY, J., AND BAKER, R. 2008. Use of unsupervised clustering to characterize learner behaviors and affective states while using an intelligent tutoring system. In Proceedings of the 16th International Conference on Computers in Education. 57–64.

SHUTE, V. J. AND VENTURA, M. 2013. Stealth assessment: Measuring and supporting learning in video games. MIT Press, Cambridge, MA.
Section
Articles