Adjusting the L Statistic when Self-Transitions are Excluded in Affect Dynamics

##plugins.themes.bootstrap3.article.main##

##plugins.themes.bootstrap3.article.sidebar##

PDF
Published Dec 29, 2020
DOI https://doi.org/10.5281/zenodo.4399681
Jeffrey Matayoshi
McGraw Hill ALEKS
Shamya Karumbaiah
University of Pennsylvania

Abstract

Affect dynamics, the investigation of how student affect transitions from one state to another, is a popular area of research in adaptive learning environments. Recently, the commonly used transition metric L has come under critical examination when applied to data that exclude self-transitions (i.e., transitions where a student remains in the same affective state on consecutive observations); in this situation, recent work has shown that L potentially overestimates the significance of certain transitions. In order to deal with the unintended side effects of removing self-transitions, a solution was proposed that shifts the chance value of L from zero to a positive value that varies based on the number of affective states in the study. Although this treatment compensates for the aforementioned issues, it could lead to misinterpretation of the results due to the counterintuitive nonzero chance value. Motivated by these issues, in this work we study a modified version of the L statistic, which we refer to as L*, with an aim to shift the chance value back to zero when self-transitions are removed. We begin by studying the mathematical and statistical properties of L*, and we also compare these properties to those of the L statistic. After analyzing the theoretical attributes of L*, we then evaluate its performance when applied to data. In our first evaluation, we compute L* values on simulated sequences of affective states, where the base rates of the states vary from being uniform to highly non-uniform. The results provide evidence that the L* values at chance are not sensitive to unequal base rates, as the computed values from our experiments are centered closely around zero. Our second evaluation applies L* to actual student data generated from students working in a digital learning environment; in this case, the use of L* seemingly gives a more coherent picture in comparison to the values returned by the L statistic. Finally, along with these analyses, we also outline a recommended procedure for applying L* to sequences of affective states.

How to Cite

Matayoshi, J., & Karumbaiah, S. (2020). Adjusting the L Statistic when Self-Transitions are Excluded in Affect Dynamics. Journal of Educational Data Mining, 12(4), 1–23. https://doi.org/10.5281/zenodo.4399681
Abstract 460 | PDF Downloads 466

##plugins.themes.bootstrap3.article.details##

Keywords

affect dynamics, L statistic, self-transitions, transition metric

References
ANDRES, J. M. L., RODRIGO, M. M. T., SUGAY, J. O., BANAWAN, M. P., PAREDES, Y. V. M., CRUZ, J. S. D., AND PALAOAG, T. D. 2015. More fun in the Philippines? Factors affecting transfer of western field methods to one developing world context. In Proceedings of the Sixth International Workshop on Culturally-Aware Tutoring Systems at the 17th International Conference on Artificial Intelligence in Education, J. Boticario and K. Muldner, Eds. CEUR Workshop Proceedings, vol. 1432. 31–40.

BAKER, R. S., D'MELLO, S. K., RODRIGO, M. T., AND GRAESSER, A. C. 2010. Better to be frustrated than bored: The incidence, persistence, and impact of learners' cognitive-affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies 68, 4, 223–241.

BENJAMINI, Y. AND HOCHBERG, Y. 1995. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological) 57, 1, 289–300.

BENJAMINI, Y. AND YEKUTIELI, D. 2001. The control of the false discovery rate in multiple testing under dependency. Annals of Statistics 29, 4, 1165–1188.

BISWAS, G., JEONG, H., KINNEBREW, J., SULCER, B., AND ROSCOE, R. D. 2010. Measuring selfregulated learning skills through social interactions in a teachable agent environment. Research and Practice in Technology Enhanced Learning 5, 2, 123–152.

BOSCH, N. AND D'MELLO, S. 2017. The affective experience of novice computer programmers. International Journal of Artificial Intelligence in Education 27, 1, 181–206.

BOSCH, N. AND D'MELLO, S. K. 2013. Sequential patterns of affective states of novice programmers. In Proceedings of the First Workshop on AI-supported Education for Computer Science at the 16th International Conference on Artificial Intelligence in Education, E. Walker and C. Looi, Eds. CEUR Workshop Proceedings, vol. 1009. 1–10.

BOSCH, N. AND PAQUETTE, L. 2020. What's next? Edge cases in measuring transitions between sequential states. Submitted for publication.

BOTELHO, A. F., BAKER, R. S., AND HEFFERNAN, N. T. 2017. Improving sensor-free affect detection using deep learning. In Proceedings of the 18th International Conference on Artificial Intelligence in Education, E. André, R. Baker, X. Hu, M. M. T. Rodrigo, and B. du Boulay, Eds. Springer International Publishing, Cham, 40–51.

BOTELHO, A. F., BAKER, R. S., OCUMPAUGH, J., AND HEFFERNAN, N. T. 2018. Studying affect dynamics and chronometry using sensor-free detectors. In Proceedings of the 11th International Conference on Educational Data Mining, K. E. Boyer and M. Yudelson, Eds. International Educational Data Mining Society, 157–166.

DE FALCO, J., ROWE, J., PAQUETTE, L., GEORGOULAS -SHERRY, V., BRAWNER, K., MOTT, B., BAKER, R., AND LESTER, J. 2018. Detecting and addressing frustration in a serious game for military training. International Journal of Artificial Intelligence in Education 28, 152–193.

D'MELLO, S. AND GRAESSER, A. 2010. Modeling cognitive-affective dynamics with hidden Markov models. In Proceedings of the 32nd Annual Meeting of the Cognitive Science Society, S. Ohlsson and R. Catrambone, Eds. Vol. 32. Cognitive Science Society, Austin, TX, 2721–2726.

D'MELLO, S. AND GRAESSER, A. 2012. Dynamics of affective states during complex learning. Learning and Instruction 22, 2, 145–157.D'MELLO, S., PERSON, N., AND LEHMAN, B. 2009. Antecedent-consequent relationships and cyclical patterns between affective states and problem solving outcomes. In Proceedings of the 14th International Conference on on Artificial Intelligence in Education. IOS Press, NLD, 57–64.

D'MELLO, S., TAYLOR, R. S., AND GRAESSER, A. 2007. Monitoring affective trajectories during complex learning. In Proceedings of the 29th Annual Cognitive Science Society, D. S. McNamara and J. G. Trafton, Eds. Cognitive Science Society, Austin, TX, 203–208.

KARUMBAIAH, S., ANDRES, J., BOTELHO, A. F., BAKER, R. S., AND OCUMPAUGH, J. S. 2018. The implications of a subtle difference in the calculation of affect dynamics. In Proceedings of the 26th International Conference on Computers in Education, J. C. Yang, M. Chang, L.-H. Wong, and M. M. T. Rodrigo, Eds. 29–38.

KARUMBAIAH, S., BAKER, R. S., AND OCUMPAUGH, J. 2019. The case of self-transitions in affective dynamics. In Proceedings of the 20th International Conference on Artificial Intelligence in Education, S. Isotani, E. Millán, A. Ogan, P. Hastings, B. McLaren, and R. Luckin, Eds. Springer International Publishing, Cham, 172–181.

KARUMBAIAH, S., LIZARRALDE, R., ALLESSIO, D., WOOLF, B. P., AND ARROYO, I. 2017. Addressing student behavior and affect with empathy and growth mindset. In Proceedings of the 10th International Conference on Educational Data Mining, X. Hu, T. Barnes, A. Hershkovitz, and L. Paquette, Eds. International Educational Data Mining Society, 96–103.

KUPPENS, P. 2015. It's about time: A special section on affect dynamics. Emotion Review 7, 4, 297–300.

MATAYOSHI, J. 2020. jmatayoshi/affect-transitions: Release 1.0.0. M C QUIGGAN, S. W. AND LESTER, J. C. 2009. Modelling affect expression and recognition in an interactive learning environment. International Journal of Learning Technology 4, 3-4, 216–233.

NYE, B. D., KARUMBAIAH, S., TOKEL, S. T., CORE, M. G., STRATOU, G., AUERBACH, D., AND GEORGILA, K. 2018. Engaging with the scenario: Affect and facial patterns from a scenariobased intelligent tutoring system. In Proceedings of the 19th International Conference on Artificial Intelligence in Education, C. Penstein Rosé, R. Martı́nez-Maldonado, H. U. Hoppe, R. Luckin, M. Mavrikis, K. Porayska-Pomsta, B. McLaren, and B. du Boulay, Eds. Springer International Publishing, Cham, 352–366.

OCUMPAUGH, J., ANDRES, J. M., BAKER, R., D E FALCO, J., PAQUETTE, L., ROWE, J., MOTT, B., LESTER, J., GEORGOULAS, V., BRAWNER, K., AND SOTTILARE, R. 2017. Affect dynamics in military trainees using vMedic: From engaged concentration to boredom to confusion. In Proceedings of the 18th International Conference on Artificial Intelligence in Education, E. André, R. Baker, X. Hu, M. M. T. Rodrigo, and B. du Boulay, Eds. Springer International Publishing, Cham, 238–249.

OCUMPAUGH, J., BAKER, R. S., AND RODRIGO, M. M. T. 2015. Baker Rodrigo Ocumpaugh Monitoring Protocol (BROMP) 2.0 Technical and Training Manual.

RODRIGO, M., ANGLO, E., SUGAY, J., AND BAKER, R. 2008. Use of unsupervised clustering to characterize learner behaviors and affective states while using an intelligent tutoring system. In Proceedings of the 16th International Conference on Computers in Education. 57–64.

SHUTE, V. J. AND VENTURA, M. 2013. Stealth assessment: Measuring and supporting learning in video games. MIT Press, Cambridge, MA.
Issue
Vol. 12 No. 4 (2020)
Section
Articles
Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Authors who publish with this journal agree to the following terms:

  1. The Author retains copyright in the Work, where the term “Work” shall include all digital objects that may result in subsequent electronic publication or distribution.
  2. Upon acceptance of the Work, the author shall grant to the Publisher the right of first publication of the Work.
  3. The Author shall grant to the Publisher and its agents the nonexclusive perpetual right and license to publish, archive, and make accessible the Work in whole or in part in all forms of media now or hereafter known under a Creative Commons 4.0 License (Attribution-Noncommercial-No Derivatives 4.0 International), or its equivalent, which, for the avoidance of doubt, allows others to copy, distribute, and transmit the Work under the following conditions:
    1. Attribution—other users must attribute the Work in the manner specified by the author as indicated on the journal Web site;
    2. Noncommercial—other users (including Publisher) may not use this Work for commercial purposes;
    3. No Derivative Works—other users (including Publisher) may not alter, transform, or build upon this Work,with the understanding that any of the above conditions can be waived with permission from the Author and that where the Work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.
  4. The Author is able to enter into separate, additional contractual arrangements for the nonexclusive distribution of the journal's published version of the Work (e.g., post it to an institutional repository or publish it in a book), as long as there is provided in the document an acknowledgement of its initial publication in this journal.
  5. Authors are permitted and encouraged to post online a pre-publication manuscript (but not the Publisher’s final formatted PDF version of the Work) in institutional repositories or on their Websites prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (see The Effect of Open Access). Any such posting made before acceptance and publication of the Work shall be updated upon publication to include a reference to the Publisher-assigned DOI (Digital Object Identifier) and a link to the online abstract for the final published Work in the Journal.
  6. Upon Publisher’s request, the Author agrees to furnish promptly to Publisher, at the Author’s own expense, written evidence of the permissions, licenses, and consents for use of third-party material included within the Work, except as determined by Publisher to be covered by the principles of Fair Use.
  7. The Author represents and warrants that:
    1. the Work is the Author’s original work;
    2. the Author has not transferred, and will not transfer, exclusive rights in the Work to any third party;
    3. the Work is not pending review or under consideration by another publisher;
    4. the Work has not previously been published;
    5. the Work contains no misrepresentation or infringement of the Work or property of other authors or third parties; and
    6. the Work contains no libel, invasion of privacy, or other unlawful matter.
  8. The Author agrees to indemnify and hold Publisher harmless from Author’s breach of the representations and warranties contained in Paragraph 6 above, as well as any claim or proceeding relating to Publisher’s use and publication of any content contained in the Work, including third-party content.