Using Machine Learning to Detect SMART Model Cognitive Operations in Mathematical Problem-Solving Process
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
Dec 18, 2022
Jiayi Zhang
Juliana Ma. Alexandra L. Andres
Stephen Hutt
Ryan S. Baker
Jaclyn Ocumpaugh
Nidhi Nasiar
Caitlin Mills
Jamiella Brooks
Sheela Sethuaman
Tyron Young
https://orcid.org/0000-0002-7041-7472
Abstract
Self-regulated learning (SRL) is a critical component of mathematics problem-solving. Students skilled in
SRL are more likely to effectively set goals, search for information, and direct their attention and cognitive
process so that they align their efforts with their objectives. An influential framework for SRL, the SMART
model (Winne, 2017), proposes that five cognitive operations (i.e., searching, monitoring, assembling,
rehearsing, and translating) play a key role in SRL. However, these categories encompass a wide range of
behaviors, making measurement challenging – often involving observing individual students and recording
their think-aloud activities or asking students to complete labor-intensive tagging activities as they work. In
the current study, to achieve better scalability, we operationalized indicators of SMART operations and
developed automated detectors using machine learning. We analyzed students’ textual responses and
interaction data collected from a mathematical learning platform where students are asked to thoroughly
explain their solutions and are scaffolded in communicating their problem-solving process. Due to the rarity
in data for one of the seven SRL indicators operationalized, we built six models to reflect students’ use of
four SMART operations. These models are found to be reliable and generalizable, with AUC ROCs ranging
from .76-.89. When applied to the full test set, these detectors are relatively robust to algorithmic bias,
performing well across different student populations and with no consistent bias against a specific group of
students.
How to Cite
Zhang, J., Andres, J. M. A. L., Hutt, S., Baker, R. S., Ocumpaugh, J., Nasiar, N., Mills, C., Brooks, J., Sethuaman, S., & Young, T. (2022). Using Machine Learning to Detect SMART Model Cognitive Operations in Mathematical Problem-Solving Process. Journal of Educational Data Mining, 14(3), 76–108. https://doi.org/10.5281/zenodo.7304763
##plugins.themes.bootstrap3.article.details##
Keywords
self-regulated learning, SMART model, automated detectors
References
AGUILAR, S.J., KARABENICK, S.A., TEASLEY, S.D., AND BAEK, C. 2021. Associations between learning analytics dashboard exposure and motivation and self-regulated learning. Computers & Education 162.104085
ALEVEN, V., MCLAREN, B., ROLL, I., AND KOEDINGER, K. 2006. Toward meta-cognitive tutoring: a model of help seeking with a cognitive tutor. International Journal of Artificial Intelligence in Education 16, 2, 101–128.
ALEVEN, V., ROLL, I., MCLAREN, B.M., AND KOEDINGER, K.R. 2016. Help helps, but only so much: research on help seeking with intelligent tutoring systems. International Journal of Artificial Intelligence in Education 26, 1, 205–223.
AZEVEDO, R., JOHNSON, A., CHAUNCEY, A., GRAESSER, A., ZIMMERMAN, B., AND SCHUNK, D. 2011. Use of hypermedia to assess and convey self-regulated learning. Handbook of Self-Regulation of Learning and Performance, B. Zimmerman and D.H. Schunk, Eds. New York, NY: Routledge. 102–121.
AZEVEDO, R., TAUB, M., AND MUDRICK, N.V. 2017. Understanding and reasoning about real-time cognitive, affective, and metacognitive processes to foster self-regulation with advanced learning technologies. Handbook of Self-Regulation of Learning and Performance, D.H. Schunk and J.A. Greene, Eds. New York, NY: Routledge. 254–270.
BAKER, R.S. AND DE CARVALHO, A.M.J.A. 2008. Labeling student behavior faster and more precisely with text replays. Proceedings of the 1st International Conference on Educational Data Mining, R.S. Baker, T. Barnes, and J.E. Beck, Eds. International Educational Data Mining Society. 38–47.
BAKER, R.S., CORBETT, A.T., AND KOEDINGER, K.R. 2004. Detecting student misuse of intelligent tutoring systems. International Conference on Intelligent Tutoring Systems, S. Lester, R.M. Vicari, and F. Paraguacu, Eds. Springer, Berlin, Heidelberg. 531–540.
BAKER, R.S., CORBETT, A.T., AND WAGNER, A.Z. 2006. Human classification of low-fidelity replays of student actions. Proceedings of the Educational Data Mining Workshop at the 8th International Conference on Intelligent Tutoring Systems, M. Ikeda, K. Ashley, and TW. Chan Eds. Springer, Berlin, Heidelberg. 29–36.
BAKER, R.S. AND HAWN, A. 2022. Algorithmic bias in education. International Journal of Artificial Intelligence in Education, 32(4), 1052-1092.
BAKER, R.S., MITROVIĆ, A., AND MATHEWS, M. 2010. Detecting gaming the system in constraint-based tutors. Proceedings of the 18th Annual Conference on User Modeling, Adaptation, and Personalization, P. Bra, A. Kobsa, and D. Chin, Eds, Springer, Berlin, Heidelberg. 267–278.
BAKER, R.S. AND OCUMPAUGH, J. 2016. Interaction-based affect detection in educational software. The Oxford Handbook of Affective Computing, R. Calvo, S. D’Mello, J. Gratch, and A. Kappas, Eds. New York, NY: Oxford University Press. 233–245.
BAKER, R.S. AND YACEF, K. 2009. The state of educational data mining in 2009: a review and future visions. Journal of Educational Data Mining 1, 1, 3–17.
BANDURA, A. 1986. Social foundations of thought and action: A social cognitive theory. Englewood Cliffs, NJ: Prentice Hall.
BISWAS, G., BAKER, R.S., AND PAQUETTE, L. 2017. Data mining methods for assessing self-regulated learning. Handbook of Self-Regulation of Learning and Performance, D.H. Schunk and J.A. Greene, Eds. New York, NY: Routledge. 388–403.
BISWAS, G., JEONG, H., KINNEBREW, J.S., SULCER, B., AND ROSCOE, R. 2010. Measuring self-regulated learning skills through social interactions in a teachable agent. Research and Practice in Technology Enhanced Learning 05, 02, 123–152.
BOEKAERTS, M. 1999. Self-regulated learning: where we are today. International Journal of Educational Research 31, 6, 445–457.
BOSCH, N., ZHANG, Y., PAQUETTE, L., BAKER, R., OCUMPAUGH, J., AND BISWAS, G. 2021. Students’ verbalized metacognition during computerized learning. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, ACM. 1–12.
BOTELHO, A.F., BAKER, R.S., OCUMPAUGH, J., AND HEFFERNAN, N.T. 2018. Studying affect dynamics and chronometry using sensor-free detectors. Proceedings of the 11th International Conference on Educational Data Mining, K.E. Boyer and M. Yudelson, Eds. International Educational Data Mining Society. 157–166.
CHARMAZ, K. 1983. The grounded theory method: An explication and interpretation. Contemporary Field Research, R. Emerson, Eds. Boston, MA: Little, Brown and Company. 109–126.
CHEN, T. AND GUESTRIN, C. 2016. XGBoost: a scalable tree boosting system. Proceedings of the 22nd ACM International Conference on Knowledge Discovery and Data Mining, ACM. 785–794.
CHO, M.-H. AND YOO, J.S. 2017. Exploring online students’ self-regulated learning with self-reported surveys and log files: a data mining approach. Interactive Learning Environments 25, 8, 970–982.
CLEARY, T.J. AND CHEN, P.P. 2009. Self-regulation, motivation, and math achievement in middle school: Variations across grade level and math context. Journal of School Psychology 47, 5, 291–314.
CONATI, C., PORAYSKA-POMSTA, K., AND MAVRIKIS, M. 2018. AI in education needs interpretable machine learning: lessons from open learner modelling. http://arxiv.org/abs/1807.00154.
DEVER, D.A., AMON, M.J., VRZAKOVA, H., WIEDBUSCH, M.D., CLOUDE, E.B., AND AZEVEDO, R. 2022. Capturing sequences of learners’ self-regulatory interactions with instructional material during game-based learning using auto-recurrence quantification analysis. Frontiers in Psychology, 13, 813677.
DEVOLDER, A., VAN BRAAK, J., AND TONDEUR, J. 2012. Supporting self‐regulated learning in computer‐based learning environments: systematic review of effects of scaffolding in the domain of science education. Journal of Computer Assisted Learning 28, 6, 557–573.
DICERBO, K.E. AND KIDWAI, K. 2013. Detecting player goals from game log files. Proceedings of the 6th International Conference on Educational Data Mining, S.K. D’Mello, R.A. Calvo, and A. Olney, Eds. International Educational Data Mining Society. 314-315.
EFKLIDES, A. 2011. Interactions of metacognition with motivation and affect in self-regulated learning: the MASRL model. Educational Psychologist 46, 1, 6–25.
FARHANA, E., POTTER, A., RUTHERFORD, T., AND LYNCH, C. F. 2021. Feedback and self-regulated learning in science reading. Proceedings of the 14th International Conference on Educational Data Mining, I.-H. Hsiao, S. Sahebi, F. Bouchet, and J.-J.Vie, Eds. International Educational Data Mining Society. 820-826
GARDNER, J., BROOKS, C., AND BAKER, R. 2019. Evaluating the fairness of predictive student models through slicing analysis. Proceedings of the 9th International Conference on Learning Analytics & Knowledge, D. Azcona and R. Chung, Eds. ACM. 225–234.
GRAESSER, A. AND MCNAMARA, D. 2010. Self-regulated learning in learning environments with pedagogical agents that interact in natural language. Educational Psychologist 45, 4, 234–244.
GRAESSER, A.C., PENUMATSA, P., VENTURA, M., CAI, Z., AND HU, X. 2007. Using LSA in autotutor: learning through mixed initiative dialogue in natural language. Handbook of Latent Semantic Analysis, T. Landauer, D. McNamara, S. Dennis, and W. Kintsch, Eds. Mahwah: Erlbaum. 243–262.
GREENE, J.A., DEEKENS, V.M., COPELAND, D.Z., AND YU, S. 2017. Capturing and modeling self-regulated learning using think-aloud protocols. Handbook of Self-Regulation of Learning and Performance, D.H. Schunk and J.A. Greene, Eds. New York, NY: Routledge. 323–337.
GREENE, J.A., ROBERTSON, J., AND COSTA, L.-J.C. 2013. Assessing self-regulated learning using think-aloud methods. Handbook of Self-Regulation of Learning and Performance, B.J. Zimmerman and D.H. Schunk, Eds. New York, NY: Routledge. 313–328.
KARUMBAIAH, S. AND BROOKS, J. 2021. How colonial continuities underlie algorithmic injustices in education. Conference on Research in Equitable and Sustained Participation in Engineering, Computing, and Technology, IEEE, 1–6.
KINNEBREW, J.S., LORETZ, K.M., AND BISWAS, G. 2013. A contextualized, differential sequence mining method to derive students’ learning behavior patterns. Journal of Educational Data Mining 5, 1, 190–219.
KIZILCEC, R.F. AND LEE, H. 2020. Algorithmic fairness in education. The Ethics of Artificial Intelligence in Education, W. Holmes and K. Porayska-Pomsta, Eds. Taylor & Francis. 174–202.
KÖCK, M. AND PARAMYTHIS, A. 2011. Activity sequence modelling and dynamic clustering for personalized e-learning. User Modeling and User-Adapted Interaction 21, 1–2, 51–97.
KOVANOVIC, V., GAŠEVIĆ, D., DAWSON, S., JOKSIMOVIC, S., AND BAKER, R. 2016. Does time-on-task estimation matter? Implications on validity of learning analytics findings. Journal of Learning Analytics 2, 3, 81–110.
LABUHN, A.S., ZIMMERMAN, B.J., AND HASSELHORN, M. 2010. Enhancing students’ self-regulation and mathematics performance: the influence of feedback and self-evaluative standards. Metacognition and Learning 5, 2, 173–194.
LEE, D.M.C., RODRIGO, MA.M.T., BAKER, R.S.J. D., SUGAY, J.O., AND CORONEL, A. 2011. Exploring the relationship between novice programmer confusion and achievement. International Conference on Affective Computing and Intelligent Interaction, S. D’Mello, A. Graesser, B. Schuller, J.C. Martin, Eds. Springer, Berlin, Heidelberg.175–184.
LUNDBERG, S.M., ERION, G.G., AND LEE, S.I. 2019. Consistent individualized feature attribution for tree ensembles. arXiv:1802.03888.
MU, T., JETTEN, A., AND BRUNSKILL, E. 2020. Towards suggesting actionable interventions for wheel-spinning students. Proceedings of the 13th International Conference on Educational Data Mining, A.N. Rafferty, J. Whitehill, V. Cavalli-Sforza, and C. Romero, Eds. International Educational Data Mining Society. 183–193.
MULDNER, K., BURLESON, W., VAN DE SANDE, B., AND VANLEHN, K. 2011. An analysis of students’ gaming behaviors in an intelligent tutoring system: predictors and impacts. User Modeling and User-Adapted Interaction 21, 1–2, 99–135.
NOTA, L., SORESI, S., AND ZIMMERMAN, B.J. 2004. Self-regulation and academic achievement and resilience: A longitudinal study. International Journal of Educational Research 41, 3, 198–215.
OCUMPAUGH, J., BAKER, R., GOWDA, S., HEFFERNAN, N., AND HEFFERNAN, C. 2014. Population validity for educational data mining models: A case study in affect detection. British Journal of Educational Technology 45, 3, 487–501.
OCUMPAUGH, J., HUTT, S., ANDRES, J.M.A.L., BAKER, R.S., BISWAS, G., BOSCH, N., PAQUETTE, L., AND MUNSHI, A. 2021. Using qualitative data from targeted interviews to inform rapid AIED development. Proceedings of the 29th International Conference on Computers in Education, M.M.T. Rodrigo, S. Iyer, A. Mitrovic, H.N.H. Cheng, D. Kohen-Vacs, C.M.A. Palalas, R. Rajenran, K. Seta, and J. Wang, Eds. Asia-Pacific Society for Computers in Education. 69–74.
OKUR, E., ASLAN, S., ALYUZ, N., ARSLAN ESME, A., AND BAKER, R.S. 2018. Role of socio-cultural differences in labeling students’ affective states. Proceedings of the 19th International Conference on Artificial Intelligence in Education, C.P. Rose, R, Martinez-Maldonado, H.U. Hoppe, R. Luckin, M. Mavrikis, K. Porayska-Pomsta, B. McLaren, B. du Boulay, Eds. Springer, Cham. 367–380.
PANADERO, E. 2017. A review of self-regulated learning: six models and four directions for research. Frontiers in Psychology 8, 422.
PAQUETTE, L. AND BAKER, R.S. 2017. Variations of gaming behaviors across populations of students and across learning environments. Proceedings of the 18th International Conference on Artificial Intelligence in Education, E. André, R. Baker, X. Hu, M.M.T. Rodrigo, and B. du Boulay, Eds. Springer Cham. 274–286.
PAQUETTE, L. AND BAKER, R.S. 2019. Comparing machine learning to knowledge engineering for student behavior modeling: a case study in gaming the system. Interactive Learning Environments 27, 5–6, 585–597.
PAQUETTE, L., DE CARVALHO, A.M.J.A., AND BAKER, R.S. 2014. Towards understanding expert coding of student disengagement in online learning. Proceedings of the 36th Annual Cognitive Science Conference, P. Bello, M. Guarini, M. McShane, and B. Scassellati, Eds. Cognitive Science Society. 1126–1131.
PAQUETTE, L., OCUMPAUGH, J., LI, Z., ANDRES, A., AND BAKER, R. 2020. Who’s learning? Using demographics in EDM research. Journal of Educational Data Mining 12, 3, 1–30.
PEDREGOSA, F., VAROQUAUX, G., GRAMFORT, A., ET AL. 2011. Scikit-learn: Machine Learning in Python. Journal of machine Learning research 12, 2825–2830.
PINTRICH, P.R. 2000. The role of goal orientation in self-regulated learning. Handbook of Self-Regulation, M. Boekaerts, P. Pintrich, and M. Zeidner, Eds. San Diego, CA: Academic Press. 451–502.
PINTRICH, P.R., SMITH, D., GARCIA, T., AND MCKEACHIE, W. 1991. A manual for the use of the motivated strategies for learning questionnaire (MSLQ).
RICHEY, J. E., ZHANG, J., DAS, R., ANDRES-BRAY, J. M., SCRUGGS, R., MOGESSIE, M., BAKER, R.S, AND MCLAREN, B. M. 2021. Gaming and confrustion explain learning advantages for a math digital learning game. Proceedings of the 22nd International Conference on Artificial Intelligence in Education, G. Biswas, S. Bull, J. Kay, and A. Mitrovic, Eds. Springer, Berlin, Heidelberg. 342–355.
ROLL, I., ALEVEN, V., MCLAREN, B.M., AND KOEDINGER, K.R. 2007. Can help seeking be tutored? Searching for the secret sauce of metacognitive tutoring. Proceedings of the 13th International Conference on Artificial Intelligence in Education, R. Luckin, K.R. Koedinger, and J. Greer, Eds. IOS Press. 203-210.
ROTH, A., OGRIN, S., AND SCHMITZ, B. 2016. Assessing self-regulated learning in higher education: a systematic literature review of self-report instruments. Educational Assessment, Evaluation and Accountability 28, 3, 225–250.
SABOURIN, J., SHORES, L.R., MOTT, B.W., AND LESTER, J.C. 2012. Predicting student self-regulation strategies in game-based learning environments. Proceedings of the 11th International Conference on Intelligent Tutoring Systems, S.A. Cerri, W.J. Clancey, G. Papadourakis, and K. Panourgia, Eds. Springer Berlin Heidelberg. 141–150.
SAO PEDRO, M.A., DE BAKER, R.S.J., GOBERT, J.D., MONTALVO, O., AND NAKAMA, A. 2013. Leveraging machine-learned detectors of systematic inquiry behavior to estimate and predict transfer of inquiry skill. User Modeling and User-Adapted Interaction 23, 1, 1–39.
SCHOOLER, J.W., OHLSSON, S., AND BROOKS, K. 1993. Thoughts beyond words: when language overshadows insight. Journal of Experimental Psychology: General 122, 2, 166–183.
SCULLEY, D., SNOEK, J., WILTSCHKO, A., AND RAHIMI, A. 2018. Winner’s curse. On pace, progress, and empirical rigor. Proceedings of the 6th International Conference on Leaning Representations, Y. Bengio and Y. LeCun, Eds. OpenReview.
SEGEDY, J.R., KINNEBREW, J.S., AND BISWAS, G. 2015. Using coherence analysis to characterize self-regulated learning behaviours in open-ended learning environments. Journal of Learning Analytics 2, 1, 13–48.
SMITH, T.F. AND WATERMAN, M.S. 1981. Identification of common molecular subsequences. Journal of Molecular Biology 147, 1, 195–197.
SWAMY, V., RADMEHR, B., KRCO, N., MARRAS, M., AND KÄSER, T. 2022. Evaluating the explainers: black-box explainable machine learning for student success prediction in MOOCs. Proceedings of the 15th International Conference on Educational Data Mining, A. Mitrovic and N. Bosch, Eds. International Educational Data Mining Society. 98-109.
TAUB, M., AZEVEDO, R., BOUCHET, F., AND KHOSRAVIFAR, B. 2014. Can the use of cognitive and metacognitive self-regulated learning strategies be predicted by learners’ levels of prior knowledge in hypermedia-learning environments? Computers in Human Behavior 39, 356–367.
WALONOSKI, J.A. AND HEFFERNAN, N.T. 2006. Prevention of off-task gaming behavior in intelligent tutoring systems. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, M. Ikeda, K.D. Ashley, and TW. Chan, Eds. Springer Berlin Heidelberg, 722–724.
WEBB, M.E., FLUCK, A., MAGENHEIM, J., MALYN-SMITH, J., WATERS, J., DESCHENES, M., AND ZAGAMI, J. 2021. Machine learning for human learners: opportunities, issues, tensions and threats. Educational Technology Research and Development 69, 4, 2109–2130.
WEINSTEIN, C., PALMER, D., AND SCHULTE, A.C. 1987. Learning and study strategies inventory (LASSI). FL: H & H Publishing.
WESTON, C., GANDELL, T., BEAUCHAMP, J., MCALPINE, L., WISEMAN, C., AND BEAUCHAMP, C. 2001. Analyzing interview data: the development and evolution of a coding system. Qualitative Sociology 24, 3, 381–400.
WIJFFELS, J., STRAKA, M., AND STRAKOVA, J. 2017. Package “udpipe”.
WINNE, P.H. 1997. Experimenting to bootstrap self-regulated learning. Journal of Educational Psychology 89, 3, 397–410.
WINNE, P.H. 2004. Students’ calibration of knowledge and learning processes: Implications for designing powerful software learning environments. International Journal of Educational Research 41, 6, 466–488.
WINNE, P.H. 2005. Key issues in modeling and applying research on self‐regulated learning. Applied Psychology 54, 2, 232–238.
WINNE, P.H. 2010a. Bootstrapping learner’s self-regulated learning. Psychological Test and Assessment Modeling 52, 4, 472.
WINNE, P.H. 2010b. Improving measurements of self-regulated learning. Educational Psychologist 45, 4, 267–276.
WINNE, P.H. 2017. Learning analytics for self-regulated learning. Handbook of Learning Analytics, C. Lang, G. Siemens, A. Wise, D. Gašević, Eds. Society for Learning Analytics and Research. 531–566.
WINNE, P.H. AND HADWIN, A.F. 1998. Studying as self-regulated learning. Metacognition in Educational Theory and Practice, D.J. Hacker, J. Dunlosky, and A.C. Graesser, Eds. Hillsdale, NJ: Erlbaum. 277–304.
WINNE, P.H. AND PERRY, N.E. 2000. Measuring self-regulated learning. Handbook of Self-Regulation, M. Boekaerts, P. Pintrich, and M. Zeidner, Eds. Orlando, FL: Academic Press. 531–566.
WINNE, P.H., TENG, K., CHANG, D., LIN M.P., MARZOUK, Z., NESBIT, J.C., PATZAK, A., RAKOVIC, M., SAMADI, D., VYTASEK, J. 2019. nStudy: software for learning analytics about processes for self-regulated learning. Journal of Learning Analytics 6, 2, 95–106.
ZIMMERMAN, B.J. 1990. Self-regulated learning and academic achievement: an overview. Educational Psychologist 25, 1, 3–17.
ZIMMERMAN, B.J. 2000. Attaining self-regulation: a social cognitive perspective. Handbook of Self-Regulation, M. Boekaerts, P. Pintrich, and M. Zeidner, Eds. Orlando, FL: Academic Press. 13–39.
ZIMMERMAN, B.J. AND SCHUNK, D.H. 2011. Self-regulated learning and performance. Handbook of Self-Regulation of Learning and Performance, B. J. Zimmerman and D. H. Schunk, Eds. New York, NY: Routledge. 1–12.
ALEVEN, V., MCLAREN, B., ROLL, I., AND KOEDINGER, K. 2006. Toward meta-cognitive tutoring: a model of help seeking with a cognitive tutor. International Journal of Artificial Intelligence in Education 16, 2, 101–128.
ALEVEN, V., ROLL, I., MCLAREN, B.M., AND KOEDINGER, K.R. 2016. Help helps, but only so much: research on help seeking with intelligent tutoring systems. International Journal of Artificial Intelligence in Education 26, 1, 205–223.
AZEVEDO, R., JOHNSON, A., CHAUNCEY, A., GRAESSER, A., ZIMMERMAN, B., AND SCHUNK, D. 2011. Use of hypermedia to assess and convey self-regulated learning. Handbook of Self-Regulation of Learning and Performance, B. Zimmerman and D.H. Schunk, Eds. New York, NY: Routledge. 102–121.
AZEVEDO, R., TAUB, M., AND MUDRICK, N.V. 2017. Understanding and reasoning about real-time cognitive, affective, and metacognitive processes to foster self-regulation with advanced learning technologies. Handbook of Self-Regulation of Learning and Performance, D.H. Schunk and J.A. Greene, Eds. New York, NY: Routledge. 254–270.
BAKER, R.S. AND DE CARVALHO, A.M.J.A. 2008. Labeling student behavior faster and more precisely with text replays. Proceedings of the 1st International Conference on Educational Data Mining, R.S. Baker, T. Barnes, and J.E. Beck, Eds. International Educational Data Mining Society. 38–47.
BAKER, R.S., CORBETT, A.T., AND KOEDINGER, K.R. 2004. Detecting student misuse of intelligent tutoring systems. International Conference on Intelligent Tutoring Systems, S. Lester, R.M. Vicari, and F. Paraguacu, Eds. Springer, Berlin, Heidelberg. 531–540.
BAKER, R.S., CORBETT, A.T., AND WAGNER, A.Z. 2006. Human classification of low-fidelity replays of student actions. Proceedings of the Educational Data Mining Workshop at the 8th International Conference on Intelligent Tutoring Systems, M. Ikeda, K. Ashley, and TW. Chan Eds. Springer, Berlin, Heidelberg. 29–36.
BAKER, R.S. AND HAWN, A. 2022. Algorithmic bias in education. International Journal of Artificial Intelligence in Education, 32(4), 1052-1092.
BAKER, R.S., MITROVIĆ, A., AND MATHEWS, M. 2010. Detecting gaming the system in constraint-based tutors. Proceedings of the 18th Annual Conference on User Modeling, Adaptation, and Personalization, P. Bra, A. Kobsa, and D. Chin, Eds, Springer, Berlin, Heidelberg. 267–278.
BAKER, R.S. AND OCUMPAUGH, J. 2016. Interaction-based affect detection in educational software. The Oxford Handbook of Affective Computing, R. Calvo, S. D’Mello, J. Gratch, and A. Kappas, Eds. New York, NY: Oxford University Press. 233–245.
BAKER, R.S. AND YACEF, K. 2009. The state of educational data mining in 2009: a review and future visions. Journal of Educational Data Mining 1, 1, 3–17.
BANDURA, A. 1986. Social foundations of thought and action: A social cognitive theory. Englewood Cliffs, NJ: Prentice Hall.
BISWAS, G., BAKER, R.S., AND PAQUETTE, L. 2017. Data mining methods for assessing self-regulated learning. Handbook of Self-Regulation of Learning and Performance, D.H. Schunk and J.A. Greene, Eds. New York, NY: Routledge. 388–403.
BISWAS, G., JEONG, H., KINNEBREW, J.S., SULCER, B., AND ROSCOE, R. 2010. Measuring self-regulated learning skills through social interactions in a teachable agent. Research and Practice in Technology Enhanced Learning 05, 02, 123–152.
BOEKAERTS, M. 1999. Self-regulated learning: where we are today. International Journal of Educational Research 31, 6, 445–457.
BOSCH, N., ZHANG, Y., PAQUETTE, L., BAKER, R., OCUMPAUGH, J., AND BISWAS, G. 2021. Students’ verbalized metacognition during computerized learning. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, ACM. 1–12.
BOTELHO, A.F., BAKER, R.S., OCUMPAUGH, J., AND HEFFERNAN, N.T. 2018. Studying affect dynamics and chronometry using sensor-free detectors. Proceedings of the 11th International Conference on Educational Data Mining, K.E. Boyer and M. Yudelson, Eds. International Educational Data Mining Society. 157–166.
CHARMAZ, K. 1983. The grounded theory method: An explication and interpretation. Contemporary Field Research, R. Emerson, Eds. Boston, MA: Little, Brown and Company. 109–126.
CHEN, T. AND GUESTRIN, C. 2016. XGBoost: a scalable tree boosting system. Proceedings of the 22nd ACM International Conference on Knowledge Discovery and Data Mining, ACM. 785–794.
CHO, M.-H. AND YOO, J.S. 2017. Exploring online students’ self-regulated learning with self-reported surveys and log files: a data mining approach. Interactive Learning Environments 25, 8, 970–982.
CLEARY, T.J. AND CHEN, P.P. 2009. Self-regulation, motivation, and math achievement in middle school: Variations across grade level and math context. Journal of School Psychology 47, 5, 291–314.
CONATI, C., PORAYSKA-POMSTA, K., AND MAVRIKIS, M. 2018. AI in education needs interpretable machine learning: lessons from open learner modelling. http://arxiv.org/abs/1807.00154.
DEVER, D.A., AMON, M.J., VRZAKOVA, H., WIEDBUSCH, M.D., CLOUDE, E.B., AND AZEVEDO, R. 2022. Capturing sequences of learners’ self-regulatory interactions with instructional material during game-based learning using auto-recurrence quantification analysis. Frontiers in Psychology, 13, 813677.
DEVOLDER, A., VAN BRAAK, J., AND TONDEUR, J. 2012. Supporting self‐regulated learning in computer‐based learning environments: systematic review of effects of scaffolding in the domain of science education. Journal of Computer Assisted Learning 28, 6, 557–573.
DICERBO, K.E. AND KIDWAI, K. 2013. Detecting player goals from game log files. Proceedings of the 6th International Conference on Educational Data Mining, S.K. D’Mello, R.A. Calvo, and A. Olney, Eds. International Educational Data Mining Society. 314-315.
EFKLIDES, A. 2011. Interactions of metacognition with motivation and affect in self-regulated learning: the MASRL model. Educational Psychologist 46, 1, 6–25.
FARHANA, E., POTTER, A., RUTHERFORD, T., AND LYNCH, C. F. 2021. Feedback and self-regulated learning in science reading. Proceedings of the 14th International Conference on Educational Data Mining, I.-H. Hsiao, S. Sahebi, F. Bouchet, and J.-J.Vie, Eds. International Educational Data Mining Society. 820-826
GARDNER, J., BROOKS, C., AND BAKER, R. 2019. Evaluating the fairness of predictive student models through slicing analysis. Proceedings of the 9th International Conference on Learning Analytics & Knowledge, D. Azcona and R. Chung, Eds. ACM. 225–234.
GRAESSER, A. AND MCNAMARA, D. 2010. Self-regulated learning in learning environments with pedagogical agents that interact in natural language. Educational Psychologist 45, 4, 234–244.
GRAESSER, A.C., PENUMATSA, P., VENTURA, M., CAI, Z., AND HU, X. 2007. Using LSA in autotutor: learning through mixed initiative dialogue in natural language. Handbook of Latent Semantic Analysis, T. Landauer, D. McNamara, S. Dennis, and W. Kintsch, Eds. Mahwah: Erlbaum. 243–262.
GREENE, J.A., DEEKENS, V.M., COPELAND, D.Z., AND YU, S. 2017. Capturing and modeling self-regulated learning using think-aloud protocols. Handbook of Self-Regulation of Learning and Performance, D.H. Schunk and J.A. Greene, Eds. New York, NY: Routledge. 323–337.
GREENE, J.A., ROBERTSON, J., AND COSTA, L.-J.C. 2013. Assessing self-regulated learning using think-aloud methods. Handbook of Self-Regulation of Learning and Performance, B.J. Zimmerman and D.H. Schunk, Eds. New York, NY: Routledge. 313–328.
KARUMBAIAH, S. AND BROOKS, J. 2021. How colonial continuities underlie algorithmic injustices in education. Conference on Research in Equitable and Sustained Participation in Engineering, Computing, and Technology, IEEE, 1–6.
KINNEBREW, J.S., LORETZ, K.M., AND BISWAS, G. 2013. A contextualized, differential sequence mining method to derive students’ learning behavior patterns. Journal of Educational Data Mining 5, 1, 190–219.
KIZILCEC, R.F. AND LEE, H. 2020. Algorithmic fairness in education. The Ethics of Artificial Intelligence in Education, W. Holmes and K. Porayska-Pomsta, Eds. Taylor & Francis. 174–202.
KÖCK, M. AND PARAMYTHIS, A. 2011. Activity sequence modelling and dynamic clustering for personalized e-learning. User Modeling and User-Adapted Interaction 21, 1–2, 51–97.
KOVANOVIC, V., GAŠEVIĆ, D., DAWSON, S., JOKSIMOVIC, S., AND BAKER, R. 2016. Does time-on-task estimation matter? Implications on validity of learning analytics findings. Journal of Learning Analytics 2, 3, 81–110.
LABUHN, A.S., ZIMMERMAN, B.J., AND HASSELHORN, M. 2010. Enhancing students’ self-regulation and mathematics performance: the influence of feedback and self-evaluative standards. Metacognition and Learning 5, 2, 173–194.
LEE, D.M.C., RODRIGO, MA.M.T., BAKER, R.S.J. D., SUGAY, J.O., AND CORONEL, A. 2011. Exploring the relationship between novice programmer confusion and achievement. International Conference on Affective Computing and Intelligent Interaction, S. D’Mello, A. Graesser, B. Schuller, J.C. Martin, Eds. Springer, Berlin, Heidelberg.175–184.
LUNDBERG, S.M., ERION, G.G., AND LEE, S.I. 2019. Consistent individualized feature attribution for tree ensembles. arXiv:1802.03888.
MU, T., JETTEN, A., AND BRUNSKILL, E. 2020. Towards suggesting actionable interventions for wheel-spinning students. Proceedings of the 13th International Conference on Educational Data Mining, A.N. Rafferty, J. Whitehill, V. Cavalli-Sforza, and C. Romero, Eds. International Educational Data Mining Society. 183–193.
MULDNER, K., BURLESON, W., VAN DE SANDE, B., AND VANLEHN, K. 2011. An analysis of students’ gaming behaviors in an intelligent tutoring system: predictors and impacts. User Modeling and User-Adapted Interaction 21, 1–2, 99–135.
NOTA, L., SORESI, S., AND ZIMMERMAN, B.J. 2004. Self-regulation and academic achievement and resilience: A longitudinal study. International Journal of Educational Research 41, 3, 198–215.
OCUMPAUGH, J., BAKER, R., GOWDA, S., HEFFERNAN, N., AND HEFFERNAN, C. 2014. Population validity for educational data mining models: A case study in affect detection. British Journal of Educational Technology 45, 3, 487–501.
OCUMPAUGH, J., HUTT, S., ANDRES, J.M.A.L., BAKER, R.S., BISWAS, G., BOSCH, N., PAQUETTE, L., AND MUNSHI, A. 2021. Using qualitative data from targeted interviews to inform rapid AIED development. Proceedings of the 29th International Conference on Computers in Education, M.M.T. Rodrigo, S. Iyer, A. Mitrovic, H.N.H. Cheng, D. Kohen-Vacs, C.M.A. Palalas, R. Rajenran, K. Seta, and J. Wang, Eds. Asia-Pacific Society for Computers in Education. 69–74.
OKUR, E., ASLAN, S., ALYUZ, N., ARSLAN ESME, A., AND BAKER, R.S. 2018. Role of socio-cultural differences in labeling students’ affective states. Proceedings of the 19th International Conference on Artificial Intelligence in Education, C.P. Rose, R, Martinez-Maldonado, H.U. Hoppe, R. Luckin, M. Mavrikis, K. Porayska-Pomsta, B. McLaren, B. du Boulay, Eds. Springer, Cham. 367–380.
PANADERO, E. 2017. A review of self-regulated learning: six models and four directions for research. Frontiers in Psychology 8, 422.
PAQUETTE, L. AND BAKER, R.S. 2017. Variations of gaming behaviors across populations of students and across learning environments. Proceedings of the 18th International Conference on Artificial Intelligence in Education, E. André, R. Baker, X. Hu, M.M.T. Rodrigo, and B. du Boulay, Eds. Springer Cham. 274–286.
PAQUETTE, L. AND BAKER, R.S. 2019. Comparing machine learning to knowledge engineering for student behavior modeling: a case study in gaming the system. Interactive Learning Environments 27, 5–6, 585–597.
PAQUETTE, L., DE CARVALHO, A.M.J.A., AND BAKER, R.S. 2014. Towards understanding expert coding of student disengagement in online learning. Proceedings of the 36th Annual Cognitive Science Conference, P. Bello, M. Guarini, M. McShane, and B. Scassellati, Eds. Cognitive Science Society. 1126–1131.
PAQUETTE, L., OCUMPAUGH, J., LI, Z., ANDRES, A., AND BAKER, R. 2020. Who’s learning? Using demographics in EDM research. Journal of Educational Data Mining 12, 3, 1–30.
PEDREGOSA, F., VAROQUAUX, G., GRAMFORT, A., ET AL. 2011. Scikit-learn: Machine Learning in Python. Journal of machine Learning research 12, 2825–2830.
PINTRICH, P.R. 2000. The role of goal orientation in self-regulated learning. Handbook of Self-Regulation, M. Boekaerts, P. Pintrich, and M. Zeidner, Eds. San Diego, CA: Academic Press. 451–502.
PINTRICH, P.R., SMITH, D., GARCIA, T., AND MCKEACHIE, W. 1991. A manual for the use of the motivated strategies for learning questionnaire (MSLQ).
RICHEY, J. E., ZHANG, J., DAS, R., ANDRES-BRAY, J. M., SCRUGGS, R., MOGESSIE, M., BAKER, R.S, AND MCLAREN, B. M. 2021. Gaming and confrustion explain learning advantages for a math digital learning game. Proceedings of the 22nd International Conference on Artificial Intelligence in Education, G. Biswas, S. Bull, J. Kay, and A. Mitrovic, Eds. Springer, Berlin, Heidelberg. 342–355.
ROLL, I., ALEVEN, V., MCLAREN, B.M., AND KOEDINGER, K.R. 2007. Can help seeking be tutored? Searching for the secret sauce of metacognitive tutoring. Proceedings of the 13th International Conference on Artificial Intelligence in Education, R. Luckin, K.R. Koedinger, and J. Greer, Eds. IOS Press. 203-210.
ROTH, A., OGRIN, S., AND SCHMITZ, B. 2016. Assessing self-regulated learning in higher education: a systematic literature review of self-report instruments. Educational Assessment, Evaluation and Accountability 28, 3, 225–250.
SABOURIN, J., SHORES, L.R., MOTT, B.W., AND LESTER, J.C. 2012. Predicting student self-regulation strategies in game-based learning environments. Proceedings of the 11th International Conference on Intelligent Tutoring Systems, S.A. Cerri, W.J. Clancey, G. Papadourakis, and K. Panourgia, Eds. Springer Berlin Heidelberg. 141–150.
SAO PEDRO, M.A., DE BAKER, R.S.J., GOBERT, J.D., MONTALVO, O., AND NAKAMA, A. 2013. Leveraging machine-learned detectors of systematic inquiry behavior to estimate and predict transfer of inquiry skill. User Modeling and User-Adapted Interaction 23, 1, 1–39.
SCHOOLER, J.W., OHLSSON, S., AND BROOKS, K. 1993. Thoughts beyond words: when language overshadows insight. Journal of Experimental Psychology: General 122, 2, 166–183.
SCULLEY, D., SNOEK, J., WILTSCHKO, A., AND RAHIMI, A. 2018. Winner’s curse. On pace, progress, and empirical rigor. Proceedings of the 6th International Conference on Leaning Representations, Y. Bengio and Y. LeCun, Eds. OpenReview.
SEGEDY, J.R., KINNEBREW, J.S., AND BISWAS, G. 2015. Using coherence analysis to characterize self-regulated learning behaviours in open-ended learning environments. Journal of Learning Analytics 2, 1, 13–48.
SMITH, T.F. AND WATERMAN, M.S. 1981. Identification of common molecular subsequences. Journal of Molecular Biology 147, 1, 195–197.
SWAMY, V., RADMEHR, B., KRCO, N., MARRAS, M., AND KÄSER, T. 2022. Evaluating the explainers: black-box explainable machine learning for student success prediction in MOOCs. Proceedings of the 15th International Conference on Educational Data Mining, A. Mitrovic and N. Bosch, Eds. International Educational Data Mining Society. 98-109.
TAUB, M., AZEVEDO, R., BOUCHET, F., AND KHOSRAVIFAR, B. 2014. Can the use of cognitive and metacognitive self-regulated learning strategies be predicted by learners’ levels of prior knowledge in hypermedia-learning environments? Computers in Human Behavior 39, 356–367.
WALONOSKI, J.A. AND HEFFERNAN, N.T. 2006. Prevention of off-task gaming behavior in intelligent tutoring systems. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, M. Ikeda, K.D. Ashley, and TW. Chan, Eds. Springer Berlin Heidelberg, 722–724.
WEBB, M.E., FLUCK, A., MAGENHEIM, J., MALYN-SMITH, J., WATERS, J., DESCHENES, M., AND ZAGAMI, J. 2021. Machine learning for human learners: opportunities, issues, tensions and threats. Educational Technology Research and Development 69, 4, 2109–2130.
WEINSTEIN, C., PALMER, D., AND SCHULTE, A.C. 1987. Learning and study strategies inventory (LASSI). FL: H & H Publishing.
WESTON, C., GANDELL, T., BEAUCHAMP, J., MCALPINE, L., WISEMAN, C., AND BEAUCHAMP, C. 2001. Analyzing interview data: the development and evolution of a coding system. Qualitative Sociology 24, 3, 381–400.
WIJFFELS, J., STRAKA, M., AND STRAKOVA, J. 2017. Package “udpipe”.
WINNE, P.H. 1997. Experimenting to bootstrap self-regulated learning. Journal of Educational Psychology 89, 3, 397–410.
WINNE, P.H. 2004. Students’ calibration of knowledge and learning processes: Implications for designing powerful software learning environments. International Journal of Educational Research 41, 6, 466–488.
WINNE, P.H. 2005. Key issues in modeling and applying research on self‐regulated learning. Applied Psychology 54, 2, 232–238.
WINNE, P.H. 2010a. Bootstrapping learner’s self-regulated learning. Psychological Test and Assessment Modeling 52, 4, 472.
WINNE, P.H. 2010b. Improving measurements of self-regulated learning. Educational Psychologist 45, 4, 267–276.
WINNE, P.H. 2017. Learning analytics for self-regulated learning. Handbook of Learning Analytics, C. Lang, G. Siemens, A. Wise, D. Gašević, Eds. Society for Learning Analytics and Research. 531–566.
WINNE, P.H. AND HADWIN, A.F. 1998. Studying as self-regulated learning. Metacognition in Educational Theory and Practice, D.J. Hacker, J. Dunlosky, and A.C. Graesser, Eds. Hillsdale, NJ: Erlbaum. 277–304.
WINNE, P.H. AND PERRY, N.E. 2000. Measuring self-regulated learning. Handbook of Self-Regulation, M. Boekaerts, P. Pintrich, and M. Zeidner, Eds. Orlando, FL: Academic Press. 531–566.
WINNE, P.H., TENG, K., CHANG, D., LIN M.P., MARZOUK, Z., NESBIT, J.C., PATZAK, A., RAKOVIC, M., SAMADI, D., VYTASEK, J. 2019. nStudy: software for learning analytics about processes for self-regulated learning. Journal of Learning Analytics 6, 2, 95–106.
ZIMMERMAN, B.J. 1990. Self-regulated learning and academic achievement: an overview. Educational Psychologist 25, 1, 3–17.
ZIMMERMAN, B.J. 2000. Attaining self-regulation: a social cognitive perspective. Handbook of Self-Regulation, M. Boekaerts, P. Pintrich, and M. Zeidner, Eds. Orlando, FL: Academic Press. 13–39.
ZIMMERMAN, B.J. AND SCHUNK, D.H. 2011. Self-regulated learning and performance. Handbook of Self-Regulation of Learning and Performance, B. J. Zimmerman and D. H. Schunk, Eds. New York, NY: Routledge. 1–12.
Issue
Section
Extended Articles from the EDM 2022 Conference
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors who publish with this journal agree to the following terms:
- The Author retains copyright in the Work, where the term “Work” shall include all digital objects that may result in subsequent electronic publication or distribution.
- Upon acceptance of the Work, the author shall grant to the Publisher the right of first publication of the Work.
- The Author shall grant to the Publisher and its agents the nonexclusive perpetual right and license to publish, archive, and make accessible the Work in whole or in part in all forms of media now or hereafter known under a Creative Commons 4.0 License (Attribution-Noncommercial-No Derivatives 4.0 International), or its equivalent, which, for the avoidance of doubt, allows others to copy, distribute, and transmit the Work under the following conditions:
- Attribution—other users must attribute the Work in the manner specified by the author as indicated on the journal Web site;
- Noncommercial—other users (including Publisher) may not use this Work for commercial purposes;
- No Derivative Works—other users (including Publisher) may not alter, transform, or build upon this Work,with the understanding that any of the above conditions can be waived with permission from the Author and that where the Work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.
- The Author is able to enter into separate, additional contractual arrangements for the nonexclusive distribution of the journal's published version of the Work (e.g., post it to an institutional repository or publish it in a book), as long as there is provided in the document an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post online a pre-publication manuscript (but not the Publisher’s final formatted PDF version of the Work) in institutional repositories or on their Websites prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (see The Effect of Open Access). Any such posting made before acceptance and publication of the Work shall be updated upon publication to include a reference to the Publisher-assigned DOI (Digital Object Identifier) and a link to the online abstract for the final published Work in the Journal.
- Upon Publisher’s request, the Author agrees to furnish promptly to Publisher, at the Author’s own expense, written evidence of the permissions, licenses, and consents for use of third-party material included within the Work, except as determined by Publisher to be covered by the principles of Fair Use.
- The Author represents and warrants that:
- the Work is the Author’s original work;
- the Author has not transferred, and will not transfer, exclusive rights in the Work to any third party;
- the Work is not pending review or under consideration by another publisher;
- the Work has not previously been published;
- the Work contains no misrepresentation or infringement of the Work or property of other authors or third parties; and
- the Work contains no libel, invasion of privacy, or other unlawful matter.
- The Author agrees to indemnify and hold Publisher harmless from Author’s breach of the representations and warranties contained in Paragraph 6 above, as well as any claim or proceeding relating to Publisher’s use and publication of any content contained in the Work, including third-party content.