Using Machine Learning to Detect SMART Model Cognitive Operations in Mathematical Problem-Solving Process

##plugins.themes.bootstrap3.article.main##

##plugins.themes.bootstrap3.article.sidebar##

Published Dec 18, 2022
Jiayi Zhang Juliana Ma. Alexandra L. Andres Stephen Hutt Ryan S. Baker Jaclyn Ocumpaugh Nidhi Nasiar Caitlin Mills Jamiella Brooks Sheela Sethuaman Tyron Young

Abstract

Self-regulated learning (SRL) is a critical component of mathematics problem-solving. Students skilled in
SRL are more likely to effectively set goals, search for information, and direct their attention and cognitive
process so that they align their efforts with their objectives. An influential framework for SRL, the SMART
model (Winne, 2017), proposes that five cognitive operations (i.e., searching, monitoring, assembling,
rehearsing, and translating) play a key role in SRL. However, these categories encompass a wide range of
behaviors, making measurement challenging – often involving observing individual students and recording
their think-aloud activities or asking students to complete labor-intensive tagging activities as they work. In
the current study, to achieve better scalability, we operationalized indicators of SMART operations and
developed automated detectors using machine learning. We analyzed students’ textual responses and
interaction data collected from a mathematical learning platform where students are asked to thoroughly
explain their solutions and are scaffolded in communicating their problem-solving process. Due to the rarity
in data for one of the seven SRL indicators operationalized, we built six models to reflect students’ use of
four SMART operations. These models are found to be reliable and generalizable, with AUC ROCs ranging
from .76-.89. When applied to the full test set, these detectors are relatively robust to algorithmic bias,
performing well across different student populations and with no consistent bias against a specific group of
students.

How to Cite

Zhang, J., Andres, J. M. A. L., Hutt, S., Baker, R. S., Ocumpaugh, J., Nasiar, N., Mills, C., Brooks, J., Sethuaman, S., & Young, T. (2022). Using Machine Learning to Detect SMART Model Cognitive Operations in Mathematical Problem-Solving Process. Journal of Educational Data Mining, 14(3), 76–108. https://doi.org/10.5281/zenodo.7304763
Abstract 670 | PDF Downloads 534

##plugins.themes.bootstrap3.article.details##

Keywords

self-regulated learning, SMART model, automated detectors

References
AGUILAR, S.J., KARABENICK, S.A., TEASLEY, S.D., AND BAEK, C. 2021. Associations between learning analytics dashboard exposure and motivation and self-regulated learning. Computers & Education 162.104085

ALEVEN, V., MCLAREN, B., ROLL, I., AND KOEDINGER, K. 2006. Toward meta-cognitive tutoring: a model of help seeking with a cognitive tutor. International Journal of Artificial Intelligence in Education 16, 2, 101–128.

ALEVEN, V., ROLL, I., MCLAREN, B.M., AND KOEDINGER, K.R. 2016. Help helps, but only so much: research on help seeking with intelligent tutoring systems. International Journal of Artificial Intelligence in Education 26, 1, 205–223.

AZEVEDO, R., JOHNSON, A., CHAUNCEY, A., GRAESSER, A., ZIMMERMAN, B., AND SCHUNK, D. 2011. Use of hypermedia to assess and convey self-regulated learning. Handbook of Self-Regulation of Learning and Performance, B. Zimmerman and D.H. Schunk, Eds. New York, NY: Routledge. 102–121.

AZEVEDO, R., TAUB, M., AND MUDRICK, N.V. 2017. Understanding and reasoning about real-time cognitive, affective, and metacognitive processes to foster self-regulation with advanced learning technologies. Handbook of Self-Regulation of Learning and Performance, D.H. Schunk and J.A. Greene, Eds. New York, NY: Routledge. 254–270.

BAKER, R.S. AND DE CARVALHO, A.M.J.A. 2008. Labeling student behavior faster and more precisely with text replays. Proceedings of the 1st International Conference on Educational Data Mining, R.S. Baker, T. Barnes, and J.E. Beck, Eds. International Educational Data Mining Society. 38–47.

BAKER, R.S., CORBETT, A.T., AND KOEDINGER, K.R. 2004. Detecting student misuse of intelligent tutoring systems. International Conference on Intelligent Tutoring Systems, S. Lester, R.M. Vicari, and F. Paraguacu, Eds. Springer, Berlin, Heidelberg. 531–540.

BAKER, R.S., CORBETT, A.T., AND WAGNER, A.Z. 2006. Human classification of low-fidelity replays of student actions. Proceedings of the Educational Data Mining Workshop at the 8th International Conference on Intelligent Tutoring Systems, M. Ikeda, K. Ashley, and TW. Chan Eds. Springer, Berlin, Heidelberg. 29–36.

BAKER, R.S. AND HAWN, A. 2022. Algorithmic bias in education. International Journal of Artificial Intelligence in Education, 32(4), 1052-1092.

BAKER, R.S., MITROVIĆ, A., AND MATHEWS, M. 2010. Detecting gaming the system in constraint-based tutors. Proceedings of the 18th Annual Conference on User Modeling, Adaptation, and Personalization, P. Bra, A. Kobsa, and D. Chin, Eds, Springer, Berlin, Heidelberg. 267–278.

BAKER, R.S. AND OCUMPAUGH, J. 2016. Interaction-based affect detection in educational software. The Oxford Handbook of Affective Computing, R. Calvo, S. D’Mello, J. Gratch, and A. Kappas, Eds. New York, NY: Oxford University Press. 233–245.

BAKER, R.S. AND YACEF, K. 2009. The state of educational data mining in 2009: a review and future visions. Journal of Educational Data Mining 1, 1, 3–17.

BANDURA, A. 1986. Social foundations of thought and action: A social cognitive theory. Englewood Cliffs, NJ: Prentice Hall.

BISWAS, G., BAKER, R.S., AND PAQUETTE, L. 2017. Data mining methods for assessing self-regulated learning. Handbook of Self-Regulation of Learning and Performance, D.H. Schunk and J.A. Greene, Eds. New York, NY: Routledge. 388–403.

BISWAS, G., JEONG, H., KINNEBREW, J.S., SULCER, B., AND ROSCOE, R. 2010. Measuring self-regulated learning skills through social interactions in a teachable agent. Research and Practice in Technology Enhanced Learning 05, 02, 123–152.

BOEKAERTS, M. 1999. Self-regulated learning: where we are today. International Journal of Educational Research 31, 6, 445–457.

BOSCH, N., ZHANG, Y., PAQUETTE, L., BAKER, R., OCUMPAUGH, J., AND BISWAS, G. 2021. Students’ verbalized metacognition during computerized learning. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, ACM. 1–12.

BOTELHO, A.F., BAKER, R.S., OCUMPAUGH, J., AND HEFFERNAN, N.T. 2018. Studying affect dynamics and chronometry using sensor-free detectors. Proceedings of the 11th International Conference on Educational Data Mining, K.E. Boyer and M. Yudelson, Eds. International Educational Data Mining Society. 157–166.

CHARMAZ, K. 1983. The grounded theory method: An explication and interpretation. Contemporary Field Research, R. Emerson, Eds. Boston, MA: Little, Brown and Company. 109–126.

CHEN, T. AND GUESTRIN, C. 2016. XGBoost: a scalable tree boosting system. Proceedings of the 22nd ACM International Conference on Knowledge Discovery and Data Mining, ACM. 785–794.

CHO, M.-H. AND YOO, J.S. 2017. Exploring online students’ self-regulated learning with self-reported surveys and log files: a data mining approach. Interactive Learning Environments 25, 8, 970–982.

CLEARY, T.J. AND CHEN, P.P. 2009. Self-regulation, motivation, and math achievement in middle school: Variations across grade level and math context. Journal of School Psychology 47, 5, 291–314.

CONATI, C., PORAYSKA-POMSTA, K., AND MAVRIKIS, M. 2018. AI in education needs interpretable machine learning: lessons from open learner modelling. http://arxiv.org/abs/1807.00154.

DEVER, D.A., AMON, M.J., VRZAKOVA, H., WIEDBUSCH, M.D., CLOUDE, E.B., AND AZEVEDO, R. 2022. Capturing sequences of learners’ self-regulatory interactions with instructional material during game-based learning using auto-recurrence quantification analysis. Frontiers in Psychology, 13, 813677.

DEVOLDER, A., VAN BRAAK, J., AND TONDEUR, J. 2012. Supporting self‐regulated learning in computer‐based learning environments: systematic review of effects of scaffolding in the domain of science education. Journal of Computer Assisted Learning 28, 6, 557–573.

DICERBO, K.E. AND KIDWAI, K. 2013. Detecting player goals from game log files. Proceedings of the 6th International Conference on Educational Data Mining, S.K. D’Mello, R.A. Calvo, and A. Olney, Eds. International Educational Data Mining Society. 314-315.

EFKLIDES, A. 2011. Interactions of metacognition with motivation and affect in self-regulated learning: the MASRL model. Educational Psychologist 46, 1, 6–25.

FARHANA, E., POTTER, A., RUTHERFORD, T., AND LYNCH, C. F. 2021. Feedback and self-regulated learning in science reading. Proceedings of the 14th International Conference on Educational Data Mining, I.-H. Hsiao, S. Sahebi, F. Bouchet, and J.-J.Vie, Eds. International Educational Data Mining Society. 820-826

GARDNER, J., BROOKS, C., AND BAKER, R. 2019. Evaluating the fairness of predictive student models through slicing analysis. Proceedings of the 9th International Conference on Learning Analytics & Knowledge, D. Azcona and R. Chung, Eds. ACM. 225–234.

GRAESSER, A. AND MCNAMARA, D. 2010. Self-regulated learning in learning environments with pedagogical agents that interact in natural language. Educational Psychologist 45, 4, 234–244.

GRAESSER, A.C., PENUMATSA, P., VENTURA, M., CAI, Z., AND HU, X. 2007. Using LSA in autotutor: learning through mixed initiative dialogue in natural language. Handbook of Latent Semantic Analysis, T. Landauer, D. McNamara, S. Dennis, and W. Kintsch, Eds. Mahwah: Erlbaum. 243–262.

GREENE, J.A., DEEKENS, V.M., COPELAND, D.Z., AND YU, S. 2017. Capturing and modeling self-regulated learning using think-aloud protocols. Handbook of Self-Regulation of Learning and Performance, D.H. Schunk and J.A. Greene, Eds. New York, NY: Routledge. 323–337.

GREENE, J.A., ROBERTSON, J., AND COSTA, L.-J.C. 2013. Assessing self-regulated learning using think-aloud methods. Handbook of Self-Regulation of Learning and Performance, B.J. Zimmerman and D.H. Schunk, Eds. New York, NY: Routledge. 313–328.

KARUMBAIAH, S. AND BROOKS, J. 2021. How colonial continuities underlie algorithmic injustices in education. Conference on Research in Equitable and Sustained Participation in Engineering, Computing, and Technology, IEEE, 1–6.

KINNEBREW, J.S., LORETZ, K.M., AND BISWAS, G. 2013. A contextualized, differential sequence mining method to derive students’ learning behavior patterns. Journal of Educational Data Mining 5, 1, 190–219.

KIZILCEC, R.F. AND LEE, H. 2020. Algorithmic fairness in education. The Ethics of Artificial Intelligence in Education, W. Holmes and K. Porayska-Pomsta, Eds. Taylor & Francis. 174–202.

KÖCK, M. AND PARAMYTHIS, A. 2011. Activity sequence modelling and dynamic clustering for personalized e-learning. User Modeling and User-Adapted Interaction 21, 1–2, 51–97.

KOVANOVIC, V., GAŠEVIĆ, D., DAWSON, S., JOKSIMOVIC, S., AND BAKER, R. 2016. Does time-on-task estimation matter? Implications on validity of learning analytics findings. Journal of Learning Analytics 2, 3, 81–110.

LABUHN, A.S., ZIMMERMAN, B.J., AND HASSELHORN, M. 2010. Enhancing students’ self-regulation and mathematics performance: the influence of feedback and self-evaluative standards. Metacognition and Learning 5, 2, 173–194.

LEE, D.M.C., RODRIGO, MA.M.T., BAKER, R.S.J. D., SUGAY, J.O., AND CORONEL, A. 2011. Exploring the relationship between novice programmer confusion and achievement. International Conference on Affective Computing and Intelligent Interaction, S. D’Mello, A. Graesser, B. Schuller, J.C. Martin, Eds. Springer, Berlin, Heidelberg.175–184.

LUNDBERG, S.M., ERION, G.G., AND LEE, S.I. 2019. Consistent individualized feature attribution for tree ensembles. arXiv:1802.03888.

MU, T., JETTEN, A., AND BRUNSKILL, E. 2020. Towards suggesting actionable interventions for wheel-spinning students. Proceedings of the 13th International Conference on Educational Data Mining, A.N. Rafferty, J. Whitehill, V. Cavalli-Sforza, and C. Romero, Eds. International Educational Data Mining Society. 183–193.

MULDNER, K., BURLESON, W., VAN DE SANDE, B., AND VANLEHN, K. 2011. An analysis of students’ gaming behaviors in an intelligent tutoring system: predictors and impacts. User Modeling and User-Adapted Interaction 21, 1–2, 99–135.

NOTA, L., SORESI, S., AND ZIMMERMAN, B.J. 2004. Self-regulation and academic achievement and resilience: A longitudinal study. International Journal of Educational Research 41, 3, 198–215.

OCUMPAUGH, J., BAKER, R., GOWDA, S., HEFFERNAN, N., AND HEFFERNAN, C. 2014. Population validity for educational data mining models: A case study in affect detection. British Journal of Educational Technology 45, 3, 487–501.

OCUMPAUGH, J., HUTT, S., ANDRES, J.M.A.L., BAKER, R.S., BISWAS, G., BOSCH, N., PAQUETTE, L., AND MUNSHI, A. 2021. Using qualitative data from targeted interviews to inform rapid AIED development. Proceedings of the 29th International Conference on Computers in Education, M.M.T. Rodrigo, S. Iyer, A. Mitrovic, H.N.H. Cheng, D. Kohen-Vacs, C.M.A. Palalas, R. Rajenran, K. Seta, and J. Wang, Eds. Asia-Pacific Society for Computers in Education. 69–74.

OKUR, E., ASLAN, S., ALYUZ, N., ARSLAN ESME, A., AND BAKER, R.S. 2018. Role of socio-cultural differences in labeling students’ affective states. Proceedings of the 19th International Conference on Artificial Intelligence in Education, C.P. Rose, R, Martinez-Maldonado, H.U. Hoppe, R. Luckin, M. Mavrikis, K. Porayska-Pomsta, B. McLaren, B. du Boulay, Eds. Springer, Cham. 367–380.

PANADERO, E. 2017. A review of self-regulated learning: six models and four directions for research. Frontiers in Psychology 8, 422.

PAQUETTE, L. AND BAKER, R.S. 2017. Variations of gaming behaviors across populations of students and across learning environments. Proceedings of the 18th International Conference on Artificial Intelligence in Education, E. André, R. Baker, X. Hu, M.M.T. Rodrigo, and B. du Boulay, Eds. Springer Cham. 274–286.

PAQUETTE, L. AND BAKER, R.S. 2019. Comparing machine learning to knowledge engineering for student behavior modeling: a case study in gaming the system. Interactive Learning Environments 27, 5–6, 585–597.

PAQUETTE, L., DE CARVALHO, A.M.J.A., AND BAKER, R.S. 2014. Towards understanding expert coding of student disengagement in online learning. Proceedings of the 36th Annual Cognitive Science Conference, P. Bello, M. Guarini, M. McShane, and B. Scassellati, Eds. Cognitive Science Society. 1126–1131.

PAQUETTE, L., OCUMPAUGH, J., LI, Z., ANDRES, A., AND BAKER, R. 2020. Who’s learning? Using demographics in EDM research. Journal of Educational Data Mining 12, 3, 1–30.

PEDREGOSA, F., VAROQUAUX, G., GRAMFORT, A., ET AL. 2011. Scikit-learn: Machine Learning in Python. Journal of machine Learning research 12, 2825–2830.

PINTRICH, P.R. 2000. The role of goal orientation in self-regulated learning. Handbook of Self-Regulation, M. Boekaerts, P. Pintrich, and M. Zeidner, Eds. San Diego, CA: Academic Press. 451–502.

PINTRICH, P.R., SMITH, D., GARCIA, T., AND MCKEACHIE, W. 1991. A manual for the use of the motivated strategies for learning questionnaire (MSLQ).

RICHEY, J. E., ZHANG, J., DAS, R., ANDRES-BRAY, J. M., SCRUGGS, R., MOGESSIE, M., BAKER, R.S, AND MCLAREN, B. M. 2021. Gaming and confrustion explain learning advantages for a math digital learning game. Proceedings of the 22nd International Conference on Artificial Intelligence in Education, G. Biswas, S. Bull, J. Kay, and A. Mitrovic, Eds. Springer, Berlin, Heidelberg. 342–355.

ROLL, I., ALEVEN, V., MCLAREN, B.M., AND KOEDINGER, K.R. 2007. Can help seeking be tutored? Searching for the secret sauce of metacognitive tutoring. Proceedings of the 13th International Conference on Artificial Intelligence in Education, R. Luckin, K.R. Koedinger, and J. Greer, Eds. IOS Press. 203-210.

ROTH, A., OGRIN, S., AND SCHMITZ, B. 2016. Assessing self-regulated learning in higher education: a systematic literature review of self-report instruments. Educational Assessment, Evaluation and Accountability 28, 3, 225–250.

SABOURIN, J., SHORES, L.R., MOTT, B.W., AND LESTER, J.C. 2012. Predicting student self-regulation strategies in game-based learning environments. Proceedings of the 11th International Conference on Intelligent Tutoring Systems, S.A. Cerri, W.J. Clancey, G. Papadourakis, and K. Panourgia, Eds. Springer Berlin Heidelberg. 141–150.

SAO PEDRO, M.A., DE BAKER, R.S.J., GOBERT, J.D., MONTALVO, O., AND NAKAMA, A. 2013. Leveraging machine-learned detectors of systematic inquiry behavior to estimate and predict transfer of inquiry skill. User Modeling and User-Adapted Interaction 23, 1, 1–39.

SCHOOLER, J.W., OHLSSON, S., AND BROOKS, K. 1993. Thoughts beyond words: when language overshadows insight. Journal of Experimental Psychology: General 122, 2, 166–183.

SCULLEY, D., SNOEK, J., WILTSCHKO, A., AND RAHIMI, A. 2018. Winner’s curse. On pace, progress, and empirical rigor. Proceedings of the 6th International Conference on Leaning Representations, Y. Bengio and Y. LeCun, Eds. OpenReview.

SEGEDY, J.R., KINNEBREW, J.S., AND BISWAS, G. 2015. Using coherence analysis to characterize self-regulated learning behaviours in open-ended learning environments. Journal of Learning Analytics 2, 1, 13–48.

SMITH, T.F. AND WATERMAN, M.S. 1981. Identification of common molecular subsequences. Journal of Molecular Biology 147, 1, 195–197.

SWAMY, V., RADMEHR, B., KRCO, N., MARRAS, M., AND KÄSER, T. 2022. Evaluating the explainers: black-box explainable machine learning for student success prediction in MOOCs. Proceedings of the 15th International Conference on Educational Data Mining, A. Mitrovic and N. Bosch, Eds. International Educational Data Mining Society. 98-109.

TAUB, M., AZEVEDO, R., BOUCHET, F., AND KHOSRAVIFAR, B. 2014. Can the use of cognitive and metacognitive self-regulated learning strategies be predicted by learners’ levels of prior knowledge in hypermedia-learning environments? Computers in Human Behavior 39, 356–367.

WALONOSKI, J.A. AND HEFFERNAN, N.T. 2006. Prevention of off-task gaming behavior in intelligent tutoring systems. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, M. Ikeda, K.D. Ashley, and TW. Chan, Eds. Springer Berlin Heidelberg, 722–724.

WEBB, M.E., FLUCK, A., MAGENHEIM, J., MALYN-SMITH, J., WATERS, J., DESCHENES, M., AND ZAGAMI, J. 2021. Machine learning for human learners: opportunities, issues, tensions and threats. Educational Technology Research and Development 69, 4, 2109–2130.

WEINSTEIN, C., PALMER, D., AND SCHULTE, A.C. 1987. Learning and study strategies inventory (LASSI). FL: H & H Publishing.

WESTON, C., GANDELL, T., BEAUCHAMP, J., MCALPINE, L., WISEMAN, C., AND BEAUCHAMP, C. 2001. Analyzing interview data: the development and evolution of a coding system. Qualitative Sociology 24, 3, 381–400.

WIJFFELS, J., STRAKA, M., AND STRAKOVA, J. 2017. Package “udpipe”.

WINNE, P.H. 1997. Experimenting to bootstrap self-regulated learning. Journal of Educational Psychology 89, 3, 397–410.

WINNE, P.H. 2004. Students’ calibration of knowledge and learning processes: Implications for designing powerful software learning environments. International Journal of Educational Research 41, 6, 466–488.

WINNE, P.H. 2005. Key issues in modeling and applying research on self‐regulated learning. Applied Psychology 54, 2, 232–238.

WINNE, P.H. 2010a. Bootstrapping learner’s self-regulated learning. Psychological Test and Assessment Modeling 52, 4, 472.

WINNE, P.H. 2010b. Improving measurements of self-regulated learning. Educational Psychologist 45, 4, 267–276.

WINNE, P.H. 2017. Learning analytics for self-regulated learning. Handbook of Learning Analytics, C. Lang, G. Siemens, A. Wise, D. Gašević, Eds. Society for Learning Analytics and Research. 531–566.

WINNE, P.H. AND HADWIN, A.F. 1998. Studying as self-regulated learning. Metacognition in Educational Theory and Practice, D.J. Hacker, J. Dunlosky, and A.C. Graesser, Eds. Hillsdale, NJ: Erlbaum. 277–304.

WINNE, P.H. AND PERRY, N.E. 2000. Measuring self-regulated learning. Handbook of Self-Regulation, M. Boekaerts, P. Pintrich, and M. Zeidner, Eds. Orlando, FL: Academic Press. 531–566.

WINNE, P.H., TENG, K., CHANG, D., LIN M.P., MARZOUK, Z., NESBIT, J.C., PATZAK, A., RAKOVIC, M., SAMADI, D., VYTASEK, J. 2019. nStudy: software for learning analytics about processes for self-regulated learning. Journal of Learning Analytics 6, 2, 95–106.

ZIMMERMAN, B.J. 1990. Self-regulated learning and academic achievement: an overview. Educational Psychologist 25, 1, 3–17.

ZIMMERMAN, B.J. 2000. Attaining self-regulation: a social cognitive perspective. Handbook of Self-Regulation, M. Boekaerts, P. Pintrich, and M. Zeidner, Eds. Orlando, FL: Academic Press. 13–39.

ZIMMERMAN, B.J. AND SCHUNK, D.H. 2011. Self-regulated learning and performance. Handbook of Self-Regulation of Learning and Performance, B. J. Zimmerman and D. H. Schunk, Eds. New York, NY: Routledge. 1–12.
Section
Extended Articles from the EDM 2022 Conference

Most read articles by the same author(s)

1 2 > >>