Advanced Knowledge Tracing: Incorporating Process Data and Curricula Information via an Attention-Based Framework for Accuracy and Interpretability
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
Oct 17, 2024
Yikai Lu
Lingbo Tong
Ying Cheng
https://orcid.org/0000-0003-4410-2589
Abstract
Knowledge tracing aims to model and predict students’ knowledge states during learning activities. Traditional methods like Bayesian Knowledge Tracing (BKT) and logistic regression have limitations in granularity and performance, while deep knowledge tracing (DKT) models often suffer from lacking transparency. This paper proposes a Transformer-based framework that emphasizes both accuracy
and interpretability. It captures the relationship between student behaviors and learning outcomes considering the associations between exam and exercise problems. We participated in the EDM Cup 2023 Contest using the proposed framework and achieved first place on the task of predicting students’ performance on end-of-unit test problems using clickstream data from previous assignments. Furthermore, the framework provides meaningful insights by analyzing user actions and visualizing attention weight matrices. These insights enable targeted interventions and personalized support, enhancing online learning experiences. We have uploaded our code, saved models, and predictions to an OSF repository: https://osf.io/mdpzc/.
How to Cite
Lu, Y., Tong, L., & Cheng , Y. (2024). Advanced Knowledge Tracing: Incorporating Process Data and Curricula Information via an Attention-Based Framework for Accuracy and Interpretability. Journal of Educational Data Mining, 16(2), 58–84. https://doi.org/10.5281/zenodo.13712553
##plugins.themes.bootstrap3.article.details##
Keywords
interpretable deep learning, deep knowledge tracing (DKT), predictive analytics, educational data mining, intelligent tutoring system (ITS), attention
References
ABDELRAHMAN, G.,WANG, Q., AND NUNES, B. 2023. Knowledge tracing: A survey. ACM Computing Surveys 55, 11.
BAKER, R. S. J. D., CORBETT, A. T., AND ALEVEN, V. 2008. More accurate student modeling through contextual estimation of slip and guess probabilities in bayesian knowledge tracing. In Intelligent Tutoring Systems, B. P.Woolf, E. Aïmeur, R. Nkambou, and S. Lajoie, Eds. Springer Berlin Heidelberg, Berlin, Heidelberg, 406–415.
BALESTRIERO, R., IBRAHIM, M., SOBAL, V., MORCOS, A., SHEKHAR, S., GOLDSTEIN, T., BORDES, F., BARDES, A., MIALON, G., TIAN, Y., SCHWARZSCHILD, A., WILSON, A. G., GEIPING, J., GARRIDO, Q., FERNANDEZ, P., BAR, A., PIRSIAVASH, H., LECUN, Y., AND GOLDBLUM, M. 2023. A Cookbook of Self-Supervised Learning. arXiv:2304.12210 [cs].
CHAWLA, N. V., BOWYER, K. W., HALL, L. O., AND KEGELMEYER, W. P. 2002. SMOTE: Synthetic Minority Over-sampling Technique. Journal of Artificial Intelligence Research 16, 321–357.
CHOI, Y., LEE, Y., CHO, J., BAEK, J., KIM, B., CHA, Y., SHIN, D., BAE, C., AND HEO, J. 2020. Towards an appropriate query, key, and value computation for knowledge tracing. In Proceedings of the Seventh ACM Conference on Learning@Scale. L@S ’20. Association for Computing Machinery, New York, NY, USA, 341–344.
CORBETT, A. T. AND ANDERSON, J. R. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction 4, 253–278.
DISESSA, A. A. 1993. Toward an Epistemology of Physics. Cognition and Instruction 10, 2/3, 105–225. Publisher: Taylor & Francis, Ltd.
GOLDBERGER, J., HINTON, G. E., ROWEIS, S., AND SALAKHUTDINOV, R. R. 2004. Neighbourhood components analysis. In Advances in Neural Information Processing Systems, L. Saul, Y. Weiss, and L. Bottou, Eds. Vol. 17. MIT Press, 513–520.
HEFFERNAN, N. T. AND HEFFERNAN, C. L. 2014. The assistments ecosystem: Building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching. International Journal of Artificial Intelligence in Education 24, 470–497.
HOCHREITER, S. AND SCHMIDHUBER, J. 1997. Long short-term memory. Neural Computation 9, 8, 1735–1780.
KOEDINGER, K. R., CORBETT, A. T., AND PERFETTI, C. 2012. The Knowledge-Learning-Instruction Framework: Bridging the Science-Practice Chasm to Enhance Robust Student Learning. Cognitive Science 36, 5, 757–798.
KÕRÖSI, G. AND FARKAS, R. 2020. MOOC Performance Prediction by Deep Learning from Raw Clickstream Data. In Advances in Computing and Data Sciences, M. Singh, P. K. Gupta, V. Tyagi, J. Flusser, T. Ören, and G. Valentino, Eds. Communications in Computer and Information Science. Springer, Singapore, 474–485.
LI, X., XIONG, H., LI, X., WU, X., ZHANG, X., LIU, J., BIAN, J., AND DOU, D. 2022. Interpretable deep learning: Interpretation, interpretability, trustworthiness, and beyond. Knowledge and Information Systems 64, 12, 3197–3234.
LIU, Q., HUANG, Z., YIN, Y., CHEN, E., XIONG, H., SU, Y., AND HU, G. 2019. Ekt: Exercise-aware knowledge tracing for student performance prediction. IEEE Transactions on Knowledge and Data Engineering 33, 1, 100–115.
LU, Y., OBER, T. M., LIU, C., AND CHENG, Y. 2022. Application of Neighborhood Components Analysis to Process and Survey Data to Predict Student Learning of Statistics. In 2022 IEEE International Conference on Advanced Learning Technologies (ICALT). IEEE, 147–151.
LU, Y., WANG, D., MENG, Q., AND CHEN, P. 2020. Towards interpretable deep learning models for knowledge tracing. In Artificial Intelligence in Education, I. I. Bittencourt, M. Cukurova, K. Muldner, R. Luckin, and E. Millán, Eds. Springer International Publishing, Cham, 185–190.
MEDSKER, L. R. AND JAIN, L. 2001. Recurrent neural networks. Design and Applications 5, 64–67.
MORENO-MARCOS, P. M., PONG, T.-C., MUÑOZ-MERINO, P. J., AND DELGADO KLOOS, C. 2020. Analysis of the Factors Influencing Learners’ Performance Prediction With Learning Analytics. IEEE Access 8, 5264–5282.
NAMOUN, A. AND ALSHANQITI, A. 2021. Predicting Student Performance Using Data Mining and Learning Analytics Techniques: A Systematic Literature Review. Applied Sciences 11, 1, 237.
PANDEY, S. AND KARYPIS, G. 2019. A self-attentive model for knowledge tracing. In Proceedings of the 12th International Conference on Educational Data Mining, C. Lynch, A. Merceron, M. Desmarais, and R. Nkambou, Eds. EDM 2019 - Proceedings of the 12th International Conference on Educational Data Mining. International Educational Data Mining Society, 384–389.
PASZKE, A., GROSS, S., MASSA, F., LERER, A., BRADBURY, J., CHANAN, G., KILLEEN, T., LIN, Z., GIMELSHEIN, N., ANTIGA, L., DESMAISON, A., KOPF, A., YANG, E., DEVITO, Z., RAISON, M., TEJANI, A., CHILAMKURTHY, S., STEINER, B., FANG, L., BAI, J., AND CHINTALA, S. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch´e-Buc, E. Fox, and R. Garnett, Eds. Vol. 32. Curran Associates, Inc., 8026–8037.
PAVLIK, P. I., CEN, H., AND KOEDINGER, K. R. 2009. Performance factors analysis –a new alternative to knowledge tracing. In Proceedings of the 2009 Conference on Artificial Intelligence in Education: Building Learning Systems That Care: From Knowledge Representation to Affective Modelling. IOS Press, NLD, 531–538.
PEDREGOSA, F., VAROQUAUX, G., GRAMFORT, A., MICHEL, V., THIRION, B., GRISEL, O., BLONDEL, M., PRETTENHOFER, P., WEISS, R., DUBOURG, V., VANDERPLAS, J., PASSOS, A., COURNAPEAU, D., BRUCHER, M., PERROT, M., AND DUCHESNAY, E. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, 2825–2830.
PIECH, C., BASSEN, J., HUANG, J., GANGULI, S., SAHAMI, M., GUIBAS, L. J., AND SOHLDICKSTEIN, J. 2015. Deep knowledge tracing. In Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, Eds. Vol. 28. Curran Associates, Inc., 505–513.
SEIN MINN. 2022. AI-assisted knowledge assessment techniques for adaptive learning environments. Computers and Education: Artificial Intelligence 3, 100050.
SHERAZI, S. W. A., BAE, J.-W., AND LEE, J. Y. 2021. A soft voting ensemble classifier for early prediction and diagnosis of occurrences of major adverse cardiovascular events for stemi and nstemi during 2-year follow-up in patients with acute coronary syndrome. PLOS ONE 16, 6 (06), 1–20.
VASWANI, A., SHAZEER, N., PARMAR, N., USZKOREIT, J., JONES, L., GOMEZ, A. N., KAISER, L. U., AND POLOSUKHIN, I. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds. Vol. 30. Curran Associates, Inc., 5998–6008.
WRIGHT, R. E. 1995. Logistic regression. In Reading and understanding multivariate statistics. American Psychological Association, Washington, DC, US, 217–244.
YEH, C., CHEN, Y., WU, A., CHEN, C., VI´E GAS, F., AND WATTENBERG, M. 2023. AttentionViz: A Global View of Transformer Attention. IEEE Transactions on Visualization and Computer Graphics, 1–11.
BAKER, R. S. J. D., CORBETT, A. T., AND ALEVEN, V. 2008. More accurate student modeling through contextual estimation of slip and guess probabilities in bayesian knowledge tracing. In Intelligent Tutoring Systems, B. P.Woolf, E. Aïmeur, R. Nkambou, and S. Lajoie, Eds. Springer Berlin Heidelberg, Berlin, Heidelberg, 406–415.
BALESTRIERO, R., IBRAHIM, M., SOBAL, V., MORCOS, A., SHEKHAR, S., GOLDSTEIN, T., BORDES, F., BARDES, A., MIALON, G., TIAN, Y., SCHWARZSCHILD, A., WILSON, A. G., GEIPING, J., GARRIDO, Q., FERNANDEZ, P., BAR, A., PIRSIAVASH, H., LECUN, Y., AND GOLDBLUM, M. 2023. A Cookbook of Self-Supervised Learning. arXiv:2304.12210 [cs].
CHAWLA, N. V., BOWYER, K. W., HALL, L. O., AND KEGELMEYER, W. P. 2002. SMOTE: Synthetic Minority Over-sampling Technique. Journal of Artificial Intelligence Research 16, 321–357.
CHOI, Y., LEE, Y., CHO, J., BAEK, J., KIM, B., CHA, Y., SHIN, D., BAE, C., AND HEO, J. 2020. Towards an appropriate query, key, and value computation for knowledge tracing. In Proceedings of the Seventh ACM Conference on Learning@Scale. L@S ’20. Association for Computing Machinery, New York, NY, USA, 341–344.
CORBETT, A. T. AND ANDERSON, J. R. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction 4, 253–278.
DISESSA, A. A. 1993. Toward an Epistemology of Physics. Cognition and Instruction 10, 2/3, 105–225. Publisher: Taylor & Francis, Ltd.
GOLDBERGER, J., HINTON, G. E., ROWEIS, S., AND SALAKHUTDINOV, R. R. 2004. Neighbourhood components analysis. In Advances in Neural Information Processing Systems, L. Saul, Y. Weiss, and L. Bottou, Eds. Vol. 17. MIT Press, 513–520.
HEFFERNAN, N. T. AND HEFFERNAN, C. L. 2014. The assistments ecosystem: Building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching. International Journal of Artificial Intelligence in Education 24, 470–497.
HOCHREITER, S. AND SCHMIDHUBER, J. 1997. Long short-term memory. Neural Computation 9, 8, 1735–1780.
KOEDINGER, K. R., CORBETT, A. T., AND PERFETTI, C. 2012. The Knowledge-Learning-Instruction Framework: Bridging the Science-Practice Chasm to Enhance Robust Student Learning. Cognitive Science 36, 5, 757–798.
KÕRÖSI, G. AND FARKAS, R. 2020. MOOC Performance Prediction by Deep Learning from Raw Clickstream Data. In Advances in Computing and Data Sciences, M. Singh, P. K. Gupta, V. Tyagi, J. Flusser, T. Ören, and G. Valentino, Eds. Communications in Computer and Information Science. Springer, Singapore, 474–485.
LI, X., XIONG, H., LI, X., WU, X., ZHANG, X., LIU, J., BIAN, J., AND DOU, D. 2022. Interpretable deep learning: Interpretation, interpretability, trustworthiness, and beyond. Knowledge and Information Systems 64, 12, 3197–3234.
LIU, Q., HUANG, Z., YIN, Y., CHEN, E., XIONG, H., SU, Y., AND HU, G. 2019. Ekt: Exercise-aware knowledge tracing for student performance prediction. IEEE Transactions on Knowledge and Data Engineering 33, 1, 100–115.
LU, Y., OBER, T. M., LIU, C., AND CHENG, Y. 2022. Application of Neighborhood Components Analysis to Process and Survey Data to Predict Student Learning of Statistics. In 2022 IEEE International Conference on Advanced Learning Technologies (ICALT). IEEE, 147–151.
LU, Y., WANG, D., MENG, Q., AND CHEN, P. 2020. Towards interpretable deep learning models for knowledge tracing. In Artificial Intelligence in Education, I. I. Bittencourt, M. Cukurova, K. Muldner, R. Luckin, and E. Millán, Eds. Springer International Publishing, Cham, 185–190.
MEDSKER, L. R. AND JAIN, L. 2001. Recurrent neural networks. Design and Applications 5, 64–67.
MORENO-MARCOS, P. M., PONG, T.-C., MUÑOZ-MERINO, P. J., AND DELGADO KLOOS, C. 2020. Analysis of the Factors Influencing Learners’ Performance Prediction With Learning Analytics. IEEE Access 8, 5264–5282.
NAMOUN, A. AND ALSHANQITI, A. 2021. Predicting Student Performance Using Data Mining and Learning Analytics Techniques: A Systematic Literature Review. Applied Sciences 11, 1, 237.
PANDEY, S. AND KARYPIS, G. 2019. A self-attentive model for knowledge tracing. In Proceedings of the 12th International Conference on Educational Data Mining, C. Lynch, A. Merceron, M. Desmarais, and R. Nkambou, Eds. EDM 2019 - Proceedings of the 12th International Conference on Educational Data Mining. International Educational Data Mining Society, 384–389.
PASZKE, A., GROSS, S., MASSA, F., LERER, A., BRADBURY, J., CHANAN, G., KILLEEN, T., LIN, Z., GIMELSHEIN, N., ANTIGA, L., DESMAISON, A., KOPF, A., YANG, E., DEVITO, Z., RAISON, M., TEJANI, A., CHILAMKURTHY, S., STEINER, B., FANG, L., BAI, J., AND CHINTALA, S. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch´e-Buc, E. Fox, and R. Garnett, Eds. Vol. 32. Curran Associates, Inc., 8026–8037.
PAVLIK, P. I., CEN, H., AND KOEDINGER, K. R. 2009. Performance factors analysis –a new alternative to knowledge tracing. In Proceedings of the 2009 Conference on Artificial Intelligence in Education: Building Learning Systems That Care: From Knowledge Representation to Affective Modelling. IOS Press, NLD, 531–538.
PEDREGOSA, F., VAROQUAUX, G., GRAMFORT, A., MICHEL, V., THIRION, B., GRISEL, O., BLONDEL, M., PRETTENHOFER, P., WEISS, R., DUBOURG, V., VANDERPLAS, J., PASSOS, A., COURNAPEAU, D., BRUCHER, M., PERROT, M., AND DUCHESNAY, E. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, 2825–2830.
PIECH, C., BASSEN, J., HUANG, J., GANGULI, S., SAHAMI, M., GUIBAS, L. J., AND SOHLDICKSTEIN, J. 2015. Deep knowledge tracing. In Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, Eds. Vol. 28. Curran Associates, Inc., 505–513.
SEIN MINN. 2022. AI-assisted knowledge assessment techniques for adaptive learning environments. Computers and Education: Artificial Intelligence 3, 100050.
SHERAZI, S. W. A., BAE, J.-W., AND LEE, J. Y. 2021. A soft voting ensemble classifier for early prediction and diagnosis of occurrences of major adverse cardiovascular events for stemi and nstemi during 2-year follow-up in patients with acute coronary syndrome. PLOS ONE 16, 6 (06), 1–20.
VASWANI, A., SHAZEER, N., PARMAR, N., USZKOREIT, J., JONES, L., GOMEZ, A. N., KAISER, L. U., AND POLOSUKHIN, I. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds. Vol. 30. Curran Associates, Inc., 5998–6008.
WRIGHT, R. E. 1995. Logistic regression. In Reading and understanding multivariate statistics. American Psychological Association, Washington, DC, US, 217–244.
YEH, C., CHEN, Y., WU, A., CHEN, C., VI´E GAS, F., AND WATTENBERG, M. 2023. AttentionViz: A Global View of Transformer Attention. IEEE Transactions on Visualization and Computer Graphics, 1–11.
Issue
Section
Special Section EDM Cup 2023
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors who publish with this journal agree to the following terms:
- The Author retains copyright in the Work, where the term “Work” shall include all digital objects that may result in subsequent electronic publication or distribution.
- Upon acceptance of the Work, the author shall grant to the Publisher the right of first publication of the Work.
- The Author shall grant to the Publisher and its agents the nonexclusive perpetual right and license to publish, archive, and make accessible the Work in whole or in part in all forms of media now or hereafter known under a Creative Commons 4.0 License (Attribution-Noncommercial-No Derivatives 4.0 International), or its equivalent, which, for the avoidance of doubt, allows others to copy, distribute, and transmit the Work under the following conditions:
- Attribution—other users must attribute the Work in the manner specified by the author as indicated on the journal Web site;
- Noncommercial—other users (including Publisher) may not use this Work for commercial purposes;
- No Derivative Works—other users (including Publisher) may not alter, transform, or build upon this Work,with the understanding that any of the above conditions can be waived with permission from the Author and that where the Work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.
- The Author is able to enter into separate, additional contractual arrangements for the nonexclusive distribution of the journal's published version of the Work (e.g., post it to an institutional repository or publish it in a book), as long as there is provided in the document an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post online a pre-publication manuscript (but not the Publisher’s final formatted PDF version of the Work) in institutional repositories or on their Websites prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (see The Effect of Open Access). Any such posting made before acceptance and publication of the Work shall be updated upon publication to include a reference to the Publisher-assigned DOI (Digital Object Identifier) and a link to the online abstract for the final published Work in the Journal.
- Upon Publisher’s request, the Author agrees to furnish promptly to Publisher, at the Author’s own expense, written evidence of the permissions, licenses, and consents for use of third-party material included within the Work, except as determined by Publisher to be covered by the principles of Fair Use.
- The Author represents and warrants that:
- the Work is the Author’s original work;
- the Author has not transferred, and will not transfer, exclusive rights in the Work to any third party;
- the Work is not pending review or under consideration by another publisher;
- the Work has not previously been published;
- the Work contains no misrepresentation or infringement of the Work or property of other authors or third parties; and
- the Work contains no libel, invasion of privacy, or other unlawful matter.
- The Author agrees to indemnify and hold Publisher harmless from Author’s breach of the representations and warranties contained in Paragraph 6 above, as well as any claim or proceeding relating to Publisher’s use and publication of any content contained in the Work, including third-party content.