Automated Search Improves Logistic Knowledge Tracing, Surpassing Deep Learning in Accuracy and Explainability

##plugins.themes.bootstrap3.article.main##

##plugins.themes.bootstrap3.article.sidebar##

Published Dec 26, 2023
Philip Irvin Pavlik Jr. Luke G. Eglington

Abstract

This paper describes how to discover simple logistic regression models that outperform more complex approaches such as Deep Knowledge Tracing (DKT) and Self-Attentive Knowledge Tracing (SAKT). Creating student models is done either by expert selection of the appropriate terms, beginning with models as simple as Item Response Theory (IRT) or Additive Factors Model (AFM) or with more “black box” approaches like DKT, in which the model discovers student features. We demonstrate how feature search approaches (i.e., stepwise selection or Least Absolute Shrinkage and Selection Operator (LASSO)) can discover superior models that are explainable. Such automatic methods of model creation offer the possibility of better student models with reduced complexity and better fit, in addition to relieving experts from the burden of searching for better models by hand with possible human error. Our new functions are part of the preexisting R package Logistic Knowledge Tracing (LKT). We demonstrate our search methods with three datasets in which research-supported features (e.g., counts of success, elapsed time, recent performance) are computed at multiple levels (student, knowledge component (KC), item) and input to stepwise regression and LASSO methods to discover the best-fitting regression models. The approach was intended to balance accuracy and explainability. However, somewhat surprisingly, both stepwise regression and LASSO found regression models that were both simpler and more accurate than DKT, SAKT, and Interpretable Knowledge Tracing (IKT) in all datasets, typically requiring multiple orders of magnitude fewer parameters than alternatives.

How to Cite

Pavlik Jr., P. I., & Eglington, L. G. (2023). Automated Search Improves Logistic Knowledge Tracing, Surpassing Deep Learning in Accuracy and Explainability. Journal of Educational Data Mining, 15(3), 58–86. https://doi.org/10.5281/zenodo.10363337
Abstract 137 | HTML Downloads 124 PDF Downloads 145

##plugins.themes.bootstrap3.article.details##

Keywords

explainable AI, transparency in AI, logistic regression, student modeling, knowledge tracing, deep knowledge tracing

References
CEN, H., KOEDINGER, K. AND JUNKER, B. 2006. Learning Factors Analysis: A General Method for Cognitive Model Evaluation and Improvement. In Proceedings of the International Conference on Intelligent Tutoring Systems, Jhongli, Taiwan, M. Ikeda, K. D. Ashley and T.-W. Chan Eds. Springer, 164-176.

CEN, H., KOEDINGER, K. R. AND JUNKER, B. 2008. Comparing two IRT models for conjunctive skills. In Proceedings of the Proceedings of the 9th International Conference on Intelligent Tutoring Systems, Montreal, Canada, B. Woolf, E. Aimer and R. Nkambou Eds. ACM, Inc., 796-798.

CHI, M., KOEDINGER, K. R., GORDON, G., JORDAN, P. AND VANLEHN, K. 2011. Instructional factors analysis: A cognitive model for multiple instructional interventions. In Proceedings of the 4th International Conference on Educational Data Mining (EDM 2011), M. Pechenizkiy, T. Calders, C. Conati, S. Ventura, C. Romero and J. Stamper Eds. International Educational Data Mining Society, Eindhoven, The Netherlands, 61- 70.

CONATI, C., PORAYSKA-POMSTA, K. AND MAVRIKIS, M. 2018. AI in Education needs interpretable machine learning: Lessons from Open Learner Modelling. arXiv preprint arXiv:1807.00154.

EGLINGTON, L. G. AND PAVLIK JR, P. I. 2020. Optimizing practice scheduling requires quantitative tracking of individual item performance. npj Science of Learning 5, 15.

FENG, M., HEFFERNAN, N. AND KOEDINGER, K. 2009. Addressing the assessment challenge with an online system that tutors as it assesses. User Modeling and User-Adapted Interaction 19, 243-266.

FISCHER, G. H. 1973. The linear logistic test model as an instrument in educational research. Acta Psychologica 37, 359-374.

FRIEDMAN, J., HASTIE, T. AND TIBSHIRANI, R. 2010. Regularization Paths for Generalized Linear Models via Coordinate Descent. J Stat Softw 33, 1-22.

FRIEDMAN, J. H., HASTIE, T. AND TIBSHIRANI, R. 2010. Regularization Paths for Generalized Linear Models via Coordinate Descent. Journal of Statistical Software 33, 1 - 22.

GALYARDT, A. AND GOLDIN, I. 2015. Move your lamp post: Recent data reflects learner knowledge better than older data. Journal of Educational Data Mining 7, 83-108.

GERVET, T., KOEDINGER, K., SCHNEIDER, J. AND MITCHELL, T. 2020. When is Deep Learning the Best Approach to Knowledge Tracing? Journal of Educational Data Mining 12, 31- 54.

GONG, Y., BECK, J. E. AND HEFFERNAN, N. T. 2011. How to construct more accurate student models: Comparing and optimizing knowledge tracing and performance factor analysis. International Journal of Artificial Intelligence in Education 21, 27-46.

HOCHREITER, S. AND SCHMIDHUBER, J. 1997. Long Short-Term Memory. Neural Computation 9, 1735-1780.

JUDD, W. A. AND GLASER, R. 1969. Response Latency as a Function of Training Method, Information Level, Acquisition, and Overlearning. Journal of Educational Psychology 60, 1-30.

KHOSRAVI, H., SHUM, S. B., CHEN, G., CONATI, C., TSAI, Y.-S., KAY, J., KNIGHT, S., MARTINEZMALDONADO, R., SADIQ, S. AND GAŠEVIĆ, D. 2022. Explainable Artificial Intelligence in education. Computers and Education: Artificial Intelligence 3, 100074.

MACLEOD, C. M. AND NELSON, T. O. 1984. Response latency and response accuracy as measures of memory. Acta Psychologica 57, 215-235.

MINN, S., VIE, J.-J., TAKEUCHI, K., KASHIMA, H. AND ZHU, F. 2022. Interpretable Knowledge Tracing: Simple and Efficient Student Modeling with Causal Relations. In Proceedings of the AAAI Conference on Artificial Intelligence AAAI Press, 12810-12818.

PANDEY, S. AND KARYPIS, G. 2019. A Self-Attentive model for Knowledge Tracing. In Proceedings of the 12th International Conference on Educational Data Mining, C. Lynch, A. Merceron, M. Desmarais and R. Nkambou Eds. International Educational Data Mining Society, 384-389.

PAPOUSEK, J., PELÁNEK, R. AND STANISLAV, V. 2014. Adaptive practice of facts in domains with varied prior knowledge. In Proceedings of the Proceedings of the 7th International Conference on Educational Data Mining (EDM 2014), J. Stamper, Z. Pardos, M. Mavrikis and B. M. Mclaren Eds. International Educational Data Mining Society, 6-13.

PAVLIK JR, P. I., EGLINGTON, L. AND ZHANG, L. 2021. Automatic Domain Model Creation and Improvement. In Proceedings of The 14th International Conference on Educational Data Mining, C. Lynch, A. Merceron, M. Desmarais and R. Nkambou Eds. International Educational Data Mining Society, 672-676.

PAVLIK JR, P. I. AND EGLINGTON, L. G. 2021. LKT: Logistic Knowledge Tracing. In CRAN R package version 1.0.

PAVLIK JR., P. I., BRAWNER, K. W., OLNEY, A. AND MITROVIC, A. 2013. A Review of Learner Models Used in Intelligent Tutoring Systems In Design Recommendations for Adaptive Intelligent Tutoring Systems: Learner Modeling, R. A. Sottilare, A. Graesser, X. Hu and H. K. Holden Eds. Army Research Labs/ University of Memphis, 39-68.

PAVLIK JR., P. I., CEN, H. AND KOEDINGER, K. R. 2009. Performance factors analysis -- A new alternative to knowledge tracing. In Proceedings of the 14th International Conference on Artificial Intelligence in Education, V. Dimitrova, R. Mizoguchi, B. D. Boulay and A. Graesser Eds. IOS Press, Brighton, England, 531–538.

PAVLIK, P. I., EGLINGTON, L. G. AND HARRELL-WILLIAMS, L. M. 2021. Logistic Knowledge Tracing: A Constrained Framework for Learner Modeling. IEEE Transactions on Learning Technologies 14, 624-639.

PELÁNEK, R. 2016. Applications of the Elo rating system in adaptive educational systems. Computers & Education 98, 169-179.

PIECH, C., BASSEN, J., HUANG, J., GANGULI, S., SAHAMI, M., GUIBAS, L. J. AND SOHL-DICKSTEIN, J. 2015. Deep Knowledge Tracing. In Proceedings of the Advances in Neural Information Processing Systems 28 (NIPS 2015), C. Cortes, N. Lawrence, D. Lee, M. Sugiyama and R. Garnett Eds. MIT Press, 1-9.

ROEDIGER, H. L. AND BUTLER, A. C. 2011. The critical role of retrieval practice in long-term retention. Trends in Cognitive Sciences 15, 20-27.

SCHMUCKER, R., WANG, J., HU, S. AND MITCHELL, T. 2022. Assessing the Performance of Online Students - New Data, New Approaches, Improved Accuracy. Journal of Educational Data Mining 14, 1-45.

SMITH, G. 2018. Step away from stepwise. Journal of Big Data 5, 32.

TAY, J. K., NARASIMHAN, B. AND HASTIE, T. 2023. Elastic Net Regularization Paths for All Generalized Linear Models. Journal of Statistical Software 106, 1 - 31.

VIE, J.-J. AND KASHIMA, H. 2019. Knowledge Tracing Machines: Factorization Machines for Knowledge Tracing. In Proceedings of the AAAI Conference on Artificial Intelligence, B. Williams, Y. Chen and J. Neville Eds. AAAI Press, 750-757.

WIXTED, J. T. AND EBBESEN, E. B. 1997. Genuine power curves in forgetting: A quantitative analysis of individual subject forgetting functions. Memory & Cognition 25, 731-739.

XIONG, X., ZHAO, S., VAN INWEGEN, E. G. AND BECK, J. E. 2016. Going Deeper with Deep Knowledge Tracing. In Proceedings of the 9th International Conference on Educational Data Mining, T. Barnes, M. Chi and M. Feng Eds. International Educational Data Mining Society, Raleigh, NC, 545-550.

YUAN, M. AND LIN, Y. 2006. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 68, 49-67.

YUDELSON, M., PAVLIK JR., P. I. AND KOEDINGER, K. R. 2011. User Modeling – A Notoriously Black Art. In User Modeling, Adaption and Personalization, J. Konstan, R. Conejo, J. Marzo and N. Oliver Eds. Springer Berlin / Heidelberg, 317-328.

ZHANG, L., XIONG, X., ZHAO, S., BOTELHO, A. AND HEFFERNAN, N. T. 2017. Incorporating Rich Features into Deep Knowledge Tracing. In Proceedings of the Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale, Cambridge, Massachusetts, USA Association for Computing Machinery, 169–172.
Section
Extended Articles from the EDM 2023 Conference