Assessing the Performance of Online Students - New Data, New Approaches, Improved Accuracy
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
Jun 24, 2022
Robin Schmucker
Jingbo Wang
Shijia Hu
Tom M. Mitchell
https://orcid.org/0000-0002-6928-8482
Abstract
We consider the problem of assessing the changing performance levels of individual students as they go
through online courses. This student performance modeling problem is a critical step for building adaptive
online teaching systems. Specifically, we conduct a study of how to utilize various types and large amounts
of log data from earlier students to train accurate machine learning models that predict the performance of
future students. This study is the first to use four very large sets of student data made available recently
from four distinct intelligent tutoring systems.
Our results include a new machine learning approach that defines a new state of the art for logistic
regression based student performance modeling, improving over earlier methods in several ways: First, we
achieve improved accuracy of student modeling by introducing new features that can be easily computed
from conventional question-response logs (e.g., features such as the pattern in the student’s most recent
answers). Second, we take advantage of features of the student history that go beyond question-response
pairs (e.g., features such as which video segments the student watched, or skipped) as well as background
information about prerequisite structure in the curriculum. Third, we train multiple specialized student
performance models for different aspects of the curriculum (e.g., specializing in early versus later segments
of the student history), then combine these specialized models to create a group prediction of the student
performance. Taken together, these innovations yield an average AUC score across these four datasets of
0.808 compared to the previous best logistic regression approach score of 0.767, and also outperforming
state-of-the-art deep neural net approaches. Importantly, we observe consistent improvements from each of
our three methodological innovations, in each diverse dataset, suggesting that our methods are of general
utility and likely to produce improvements for other online tutoring systems as well.
How to Cite
Schmucker, R., Wang, J., Hu, S., & Mitchell, T. (2022). Assessing the Performance of Online Students - New Data, New Approaches, Improved Accuracy. Journal of Educational Data Mining, 14(1), 1–45. https://doi.org/10.5281/zenodo.6450190
##plugins.themes.bootstrap3.article.details##
Keywords
performance modeling, knowledge tracing, logistic regression, deep learning, features
References
AIKENS, N. L. AND BARBARIN, O. 2008. Socioeconomic differences in reading trajectories: The contribution of family, neighborhood, and school contexts. Journal of Educational Psychology 100, 2, 235.
BADRINATH, A., WANG, F., AND PARDOS, Z. 2021. pyBKT: An accessible python library of bayesian knowledge tracing models. In Proceedings of the 14th International Conference on Educational Data Mining. EDM, Paris, France, 468–474.
BAKER, R., CORBETT, A. T., AND ALEVEN, V. 2008. More accurate student modeling through contextual estimation of slip and guess probabilities in bayesian knowledge tracing. In Proceedings of the 9th International Conference on Intelligent Tutoring Systems. Springer, Berlin, Germany, 406–415.
BREIMAN, L. 2001. Random forests. Machine learning 45, 1, 5–32.
CEN, H., KOEDINGER, K., AND JUNKER, B. 2006. Learning factors analysis–a general method for cognitive model evaluation and improvement. In Proceedings of the 8th International Conference on Intelligent Tutoring Systems. Springer, Berlin, Germany, 164–175.
CEPEDA, N. J., VUL, E., ROHRER, D., WIXTED, J. T., AND PASHLER, H. 2008. Spacing effects in learning: A temporal ridgeline of optimal retention. Psychological science 19, 11, 1095–1102.
CHANG, H.-S., HSU, H.-J., AND CHEN, K.-T. 2015. Modeling exercise relationships in e-learning: A unified approach. In Proceedings of the 12th International Conference on Educational Data Mining. EDM, Madrid, Spain, 532–535.
CHOFFIN, B., POPINEAU, F., BOURDA, Y., AND VIE, J.-J. 2019. DAS3H: Modeling student learning and forgetting for optimally scheduling distributed practice of skills. In Proceedings of the 12th International Conference on Educational Data Mining. EDM, Montr´eal, QC, Canada, 29–38.
CHOI, Y., LEE, Y., CHO, J., BAEK, J., KIM, B., CHA, Y., SHIN, D., BAE, C., AND HEO, J. 2020. Towards an appropriate query, key, and value computation for knowledge tracing. In Proceedings of 7th ACM Conference on Learning @ Scale. L@S ’20. ACM, New York, NY, USA, 341–344.
CHOI, Y., LEE, Y., SHIN, D., CHO, J., PARK, S., LEE, S., BAEK, J., BAE, C., KIM, B., AND HEO, J. 2020. EdNet: A large-scale hierarchical dataset in education. In Proceedings of the 21st International Conference on Artificial Intelligence in Education. Springer International Publishing, Cham, Switzerland, 69–73.
CORBETT, A. T. AND ANDERSON, J. R. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction 4, 4, 253–278.
DING, X. AND LARSON, E. C. 2019. Why deep knowledge tracing has less depth than anticipated. In Proceedings of the 12th International Conference on Educational Data Mining. EDM, Montr´eal, QC, Canada, 282–287.
DING, X. AND LARSON, E. C. 2021. On the interpretability of deep learning based models for knowledge tracing. CoRR abs/2101.11335.
EGLINGTON, L. G. AND PAVLIK, JR, P. I. 2019. Predictiveness of prior failures is improved by incorporating trial duration. Journal of Educational Data Mining 11, 2 (9), 1–19.
FENG, M., HEFFERNAN, N., AND KOEDINGER, K. 2009. Addressing the assessment challenge with an online system that tutors as it assesses. User Modeling and User-Adapted Interaction 19, 3, 243–266.
GALYARDT, A. AND GOLDIN, I. 2015. Move your lamp post: Recent data reflects learner knowledge better than older data. Journal of Educational Data Mining 7, 2, 83–108.
GERVET, T., KOEDINGER, K., SCHNEIDER, J., AND MITCHELL, T. 2020. When is deep learning the best approach to knowledge tracing? Journal of Educational Data Mining 12, 3, 31–54.
GHOSH, A., HEFFERNAN, N., AND LAN, A. S. 2020. Context-aware attentive knowledge tracing. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. KDD ’20. ACM, New York, NY, USA, 2330–2339.
GONZÁLEZ-BRENES, J., HUANG, Y., AND BRUSILOVSKY, P. 2014. General features in knowledge tracing: Applications to multiple subskills, temporal item response theory, and expert knowledge. In Proceedings of 7th International Conference on Educational Data Mining. EDM, London, UK, 84–91.
HOCHREITER, S. AND SCHMIDHUBER, J. 1997. Long short-term memory. Neural computation 9, 8, 1735–1780.
HUANG, X., CRAIG, S. D., XIE, J., GRAESSER, A., AND HU, X. 2016. Intelligent tutoring systems work as a math gap reducer in 6th grade after-school program. Learning and Individual Differences 47, 258–265.
KHAJAH, M., LINDSEY, R. V., AND MOZER, M. C. 2016. How deep is knowledge tracing? In Proceedings of the 9th International Conference on Educational Data Mining. EDM, Raleigh, NC, USA, 94–101.
KOEDINGER, K., SCHEINES, R., AND SCHALDENBRAND, P. 2018. Is the doer effect robust across multiple data sets? In Proceedings of the 11th International Conference on Educational Data Mining. EDM, Buffalo, NY, USA, 369–375.
KOEDINGER, K. R., KIM, J., JIA, J. Z., MCLAUGHLIN, E. A., AND BIER, N. L. 2015. Learning is not a spectator sport: Doing is better than watching for learning from a mooc. In Proceedings of the 2nd ACM Conference on Learning @ Scale. L@S ’15. ACM, New York, NY, USA, 111–120.
KOEDINGER, K. R., MCLAUGHLIN, E. A., JIA, J. Z., AND BIER, N. L. 2016. Is the doer effect a causal relationship? how can we tell and why it’s important. In Proceedings of the 6th International Conference on Learning Analytics & Knowledge. LAK ’16. ACM, New York, NY, USA, 388–397.
KÄSER, T., KLINGLER, S., SCHWING, A. G., AND GROSS, M. 2017. Dynamic bayesian networks for student modeling. IEEE Transactions on Learning Technologies 10, 4, 450–462.
LECUN, Y., HAFFNER, P., BOTTOU, L., AND BENGIO, Y. 1999. Object Recognition with Gradient-Based Learning. Springer, Berlin, Germany, 319–345.
LINDSEY, R. V., SHROYER, J. D., PASHLER, H., AND MOZER, M. C. 2014. Improving students’ long-term knowledge retention through personalized review. Psychological Science 25, 3, 639–647.
LIU, Q., HUANG, Z., YIN, Y., CHEN, E., XIONG, H., SU, Y., AND HU, G. 2019. Ekt: Exercise-aware knowledge tracing for student performance prediction. IEEE Transactions on Knowledge and Data Engineering 33, 1, 100–115.
LIU, Q., SHEN, S., HUANG, Z., CHEN, E., AND ZHENG, Y. 2021. A survey of knowledge tracing. CoRR abs/2105.15106.
MANNING, C. D. AND SCH¨UTZE, H. 1999. Foundations of Statistical Natural Language Processing. MIT Press, Cambridge, MA, USA.
MONTERO, S., ARORA, A., KELLY, S., MILNE, B., AND MOZER, M. 2018. Does deep knowledge tracing model interactions among skills? In Proceedings of the 11th International Conference on Educational Data Mining. EDM, Buffalo, NY, USA, 462–466.
MOZER, M. C. AND LINDSEY, R. V. 2016. Predicting and Improving Memory Retention: Psychological Theory Matters in the Big Data Era. Routledge/Taylor & Francis Group, London, UK.
NAKAGAWA, H., IWASAWA, Y., AND MATSUO, Y. 2019. Graph-based knowledge tracing: Modeling student proficiency using graph neural network. In IEEE/WIC/ACM International Conference on Web Intelligence. IEEE, New York, NY, USA, 156–163.
PANDEY, S. AND KARYPIS, G. 2019. A self-attentive model for knowledge tracing. In Proceedings of the 12th International Conference on Educational Data Mining. EDM, Montr´eal, QC, Canada, 384–389.
PARDOS, Z. A. AND HEFFERNAN, N. T. 2010. Modeling individualization in a bayesian networks implementation of knowledge tracing. In Proceedings of the 18th International Conference on User Modeling, Adaptation, and Personalization. Springer, Berlin, Germany, 255–266.
PARDOS, Z. A. AND HEFFERNAN, N. T. 2011. KT-IDEM: Introducing item difficulty to the knowledge tracing model. In Proceedings of the 19th International Conference on User Modeling, Adaptation, and Personalization. Springer, Berlin, Germany, 243–254.
PASZKE, A., GROSS, S., MASSA, F., LERER, A., BRADBURY, J., CHANAN, G., KILLEEN, T., LIN, Z., GIMELSHEIN, N., ANTIGA, L., ET AL. 2019. PyTorch: An imperative style, high-performance deep learning library. In Proceedings of the 32th International Conference on Advances in Neural Information Processing Systems. Vol. 32. Curran Associates, Inc., Vancouver, BC, Canada, 8026–8037.
PAVLIK, P. I., EGLINGTON, L. G., AND HARRELL-WILLIAMS, L. M. 2021. Logistic knowledge tracing: A constrained framework for learner modeling. IEEE Transactions on Learning Technologies 14, 5, 624–639.
PAVLIK JR, P., CEN, H., AND KOEDINGER, K. 2009. Performance factors analysis–a new alternative to knowledge tracing. In Frontiers in Artificial Intelligence and Applications. Vol. 200. IOS Press, Amsterdam, Netherlands, 531–538.
PEDREGOSA, F., VAROQUAUX, G., GRAMFORT, A., MICHEL, V., THIRION, B., GRISEL, O., BLONDEL, M., PRETTENHOFER, P., WEISS, R., DUBOURG, V., ET AL. 2011. Scikit-learn: Machine learning in python. The Journal of Machine Learning Research 12, 2825–2830.
PIECH, C., BASSEN, J., HUANG, J., GANGULI, S., SAHAMI, M., GUIBAS, L. J., AND SOHL-DICKSTEIN, J. 2015. Deep knowledge tracing. In Proceedings of the 28th Conference on Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, Eds. Vol. 28th. Curran Associates, Inc., Montréal, QC, Canada.
POJEN, C., MINGEN, H., AND TZUYANG, T. 2020. Junyi academy online learning activity dataset: A large-scale public online learning activity dataset from elementary to senior high school students. [Available online at: https://www.kaggle.com/datasets/junyiacademy/learning-activity-public-datasetby- junyi-academy; accessed 28-March-2022].
QIU, Y., QI, Y., LU, H., PARDOS, Z. A., AND HEFFERNAN, N. T. 2011. Does time matter? modeling the effect of time with bayesian knowledge tracing. In Proceedings of the 4th International Conference on Educational Data Mining. Eindhoven University of Technology, Eindhoven, Netherlands, 139–148.
RASCH, G. 1960. Probabilistic Models for Some Intelligence and Attainment Tests. Studies in mathematical psychology. Danmarks Paedagogiske Institut, Aarhus, Denmark.
SAO PEDRO, M., BAKER, R., AND GOBERT, J. 2013. Incorporating scaffolding and tutor context into bayesian knowledge tracing to predict inquiry skill acquisition. In Proceedings of the 6th International Conference on Educational Data Mining. EDM, Memphis, TN, USA.
SCRUGGS, R., BAKER, R., AND MCLAREN, B. 2020. Extending deep knowledge tracing: Inferring interpretable knowledge and predicting post system performance. In Proceedings of the 28th International Conference on Computers in Education. APSCE, Jhongli City, Taiwan, 195–204.
SHEN, S., LIU, Q., CHEN, E., WU, H., HUANG, Z., ZHAO, W., SU, Y., MA, H., AND WANG, S. 2020. Convolutional knowledge tracing: Modeling individualization in student learning process. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, USA, 1857–1860.
SHIN, D., SHIM, Y., YU, H., LEE, S., KIM, B., AND CHOI, Y. 2021. Saint+: Integrating temporal features for EdNet correctness prediction. In Proceedings of the 11th International Learning Analytics and Knowledge Conference. ACM, New York, NY, USA, 490–496.
STAMPER, J., NICULESCU-MIZIL, A., RITTER, S., GORDON, G., AND KOEDINGER, K. 2010. Bridge to Algebra 2006-2007. development data set from KDD Cup 2010 educational data mining challenge. [Available online at: https://pslcdatashop.web.cmu.edu/KDDCup/downloads.jsp; accessed 28-March- 2022].
TONG, H., WANG, Z., LIU, Q., ZHOU, Y., AND HAN, W. 2020. HGKT : Introducing problem schema with hierarchical exercise graph for knowledge tracing. CoRR abs/2006.16915.
TSUTSUMI, E., KINOSHITA, R., AND UENO, M. 2021. Deep-IRT with independent student and item networks. In Proceedings of the 14th International Conference on Educational Data Mining. EDM, Paris, France, 510–517.
VAN CAMPENHOUT, R., JOHNSON, B., AND OLSEN, J. 2021. The doer effect: Replicating findings that doing causes learning. In Proceedings of the 13th International Conference on Mobile, Hybrid, and On-line Learning. IARIA, France, 1–6.
VAN DER LINDEN, W. J. AND HAMBLETON, R. K. 2013. Handbook of Modern Item Response Theory. Springer, New York, NY, USA.
VASWANI, A., SHAZEER, N., PARMAR, N., USZKOREIT, J., JONES, L., GOMEZ, A. N., KAISER, L. U., AND POLOSUKHIN, I. 2017. Attention is all you need. In Proceedings of the 31th Conference on Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds. Vol. 30. Curran Associates, Inc., Long Beach, CA, USA.
VON STUMM, S. 2017. Socioeconomic status amplifies the achievement gap throughout compulsory education independent of intelligence. Intelligence 60, 57–62.
WALSH, M. M., GLUCK, K. A., GUNZELMANN, G., JASTRZEMBSKI, T., KRUSMARK, M., MYUNG, J. I., PITT, M. A., AND ZHOU, R. 2018. Mechanisms underlying the spacing effect in learning: A comparison of three computational models. Journal of Experimental Psychology 147, 9, 1325.
WANG, Z., LAMB, A., SAVELIEV, E., CAMERON, P., ZAYKOV, J., HERNANDEZ-LOBATO, J. M., TURNER, R. E., BARANIUK, R. G., BARTON, C., JONES, S. P., ET AL. 2021. Results and insights from diagnostic questions: The neurips 2020 education challenge. In NeurIPS 2020 Competition and Demonstration Track. PMLR, Virtual, 191–205.
WANG, Z., LAMB, A., SAVELIEV, E., CAMERON, P., ZAYKOV, Y., HERN´A NDEZ-LOBATO, J. M., TURNER, R. E., BARANIUK, R. G., BARTON, C., AND JONES. 2020. Diagnostic questions: The NeurIPS 2020 education challenge. CoRR abs/2007.12061.
WHITE, K. R. 1982. The relation between socioeconomic status and academic achievement. Psychological Bulletin 91, 3, 461.
WILSON, K. H., XIONG, X., KHAJAH, M., LINDSEY, R. V., ZHAO, S., KARKLIN, Y., VAN INWEGEN, E. G., HAN, B., EKANADHAM, C., BECK, J. E., ET AL. 2016. Estimating student proficiency: Deep learning is not the panacea. In Neural Information Processing Systems, Workshop on Machine Learning for Education. NIPS, Barcelona Spain, 3–7.
WOLPERT, D. H. 1992. Stacked generalization. Neural Networks 5, 2, 241–259.
YANG, H. AND CHEUNG, L. P. 2018. Implicit heterogeneous features embedding in deep knowledge tracing. Cognitive Computation 10, 1, 3–14.
YANG, Y., SHEN, J., QU, Y., LIU, Y., WANG, K., ZHU, Y., ZHANG, W., AND YU, Y. 2021. Gikt: A graph-based interaction model for knowledge tracing. In Machine Learning and Knowledge Discovery in Databases. Springer International Publishing, Cham, Switzerland, 299–315.
YEUNG, C.-K. 2019. Deep-IRT: Make deep learning based knowledge tracing explainable using item response theory. CoRR abs/1904.11738.
YUDELSON, M. V., KOEDINGER, K. R., AND GORDON, G. J. 2013. Individualized bayesian knowledge tracing models. In Proceedings of the 16th International Conference on Artificial Intelligence in Education. Springer, Berlin, Germany, 171–180.
ZHANG, C., JIANG, Y., ZHANG, W., AND GU, C. 2021. Muse: Multi-scale temporal features evolution for knowledge tracing. CoRR abs/2102.00228.
ZHANG, J., DAS, R., BAKER, R., AND SCRUGGS, R. 2021. Knowledge tracing models’ predictive performance when a student starts a skill. In Proceedings of the 14th International Conference on Educational Data Mining. EDM, Paris, France, 625–629.
ZHANG, J., SHI, X., KING, I., AND YEUNG, D.-Y. 2017. Dynamic key-value memory networks for knowledge tracing. In Proceedings of the 26th International Conference on World Wide Web. WWW ’17. IW3C2, Geneva, Switzerland, 765–774.
ZHANG, L., XIONG, X., ZHAO, S., BOTELHO, A., AND HEFFERNAN, N. T. 2017. Incorporating rich features into deep knowledge tracing. In Proceedings of 7th ACM Conference on Learning @ Scale. L@S ’17. ACM, New York, NY, USA, 169–172.
ZHAO, S., WANG, C., AND SAHEBI, S. 2020. Modeling knowledge acquisition from multiple learning resource types. In Proceedings of the 13th International Conference on Educational Data Mining. EDM, Virtual, 313–324.
ZHOU, Y., LI, X., CAO, Y., ZHAO, X., YE, Q., AND LV, J. 2021. Lana: Towards personalized deep knowledge tracing through distinguishable interactive sequences. In Proceedings of the 14th International Conference on Educational Data Mining. EDM, Paris, France, 602–608.
BADRINATH, A., WANG, F., AND PARDOS, Z. 2021. pyBKT: An accessible python library of bayesian knowledge tracing models. In Proceedings of the 14th International Conference on Educational Data Mining. EDM, Paris, France, 468–474.
BAKER, R., CORBETT, A. T., AND ALEVEN, V. 2008. More accurate student modeling through contextual estimation of slip and guess probabilities in bayesian knowledge tracing. In Proceedings of the 9th International Conference on Intelligent Tutoring Systems. Springer, Berlin, Germany, 406–415.
BREIMAN, L. 2001. Random forests. Machine learning 45, 1, 5–32.
CEN, H., KOEDINGER, K., AND JUNKER, B. 2006. Learning factors analysis–a general method for cognitive model evaluation and improvement. In Proceedings of the 8th International Conference on Intelligent Tutoring Systems. Springer, Berlin, Germany, 164–175.
CEPEDA, N. J., VUL, E., ROHRER, D., WIXTED, J. T., AND PASHLER, H. 2008. Spacing effects in learning: A temporal ridgeline of optimal retention. Psychological science 19, 11, 1095–1102.
CHANG, H.-S., HSU, H.-J., AND CHEN, K.-T. 2015. Modeling exercise relationships in e-learning: A unified approach. In Proceedings of the 12th International Conference on Educational Data Mining. EDM, Madrid, Spain, 532–535.
CHOFFIN, B., POPINEAU, F., BOURDA, Y., AND VIE, J.-J. 2019. DAS3H: Modeling student learning and forgetting for optimally scheduling distributed practice of skills. In Proceedings of the 12th International Conference on Educational Data Mining. EDM, Montr´eal, QC, Canada, 29–38.
CHOI, Y., LEE, Y., CHO, J., BAEK, J., KIM, B., CHA, Y., SHIN, D., BAE, C., AND HEO, J. 2020. Towards an appropriate query, key, and value computation for knowledge tracing. In Proceedings of 7th ACM Conference on Learning @ Scale. L@S ’20. ACM, New York, NY, USA, 341–344.
CHOI, Y., LEE, Y., SHIN, D., CHO, J., PARK, S., LEE, S., BAEK, J., BAE, C., KIM, B., AND HEO, J. 2020. EdNet: A large-scale hierarchical dataset in education. In Proceedings of the 21st International Conference on Artificial Intelligence in Education. Springer International Publishing, Cham, Switzerland, 69–73.
CORBETT, A. T. AND ANDERSON, J. R. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction 4, 4, 253–278.
DING, X. AND LARSON, E. C. 2019. Why deep knowledge tracing has less depth than anticipated. In Proceedings of the 12th International Conference on Educational Data Mining. EDM, Montr´eal, QC, Canada, 282–287.
DING, X. AND LARSON, E. C. 2021. On the interpretability of deep learning based models for knowledge tracing. CoRR abs/2101.11335.
EGLINGTON, L. G. AND PAVLIK, JR, P. I. 2019. Predictiveness of prior failures is improved by incorporating trial duration. Journal of Educational Data Mining 11, 2 (9), 1–19.
FENG, M., HEFFERNAN, N., AND KOEDINGER, K. 2009. Addressing the assessment challenge with an online system that tutors as it assesses. User Modeling and User-Adapted Interaction 19, 3, 243–266.
GALYARDT, A. AND GOLDIN, I. 2015. Move your lamp post: Recent data reflects learner knowledge better than older data. Journal of Educational Data Mining 7, 2, 83–108.
GERVET, T., KOEDINGER, K., SCHNEIDER, J., AND MITCHELL, T. 2020. When is deep learning the best approach to knowledge tracing? Journal of Educational Data Mining 12, 3, 31–54.
GHOSH, A., HEFFERNAN, N., AND LAN, A. S. 2020. Context-aware attentive knowledge tracing. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. KDD ’20. ACM, New York, NY, USA, 2330–2339.
GONZÁLEZ-BRENES, J., HUANG, Y., AND BRUSILOVSKY, P. 2014. General features in knowledge tracing: Applications to multiple subskills, temporal item response theory, and expert knowledge. In Proceedings of 7th International Conference on Educational Data Mining. EDM, London, UK, 84–91.
HOCHREITER, S. AND SCHMIDHUBER, J. 1997. Long short-term memory. Neural computation 9, 8, 1735–1780.
HUANG, X., CRAIG, S. D., XIE, J., GRAESSER, A., AND HU, X. 2016. Intelligent tutoring systems work as a math gap reducer in 6th grade after-school program. Learning and Individual Differences 47, 258–265.
KHAJAH, M., LINDSEY, R. V., AND MOZER, M. C. 2016. How deep is knowledge tracing? In Proceedings of the 9th International Conference on Educational Data Mining. EDM, Raleigh, NC, USA, 94–101.
KOEDINGER, K., SCHEINES, R., AND SCHALDENBRAND, P. 2018. Is the doer effect robust across multiple data sets? In Proceedings of the 11th International Conference on Educational Data Mining. EDM, Buffalo, NY, USA, 369–375.
KOEDINGER, K. R., KIM, J., JIA, J. Z., MCLAUGHLIN, E. A., AND BIER, N. L. 2015. Learning is not a spectator sport: Doing is better than watching for learning from a mooc. In Proceedings of the 2nd ACM Conference on Learning @ Scale. L@S ’15. ACM, New York, NY, USA, 111–120.
KOEDINGER, K. R., MCLAUGHLIN, E. A., JIA, J. Z., AND BIER, N. L. 2016. Is the doer effect a causal relationship? how can we tell and why it’s important. In Proceedings of the 6th International Conference on Learning Analytics & Knowledge. LAK ’16. ACM, New York, NY, USA, 388–397.
KÄSER, T., KLINGLER, S., SCHWING, A. G., AND GROSS, M. 2017. Dynamic bayesian networks for student modeling. IEEE Transactions on Learning Technologies 10, 4, 450–462.
LECUN, Y., HAFFNER, P., BOTTOU, L., AND BENGIO, Y. 1999. Object Recognition with Gradient-Based Learning. Springer, Berlin, Germany, 319–345.
LINDSEY, R. V., SHROYER, J. D., PASHLER, H., AND MOZER, M. C. 2014. Improving students’ long-term knowledge retention through personalized review. Psychological Science 25, 3, 639–647.
LIU, Q., HUANG, Z., YIN, Y., CHEN, E., XIONG, H., SU, Y., AND HU, G. 2019. Ekt: Exercise-aware knowledge tracing for student performance prediction. IEEE Transactions on Knowledge and Data Engineering 33, 1, 100–115.
LIU, Q., SHEN, S., HUANG, Z., CHEN, E., AND ZHENG, Y. 2021. A survey of knowledge tracing. CoRR abs/2105.15106.
MANNING, C. D. AND SCH¨UTZE, H. 1999. Foundations of Statistical Natural Language Processing. MIT Press, Cambridge, MA, USA.
MONTERO, S., ARORA, A., KELLY, S., MILNE, B., AND MOZER, M. 2018. Does deep knowledge tracing model interactions among skills? In Proceedings of the 11th International Conference on Educational Data Mining. EDM, Buffalo, NY, USA, 462–466.
MOZER, M. C. AND LINDSEY, R. V. 2016. Predicting and Improving Memory Retention: Psychological Theory Matters in the Big Data Era. Routledge/Taylor & Francis Group, London, UK.
NAKAGAWA, H., IWASAWA, Y., AND MATSUO, Y. 2019. Graph-based knowledge tracing: Modeling student proficiency using graph neural network. In IEEE/WIC/ACM International Conference on Web Intelligence. IEEE, New York, NY, USA, 156–163.
PANDEY, S. AND KARYPIS, G. 2019. A self-attentive model for knowledge tracing. In Proceedings of the 12th International Conference on Educational Data Mining. EDM, Montr´eal, QC, Canada, 384–389.
PARDOS, Z. A. AND HEFFERNAN, N. T. 2010. Modeling individualization in a bayesian networks implementation of knowledge tracing. In Proceedings of the 18th International Conference on User Modeling, Adaptation, and Personalization. Springer, Berlin, Germany, 255–266.
PARDOS, Z. A. AND HEFFERNAN, N. T. 2011. KT-IDEM: Introducing item difficulty to the knowledge tracing model. In Proceedings of the 19th International Conference on User Modeling, Adaptation, and Personalization. Springer, Berlin, Germany, 243–254.
PASZKE, A., GROSS, S., MASSA, F., LERER, A., BRADBURY, J., CHANAN, G., KILLEEN, T., LIN, Z., GIMELSHEIN, N., ANTIGA, L., ET AL. 2019. PyTorch: An imperative style, high-performance deep learning library. In Proceedings of the 32th International Conference on Advances in Neural Information Processing Systems. Vol. 32. Curran Associates, Inc., Vancouver, BC, Canada, 8026–8037.
PAVLIK, P. I., EGLINGTON, L. G., AND HARRELL-WILLIAMS, L. M. 2021. Logistic knowledge tracing: A constrained framework for learner modeling. IEEE Transactions on Learning Technologies 14, 5, 624–639.
PAVLIK JR, P., CEN, H., AND KOEDINGER, K. 2009. Performance factors analysis–a new alternative to knowledge tracing. In Frontiers in Artificial Intelligence and Applications. Vol. 200. IOS Press, Amsterdam, Netherlands, 531–538.
PEDREGOSA, F., VAROQUAUX, G., GRAMFORT, A., MICHEL, V., THIRION, B., GRISEL, O., BLONDEL, M., PRETTENHOFER, P., WEISS, R., DUBOURG, V., ET AL. 2011. Scikit-learn: Machine learning in python. The Journal of Machine Learning Research 12, 2825–2830.
PIECH, C., BASSEN, J., HUANG, J., GANGULI, S., SAHAMI, M., GUIBAS, L. J., AND SOHL-DICKSTEIN, J. 2015. Deep knowledge tracing. In Proceedings of the 28th Conference on Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, Eds. Vol. 28th. Curran Associates, Inc., Montréal, QC, Canada.
POJEN, C., MINGEN, H., AND TZUYANG, T. 2020. Junyi academy online learning activity dataset: A large-scale public online learning activity dataset from elementary to senior high school students. [Available online at: https://www.kaggle.com/datasets/junyiacademy/learning-activity-public-datasetby- junyi-academy; accessed 28-March-2022].
QIU, Y., QI, Y., LU, H., PARDOS, Z. A., AND HEFFERNAN, N. T. 2011. Does time matter? modeling the effect of time with bayesian knowledge tracing. In Proceedings of the 4th International Conference on Educational Data Mining. Eindhoven University of Technology, Eindhoven, Netherlands, 139–148.
RASCH, G. 1960. Probabilistic Models for Some Intelligence and Attainment Tests. Studies in mathematical psychology. Danmarks Paedagogiske Institut, Aarhus, Denmark.
SAO PEDRO, M., BAKER, R., AND GOBERT, J. 2013. Incorporating scaffolding and tutor context into bayesian knowledge tracing to predict inquiry skill acquisition. In Proceedings of the 6th International Conference on Educational Data Mining. EDM, Memphis, TN, USA.
SCRUGGS, R., BAKER, R., AND MCLAREN, B. 2020. Extending deep knowledge tracing: Inferring interpretable knowledge and predicting post system performance. In Proceedings of the 28th International Conference on Computers in Education. APSCE, Jhongli City, Taiwan, 195–204.
SHEN, S., LIU, Q., CHEN, E., WU, H., HUANG, Z., ZHAO, W., SU, Y., MA, H., AND WANG, S. 2020. Convolutional knowledge tracing: Modeling individualization in student learning process. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, USA, 1857–1860.
SHIN, D., SHIM, Y., YU, H., LEE, S., KIM, B., AND CHOI, Y. 2021. Saint+: Integrating temporal features for EdNet correctness prediction. In Proceedings of the 11th International Learning Analytics and Knowledge Conference. ACM, New York, NY, USA, 490–496.
STAMPER, J., NICULESCU-MIZIL, A., RITTER, S., GORDON, G., AND KOEDINGER, K. 2010. Bridge to Algebra 2006-2007. development data set from KDD Cup 2010 educational data mining challenge. [Available online at: https://pslcdatashop.web.cmu.edu/KDDCup/downloads.jsp; accessed 28-March- 2022].
TONG, H., WANG, Z., LIU, Q., ZHOU, Y., AND HAN, W. 2020. HGKT : Introducing problem schema with hierarchical exercise graph for knowledge tracing. CoRR abs/2006.16915.
TSUTSUMI, E., KINOSHITA, R., AND UENO, M. 2021. Deep-IRT with independent student and item networks. In Proceedings of the 14th International Conference on Educational Data Mining. EDM, Paris, France, 510–517.
VAN CAMPENHOUT, R., JOHNSON, B., AND OLSEN, J. 2021. The doer effect: Replicating findings that doing causes learning. In Proceedings of the 13th International Conference on Mobile, Hybrid, and On-line Learning. IARIA, France, 1–6.
VAN DER LINDEN, W. J. AND HAMBLETON, R. K. 2013. Handbook of Modern Item Response Theory. Springer, New York, NY, USA.
VASWANI, A., SHAZEER, N., PARMAR, N., USZKOREIT, J., JONES, L., GOMEZ, A. N., KAISER, L. U., AND POLOSUKHIN, I. 2017. Attention is all you need. In Proceedings of the 31th Conference on Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds. Vol. 30. Curran Associates, Inc., Long Beach, CA, USA.
VON STUMM, S. 2017. Socioeconomic status amplifies the achievement gap throughout compulsory education independent of intelligence. Intelligence 60, 57–62.
WALSH, M. M., GLUCK, K. A., GUNZELMANN, G., JASTRZEMBSKI, T., KRUSMARK, M., MYUNG, J. I., PITT, M. A., AND ZHOU, R. 2018. Mechanisms underlying the spacing effect in learning: A comparison of three computational models. Journal of Experimental Psychology 147, 9, 1325.
WANG, Z., LAMB, A., SAVELIEV, E., CAMERON, P., ZAYKOV, J., HERNANDEZ-LOBATO, J. M., TURNER, R. E., BARANIUK, R. G., BARTON, C., JONES, S. P., ET AL. 2021. Results and insights from diagnostic questions: The neurips 2020 education challenge. In NeurIPS 2020 Competition and Demonstration Track. PMLR, Virtual, 191–205.
WANG, Z., LAMB, A., SAVELIEV, E., CAMERON, P., ZAYKOV, Y., HERN´A NDEZ-LOBATO, J. M., TURNER, R. E., BARANIUK, R. G., BARTON, C., AND JONES. 2020. Diagnostic questions: The NeurIPS 2020 education challenge. CoRR abs/2007.12061.
WHITE, K. R. 1982. The relation between socioeconomic status and academic achievement. Psychological Bulletin 91, 3, 461.
WILSON, K. H., XIONG, X., KHAJAH, M., LINDSEY, R. V., ZHAO, S., KARKLIN, Y., VAN INWEGEN, E. G., HAN, B., EKANADHAM, C., BECK, J. E., ET AL. 2016. Estimating student proficiency: Deep learning is not the panacea. In Neural Information Processing Systems, Workshop on Machine Learning for Education. NIPS, Barcelona Spain, 3–7.
WOLPERT, D. H. 1992. Stacked generalization. Neural Networks 5, 2, 241–259.
YANG, H. AND CHEUNG, L. P. 2018. Implicit heterogeneous features embedding in deep knowledge tracing. Cognitive Computation 10, 1, 3–14.
YANG, Y., SHEN, J., QU, Y., LIU, Y., WANG, K., ZHU, Y., ZHANG, W., AND YU, Y. 2021. Gikt: A graph-based interaction model for knowledge tracing. In Machine Learning and Knowledge Discovery in Databases. Springer International Publishing, Cham, Switzerland, 299–315.
YEUNG, C.-K. 2019. Deep-IRT: Make deep learning based knowledge tracing explainable using item response theory. CoRR abs/1904.11738.
YUDELSON, M. V., KOEDINGER, K. R., AND GORDON, G. J. 2013. Individualized bayesian knowledge tracing models. In Proceedings of the 16th International Conference on Artificial Intelligence in Education. Springer, Berlin, Germany, 171–180.
ZHANG, C., JIANG, Y., ZHANG, W., AND GU, C. 2021. Muse: Multi-scale temporal features evolution for knowledge tracing. CoRR abs/2102.00228.
ZHANG, J., DAS, R., BAKER, R., AND SCRUGGS, R. 2021. Knowledge tracing models’ predictive performance when a student starts a skill. In Proceedings of the 14th International Conference on Educational Data Mining. EDM, Paris, France, 625–629.
ZHANG, J., SHI, X., KING, I., AND YEUNG, D.-Y. 2017. Dynamic key-value memory networks for knowledge tracing. In Proceedings of the 26th International Conference on World Wide Web. WWW ’17. IW3C2, Geneva, Switzerland, 765–774.
ZHANG, L., XIONG, X., ZHAO, S., BOTELHO, A., AND HEFFERNAN, N. T. 2017. Incorporating rich features into deep knowledge tracing. In Proceedings of 7th ACM Conference on Learning @ Scale. L@S ’17. ACM, New York, NY, USA, 169–172.
ZHAO, S., WANG, C., AND SAHEBI, S. 2020. Modeling knowledge acquisition from multiple learning resource types. In Proceedings of the 13th International Conference on Educational Data Mining. EDM, Virtual, 313–324.
ZHOU, Y., LI, X., CAO, Y., ZHAO, X., YE, Q., AND LV, J. 2021. Lana: Towards personalized deep knowledge tracing through distinguishable interactive sequences. In Proceedings of the 14th International Conference on Educational Data Mining. EDM, Paris, France, 602–608.
Issue
Section
Articles
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors who publish with this journal agree to the following terms:
- The Author retains copyright in the Work, where the term “Work” shall include all digital objects that may result in subsequent electronic publication or distribution.
- Upon acceptance of the Work, the author shall grant to the Publisher the right of first publication of the Work.
- The Author shall grant to the Publisher and its agents the nonexclusive perpetual right and license to publish, archive, and make accessible the Work in whole or in part in all forms of media now or hereafter known under a Creative Commons 4.0 License (Attribution-Noncommercial-No Derivatives 4.0 International), or its equivalent, which, for the avoidance of doubt, allows others to copy, distribute, and transmit the Work under the following conditions:
- Attribution—other users must attribute the Work in the manner specified by the author as indicated on the journal Web site;
- Noncommercial—other users (including Publisher) may not use this Work for commercial purposes;
- No Derivative Works—other users (including Publisher) may not alter, transform, or build upon this Work,with the understanding that any of the above conditions can be waived with permission from the Author and that where the Work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.
- The Author is able to enter into separate, additional contractual arrangements for the nonexclusive distribution of the journal's published version of the Work (e.g., post it to an institutional repository or publish it in a book), as long as there is provided in the document an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post online a pre-publication manuscript (but not the Publisher’s final formatted PDF version of the Work) in institutional repositories or on their Websites prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (see The Effect of Open Access). Any such posting made before acceptance and publication of the Work shall be updated upon publication to include a reference to the Publisher-assigned DOI (Digital Object Identifier) and a link to the online abstract for the final published Work in the Journal.
- Upon Publisher’s request, the Author agrees to furnish promptly to Publisher, at the Author’s own expense, written evidence of the permissions, licenses, and consents for use of third-party material included within the Work, except as determined by Publisher to be covered by the principles of Fair Use.
- The Author represents and warrants that:
- the Work is the Author’s original work;
- the Author has not transferred, and will not transfer, exclusive rights in the Work to any third party;
- the Work is not pending review or under consideration by another publisher;
- the Work has not previously been published;
- the Work contains no misrepresentation or infringement of the Work or property of other authors or third parties; and
- the Work contains no libel, invasion of privacy, or other unlawful matter.
- The Author agrees to indemnify and hold Publisher harmless from Author’s breach of the representations and warranties contained in Paragraph 6 above, as well as any claim or proceeding relating to Publisher’s use and publication of any content contained in the Work, including third-party content.