Assessing the Performance of Online Students - New Data, New Approaches, Improved Accuracy

##plugins.themes.bootstrap3.article.main##

##plugins.themes.bootstrap3.article.sidebar##

Published Jun 24, 2022
Robin Schmucker Jingbo Wang Shijia Hu Tom M. Mitchell

Abstract

We consider the problem of assessing the changing performance levels of individual students as they go
through online courses. This student performance modeling problem is a critical step for building adaptive
online teaching systems. Specifically, we conduct a study of how to utilize various types and large amounts
of log data from earlier students to train accurate machine learning models that predict the performance of
future students. This study is the first to use four very large sets of student data made available recently
from four distinct intelligent tutoring systems.
Our results include a new machine learning approach that defines a new state of the art for logistic
regression based student performance modeling, improving over earlier methods in several ways: First, we
achieve improved accuracy of student modeling by introducing new features that can be easily computed
from conventional question-response logs (e.g., features such as the pattern in the student’s most recent
answers). Second, we take advantage of features of the student history that go beyond question-response
pairs (e.g., features such as which video segments the student watched, or skipped) as well as background
information about prerequisite structure in the curriculum. Third, we train multiple specialized student
performance models for different aspects of the curriculum (e.g., specializing in early versus later segments
of the student history), then combine these specialized models to create a group prediction of the student
performance. Taken together, these innovations yield an average AUC score across these four datasets of
0.808 compared to the previous best logistic regression approach score of 0.767, and also outperforming
state-of-the-art deep neural net approaches. Importantly, we observe consistent improvements from each of
our three methodological innovations, in each diverse dataset, suggesting that our methods are of general
utility and likely to produce improvements for other online tutoring systems as well.

How to Cite

Schmucker, R., Wang, J., Hu, S., & Mitchell, T. (2022). Assessing the Performance of Online Students - New Data, New Approaches, Improved Accuracy. Journal of Educational Data Mining, 14(1), 1–45. https://doi.org/10.5281/zenodo.6450190
Abstract 430 | PDF Downloads 332

##plugins.themes.bootstrap3.article.details##

Keywords

performance modeling, knowledge tracing, logistic regression, deep learning, features

References
AIKENS, N. L. AND BARBARIN, O. 2008. Socioeconomic differences in reading trajectories: The contribution of family, neighborhood, and school contexts. Journal of Educational Psychology 100, 2, 235.

BADRINATH, A., WANG, F., AND PARDOS, Z. 2021. pyBKT: An accessible python library of bayesian knowledge tracing models. In Proceedings of the 14th International Conference on Educational Data Mining. EDM, Paris, France, 468–474.

BAKER, R., CORBETT, A. T., AND ALEVEN, V. 2008. More accurate student modeling through contextual estimation of slip and guess probabilities in bayesian knowledge tracing. In Proceedings of the 9th International Conference on Intelligent Tutoring Systems. Springer, Berlin, Germany, 406–415.

BREIMAN, L. 2001. Random forests. Machine learning 45, 1, 5–32.

CEN, H., KOEDINGER, K., AND JUNKER, B. 2006. Learning factors analysis–a general method for cognitive model evaluation and improvement. In Proceedings of the 8th International Conference on Intelligent Tutoring Systems. Springer, Berlin, Germany, 164–175.

CEPEDA, N. J., VUL, E., ROHRER, D., WIXTED, J. T., AND PASHLER, H. 2008. Spacing effects in learning: A temporal ridgeline of optimal retention. Psychological science 19, 11, 1095–1102.

CHANG, H.-S., HSU, H.-J., AND CHEN, K.-T. 2015. Modeling exercise relationships in e-learning: A unified approach. In Proceedings of the 12th International Conference on Educational Data Mining. EDM, Madrid, Spain, 532–535.

CHOFFIN, B., POPINEAU, F., BOURDA, Y., AND VIE, J.-J. 2019. DAS3H: Modeling student learning and forgetting for optimally scheduling distributed practice of skills. In Proceedings of the 12th International Conference on Educational Data Mining. EDM, Montr´eal, QC, Canada, 29–38.

CHOI, Y., LEE, Y., CHO, J., BAEK, J., KIM, B., CHA, Y., SHIN, D., BAE, C., AND HEO, J. 2020. Towards an appropriate query, key, and value computation for knowledge tracing. In Proceedings of 7th ACM Conference on Learning @ Scale. L@S ’20. ACM, New York, NY, USA, 341–344.

CHOI, Y., LEE, Y., SHIN, D., CHO, J., PARK, S., LEE, S., BAEK, J., BAE, C., KIM, B., AND HEO, J. 2020. EdNet: A large-scale hierarchical dataset in education. In Proceedings of the 21st International Conference on Artificial Intelligence in Education. Springer International Publishing, Cham, Switzerland, 69–73.

CORBETT, A. T. AND ANDERSON, J. R. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction 4, 4, 253–278.

DING, X. AND LARSON, E. C. 2019. Why deep knowledge tracing has less depth than anticipated. In Proceedings of the 12th International Conference on Educational Data Mining. EDM, Montr´eal, QC, Canada, 282–287.

DING, X. AND LARSON, E. C. 2021. On the interpretability of deep learning based models for knowledge tracing. CoRR abs/2101.11335.

EGLINGTON, L. G. AND PAVLIK, JR, P. I. 2019. Predictiveness of prior failures is improved by incorporating trial duration. Journal of Educational Data Mining 11, 2 (9), 1–19.

FENG, M., HEFFERNAN, N., AND KOEDINGER, K. 2009. Addressing the assessment challenge with an online system that tutors as it assesses. User Modeling and User-Adapted Interaction 19, 3, 243–266.

GALYARDT, A. AND GOLDIN, I. 2015. Move your lamp post: Recent data reflects learner knowledge better than older data. Journal of Educational Data Mining 7, 2, 83–108.

GERVET, T., KOEDINGER, K., SCHNEIDER, J., AND MITCHELL, T. 2020. When is deep learning the best approach to knowledge tracing? Journal of Educational Data Mining 12, 3, 31–54.

GHOSH, A., HEFFERNAN, N., AND LAN, A. S. 2020. Context-aware attentive knowledge tracing. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. KDD ’20. ACM, New York, NY, USA, 2330–2339.

GONZÁLEZ-BRENES, J., HUANG, Y., AND BRUSILOVSKY, P. 2014. General features in knowledge tracing: Applications to multiple subskills, temporal item response theory, and expert knowledge. In Proceedings of 7th International Conference on Educational Data Mining. EDM, London, UK, 84–91.

HOCHREITER, S. AND SCHMIDHUBER, J. 1997. Long short-term memory. Neural computation 9, 8, 1735–1780.

HUANG, X., CRAIG, S. D., XIE, J., GRAESSER, A., AND HU, X. 2016. Intelligent tutoring systems work as a math gap reducer in 6th grade after-school program. Learning and Individual Differences 47, 258–265.

KHAJAH, M., LINDSEY, R. V., AND MOZER, M. C. 2016. How deep is knowledge tracing? In Proceedings of the 9th International Conference on Educational Data Mining. EDM, Raleigh, NC, USA, 94–101.

KOEDINGER, K., SCHEINES, R., AND SCHALDENBRAND, P. 2018. Is the doer effect robust across multiple data sets? In Proceedings of the 11th International Conference on Educational Data Mining. EDM, Buffalo, NY, USA, 369–375.

KOEDINGER, K. R., KIM, J., JIA, J. Z., MCLAUGHLIN, E. A., AND BIER, N. L. 2015. Learning is not a spectator sport: Doing is better than watching for learning from a mooc. In Proceedings of the 2nd ACM Conference on Learning @ Scale. L@S ’15. ACM, New York, NY, USA, 111–120.

KOEDINGER, K. R., MCLAUGHLIN, E. A., JIA, J. Z., AND BIER, N. L. 2016. Is the doer effect a causal relationship? how can we tell and why it’s important. In Proceedings of the 6th International Conference on Learning Analytics & Knowledge. LAK ’16. ACM, New York, NY, USA, 388–397.

KÄSER, T., KLINGLER, S., SCHWING, A. G., AND GROSS, M. 2017. Dynamic bayesian networks for student modeling. IEEE Transactions on Learning Technologies 10, 4, 450–462.

LECUN, Y., HAFFNER, P., BOTTOU, L., AND BENGIO, Y. 1999. Object Recognition with Gradient-Based Learning. Springer, Berlin, Germany, 319–345.

LINDSEY, R. V., SHROYER, J. D., PASHLER, H., AND MOZER, M. C. 2014. Improving students’ long-term knowledge retention through personalized review. Psychological Science 25, 3, 639–647.

LIU, Q., HUANG, Z., YIN, Y., CHEN, E., XIONG, H., SU, Y., AND HU, G. 2019. Ekt: Exercise-aware knowledge tracing for student performance prediction. IEEE Transactions on Knowledge and Data Engineering 33, 1, 100–115.

LIU, Q., SHEN, S., HUANG, Z., CHEN, E., AND ZHENG, Y. 2021. A survey of knowledge tracing. CoRR abs/2105.15106.

MANNING, C. D. AND SCH¨UTZE, H. 1999. Foundations of Statistical Natural Language Processing. MIT Press, Cambridge, MA, USA.

MONTERO, S., ARORA, A., KELLY, S., MILNE, B., AND MOZER, M. 2018. Does deep knowledge tracing model interactions among skills? In Proceedings of the 11th International Conference on Educational Data Mining. EDM, Buffalo, NY, USA, 462–466.

MOZER, M. C. AND LINDSEY, R. V. 2016. Predicting and Improving Memory Retention: Psychological Theory Matters in the Big Data Era. Routledge/Taylor & Francis Group, London, UK.

NAKAGAWA, H., IWASAWA, Y., AND MATSUO, Y. 2019. Graph-based knowledge tracing: Modeling student proficiency using graph neural network. In IEEE/WIC/ACM International Conference on Web Intelligence. IEEE, New York, NY, USA, 156–163.

PANDEY, S. AND KARYPIS, G. 2019. A self-attentive model for knowledge tracing. In Proceedings of the 12th International Conference on Educational Data Mining. EDM, Montr´eal, QC, Canada, 384–389.

PARDOS, Z. A. AND HEFFERNAN, N. T. 2010. Modeling individualization in a bayesian networks implementation of knowledge tracing. In Proceedings of the 18th International Conference on User Modeling, Adaptation, and Personalization. Springer, Berlin, Germany, 255–266.

PARDOS, Z. A. AND HEFFERNAN, N. T. 2011. KT-IDEM: Introducing item difficulty to the knowledge tracing model. In Proceedings of the 19th International Conference on User Modeling, Adaptation, and Personalization. Springer, Berlin, Germany, 243–254.

PASZKE, A., GROSS, S., MASSA, F., LERER, A., BRADBURY, J., CHANAN, G., KILLEEN, T., LIN, Z., GIMELSHEIN, N., ANTIGA, L., ET AL. 2019. PyTorch: An imperative style, high-performance deep learning library. In Proceedings of the 32th International Conference on Advances in Neural Information Processing Systems. Vol. 32. Curran Associates, Inc., Vancouver, BC, Canada, 8026–8037.

PAVLIK, P. I., EGLINGTON, L. G., AND HARRELL-WILLIAMS, L. M. 2021. Logistic knowledge tracing: A constrained framework for learner modeling. IEEE Transactions on Learning Technologies 14, 5, 624–639.

PAVLIK JR, P., CEN, H., AND KOEDINGER, K. 2009. Performance factors analysis–a new alternative to knowledge tracing. In Frontiers in Artificial Intelligence and Applications. Vol. 200. IOS Press, Amsterdam, Netherlands, 531–538.

PEDREGOSA, F., VAROQUAUX, G., GRAMFORT, A., MICHEL, V., THIRION, B., GRISEL, O., BLONDEL, M., PRETTENHOFER, P., WEISS, R., DUBOURG, V., ET AL. 2011. Scikit-learn: Machine learning in python. The Journal of Machine Learning Research 12, 2825–2830.

PIECH, C., BASSEN, J., HUANG, J., GANGULI, S., SAHAMI, M., GUIBAS, L. J., AND SOHL-DICKSTEIN, J. 2015. Deep knowledge tracing. In Proceedings of the 28th Conference on Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, Eds. Vol. 28th. Curran Associates, Inc., Montréal, QC, Canada.

POJEN, C., MINGEN, H., AND TZUYANG, T. 2020. Junyi academy online learning activity dataset: A large-scale public online learning activity dataset from elementary to senior high school students. [Available online at: https://www.kaggle.com/datasets/junyiacademy/learning-activity-public-datasetby- junyi-academy; accessed 28-March-2022].

QIU, Y., QI, Y., LU, H., PARDOS, Z. A., AND HEFFERNAN, N. T. 2011. Does time matter? modeling the effect of time with bayesian knowledge tracing. In Proceedings of the 4th International Conference on Educational Data Mining. Eindhoven University of Technology, Eindhoven, Netherlands, 139–148.

RASCH, G. 1960. Probabilistic Models for Some Intelligence and Attainment Tests. Studies in mathematical psychology. Danmarks Paedagogiske Institut, Aarhus, Denmark.

SAO PEDRO, M., BAKER, R., AND GOBERT, J. 2013. Incorporating scaffolding and tutor context into bayesian knowledge tracing to predict inquiry skill acquisition. In Proceedings of the 6th International Conference on Educational Data Mining. EDM, Memphis, TN, USA.

SCRUGGS, R., BAKER, R., AND MCLAREN, B. 2020. Extending deep knowledge tracing: Inferring interpretable knowledge and predicting post system performance. In Proceedings of the 28th International Conference on Computers in Education. APSCE, Jhongli City, Taiwan, 195–204.

SHEN, S., LIU, Q., CHEN, E., WU, H., HUANG, Z., ZHAO, W., SU, Y., MA, H., AND WANG, S. 2020. Convolutional knowledge tracing: Modeling individualization in student learning process. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, USA, 1857–1860.

SHIN, D., SHIM, Y., YU, H., LEE, S., KIM, B., AND CHOI, Y. 2021. Saint+: Integrating temporal features for EdNet correctness prediction. In Proceedings of the 11th International Learning Analytics and Knowledge Conference. ACM, New York, NY, USA, 490–496.

STAMPER, J., NICULESCU-MIZIL, A., RITTER, S., GORDON, G., AND KOEDINGER, K. 2010. Bridge to Algebra 2006-2007. development data set from KDD Cup 2010 educational data mining challenge. [Available online at: https://pslcdatashop.web.cmu.edu/KDDCup/downloads.jsp; accessed 28-March- 2022].

TONG, H., WANG, Z., LIU, Q., ZHOU, Y., AND HAN, W. 2020. HGKT : Introducing problem schema with hierarchical exercise graph for knowledge tracing. CoRR abs/2006.16915.

TSUTSUMI, E., KINOSHITA, R., AND UENO, M. 2021. Deep-IRT with independent student and item networks. In Proceedings of the 14th International Conference on Educational Data Mining. EDM, Paris, France, 510–517.

VAN CAMPENHOUT, R., JOHNSON, B., AND OLSEN, J. 2021. The doer effect: Replicating findings that doing causes learning. In Proceedings of the 13th International Conference on Mobile, Hybrid, and On-line Learning. IARIA, France, 1–6.

VAN DER LINDEN, W. J. AND HAMBLETON, R. K. 2013. Handbook of Modern Item Response Theory. Springer, New York, NY, USA.

VASWANI, A., SHAZEER, N., PARMAR, N., USZKOREIT, J., JONES, L., GOMEZ, A. N., KAISER, L. U., AND POLOSUKHIN, I. 2017. Attention is all you need. In Proceedings of the 31th Conference on Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds. Vol. 30. Curran Associates, Inc., Long Beach, CA, USA.

VON STUMM, S. 2017. Socioeconomic status amplifies the achievement gap throughout compulsory education independent of intelligence. Intelligence 60, 57–62.

WALSH, M. M., GLUCK, K. A., GUNZELMANN, G., JASTRZEMBSKI, T., KRUSMARK, M., MYUNG, J. I., PITT, M. A., AND ZHOU, R. 2018. Mechanisms underlying the spacing effect in learning: A comparison of three computational models. Journal of Experimental Psychology 147, 9, 1325.

WANG, Z., LAMB, A., SAVELIEV, E., CAMERON, P., ZAYKOV, J., HERNANDEZ-LOBATO, J. M., TURNER, R. E., BARANIUK, R. G., BARTON, C., JONES, S. P., ET AL. 2021. Results and insights from diagnostic questions: The neurips 2020 education challenge. In NeurIPS 2020 Competition and Demonstration Track. PMLR, Virtual, 191–205.

WANG, Z., LAMB, A., SAVELIEV, E., CAMERON, P., ZAYKOV, Y., HERN´A NDEZ-LOBATO, J. M., TURNER, R. E., BARANIUK, R. G., BARTON, C., AND JONES. 2020. Diagnostic questions: The NeurIPS 2020 education challenge. CoRR abs/2007.12061.

WHITE, K. R. 1982. The relation between socioeconomic status and academic achievement. Psychological Bulletin 91, 3, 461.

WILSON, K. H., XIONG, X., KHAJAH, M., LINDSEY, R. V., ZHAO, S., KARKLIN, Y., VAN INWEGEN, E. G., HAN, B., EKANADHAM, C., BECK, J. E., ET AL. 2016. Estimating student proficiency: Deep learning is not the panacea. In Neural Information Processing Systems, Workshop on Machine Learning for Education. NIPS, Barcelona Spain, 3–7.

WOLPERT, D. H. 1992. Stacked generalization. Neural Networks 5, 2, 241–259.

YANG, H. AND CHEUNG, L. P. 2018. Implicit heterogeneous features embedding in deep knowledge tracing. Cognitive Computation 10, 1, 3–14.

YANG, Y., SHEN, J., QU, Y., LIU, Y., WANG, K., ZHU, Y., ZHANG, W., AND YU, Y. 2021. Gikt: A graph-based interaction model for knowledge tracing. In Machine Learning and Knowledge Discovery in Databases. Springer International Publishing, Cham, Switzerland, 299–315.

YEUNG, C.-K. 2019. Deep-IRT: Make deep learning based knowledge tracing explainable using item response theory. CoRR abs/1904.11738.

YUDELSON, M. V., KOEDINGER, K. R., AND GORDON, G. J. 2013. Individualized bayesian knowledge tracing models. In Proceedings of the 16th International Conference on Artificial Intelligence in Education. Springer, Berlin, Germany, 171–180.

ZHANG, C., JIANG, Y., ZHANG, W., AND GU, C. 2021. Muse: Multi-scale temporal features evolution for knowledge tracing. CoRR abs/2102.00228.

ZHANG, J., DAS, R., BAKER, R., AND SCRUGGS, R. 2021. Knowledge tracing models’ predictive performance when a student starts a skill. In Proceedings of the 14th International Conference on Educational Data Mining. EDM, Paris, France, 625–629.

ZHANG, J., SHI, X., KING, I., AND YEUNG, D.-Y. 2017. Dynamic key-value memory networks for knowledge tracing. In Proceedings of the 26th International Conference on World Wide Web. WWW ’17. IW3C2, Geneva, Switzerland, 765–774.

ZHANG, L., XIONG, X., ZHAO, S., BOTELHO, A., AND HEFFERNAN, N. T. 2017. Incorporating rich features into deep knowledge tracing. In Proceedings of 7th ACM Conference on Learning @ Scale. L@S ’17. ACM, New York, NY, USA, 169–172.

ZHAO, S., WANG, C., AND SAHEBI, S. 2020. Modeling knowledge acquisition from multiple learning resource types. In Proceedings of the 13th International Conference on Educational Data Mining. EDM, Virtual, 313–324.

ZHOU, Y., LI, X., CAO, Y., ZHAO, X., YE, Q., AND LV, J. 2021. Lana: Towards personalized deep knowledge tracing through distinguishable interactive sequences. In Proceedings of the 14th International Conference on Educational Data Mining. EDM, Paris, France, 602–608.
Section
Articles

Most read articles by the same author(s)