dAFM: Fusing Psychometric and Connectionist Modeling for Q-matrix Refinement

##plugins.themes.bootstrap3.article.main##

##plugins.themes.bootstrap3.article.sidebar##

Published Oct 25, 2018
Zachary A. Pardos Anant Dadu

Abstract

We introduce a model which combines principles from psychometric and connectionist paradigms to allow direct Q-matrix refinement via backpropagation. We call this model dAFM, based on augmentation of the original Additive Factors Model (AFM), whose calculations and constraints we show can be exactly replicated within the framework of neural networks. In order to parameterize the Q-matrix definition in the model, the associations between questions and knowledge components (KC) need to be represented by adjustable weights. Furthermore, student KC opportunity counts, instead of serving as fixed inputs, need to be calculated dynamically as the Q-matrix changes during training. We describe our solutions to these two modeling challenges and evaluate several variants of our fully realized model on datasets from the Cognitive Tutor and ASSISTments. We compare learning the Q-matrix from scratch vs. refining an expert specified KC model and evaluate various procedures for refinement. In our quantitative predictive analysis, we find that dAFM learns a better generalizing Q-matrix than the original expert model in all our primary datasets. Using a development set, we also find that the dAFM Q-matrix is superior to KC representations extracted from trained Deep Knowledge Tracing and skip-gram models. Examples are shown of questions whose fit was improved by dAFM with depictions of their original and refined KC associations. We consistently find in our experiments that our dAFM variant which attempted to learn the Q-matrix from scratch underperformed models which started with an expert defined Q-matrix that was then refined. This observation continues a theme in EDM of utility found in the enduring value of expert domain knowledge enhanced through data-driven refinement.

How to Cite

Pardos, Z. A., & Dadu, A. (2018). dAFM: Fusing Psychometric and Connectionist Modeling for Q-matrix Refinement. Journal of Educational Data Mining, 10(2), 1–27. https://doi.org/10.5281/zenodo.3554689
Abstract 993 | PDF Downloads 1286

##plugins.themes.bootstrap3.article.details##

Keywords

neural networks, IRT, Q-matrix, KC model, DKT, skip-gram, learning, measurement

References
BURRELL, J. 2016. How the machine thinks: Understanding opacity in machine learning algorithms. Big Data & Society 3, 1.

CEN, H., KOEDINGER, K., AND JUNKER, B. 2005. Automating cognitive model improvement by A* search and logistic regression. In Proceedings of AAAI 2005 Educational Data Mining Workshop, J. E. Beck, Ed.

CEN, H., KOEDINGER, K., AND JUNKER, B. 2006. Learning factors analysis: A general method for cognitive model evaluation and improvement. In Proceedings of the 8th International Conference on Intelligent Tutoring Systems, M. Ikeda, K. Ashley, and T.-W. Chan, Eds. Springer-Verlag, 164–175.

CEN, H., KOEDINGER, K. R., AND JUNKER, B. 2007. Is over practice necessary? Improving learning efficiency with the Cognitive Tutor through educational data mining. In Proceedings of 13th International Conference on Artificial Intelligence in Education, K. Ashley and T. van Engers, Eds. IOS Press, 511–518.

CHEN, Y., GONZ´A LEZ-BRENES, J. P., AND TIAN, J. 2016. Joint discovery of skill prerequisite graphs and student models. In Proceedings of the 9th International Conference on Educational Data Mining, M. Chi and M. Feng, Eds. 46–53.

CHIU, C.-Y. 2013. Statistical refinement of the Q-matrix in cognitive diagnosis. Applied Psychological Measurement 37, 8, 598–618.

CHOROMANSKA, A., HENAFF, M., MATHIEU, M., AROUS, G. B., AND LECUN, Y. 2015. The loss surfaces of multilayer networks. In Artificial Intelligence and Statistics. 192–204.

CLARK, R., FELDON, D., VANMERRIENBOER, J., YATES, K., AND EARLY, S. 2006. Cognitive task analysis. Handbook of research on educational communications and technology 3.

CORBETT, A. 2001. Cognitive computer tutors: Solving the two-sigma problem. In Proceedings of the 8th International Conference on User Modeling, M. Bauer, P. J. Gmytrasiewicz, and J. Vassileva, Eds. Springer, 137–147.

CORBETT, A. T. AND ANDERSON, J. R. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction 4, 4, 253–278.

DESMARAIS, M. ET AL. 2011. Conditions for effectively deriving a Q-matrix from data with nonnegative matrix factorization. In Proceedings of the 4th International Conference on Educational Data Mining, C. Conati and S. Ventura, Eds. 41–50.

DESMARAIS, M. C. AND NACEUR, R. 2013. A matrix factorization method for mapping items to skills and for enhancing expert-based Q-matrices. In Proceedings of the 6th International Conference on Artificial Intelligence in Education, Y. B. Kafai, W. A. Sandoval, and N. Enyedy, Eds. Springer, 441–450.

FODOR, J. A. AND PYLYSHYN, Z. W. 1988. Connectionism and cognitive architecture: A critical analysis. Cognition 28, 1-2, 3–71. GONZ´A LEZ-BRENES, J., HUANG, Y., AND BRUSILOVSKY, P. 2014. General features in knowledge tracing to model multiple subskills, temporal item response theory, and expert knowledge. In Proceedings of the 7th International Conference on Educational Data Mining, J. Stamper and Z. A. Pardos, Eds. 84–91.

GONZ´A LEZ-BRENES, J. P. AND MOSTOW, J. 2012. Dynamic cognitive tracing: Towards unified discovery of student and cognitive models. In Proceedings of the 5th International Conference on Educational Data Mining, K. Yacef and O. Zaiane, Eds. ERIC.

HART, P. E., NILSSON, N. J., AND RAPHAEL, B. 1968. A formal basis for the heuristic determination of minimum cost paths. IEEE transactions on Systems Science and Cybernetics 4, 2, 100–107.

HEFFERNAN, N. T. AND HEFFERNAN, C. L. 2014. The ASSISTments ecosystem: Building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching. International Journal of Artificial Intelligence in Education 24, 4, 470–497.

KHAJAH, M., WING, R., LINDSEY, R., AND MOZER, M. 2014. Incorporating latent factors into knowledge tracing to predict individual differences in learning. In Proceedings of the 7th International Conference on Educational Data Mining, J. Stamper and Z. A. Pardos, Eds. 99–106.

KOEDINGER, K. R., D’MELLO, S., MCLAUGHLIN, E. A., PARDOS, Z. A., AND ROS´E, C. P. 2015. Data mining and education. Wiley Interdisciplinary Reviews: Cognitive Science 6, 4, 333–353.

KOEDINGER, K. R., MCLAUGHLIN, E. A., AND STAMPER, J. C. 2012. Automated student model improvement. In Proceedings of the 5th International Conference on Educational Data Mining, K. Yacef and O. Zaiane, Eds. 17–24.

KRIZHEVSKY, A., SUTSKEVER, I., AND HINTON, G. E. 2012. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, L. B. F. Pereira, C. J. C. Burges and K. Q. Weinberger, Eds. 1097–1105.

LECUN, Y., BENGIO, Y., AND HINTON, G. 2015. Deep learning. Nature 521, 7553, 436–444.

LEVY, O. AND GOLDBERG, Y. 2014. Linguistic regularities in sparse and explicit word representations. In Proceedings of the 18th Conference on Computational Natural Language Learning, M. Kay, Ed. 171–180.

LIU, J., XU, G., AND YING, Z. 2012. Data-driven learning of Q-matrix. Applied psychological measurement 36, 7, 548–564.

LIU, R., PATEL, R., AND KOEDINGER, K. R. 2016. Modeling common misconceptions in learning process data. In Proceedings of the 6th International Conference on Learning Analytics & Knowledge, S. Dawson, H. Drachsler, and C. P. Ros´e, Eds. ACM, 369–377.

MARTIN, B., MITROVIC, A., KOEDINGER, K. R., AND MATHAN, S. 2011. Evaluating and improving adaptive educational systems with learning curves. User Modeling and User-Adapted Interaction 21, 3, 249–283.

MIKOLOV, T., SUTSKEVER, I., CHEN, K., CORRADO, G. S., AND DEAN, J. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, Eds. 3111–3119.

NATHAN, M. J., KOEDINGER, K. R., AND ALIBALI, M. W. 2001. Expert blind spot: When content knowledge eclipses pedagogical content knowledge. In Proceedings of the 3rd International Conference on Cognitive Science, J. Mu, Q. Liang, W. Wang, B. Zhang, and Y. Pi, Eds. Beijing: University of Science and Technology of China Press, 644–648.

PARDOS, Z. A. 2017. Big data in education and the models that love them. Current Opinion in Behavioral Sciences 18, 107–113.

PARDOS, Z. A. AND DADU, A. 2017. Imputing KCs with representations of problem content and context. In Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization, F. Cena and M. Desmarias, Eds. ACM, 148–155.

PAVLIK JR, P. I., CEN, H., AND KOEDINGER, K. R. 2009a. Learning factors transfer analysis: Using learning curve analysis to automatically generate domain models. In Proceedings of the 2nd International Conference on Educational Data Mining, T. Barnes and M. Desmarias, Eds. 121–130.

PAVLIK JR, P. I., CEN, H., AND KOEDINGER, K. R. 2009b. Performance factors analysis–a new alternative to knowledge tracing. In Proceedings of the 14th International Conference on Artificial Intelligence in Education, D. Dicheva and D. Dochev, Eds. IOS Press, 531–538.

PEL´ANEK, R. 2015. Metrics for evaluation of student models. Journal of Educational Data Mining 7, 2, 1–19.

PEL´ANEK, R. 2017. Bayesian knowledge tracing, logistic models, and beyond: an overview of learner modeling techniques. User Modeling and User-Adapted Interaction 27, 3-5, 313–350.

PIECH, C., BASSEN, J., HUANG, J., GANGULI, S., SAHAMI, M., GUIBAS, L. J., AND SOHLDICKSTEIN, J. 2015. Deep knowledge tracing. In Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, Eds. 505–513.

RASCH, G. 1961. On general laws and the meaning of measurement in psychology. In Proceedings of the 4th Berkeley Symposium on Mathematical Statistics and Probability, J. Neyman, Ed. Vol. 4. 321–333.

RITTER, F. E. AND SCHOOLER, L. J. 2001. The learning curve. International Encyclopedia of the Social and Behavioral Sciences 13, 8602–8605.

RITTER, S., ANDERSON, J. R., KOEDINGER, K. R., AND CORBETT, A. 2007. Cognitive Tutor: Applied research in mathematics education. Psychonomic Bulletin & Review 14, 2, 249–255.

ROS´E , C. P., DONMEZ, P., GWEON, G., KNIGHT, A., JUNKER, B., COHEN, W. W., KOEDINGER, K. R., AND HEFFERNAN, N. T. 2005. Automatic and semi-automatic skill coding with a view towards supporting on-line assessment. In Proceedings of the 12th International Conference on Artificial Intelligence in Education, G. McCalla and C. Looi, Eds. IOS Press, 571–578.

RUMELHART, D. E., HINTON, G. E., AND WILLIAMS, R. J. 1986. Learning representations by backpropagating errors. Nature 323, 533–536.

SAHEBI, S., LIN, Y.-R., AND BRUSILOVSKY, P. 2016. Tensor factorization for student modeling and performance prediction in unstructured domain. In Proceedings of the 9th International Conference on Educational Data Mining, M. Chi and M. Feng, Eds. 502–506.

SCHEINES, R., SILVER, E., AND GOLDIN, I. M. 2014. Discovering prerequisite relationships among knowledge components. In Proceedings of the 7th International Conference on Educational Data Mining, J. Stamper and Z. A. Pardos, Eds. 355–356.

SHAVER, K. G. 2012. The attribution of blame: Causality, responsibility, and blameworthiness. Springer Science & Business Media.

STAMPER, J. AND PARDOS, Z. A. 2016. The 2010 KDD cup competition dataset: Engaging the machine learning community in predictive learning analytics. Journal of Learning Analytics 3, 2, 312–316.

SUN, Y., YE, S., INOUE, S., AND SUN, Y. 2014. Alternating recursive method for Q-matrix learning. In Proceedings of the 7th International Conference on Educational Data Mining, J. Stamper and Z. A. Pardos, Eds. 14–20.

TATSUOKA, K. K. 1983. Rule space: An approach for dealing with misconceptions based on item response theory. Journal of educational measurement 20, 4, 345–354.

WILLIAMS, R. J. AND ZIPSER, D. 1989. A learning algorithm for continually running fully recurrent neural networks. Neural Computation 1, 2, 270–280.

YUDELSON, M. V., KOEDINGER, K. R., AND GORDON, G. J. 2013. Individualized Bayesian knowledge tracing models. In Proceedings of 16th International Conference on Artificial Intelligence in Education, H. Lane, K. Yacef, J. Mostow, and P. Pavlik, Eds. Springer, 171–180.
Section
EDM 2018 Journal Track