Tracing Systematic Errors to Personalize Recommendations in Single Digit Multiplication and Beyond



Published Dec 28, 2021
Alexander O. Savi Benjamin E. Deonovic Maria Bolsinova Han L. J. van der Maas Gunter K. J. Maris


In learning, errors are ubiquitous and inevitable. As these errors may signal otherwise latent cognitive processes, tutors - and students alike - can greatly benefit from the information they provide. In this paper, we introduce and evaluate the Systematic Error Tracing (SET) model that identifies the possible causes of systematically observed errors in domains where items are susceptible to most or all causes and errors can be explained by multiple causes. We apply the model to single-digit multiplication, a domain that is very suitable for the model, is well-studied, and allows us to analyze over 25,000 error responses from 335 learners. The model, derived from the Ising model popular in physics, makes use of a bigraph that links errors to causes. The error responses were taken from Math Garden, a computerized adaptive practice environment for arithmetic that is widely used in the Netherlands. We discuss and evaluate various model configurations with respect to the ranking of recommendations and calibration of probability estimates. The results show that the SET model outranks a majority vote baseline model when more than a single recommendation is considered. Finally, we contrast the SET model to similar approaches and discuss limitations and implications.

How to Cite

Savi, A. O., Deonovic, B. E., Bolsinova, M., van der Maas, H. L. J., & Maris, G. K. J. (2021). Tracing Systematic Errors to Personalize Recommendations in Single Digit Multiplication and Beyond. Journal of Educational Data Mining, 13(4), 1–30.
Abstract 721 | PDF Downloads 436



computerized adaptive practice, Ising model, learning diagnosis, recommendation system, ranking and calibration evaluation

BEN-ZEEV, T. 1995. The nature and origin of rational errors in arithmetic thinking: Induction from examples and prior knowledge. Cognitive Science 19, 3, 341–376.

BEN-ZEEV, T. 1998. Rational errors and the mathematical mind. Review of General Psychology 2, 4, 366–383.

BRADSHAW, L. AND TEMPLIN, J. L. 2013. Combining item response theory and diagnostic classification models: A psychometric model for scaling ability and diagnosing misconceptions. Psychometrika 79, 3, 403–425.

BRAITHWAITE, D. W., PYKE, A. A., AND SIEGLER, R. S. 2017. A computational model of fraction arithmetic. Psychological Review 124, 5, 603–625.

BRIER, G. W. 1950. Verification of forecasts expressed in terms of probability. Monthly Weather Re-view 78, 1, 1–3.

BRINKHUIS, M., SAVI, A., HOFMAN, A. D., COOMANS, F., VAN DER MAAS, H. L. J., AND MARIS, G. 2018. Learning as it happens: A decade of analyzing and shaping a large-scale online learning system. Journal of Learning Analytics 5, 2, 29–46.

BROWN, J. S. AND BURTON, R. R. 1978. Diagnostic models for procedural bugs in basic mathematical skills. Cognitive Science 2, 2, 155–192.

BUWALDA, T., BORST, J., VAN DER MAAS, H. L. J., AND TAATGEN, N. 2016. Explaining mistakes in single digit multiplication: A cognitive model. In Proceedings of the 14th International Conference on Cognitive Modeling, D. Reitter and F. E. Ritter, Eds. 131–136.

CHEN, J. AND DE LA TORRE, J. 2018. Introducing the general polytomous diagnosis modeling frame-work. Frontiers in Psychology 9, 1474.

CONATI, C., GERTNER, A., AND VANLEHN, K. 2002. Using Bayesian networks to manage uncertainty in student modeling. User Modeling and User-Adapted Interaction 12, 4, 371–417.

CONWAY, J. R., LEX, A., AND GEHLENBORG, N. 2017. UpSetR: an R package for the visualization of intersecting sets and their properties. Bioinformatics 33, 18, 2938–2940.

CORBETT, A. T. AND ANDERSON, J. R. 1995. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction 4, 4, 253–278.

COWELL, R. G., DAWID, P., LAURITZEN, S. L., AND SPIEGELHALTER, D. J. 1999. Probabilistic Networks and Expert Systems. Information Science and Statistics. Springer-Verlag New York.

COX, D. R. AND WERMUTH, N. 1994. A note on the quadratic exponential binary distribution. Biometrika 81, 2, 403–408.

DE MOOIJ, S. M. M., RAIJMAKERS, M. E. J., DUMONTHEIL, I., KIRKHAM, N. Z., AND VAN DERMAAS, H. L. J. 2021. Error detection through mouse movement in an online adaptive learning environment. Journal of Computer Assisted Learning 37, 1, 242–252.

DESMARAIS, M. C. AND BAKER, R. S. J. D. 2011. A review of recent advances in learner and skill modeling in intelligent learning environments. User Modeling and User-Adapted Interaction 22, 1-2, 9–38.

GU, Y. AND XU, G. 2018. The sufficient and necessary condition for the identifiability and estimability of the DINA model. Psychometrika 84, 2, 468–483.

HAERTEL, E. H. 1989. Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement 26, 4, 301–321.

ISING, E. 1925. Beitrag zur theorie des ferromagnetismus. Zeitschrift fur Physik 31, 1, 253–258.

KHAJAH, M., LINDSEY, R. V., AND MOZER, M. C. 2016. How deep is knowledge tracing? In Proceedings of the 9th International Conference on Educational Data Mining, T. Barnes, M. Chi, and M. Feng, Eds. 94–101.

KLINKENBERG, S., STRAATEMEIER, M., AND VAN DER MAAS, H. L. J. 2011. Computer adaptive practice of maths ability using a new item response model for on the fly ability and difficulty estimation. Computers & Education 57, 2, 1813–1824.

KRUIS, J. AND MARIS, G. 2016. Three representations of the Ising model. Scientific Reports 6, 1, 1–11.

KUO, B.-C., CHEN, C.-H., AND DE LA TORRE, J. 2017. A cognitive diagnosis model for identifying coexisting skills and misconceptions. Applied Psychological Measurement 42, 3, 179–191.

KUO, B.-C., CHEN, C.-H., YANG, C.-W., AND MOK, M. M. C. 2016. Cognitive diagnostic models for tests with multiple-choice and constructed-response items. Educational Psychology 36, 6, 1115–1133.

LIU, J., XU, G., AND YING, Z. 2012. Data-driven learning of q-matrix. 36, 7, 548–564.

LUCE, R. D. 2005. Individual choice behavior: A theoretical analysis. Dover Publications.

MARIS, G. AND VAN DER MAAS, H. L. J. 2012. Speed-accuracy response models: Scoring rules based on response time and accuracy. Psychometrika 77, 4, 615–633.

MCCLOSKEY, M., HARLEY, W., AND SOKOL, S. M. 1991. Models of arithmetic fact retrieval: An evaluation in light of findings from normal and brain-damaged subjects. Journal of Experimental Psychology: Learning, Memory, and Cognition 17, 3, 377–397.

MITROVIC, A. 2011. Fifteen years of constraint-based tutors: what we have achieved and where we are going. User Modeling and User-Adapted Interaction 22, 1-2, 39–72.

MULLER, D. A., BEWES, J., SHARMA, M. D., AND REIMANN, P. 2007a. Saying the wrong thing: improving learning with multimedia by including misconceptions. Journal of Computer Assisted Learning 24, 2, 144–155.

MULLER, D. A., SHARMA, M. D., EKLUND, J., AND REIMANN, P. 2007b. Conceptual change through vicarious learning in an authentic physics setting. Instructional Science 35, 6, 519–533.

NORMAN, D. A. 1981. Categorization of action slips. Psychological Review 88, 1, 1–15.

PEARL, J. 1988. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann.

PELANEK, R. 2017. Bayesian knowledge tracing, logistic models, and beyond: An overview of learner modeling techniques. User Modeling and User-Adapted Interaction 27, 3-5, 313–350.

PIECH, C., BASSEN, J., HUANG, J., GANGULI, S., SAHAMI, M., GUIBAS, L. J., AND SOHL-DICKSTEIN, J. 2015. Deep knowledge tracing. In Advances in Neural Information Processing Systems 28, C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, Eds. Curran Associates,Inc., 505–513.

REBER, R., BRUN, M., AND MITTERNDORFER, K. 2008. The use of heuristics in intuitive mathematical judgment. Psychonomic Bulletin & Review 15, 6, 1174–1178.

SHAKI, S. AND FISCHER, M. H. 2017. Competing biases in mental arithmetic: When division is more and multiplication is less. Frontiers in Human Neuroscience 11, 37.

STRAATEMEIER, M. 2014. Math garden: A new educational and scientific instrument. Ph. D. thesis.

TARAGHI, B., FREY, M., SARANTI, A., EBNER, M., M¨ULLER, V., AND GROSSMANN, A. 2015. Determining the causing factors of errors for multiplication problems. In Communications in Computer and Information Science. Springer International Publishing, 27–38.

TARAGHI, B., SARANTI, A., LEGENSTEIN, R., AND EBNER, M. 2016. Bayesian modeling of student misconceptions in the one-digit multiplication with probabilistic programming. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge. ACM Press, 449–453.

TATSUOKA, K. K. 1983. Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement 20, 4, 345–354.

TEMPLIN, J. L. AND HENSON, R. A. 2006. Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods 11, 3, 287–305.

VAN DER VEN, S. H. G., KLAIBER, J. D., AND VAN DER MAAS, H. L. J. 2017. Four and twenty blackbirds: How transcoding ability mediates the relationship between visuospatial working memory and math in a language with inversion. Educational Psychology 37, 4, 1–24.

VANLEHN, K. 1986. Arithmetic procedures are induced from examples. In Conceptual and procedural knowledge: The case of mathematics, J. Hiebert, Ed. Hillsdale, NJ: Lawrence Erlbaum Associates,133–179.

VOMLEL, J. 2004. Bayesian networks in educational testing. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 12, supp01, 83–100.

XU, Y. AND MOSTOW, J. 2011. Using logistic regression to trace multiple sub-skills in a dynamic Bayes net. In Proceedings of the 4th International Conference on Educational Data Mining, M. Pechenizkiy,T. Calders, C. Conati, S. Ventura, C. Romero, and J. Stamper, Eds. 241–246.