Tracing Systematic Errors to Personalize Recommendations in Single Digit Multiplication and Beyond
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
Dec 28, 2021
Alexander O. Savi
Benjamin E. Deonovic
Maria Bolsinova
Han L. J. van der Maas
Gunter K. J. Maris
https://orcid.org/0000-0002-9271-7476
Abstract
In learning, errors are ubiquitous and inevitable. As these errors may signal otherwise latent cognitive processes, tutors - and students alike - can greatly benefit from the information they provide. In this paper, we introduce and evaluate the Systematic Error Tracing (SET) model that identifies the possible causes of systematically observed errors in domains where items are susceptible to most or all causes and errors can be explained by multiple causes. We apply the model to single-digit multiplication, a domain that is very suitable for the model, is well-studied, and allows us to analyze over 25,000 error responses from 335 learners. The model, derived from the Ising model popular in physics, makes use of a bigraph that links errors to causes. The error responses were taken from Math Garden, a computerized adaptive practice environment for arithmetic that is widely used in the Netherlands. We discuss and evaluate various model configurations with respect to the ranking of recommendations and calibration of probability estimates. The results show that the SET model outranks a majority vote baseline model when more than a single recommendation is considered. Finally, we contrast the SET model to similar approaches and discuss limitations and implications.
How to Cite
Savi, A. O., Deonovic, B. E., Bolsinova, M., van der Maas, H. L. J., & Maris, G. K. J. (2021). Tracing Systematic Errors to Personalize Recommendations in Single Digit Multiplication and Beyond. Journal of Educational Data Mining, 13(4), 1–30. https://doi.org/10.5281/zenodo.5806832
##plugins.themes.bootstrap3.article.details##
Keywords
computerized adaptive practice, Ising model, learning diagnosis, recommendation system, ranking and calibration evaluation
References
BEN-ZEEV, T. 1995. The nature and origin of rational errors in arithmetic thinking: Induction from examples and prior knowledge. Cognitive Science 19, 3, 341–376.
BEN-ZEEV, T. 1998. Rational errors and the mathematical mind. Review of General Psychology 2, 4, 366–383.
BRADSHAW, L. AND TEMPLIN, J. L. 2013. Combining item response theory and diagnostic classification models: A psychometric model for scaling ability and diagnosing misconceptions. Psychometrika 79, 3, 403–425.
BRAITHWAITE, D. W., PYKE, A. A., AND SIEGLER, R. S. 2017. A computational model of fraction arithmetic. Psychological Review 124, 5, 603–625.
BRIER, G. W. 1950. Verification of forecasts expressed in terms of probability. Monthly Weather Re-view 78, 1, 1–3.
BRINKHUIS, M., SAVI, A., HOFMAN, A. D., COOMANS, F., VAN DER MAAS, H. L. J., AND MARIS, G. 2018. Learning as it happens: A decade of analyzing and shaping a large-scale online learning system. Journal of Learning Analytics 5, 2, 29–46.
BROWN, J. S. AND BURTON, R. R. 1978. Diagnostic models for procedural bugs in basic mathematical skills. Cognitive Science 2, 2, 155–192.
BUWALDA, T., BORST, J., VAN DER MAAS, H. L. J., AND TAATGEN, N. 2016. Explaining mistakes in single digit multiplication: A cognitive model. In Proceedings of the 14th International Conference on Cognitive Modeling, D. Reitter and F. E. Ritter, Eds. 131–136.
CHEN, J. AND DE LA TORRE, J. 2018. Introducing the general polytomous diagnosis modeling frame-work. Frontiers in Psychology 9, 1474.
CONATI, C., GERTNER, A., AND VANLEHN, K. 2002. Using Bayesian networks to manage uncertainty in student modeling. User Modeling and User-Adapted Interaction 12, 4, 371–417.
CONWAY, J. R., LEX, A., AND GEHLENBORG, N. 2017. UpSetR: an R package for the visualization of intersecting sets and their properties. Bioinformatics 33, 18, 2938–2940.
CORBETT, A. T. AND ANDERSON, J. R. 1995. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction 4, 4, 253–278.
COWELL, R. G., DAWID, P., LAURITZEN, S. L., AND SPIEGELHALTER, D. J. 1999. Probabilistic Networks and Expert Systems. Information Science and Statistics. Springer-Verlag New York.
COX, D. R. AND WERMUTH, N. 1994. A note on the quadratic exponential binary distribution. Biometrika 81, 2, 403–408.
DE MOOIJ, S. M. M., RAIJMAKERS, M. E. J., DUMONTHEIL, I., KIRKHAM, N. Z., AND VAN DERMAAS, H. L. J. 2021. Error detection through mouse movement in an online adaptive learning environment. Journal of Computer Assisted Learning 37, 1, 242–252.
DESMARAIS, M. C. AND BAKER, R. S. J. D. 2011. A review of recent advances in learner and skill modeling in intelligent learning environments. User Modeling and User-Adapted Interaction 22, 1-2, 9–38.
GU, Y. AND XU, G. 2018. The sufficient and necessary condition for the identifiability and estimability of the DINA model. Psychometrika 84, 2, 468–483.
HAERTEL, E. H. 1989. Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement 26, 4, 301–321.
ISING, E. 1925. Beitrag zur theorie des ferromagnetismus. Zeitschrift fur Physik 31, 1, 253–258.
KHAJAH, M., LINDSEY, R. V., AND MOZER, M. C. 2016. How deep is knowledge tracing? In Proceedings of the 9th International Conference on Educational Data Mining, T. Barnes, M. Chi, and M. Feng, Eds. 94–101.
KLINKENBERG, S., STRAATEMEIER, M., AND VAN DER MAAS, H. L. J. 2011. Computer adaptive practice of maths ability using a new item response model for on the fly ability and difficulty estimation. Computers & Education 57, 2, 1813–1824.
KRUIS, J. AND MARIS, G. 2016. Three representations of the Ising model. Scientific Reports 6, 1, 1–11.
KUO, B.-C., CHEN, C.-H., AND DE LA TORRE, J. 2017. A cognitive diagnosis model for identifying coexisting skills and misconceptions. Applied Psychological Measurement 42, 3, 179–191.
KUO, B.-C., CHEN, C.-H., YANG, C.-W., AND MOK, M. M. C. 2016. Cognitive diagnostic models for tests with multiple-choice and constructed-response items. Educational Psychology 36, 6, 1115–1133.
LIU, J., XU, G., AND YING, Z. 2012. Data-driven learning of q-matrix. 36, 7, 548–564.
LUCE, R. D. 2005. Individual choice behavior: A theoretical analysis. Dover Publications.
MARIS, G. AND VAN DER MAAS, H. L. J. 2012. Speed-accuracy response models: Scoring rules based on response time and accuracy. Psychometrika 77, 4, 615–633.
MCCLOSKEY, M., HARLEY, W., AND SOKOL, S. M. 1991. Models of arithmetic fact retrieval: An evaluation in light of findings from normal and brain-damaged subjects. Journal of Experimental Psychology: Learning, Memory, and Cognition 17, 3, 377–397.
MITROVIC, A. 2011. Fifteen years of constraint-based tutors: what we have achieved and where we are going. User Modeling and User-Adapted Interaction 22, 1-2, 39–72.
MULLER, D. A., BEWES, J., SHARMA, M. D., AND REIMANN, P. 2007a. Saying the wrong thing: improving learning with multimedia by including misconceptions. Journal of Computer Assisted Learning 24, 2, 144–155.
MULLER, D. A., SHARMA, M. D., EKLUND, J., AND REIMANN, P. 2007b. Conceptual change through vicarious learning in an authentic physics setting. Instructional Science 35, 6, 519–533.
NORMAN, D. A. 1981. Categorization of action slips. Psychological Review 88, 1, 1–15.
PEARL, J. 1988. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann.
PELANEK, R. 2017. Bayesian knowledge tracing, logistic models, and beyond: An overview of learner modeling techniques. User Modeling and User-Adapted Interaction 27, 3-5, 313–350.
PIECH, C., BASSEN, J., HUANG, J., GANGULI, S., SAHAMI, M., GUIBAS, L. J., AND SOHL-DICKSTEIN, J. 2015. Deep knowledge tracing. In Advances in Neural Information Processing Systems 28, C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, Eds. Curran Associates,Inc., 505–513.
REBER, R., BRUN, M., AND MITTERNDORFER, K. 2008. The use of heuristics in intuitive mathematical judgment. Psychonomic Bulletin & Review 15, 6, 1174–1178.
SHAKI, S. AND FISCHER, M. H. 2017. Competing biases in mental arithmetic: When division is more and multiplication is less. Frontiers in Human Neuroscience 11, 37.
STRAATEMEIER, M. 2014. Math garden: A new educational and scientific instrument. Ph. D. thesis.
TARAGHI, B., FREY, M., SARANTI, A., EBNER, M., M¨ULLER, V., AND GROSSMANN, A. 2015. Determining the causing factors of errors for multiplication problems. In Communications in Computer and Information Science. Springer International Publishing, 27–38.
TARAGHI, B., SARANTI, A., LEGENSTEIN, R., AND EBNER, M. 2016. Bayesian modeling of student misconceptions in the one-digit multiplication with probabilistic programming. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge. ACM Press, 449–453.
TATSUOKA, K. K. 1983. Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement 20, 4, 345–354.
TEMPLIN, J. L. AND HENSON, R. A. 2006. Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods 11, 3, 287–305.
VAN DER VEN, S. H. G., KLAIBER, J. D., AND VAN DER MAAS, H. L. J. 2017. Four and twenty blackbirds: How transcoding ability mediates the relationship between visuospatial working memory and math in a language with inversion. Educational Psychology 37, 4, 1–24.
VANLEHN, K. 1986. Arithmetic procedures are induced from examples. In Conceptual and procedural knowledge: The case of mathematics, J. Hiebert, Ed. Hillsdale, NJ: Lawrence Erlbaum Associates,133–179.
VOMLEL, J. 2004. Bayesian networks in educational testing. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 12, supp01, 83–100.
XU, Y. AND MOSTOW, J. 2011. Using logistic regression to trace multiple sub-skills in a dynamic Bayes net. In Proceedings of the 4th International Conference on Educational Data Mining, M. Pechenizkiy,T. Calders, C. Conati, S. Ventura, C. Romero, and J. Stamper, Eds. 241–246.
BEN-ZEEV, T. 1998. Rational errors and the mathematical mind. Review of General Psychology 2, 4, 366–383.
BRADSHAW, L. AND TEMPLIN, J. L. 2013. Combining item response theory and diagnostic classification models: A psychometric model for scaling ability and diagnosing misconceptions. Psychometrika 79, 3, 403–425.
BRAITHWAITE, D. W., PYKE, A. A., AND SIEGLER, R. S. 2017. A computational model of fraction arithmetic. Psychological Review 124, 5, 603–625.
BRIER, G. W. 1950. Verification of forecasts expressed in terms of probability. Monthly Weather Re-view 78, 1, 1–3.
BRINKHUIS, M., SAVI, A., HOFMAN, A. D., COOMANS, F., VAN DER MAAS, H. L. J., AND MARIS, G. 2018. Learning as it happens: A decade of analyzing and shaping a large-scale online learning system. Journal of Learning Analytics 5, 2, 29–46.
BROWN, J. S. AND BURTON, R. R. 1978. Diagnostic models for procedural bugs in basic mathematical skills. Cognitive Science 2, 2, 155–192.
BUWALDA, T., BORST, J., VAN DER MAAS, H. L. J., AND TAATGEN, N. 2016. Explaining mistakes in single digit multiplication: A cognitive model. In Proceedings of the 14th International Conference on Cognitive Modeling, D. Reitter and F. E. Ritter, Eds. 131–136.
CHEN, J. AND DE LA TORRE, J. 2018. Introducing the general polytomous diagnosis modeling frame-work. Frontiers in Psychology 9, 1474.
CONATI, C., GERTNER, A., AND VANLEHN, K. 2002. Using Bayesian networks to manage uncertainty in student modeling. User Modeling and User-Adapted Interaction 12, 4, 371–417.
CONWAY, J. R., LEX, A., AND GEHLENBORG, N. 2017. UpSetR: an R package for the visualization of intersecting sets and their properties. Bioinformatics 33, 18, 2938–2940.
CORBETT, A. T. AND ANDERSON, J. R. 1995. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction 4, 4, 253–278.
COWELL, R. G., DAWID, P., LAURITZEN, S. L., AND SPIEGELHALTER, D. J. 1999. Probabilistic Networks and Expert Systems. Information Science and Statistics. Springer-Verlag New York.
COX, D. R. AND WERMUTH, N. 1994. A note on the quadratic exponential binary distribution. Biometrika 81, 2, 403–408.
DE MOOIJ, S. M. M., RAIJMAKERS, M. E. J., DUMONTHEIL, I., KIRKHAM, N. Z., AND VAN DERMAAS, H. L. J. 2021. Error detection through mouse movement in an online adaptive learning environment. Journal of Computer Assisted Learning 37, 1, 242–252.
DESMARAIS, M. C. AND BAKER, R. S. J. D. 2011. A review of recent advances in learner and skill modeling in intelligent learning environments. User Modeling and User-Adapted Interaction 22, 1-2, 9–38.
GU, Y. AND XU, G. 2018. The sufficient and necessary condition for the identifiability and estimability of the DINA model. Psychometrika 84, 2, 468–483.
HAERTEL, E. H. 1989. Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement 26, 4, 301–321.
ISING, E. 1925. Beitrag zur theorie des ferromagnetismus. Zeitschrift fur Physik 31, 1, 253–258.
KHAJAH, M., LINDSEY, R. V., AND MOZER, M. C. 2016. How deep is knowledge tracing? In Proceedings of the 9th International Conference on Educational Data Mining, T. Barnes, M. Chi, and M. Feng, Eds. 94–101.
KLINKENBERG, S., STRAATEMEIER, M., AND VAN DER MAAS, H. L. J. 2011. Computer adaptive practice of maths ability using a new item response model for on the fly ability and difficulty estimation. Computers & Education 57, 2, 1813–1824.
KRUIS, J. AND MARIS, G. 2016. Three representations of the Ising model. Scientific Reports 6, 1, 1–11.
KUO, B.-C., CHEN, C.-H., AND DE LA TORRE, J. 2017. A cognitive diagnosis model for identifying coexisting skills and misconceptions. Applied Psychological Measurement 42, 3, 179–191.
KUO, B.-C., CHEN, C.-H., YANG, C.-W., AND MOK, M. M. C. 2016. Cognitive diagnostic models for tests with multiple-choice and constructed-response items. Educational Psychology 36, 6, 1115–1133.
LIU, J., XU, G., AND YING, Z. 2012. Data-driven learning of q-matrix. 36, 7, 548–564.
LUCE, R. D. 2005. Individual choice behavior: A theoretical analysis. Dover Publications.
MARIS, G. AND VAN DER MAAS, H. L. J. 2012. Speed-accuracy response models: Scoring rules based on response time and accuracy. Psychometrika 77, 4, 615–633.
MCCLOSKEY, M., HARLEY, W., AND SOKOL, S. M. 1991. Models of arithmetic fact retrieval: An evaluation in light of findings from normal and brain-damaged subjects. Journal of Experimental Psychology: Learning, Memory, and Cognition 17, 3, 377–397.
MITROVIC, A. 2011. Fifteen years of constraint-based tutors: what we have achieved and where we are going. User Modeling and User-Adapted Interaction 22, 1-2, 39–72.
MULLER, D. A., BEWES, J., SHARMA, M. D., AND REIMANN, P. 2007a. Saying the wrong thing: improving learning with multimedia by including misconceptions. Journal of Computer Assisted Learning 24, 2, 144–155.
MULLER, D. A., SHARMA, M. D., EKLUND, J., AND REIMANN, P. 2007b. Conceptual change through vicarious learning in an authentic physics setting. Instructional Science 35, 6, 519–533.
NORMAN, D. A. 1981. Categorization of action slips. Psychological Review 88, 1, 1–15.
PEARL, J. 1988. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann.
PELANEK, R. 2017. Bayesian knowledge tracing, logistic models, and beyond: An overview of learner modeling techniques. User Modeling and User-Adapted Interaction 27, 3-5, 313–350.
PIECH, C., BASSEN, J., HUANG, J., GANGULI, S., SAHAMI, M., GUIBAS, L. J., AND SOHL-DICKSTEIN, J. 2015. Deep knowledge tracing. In Advances in Neural Information Processing Systems 28, C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, Eds. Curran Associates,Inc., 505–513.
REBER, R., BRUN, M., AND MITTERNDORFER, K. 2008. The use of heuristics in intuitive mathematical judgment. Psychonomic Bulletin & Review 15, 6, 1174–1178.
SHAKI, S. AND FISCHER, M. H. 2017. Competing biases in mental arithmetic: When division is more and multiplication is less. Frontiers in Human Neuroscience 11, 37.
STRAATEMEIER, M. 2014. Math garden: A new educational and scientific instrument. Ph. D. thesis.
TARAGHI, B., FREY, M., SARANTI, A., EBNER, M., M¨ULLER, V., AND GROSSMANN, A. 2015. Determining the causing factors of errors for multiplication problems. In Communications in Computer and Information Science. Springer International Publishing, 27–38.
TARAGHI, B., SARANTI, A., LEGENSTEIN, R., AND EBNER, M. 2016. Bayesian modeling of student misconceptions in the one-digit multiplication with probabilistic programming. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge. ACM Press, 449–453.
TATSUOKA, K. K. 1983. Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement 20, 4, 345–354.
TEMPLIN, J. L. AND HENSON, R. A. 2006. Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods 11, 3, 287–305.
VAN DER VEN, S. H. G., KLAIBER, J. D., AND VAN DER MAAS, H. L. J. 2017. Four and twenty blackbirds: How transcoding ability mediates the relationship between visuospatial working memory and math in a language with inversion. Educational Psychology 37, 4, 1–24.
VANLEHN, K. 1986. Arithmetic procedures are induced from examples. In Conceptual and procedural knowledge: The case of mathematics, J. Hiebert, Ed. Hillsdale, NJ: Lawrence Erlbaum Associates,133–179.
VOMLEL, J. 2004. Bayesian networks in educational testing. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 12, supp01, 83–100.
XU, Y. AND MOSTOW, J. 2011. Using logistic regression to trace multiple sub-skills in a dynamic Bayes net. In Proceedings of the 4th International Conference on Educational Data Mining, M. Pechenizkiy,T. Calders, C. Conati, S. Ventura, C. Romero, and J. Stamper, Eds. 241–246.
Issue
Section
Articles
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors who publish with this journal agree to the following terms:
- The Author retains copyright in the Work, where the term “Work” shall include all digital objects that may result in subsequent electronic publication or distribution.
- Upon acceptance of the Work, the author shall grant to the Publisher the right of first publication of the Work.
- The Author shall grant to the Publisher and its agents the nonexclusive perpetual right and license to publish, archive, and make accessible the Work in whole or in part in all forms of media now or hereafter known under a Creative Commons 4.0 License (Attribution-Noncommercial-No Derivatives 4.0 International), or its equivalent, which, for the avoidance of doubt, allows others to copy, distribute, and transmit the Work under the following conditions:
- Attribution—other users must attribute the Work in the manner specified by the author as indicated on the journal Web site;
- Noncommercial—other users (including Publisher) may not use this Work for commercial purposes;
- No Derivative Works—other users (including Publisher) may not alter, transform, or build upon this Work,with the understanding that any of the above conditions can be waived with permission from the Author and that where the Work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.
- The Author is able to enter into separate, additional contractual arrangements for the nonexclusive distribution of the journal's published version of the Work (e.g., post it to an institutional repository or publish it in a book), as long as there is provided in the document an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post online a pre-publication manuscript (but not the Publisher’s final formatted PDF version of the Work) in institutional repositories or on their Websites prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (see The Effect of Open Access). Any such posting made before acceptance and publication of the Work shall be updated upon publication to include a reference to the Publisher-assigned DOI (Digital Object Identifier) and a link to the online abstract for the final published Work in the Journal.
- Upon Publisher’s request, the Author agrees to furnish promptly to Publisher, at the Author’s own expense, written evidence of the permissions, licenses, and consents for use of third-party material included within the Work, except as determined by Publisher to be covered by the principles of Fair Use.
- The Author represents and warrants that:
- the Work is the Author’s original work;
- the Author has not transferred, and will not transfer, exclusive rights in the Work to any third party;
- the Work is not pending review or under consideration by another publisher;
- the Work has not previously been published;
- the Work contains no misrepresentation or infringement of the Work or property of other authors or third parties; and
- the Work contains no libel, invasion of privacy, or other unlawful matter.
- The Author agrees to indemnify and hold Publisher harmless from Author’s breach of the representations and warranties contained in Paragraph 6 above, as well as any claim or proceeding relating to Publisher’s use and publication of any content contained in the Work, including third-party content.