Extending Adaptive Spacing Heuristics to Multi-Skill Items



Published Oct 31, 2021
Benoît Choffin Fabrice Popineau Yolaine Bourda


Adaptive spacing algorithms are powerful tools for helping learners manage their study time efficiently. By personalizing the temporal distribution of retrieval practice of a given piece of knowledge, they improve learners' long-term memory retention compared to fixed review schedules. However, such algorithms are generally designed for the pure memorization of single items, such as vocabulary words. Yet, the spacing effect has been shown to extend to more complex knowledge, such as the practice of mathematical skills. In this article, we extend three adaptive spacing heuristics from the literature for selecting the best skill to review at any timestamp given a student's past study history. In real-world educational settings, items generally involve multiple skills at the same time. Thus, we also propose a multi-skill version for two of these heuristics: instead of selecting one single skill, they select with a greedy procedure the most promising subset of skills to review. To compare these five heuristics, we develop a synthetic experimental framework that simulates student learning and forgetting trajectories with a student model. We run multiple synthetic experiments on large cohorts of 500 simulated students and publicly release the code for these experiments. Our results highlight the strengths and weaknesses of each heuristic in terms of performance, robustness, and complexity. Finally, we find evidence that selecting the best subset of skills yields better retention compared to selecting the single best skill to review.

How to Cite

Choffin, B., Popineau, F., & Bourda, Y. (2021). Extending Adaptive Spacing Heuristics to Multi-Skill Items. Journal of Educational Data Mining, 13(3), 69–102. https://doi.org/10.5281/zenodo.5634220
Abstract 63 | PDF Downloads 51



adaptive spacing, skill selection heuristics, knowledge components, multi-skill learning items

BARZAGAR NAZARI, K. AND EBERSBACH, M. 2019. Distributing mathematical practice of third and seventh graders: Applicability of the spacing effect in the classroom. Applied Cognitive Psychology 33, 2, 288–298.

BJORK, R. 1994. Memory and meta-memory considerations in the training of human beings. In Metacognition: Knowing about knowing, J. Metcalfe & A. Shimamura (Eds.), Ed. MIT Press, Cambridge, MA, US, 185–205.

CEPEDA, N. J., VUL, E., ROHRER, D., WIXTED, J. T., AND PASHLER, H. 2008. Spacing effects in learning: A temporal ridgeline of optimal retention. Psychological Science 19, 11, 1095–1102.

CHOFFIN, B., POPINEAU, F., BOURDA, Y., AND VIE, J. 2019. DAS3H: modeling student learning and forgetting for optimally scheduling distributed practice of skills. In Proceedings of the 12th International Conference on Educational Data Mining, M. C. Desmarais, C. F. Lynch, A. Merceron, and R. Nkambou, Eds. International Educational Data Mining Society, 29-38.

DOROUDI, S., ALEVEN, V., AND BRUNSKILL, E. 2017. Robust evaluation matrix: Towards a more principled offline exploration of instructional policies. In Proceedings of the Fourth ACM Conference on Learning@Scale. ACM, 3–12.

DOROUDI, S., ALEVEN, V., AND BRUNSKILL, E. 2019. Where’s the reward? International Journal of Artificial Intelligence in Education 29, 4, 568–620.

DUNLOSKY, J., RAWSON, K. A., MARSH, E. J., NATHAN, M. J., AND WILLINGHAM, D. T. 2013. Improving students’ learning with effective learning techniques: Promising directions from cognitive and educational psychology. Psychological Science in the Public Interest 14, 1, 4–58.

EBBINGHAUS, H. 1885. Ueber das Gedächtnis. Duncker and Humblot.

EFFENBERGER, T., PELÁNEK, R., AND ČECHÁK, J. 2020. Exploration of the robustness and generalizability of the additive factors model. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (LAK 2020). ACM, 472–479.

EGLINGTON, L. G. AND PAVLIK, JR, P. I. 2020. Optimizing practice scheduling requires quantitative tracking of individual item performance. npj Science of Learning 5, 1, 15.

GOUTTE, C., DURAND, G., AND LÉGER, S. 2018. On the learning curve attrition bias in additive factor modeling. In Proceedings of the 19th International Conference on Artificial Intelligence in Education. LNCS, vol. 10948. Springer, 109–113.

HUNZIKER, A., CHEN, Y., MAC AODHA, O., RODRIGUEZ, M. G., KRAUSE, A., PERONA, P., YUE, Y., AND SINGLA, A. 2019. Teaching multiple concepts to a forgetful learner. In Advances in Neural Information Processing Systems. 4050–4060.

KANG, S. H., LINDSEY, R. V., MOZER, M. C., AND PASHLER, H. 2014. Retrieval practice over the long term: Should spacing be expanding or equal-interval? Psychonomic Bulletin & Review 21, 6, 1544–1550.

KHAJAH, M. M., LINDSEY, R. V., AND MOZER, M. C. 2014. Maximizing students’ retention via spaced review: Practical guidance from computational models of memory. Topics in Cognitive Science 6, 1, 157–169.

LEITNER, S. 1972. So Lernt Man Lernen. Angewandte Lernpsychologie - ein Weg zum Erfolg. Herder.

LINDSEY, R. V., SHROYER, J. D., PASHLER, H., AND MOZER, M. C. 2014. Improving students’ long-term knowledge retention through personalized review. Psychological Science 25, 3, 639–647.

METTLER, E., MASSEY, C. M., AND KELLMAN, P. J. 2016. A comparison of adaptive and fixed schedules of practice. Journal of Experimental Psychology: General 145, 7, 897–917.

METZLER-BADDELEY, C. AND BADDELEY, R. J. 2009. Does adaptive training work? Applied Cognitive Psychology 23, 2, 254–266.

PASHLER, H., CEPEDA, N., LINDSEY, R. V., VUL, E., AND MOZER, M. C. 2009. Predicting the optimal spacing of study: A multiscale context model of memory. In Advances in Neural Information Processing Systems 22. Curran Associates, Inc., 1321–1329.

PAVLIK, P., BOLSTER, T., WU, S.-M., KOEDINGER, K., AND MACWHINNEY, B. 2008. Using optimally selected drill practice to train basic facts. In International Conference on Intelligent Tutoring Systems (ITS 2008). LNCS, vol. 5091. Springer, 593–602.

PAVLIK, P. I. AND ANDERSON, J. R. 2008. Using a model to compute the optimal schedule of practice. Journal of Experimental Psychology: Applied 14, 2, 101–117.

PIMSLEUR, P. 1967. A memory schedule. The Modern Language Journal 51, 2, 73–75.

REDDY, S., LABUTOV, I., BANERJEE, S., AND JOACHIMS, T. 2016. Unbounded human learning: Optimal scheduling for spaced repetition. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, B. Krishnapuram, M. Shah, A. J. Smola, C. C. Aggarwal, D. Shen, and R. Rastogi, Eds. ACM, 1815–1824.

REDDY, S., LEVINE, S., AND DRAGAN, A. 2017. Accelerating human learning with deep reinforcement learning. In NIPS’17 Workshop: Teaching Machines, Robots, and Humans.

ROEDIGER III, H. L. AND KARPICKE, J. D. 2011. Intricacies of spaced retrieval: A resolution. In Successful Remembering and Successful Forgetting, A. Benjamin, Ed. Psychology Press, 41–66.

ROLLINSON, J. AND BRUNSKILL, E. 2015. From Predictive Models to Instructional Policies. In Proceedings of the Eighth International Conference on Educational Data Mining, O. C. Santos, J. Boticario, C. Romero, M. Pechenizkiy, A. Merceron, P. Mitros, J. M. Luna, M. C. Mihaescu, P. Moreno, A. Hershkovitz, S. Ventura, and M. C. Desmarais, Eds. International Educational Data MiningSociety, 179–186.

ROWLAND, C. A. 2014. The effect of testing versus restudy on retention: a meta-analytic review of the testing effect. Psychological Bulletin 140, 6, 1432–1463.

STAMPER, J., NICULESCU-MIZIL, A., RITTER, S., GORDON, G., AND KOEDINGER, K. 2010. Algebra I 2005-2006 and Bridge to Algebra 2006-2007. Development data sets from KDD Cup 2010 Educational DataMining Challenge. Find them at http://pslcdatashop.web.cmu.edu/KDDCup/downloads.jsp.

TABIBIAN, B., UPADHYAY, U., DE, A., ZAREZADE, A., SCHÖLKOPF, B., AND GOMEZ-RODRIGUEZ, M. 2019. Enhancing human learning via spaced repetition optimization. Proceedings of the National Academy of Sciences 116, 10, 3988–3993.

TATSUOKA, K. K. 1983. Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement 20, 4, 345–354.

UPADHYAY, U., DE, A., AND RODRIGUEZ, M. G. 2018. Deep reinforcement learning of marked temporal point processes. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. Curran Associates Inc., 3168–3178.

VAN GOG, T. AND SWELLER, J. 2015. Not new, but nearly forgotten: the testing effect decreases or even disappears as the complexity of learning materials increases. Educational Psychology Review 27, 2, 247–264.

VAN RIJN, H., VAN MAANEN, L., AND VAN WOUDENBERG, M. 2009. Passing the test: Improving learning gains by balancing spacing and testing effects. In Proceedings of the 9th International Conference of Cognitive Modeling. Vol. 2. 7–6.

VIE, J. AND KASHIMA, H. 2019. Knowledge tracing machines: Factorization machines for knowledge tracing. In Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. AAAI Press, 750–757.

VLACH, H. A. AND SANDHOFER, C. M. 2012. Distributing learning over time: the spacing effect in children’s acquisition and generalization of science concepts. Child Development 83, 4, 1137–1144.

WANG, Z., LAMB, A., SAVELIEV, E., CAMERON, P., ZAYKOV, Y., HERNÁNDEZ-LOBATO, J. M., TURNER,R. E., BARANIUK, R. G., BARTON, C., JONES, S. P., WOODHEAD, S., AND ZHANG, C. 2020. Diagnostic questions: The NeurIPS 2020 Education Challenge.

WEINSTEIN, Y., MADAN, C. R., AND SUMERACKI, M. A. 2018. Teaching the science of learning. Cognitive Research: Principles and Implications 3, 1, 2.

WOZNIAK, P. AND GORZELANCZYK, E. J. 1994. Optimization of repetition spacing in the practice of learning. Acta Neurobiologiae Experimentalis 54, 59–59.

YANG, Z., SHEN, J., LIU, Y., YANG, Y., ZHANG, W., AND YU, Y. 2020. TADS: learning time-aware scheduling policy with dyna-style planning for spaced repetition. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, J. Huang, Y. Chang, X. Cheng, J. Kamps, V. Murdock, J. Wen, and Y. Liu, Eds. ACM, 1917–1920.
EDM 2021 Journal Track