Predictiveness of Prior Failures is Improved by Incorporating Trial Duration



Published Sep 30, 2019
Luke Glenn Eglington Philip I. Pavlik, Jr


In recent years, there has been a proliferation of adaptive learner models that seek to predict student correctness. Improvements on earlier models have shown that separate predictors for prior successes, failures, and recent performance further improve fit while remaining interpretable. However, students who engage in “gaming” or other off-task behaviors may reduce the predictiveness of learner models that treat counts of prior performance equivalently across gaming and non-gaming student populations. The present research evaluated how sub-groups of students that varied in their potential gaming behavior were differently fit by a logistic learner model, and whether any observed differences between sub-groups could inspire the creation of new predictors that might improve model fit. Student data extracted from a college-level online learning application were clustered according to speed and accuracy using Gaussian mixture modeling. Distinct clusters were found, with similar cluster patterns detected in three separate datasets. Subsequently, each cluster was separately fit to a Performance Factors Analysis model (PFA). Significantly different parameter coefficients across clusters implied that students more likely to have been gaming benefitted less from prior failures. These differences inspired new and modified predictors that were found to improve overall model fit - an improvement that varied in magnitude across clusters. The present findings indicate that incorporating trial duration into counts of prior failures can improve the predictive power of learning models.

How to Cite

Eglington, L. G., & Pavlik, Jr, P. I. (2019). Predictiveness of Prior Failures is Improved by Incorporating Trial Duration. JEDM | Journal of Educational Data Mining, 11(2), 1-19.
Abstract 318 | PDF Downloads 287



learning, learner models, clustering, student variability, feature engineering, Performance Factors Analysis

ALEVEN, V. AND KOEDINGER, K. R. 2001. Investigations into help seeking and learning with a cognitive tutor. In Proceedings of the Papers of the AIED-2001 workshop on help provision and help seeking in interactive learning environments, 47-58.

AYERS, E. AND JUNKER, B. 2006. Do skills combine additively to predict task difficulty in eighth grade mathematics? In Educational Data Mining: Papers from the AAAI Workshop, J. Beck, E. Aimeur and T. Barnes Eds. AAAI Press, Menlo Park, CA, 14-20.

BAKER, R., WALONOSKI, J., HEFFERNAN, N., ROLL, I., CORBETT, A. AND KOEDINGER, K. 2008. Why Students Engage in “Gaming the System” Behavior in Interactive Learning Environments. Journal of Interactive Learning Research 19, 185-224.

BAKER, R. S., CORBETT, A. T. AND KOEDINGER, K. R. 2004. Detecting Student Misuse of Intelligent Tutoring Systems. In Proceedings of the Intelligent Tutoring Systems, Berlin, Heidelberg, J. C. Lester, R. M. Vicari and F. Paraguaçu Eds. Springer Berlin Heidelberg, 531-540.

BAKER, R. S. J. D., CORBETT, A. T., KOEDINGER, K. R., EVENSON, S., ROLL, I., WAGNER, A. Z., NAIM, M., RASPAT, J., BAKER, D. J. AND BECK, J. E. 2006. Adapting to When Students Game an Intelligent Tutoring System. In Proceedings of the Intelligent Tutoring Systems, Berlin, Heidelberg, M. Ikeda, K. D. Ashley and T.-W. Chan Eds. Springer Berlin Heidelberg, 392-401.

BAKER, R. S. J. D., GOLDSTEIN, A. B. AND HEFFERNAN, N. T. 2011. Detecting learning moment-by-moment. International Journal of Artificial Intelligence and Education 21, 5-25.

BATES, D., MÄCHLER, M., BOLKER, B. AND WALKER, S. 2014. Fitting linear mixed-effects models using lme4. arXiv preprint arXiv:1506.05908.

BUTLER, A. C., FAZIO, L. K. AND MARSH, E. J. 2011. The hypercorrection effect persists over a week, but high-confidence errors return. Psychonomic Bulletin & Review 18, 1238-1244.

CARRIER, M. AND PASHLER, H. 1992. The influence of retrieval on retention. Memory & Cognition 20, 633-642.

CEN, H., KOEDINGER, K. R. AND JUNKER, B. 2006. Learning Factors Analysis - A general method for cognitive model evaluation and improvement. In Proceedings of the 8th International Conference on Intelligent Tutoring Systems Springer Berlin / Heidelberg, 164-175.

CHI, M., KOEDINGER, K. R., GORDON, G., JORDAN, P. AND VANLEHN, K. 2011. Instructional Factors Analysis: A cognitive model for multiple instructional interventions. In Proceedings of the 4th International Conference on Educational Data Mining, Eindhoven, The Netherlands, M. Pechenizkiy, T. Calders, C. Conati, S. Ventura, C. Romero and J. Stamper Eds., 61-70.

CORBETT, A. T. AND ANDERSON, J. R. 1992. Student modeling and mastery learning in a computer-based programming tutor. In Intelligent Tutoring Systems: Second International Conference on Intelligent Tutoring Systems, C. Frasson, G. Gauthier and G. Mccalla Eds. Springer-Verlag, New York, 413-420.

DE BOECK, P., BAKKER, M., ZWITSER, R., NIVARD, M., HOFMAN, A., TUERLINCKX, F. AND PARTCHEV, I. 2011. The estimation of item response models with the lmer function from the lme4 package in R. Journal of Statistical Software 39, 1-28.

DESMARAIS, M. C. AND LEMIEUX, F. 2013. Clustering and Visualizing Study State Sequences. In Proceedings of the 6th International Conference of Educational Datamining, S. K. D'mello, R. A. Calvo and A. Olney Eds., Memphis. TN, 224-227.

FISCHER, G. H. 1973. The linear logistic test model as an instrument in educational research. Acta Psychologica 37, 359-374.

FRALEY, C., RAFTERY, A. E., MURPHY, T. B. AND SCRUCCA, L. 2012. mclust Version 4 for R: Normal Mixture Modeling for Model-Based Clustering, Classification, and Density Estimation.

GALYARDT, A. AND GOLDIN, I. 2015. Move your lamp post: Recent data reflects learner knowledge better than older data. Journal of Educational Data Mining 7, 83-108.

GONG, Y., BECK, J. E. AND HEFFERNAN, N. T. 2011. How to Construct More Accurate Student Models: Comparing and Optimizing Knowledge Tracing and Performance Factor Analysis. International Journal of Artificial Intelligence in Education 21, 27-46.

GRIMALDI, P. J. AND KARPICKE, J. D. 2014. Guided retrieval practice of educational materials using automated scoring. Journal of Educational Psychology 106, 58-68.

IZAWA, C. 1970. Optimal potentiating effects and forgetting-prevention effects of tests in paired-associate learning. Journal of Experimental Psychology 83, 340-344.

JOHNSON, P. C. D. 2014. Extension of Nakagawa & Schielzeth's R2GLMM to random slopes models. Methods in Ecology and Evolution 5, 944-946.

KANG, S. H. K. AND PASHLER, H. 2014. Is the benefit of retrieval practice modulated by motivation? Journal of Applied Research in Memory and Cognition 3, 183-188.

KELLY, J. W., CARPENTER, S. K. AND SJOLUND, L. A. 2015. Retrieval enhances route knowledge acquisition, but only when movement errors are prevented. Journal of Experimental Psychology: Learning, Memory, and Cognition 41, 1540-1547.

KORNELL, N. AND BJORK, R. A. 2007. The promise and perils of self-regulated study. Psychonomic Bulletin & Review 14, 219-224.

KORNELL, N., HAYS, M. J. AND BJORK, R. A. 2009. Unsuccessful retrieval attempts enhance subsequent learning. Journal of Experimental Psychology: Learning, Memory, & Cognition 35, 989-998.

MCCABE, J. 2011. Metacognitive awareness of learning strategies in undergraduates. Memory & Cognition 39, 462-476.

MORRIS, C. D., BRANSFORD, J. D. AND FRANKS, J. J. 1977. Levels of processing versus transfer appropriate processing. Journal of Verbal Learning and Verbal Behavior 16, 519-533.

NAKAGAWA, S., JOHNSON, P. C. D. AND SCHIELZETH, H. 2017. The coefficient of determination R(2) and intra-class correlation coefficient from generalized linear mixed-effects models revisited and expanded. J R Soc Interface 14.

NAKAGAWA, S. AND SCHIELZETH, H. 2013. A general and simple method for obtaining R2 from generalized linear mixed-effects models. Methods in Ecology and Evolution 4, 133-142.

NEWELL, A. AND ROSENBLOOM, P. S. 1981. Mechanisms of skill acquisition and the law of practice. Cognitive skills and their acquisition 1, 1-55.

PARDOS, Z. A., TRIVEDI, S., HEFFERNAN, N. T. AND SÁRKÖZY, G. N. 2012. Clustered Knowledge Tracing. In Proceedings of the Intelligent Tutoring Systems, Berlin, Heidelberg, S. A. Cerri, W. J. Clancey, G. Papadourakis and K. Panourgia Eds. Springer Berlin Heidelberg, 405-410.

PAVLIK JR., P. I. 2007. Understanding and applying the dynamics of test practice and study practice. Instructional Science 35, 407–441.

PAVLIK JR., P. I. AND ANDERSON, J. R. 2008. Using a model to compute the optimal schedule of practice. Journal of Experimental Psychology: Applied 14, 101–117.

PAVLIK JR., P. I., CEN, H. AND KOEDINGER, K. R. 2009. Performance factors analysis -- A new alternative to knowledge tracing. In Proceedings of the 14th International Conference on Artificial Intelligence in Education, V. Dimitrova, R. Mizoguchi, B. D. Boulay and A. Graesser Eds., Brighton, England, 531–538.

PAVLIK JR., P. I., YUDELSON, M. AND KOEDINGER, K. R. 2011. Using contextual factors analysis to explain transfer of least common multiple skills. In Artificial Intelligence in Education, G. Biswas, S. Bull, J. Kay and A. Mitrovic Eds. Springer, Berlin, Germany, 256–263.

PIECH, C., SPENCER, J., HUANG, J., GANGULI, S., SAHAMI, M., GUIBAS, L. AND SOHL-DICKSTEIN, J. 2015. Deep Knowledge Tracing. arXiv preprint arXiv:1506.05908.

RASCH, G. 1961. On general laws and the meaning of measurement in psychology. In Proceedings of the Proceedings of the fourth Berkeley symposium on mathematical statistics and probability University of California Press Berkeley, CA, 321-333.

RIHÁK, J. 2015. Use of Time Information in Models behind Adaptive System for Building Fluency in Mathematics. In Proceedings of the 8th International Conference on Educational Data Mining, O. C. Santos, J. G. Boticario, C. Romero, M. Pechenizkiy, A. Merceron, P. Mitros, . . . M. Desmarais Eds., Madrid, Spain, 642-644.

ROEDIGER, H. L. AND BUTLER, A. C. 2011. The critical role of retrieval practice in long-term retention. Trends in Cognitive Sciences 15, 20-27.

ROWLAND, C. A. 2014. The effect of testing versus restudy on retention: A meta-analytic review of the testing effect. Psychological Bulletin 140, 1432-1463.

SHIH, B., KOEDINGER, K. R. AND SCHEINES, R. 2008. A response time model for bottom-out hints as worked examples. In Proceedings of the 1st International Conference on Educational Data Mining, R. S. Baker and J. E. Beck Eds., Montreal, Canada, 117–126.

WOOD, H. AND WOOD, D. 1999. Help seeking, learning and contingent tutoring. Computers & Education 33, 153-169.

YAN, V. X., BJORK, E. L. AND BJORK, R. A. 2016. On the difficulty of mending metacognitive illusions: A priori theories, fluency effects, and misattributions of the interleaving benefit. J Exp Psychol Gen 145, 918-933.
EDM 2019 Journal Track