Predictiveness of Prior Failures is Improved by Incorporating Trial Duration
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Abstract
In recent years, there has been a proliferation of adaptive learner models that seek to predict student correctness. Improvements on earlier models have shown that separate predictors for prior successes, failures, and recent performance further improve fit while remaining interpretable. However, students who engage in “gaming” or other off-task behaviors may reduce the predictiveness of learner models that treat counts of prior performance equivalently across gaming and non-gaming student populations. The present research evaluated how sub-groups of students that varied in their potential gaming behavior were differently fit by a logistic learner model, and whether any observed differences between sub-groups could inspire the creation of new predictors that might improve model fit. Student data extracted from a college-level online learning application were clustered according to speed and accuracy using Gaussian mixture modeling. Distinct clusters were found, with similar cluster patterns detected in three separate datasets. Subsequently, each cluster was separately fit to a Performance Factors Analysis model (PFA). Significantly different parameter coefficients across clusters implied that students more likely to have been gaming benefitted less from prior failures. These differences inspired new and modified predictors that were found to improve overall model fit - an improvement that varied in magnitude across clusters. The present findings indicate that incorporating trial duration into counts of prior failures can improve the predictive power of learning models.
How to Cite
##plugins.themes.bootstrap3.article.details##
learning, learner models, clustering, student variability, feature engineering, Performance Factors Analysis
AYERS, E. AND JUNKER, B. 2006. Do skills combine additively to predict task difficulty in eighth grade mathematics? In Educational Data Mining: Papers from the AAAI Workshop, J. Beck, E. Aimeur and T. Barnes Eds. AAAI Press, Menlo Park, CA, 14-20.
BAKER, R., WALONOSKI, J., HEFFERNAN, N., ROLL, I., CORBETT, A. AND KOEDINGER, K. 2008. Why Students Engage in “Gaming the System” Behavior in Interactive Learning Environments. Journal of Interactive Learning Research 19, 185-224.
BAKER, R. S., CORBETT, A. T. AND KOEDINGER, K. R. 2004. Detecting Student Misuse of Intelligent Tutoring Systems. In Proceedings of the Intelligent Tutoring Systems, Berlin, Heidelberg, J. C. Lester, R. M. Vicari and F. Paraguaçu Eds. Springer Berlin Heidelberg, 531-540.
BAKER, R. S. J. D., CORBETT, A. T., KOEDINGER, K. R., EVENSON, S., ROLL, I., WAGNER, A. Z., NAIM, M., RASPAT, J., BAKER, D. J. AND BECK, J. E. 2006. Adapting to When Students Game an Intelligent Tutoring System. In Proceedings of the Intelligent Tutoring Systems, Berlin, Heidelberg, M. Ikeda, K. D. Ashley and T.-W. Chan Eds. Springer Berlin Heidelberg, 392-401.
BAKER, R. S. J. D., GOLDSTEIN, A. B. AND HEFFERNAN, N. T. 2011. Detecting learning moment-by-moment. International Journal of Artificial Intelligence and Education 21, 5-25.
BATES, D., MÄCHLER, M., BOLKER, B. AND WALKER, S. 2014. Fitting linear mixed-effects models using lme4. arXiv preprint arXiv:1506.05908.
BUTLER, A. C., FAZIO, L. K. AND MARSH, E. J. 2011. The hypercorrection effect persists over a week, but high-confidence errors return. Psychonomic Bulletin & Review 18, 1238-1244.
CARRIER, M. AND PASHLER, H. 1992. The influence of retrieval on retention. Memory & Cognition 20, 633-642.
CEN, H., KOEDINGER, K. R. AND JUNKER, B. 2006. Learning Factors Analysis - A general method for cognitive model evaluation and improvement. In Proceedings of the 8th International Conference on Intelligent Tutoring Systems Springer Berlin / Heidelberg, 164-175.
CHI, M., KOEDINGER, K. R., GORDON, G., JORDAN, P. AND VANLEHN, K. 2011. Instructional Factors Analysis: A cognitive model for multiple instructional interventions. In Proceedings of the 4th International Conference on Educational Data Mining, Eindhoven, The Netherlands, M. Pechenizkiy, T. Calders, C. Conati, S. Ventura, C. Romero and J. Stamper Eds., 61-70.
CORBETT, A. T. AND ANDERSON, J. R. 1992. Student modeling and mastery learning in a computer-based programming tutor. In Intelligent Tutoring Systems: Second International Conference on Intelligent Tutoring Systems, C. Frasson, G. Gauthier and G. Mccalla Eds. Springer-Verlag, New York, 413-420.
DE BOECK, P., BAKKER, M., ZWITSER, R., NIVARD, M., HOFMAN, A., TUERLINCKX, F. AND PARTCHEV, I. 2011. The estimation of item response models with the lmer function from the lme4 package in R. Journal of Statistical Software 39, 1-28.
DESMARAIS, M. C. AND LEMIEUX, F. 2013. Clustering and Visualizing Study State Sequences. In Proceedings of the 6th International Conference of Educational Datamining, S. K. D'mello, R. A. Calvo and A. Olney Eds., Memphis. TN, 224-227.
FISCHER, G. H. 1973. The linear logistic test model as an instrument in educational research. Acta Psychologica 37, 359-374.
FRALEY, C., RAFTERY, A. E., MURPHY, T. B. AND SCRUCCA, L. 2012. mclust Version 4 for R: Normal Mixture Modeling for Model-Based Clustering, Classification, and Density Estimation.
GALYARDT, A. AND GOLDIN, I. 2015. Move your lamp post: Recent data reflects learner knowledge better than older data. Journal of Educational Data Mining 7, 83-108.
GONG, Y., BECK, J. E. AND HEFFERNAN, N. T. 2011. How to Construct More Accurate Student Models: Comparing and Optimizing Knowledge Tracing and Performance Factor Analysis. International Journal of Artificial Intelligence in Education 21, 27-46.
GRIMALDI, P. J. AND KARPICKE, J. D. 2014. Guided retrieval practice of educational materials using automated scoring. Journal of Educational Psychology 106, 58-68.
IZAWA, C. 1970. Optimal potentiating effects and forgetting-prevention effects of tests in paired-associate learning. Journal of Experimental Psychology 83, 340-344.
JOHNSON, P. C. D. 2014. Extension of Nakagawa & Schielzeth's R2GLMM to random slopes models. Methods in Ecology and Evolution 5, 944-946.
KANG, S. H. K. AND PASHLER, H. 2014. Is the benefit of retrieval practice modulated by motivation? Journal of Applied Research in Memory and Cognition 3, 183-188.
KELLY, J. W., CARPENTER, S. K. AND SJOLUND, L. A. 2015. Retrieval enhances route knowledge acquisition, but only when movement errors are prevented. Journal of Experimental Psychology: Learning, Memory, and Cognition 41, 1540-1547.
KORNELL, N. AND BJORK, R. A. 2007. The promise and perils of self-regulated study. Psychonomic Bulletin & Review 14, 219-224.
KORNELL, N., HAYS, M. J. AND BJORK, R. A. 2009. Unsuccessful retrieval attempts enhance subsequent learning. Journal of Experimental Psychology: Learning, Memory, & Cognition 35, 989-998.
MCCABE, J. 2011. Metacognitive awareness of learning strategies in undergraduates. Memory & Cognition 39, 462-476.
MORRIS, C. D., BRANSFORD, J. D. AND FRANKS, J. J. 1977. Levels of processing versus transfer appropriate processing. Journal of Verbal Learning and Verbal Behavior 16, 519-533.
NAKAGAWA, S., JOHNSON, P. C. D. AND SCHIELZETH, H. 2017. The coefficient of determination R(2) and intra-class correlation coefficient from generalized linear mixed-effects models revisited and expanded. J R Soc Interface 14.
NAKAGAWA, S. AND SCHIELZETH, H. 2013. A general and simple method for obtaining R2 from generalized linear mixed-effects models. Methods in Ecology and Evolution 4, 133-142.
NEWELL, A. AND ROSENBLOOM, P. S. 1981. Mechanisms of skill acquisition and the law of practice. Cognitive skills and their acquisition 1, 1-55.
PARDOS, Z. A., TRIVEDI, S., HEFFERNAN, N. T. AND SÁRKÖZY, G. N. 2012. Clustered Knowledge Tracing. In Proceedings of the Intelligent Tutoring Systems, Berlin, Heidelberg, S. A. Cerri, W. J. Clancey, G. Papadourakis and K. Panourgia Eds. Springer Berlin Heidelberg, 405-410.
PAVLIK JR., P. I. 2007. Understanding and applying the dynamics of test practice and study practice. Instructional Science 35, 407–441.
PAVLIK JR., P. I. AND ANDERSON, J. R. 2008. Using a model to compute the optimal schedule of practice. Journal of Experimental Psychology: Applied 14, 101–117.
PAVLIK JR., P. I., CEN, H. AND KOEDINGER, K. R. 2009. Performance factors analysis -- A new alternative to knowledge tracing. In Proceedings of the 14th International Conference on Artificial Intelligence in Education, V. Dimitrova, R. Mizoguchi, B. D. Boulay and A. Graesser Eds., Brighton, England, 531–538.
PAVLIK JR., P. I., YUDELSON, M. AND KOEDINGER, K. R. 2011. Using contextual factors analysis to explain transfer of least common multiple skills. In Artificial Intelligence in Education, G. Biswas, S. Bull, J. Kay and A. Mitrovic Eds. Springer, Berlin, Germany, 256–263.
PIECH, C., SPENCER, J., HUANG, J., GANGULI, S., SAHAMI, M., GUIBAS, L. AND SOHL-DICKSTEIN, J. 2015. Deep Knowledge Tracing. arXiv preprint arXiv:1506.05908.
RASCH, G. 1961. On general laws and the meaning of measurement in psychology. In Proceedings of the Proceedings of the fourth Berkeley symposium on mathematical statistics and probability University of California Press Berkeley, CA, 321-333.
RIHÁK, J. 2015. Use of Time Information in Models behind Adaptive System for Building Fluency in Mathematics. In Proceedings of the 8th International Conference on Educational Data Mining, O. C. Santos, J. G. Boticario, C. Romero, M. Pechenizkiy, A. Merceron, P. Mitros, . . . M. Desmarais Eds., Madrid, Spain, 642-644.
ROEDIGER, H. L. AND BUTLER, A. C. 2011. The critical role of retrieval practice in long-term retention. Trends in Cognitive Sciences 15, 20-27.
ROWLAND, C. A. 2014. The effect of testing versus restudy on retention: A meta-analytic review of the testing effect. Psychological Bulletin 140, 1432-1463.
SHIH, B., KOEDINGER, K. R. AND SCHEINES, R. 2008. A response time model for bottom-out hints as worked examples. In Proceedings of the 1st International Conference on Educational Data Mining, R. S. Baker and J. E. Beck Eds., Montreal, Canada, 117–126.
WOOD, H. AND WOOD, D. 1999. Help seeking, learning and contingent tutoring. Computers & Education 33, 153-169.
YAN, V. X., BJORK, E. L. AND BJORK, R. A. 2016. On the difficulty of mending metacognitive illusions: A priori theories, fluency effects, and misattributions of the interleaving benefit. J Exp Psychol Gen 145, 918-933.
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors who publish with this journal agree to the following terms:
- The Author retains copyright in the Work, where the term “Work” shall include all digital objects that may result in subsequent electronic publication or distribution.
- Upon acceptance of the Work, the author shall grant to the Publisher the right of first publication of the Work.
- The Author shall grant to the Publisher and its agents the nonexclusive perpetual right and license to publish, archive, and make accessible the Work in whole or in part in all forms of media now or hereafter known under a Creative Commons 4.0 License (Attribution-Noncommercial-No Derivatives 4.0 International), or its equivalent, which, for the avoidance of doubt, allows others to copy, distribute, and transmit the Work under the following conditions:
- Attribution—other users must attribute the Work in the manner specified by the author as indicated on the journal Web site;
- Noncommercial—other users (including Publisher) may not use this Work for commercial purposes;
- No Derivative Works—other users (including Publisher) may not alter, transform, or build upon this Work,with the understanding that any of the above conditions can be waived with permission from the Author and that where the Work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.
- The Author is able to enter into separate, additional contractual arrangements for the nonexclusive distribution of the journal's published version of the Work (e.g., post it to an institutional repository or publish it in a book), as long as there is provided in the document an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post online a pre-publication manuscript (but not the Publisher’s final formatted PDF version of the Work) in institutional repositories or on their Websites prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (see The Effect of Open Access). Any such posting made before acceptance and publication of the Work shall be updated upon publication to include a reference to the Publisher-assigned DOI (Digital Object Identifier) and a link to the online abstract for the final published Work in the Journal.
- Upon Publisher’s request, the Author agrees to furnish promptly to Publisher, at the Author’s own expense, written evidence of the permissions, licenses, and consents for use of third-party material included within the Work, except as determined by Publisher to be covered by the principles of Fair Use.
- The Author represents and warrants that:
- the Work is the Author’s original work;
- the Author has not transferred, and will not transfer, exclusive rights in the Work to any third party;
- the Work is not pending review or under consideration by another publisher;
- the Work has not previously been published;
- the Work contains no misrepresentation or infringement of the Work or property of other authors or third parties; and
- the Work contains no libel, invasion of privacy, or other unlawful matter.
- The Author agrees to indemnify and hold Publisher harmless from Author’s breach of the representations and warranties contained in Paragraph 6 above, as well as any claim or proceeding relating to Publisher’s use and publication of any content contained in the Work, including third-party content.