Move Your Lamp Post: Recent Data Reflects Learner Knowledge Better Than Older Data

##plugins.themes.bootstrap3.article.main##

##plugins.themes.bootstrap3.article.sidebar##

PDF
Published Sep 25, 2015
DOI https://doi.org/10.5281/zenodo.3554671
April Galyardt
University of Georgia
Ilya Goldin
Pearson

Abstract

In educational technology and learning sciences, there are multiple uses for a predictive model of whether a student will perform a task correctly or not. For example, an intelligent tutoring system may use such a model to estimate whether or not a student has mastered a skill. We analyze the significance of data recency in making such predictions, i.e., asking whether relatively more recent observations of a student’s performance matter more than relatively older observations. We investigate several representations of recency, such as the count of prior practice in the AFM model, and the proportion of recent successes with exponential and box kernels. We find that an exponential decay of a proportion of successes provides the summary of recent practice with the highest predictive accuracy over alternative models. As a secondary contribution, we develop a new logistic regression model, Recent-Performance Factors Analysis, that leverages this representation of recent performance, and has higher predictive accuracy than existing logistic regression models.

How to Cite

Galyardt, A., & Goldin, I. (2015). Move Your Lamp Post: Recent Data Reflects Learner Knowledge Better Than Older Data. Journal of Educational Data Mining, 7(2), 83–108. https://doi.org/10.5281/zenodo.3554671
Abstract 736 | PDF Downloads 399

##plugins.themes.bootstrap3.article.details##

Keywords

instance weighting, adaptive learning, student modeling, linear logistic test model

References
ADAMS, R. J.,WILSON, M., AND WANG, W.-C. 1997. The multidimensional random coefficients multinomial logit model. Applied psychological measurement 21, 1, 1–23.

AKAIKE, H. 1973. Information theory and an extension of the maximum likelihood principle. In Proceedings of Second International Symposium on Information Theory, B. Petrov and F. Caski, Eds. Akademiai Kiado, Budapest, 267–281.

AKAIKE, H. 1985. Prediction and entropy. In A Celebration of Statistics, A. Atkinson and S. Fienberg, Eds. Springer: New York, 1–24.

ALEVEN, V. AND KOEDINGER, K. R. 2000. Limitations of student control: Do students know when they need help? In Intelligent Tutoring Systems, G. Gauthier, C. Frasson, and K. VanLehn, Eds. Vol. 1839. Springer Berlin Heidelberg, Berlin, Heidelberg, 292–303.

BAKER, R. S., CORBETT, A. T., KOEDINGER, K. R., AND WAGNER, A. Z. 2004. Off-task behavior in the cognitive tutor classroom: when students game the system. In Proceedings of SIGCHI conference on Human factors in computing systems. ACM, 383–390.

BAKER, R. S., GOLDSTEIN, A. B., AND HEFFERNAN, N. T. 2011. Detecting learning moment-bymoment. In IJAIED. Vol. 21. 5–25.

BATES, D., MAECHLER, M., BOLKER, B., AND WALKER, S. 2013. lme4: Linear mixed-effects models using eigen and s4. Computer Program.

BECK, J. E. AND CHANG, K.-M. 2007. Identifiability: A fundamental problem of student modeling. In User Modeling 2007, C. Conati, K. McCoy, and G. Paliouras, Eds. Number 4511 in Lecture Notes in Computer Science. Springer Berlin Heidelberg, 137–146.

BECK, J. E., CHANG, K.-M., MOSTOW, J., AND CORBETT, A. 2008. Does help help? Introducing the Bayesian evaluation and assessment methodology. In Intelligent Tutoring Systems, B. P. Woolf, E. A¨ımeur, R. Nkambou, and S. Lajoie, Eds. Vol. 5091. Springer, Berlin, 383–394.

BOOTS, B., SIDDIQI, S. M., AND GORDON, G. J. 2011. Closing the learning-planning loop with predictive state representations. The International Journal of Robotics Research 30, 7 (June), 954–966.

CEN, H., KOEDINGER, K., AND JUNKER, B. 2006. Learning Factors Analysis – a general method for cognitive model evaluation and improvement. In Proc of 8th ITS Conf, M. Ikeda, K. D. Ashley, and T.-W. Chan, Eds. Vol. 4053. Springer Berlin Heidelberg, Berlin, Heidelberg, 164–175.

CHI, M., KOEDINGER, K., GORDON, G., JORDAN, P., AND VANLEHN, K. 2011. Instructional factors analysis: A cognitive model for multiple instructional interventions. In Proceedings of 4th International Conference on Educational Data Mining.

CORBETT, A. T. AND ANDERSON, J. R. 1995. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction 4, 4, 253–278. DE BOECK, P. AND WILSON, M., Eds. 2004. Explanatory item response models: a generalized linear and nonlinear approach. Springer, New York.

FALAKMASIR, M. H., PARDOS, Z. A., GORDON, G. J., AND BRUSILOVSKY, P. 2013. A spectral learning approach to knowledge tracing. In Proceedings of 6th International Conference on Educational Data Mining., S. K. D’Mello, R. A. Calvo, and A. Olney, Eds. Memphis, TN, 28–34.

FALAKMASIR, M. H., YUDELSON, M., RITTER, S., AND KOEDINGER, K. 2015. Spectral bayesian knowledge tracing. In Proceedings of the 8th International Conference on Educational Data Mining., O. C. Santos, J. G. Boticario, C. Romero, M. Pechenizkiy, A. Merceron, P. Mitros, J. M. Luna, C. Mihaescu, P. Moreno, A. Hershkovitz, S. Ventura, and M. Desmarais, Eds. Madrid, Spain, 360– 364.

FISCHER, G. H. 1973. The linear logistic test model as an instrument in educational research. Acta Psychologica 37, 359–374.

GALYARDT, A. AND GOLDIN, I. 2014. Recent-Performance Factors Analysis. In Proceedings of 7th International Conference on Educational Data Mining, J. Stamper, Z. Pardos, M. Mavrikis, and B. McLaren, Eds. 411–412. (Poster paper).

GOLDIN, I. AND GALYARDT, A. 2015a. Viz-R: Using Recency to Improve Student and Domain Models. In Proceedings of Second (2015) ACM Conference on Learning@Scale. L@S ’15. ACM, New York, NY, USA, 417–420.

GOLDIN, I. M. AND GALYARDT, A. 2015b. Convergent validity of a student model: Recent-Performance Factors Analysis. In Proceedings of 8th International Conference on Educational Data Mining. Madrid, Spain.

GOLDIN, I. M., KOEDINGER, K. R., AND ALEVEN, V. A. W. M. M. 2012. Learner Differences in Hint Processing. In Proceedings of 5th International Conference on Educational Data Mining, K. Yacef, O. Za¨ıane, A. Hershkovitz, M. Yudelson, and J. Stamper, Eds. Chania, Greece, 73–80.

GONG, Y., BECK, J. E., AND HEFFERNAN, N. T. 2011. How to construct more accurate student models: Comparing and optimizing knowledge tracing and performance factor analysis. International Journal of Artificial Intelligence in Education 21, 1, 27–46.

GONZ´A LEZ-BRENES, J., HUANG, Y., AND BRUSILOVSKY, P. 2014. General features in knowledge tracing: Applications to multiple subskills, temporal item response theory, and expert knowledge. In Proceedings of 7th International Conference on Educational Data Mining, J. Stamper, Z. Pardos, M. Mavrikis, and B. McLaren, Eds. London, England, 84–91.

HEFFERNAN, N. T. AND HEFFERNAN, C. L. 2014. The ASSISTments ecosystem. IJAIE 24, 4 (Dec.), 470–497.

JUNKER, B. W. 2011. Modeling hierarchy and dependence among task responses in educational data mining. In Handbook of Educational Data Mining. Chapman & Hall/CRC.

KASER, T., KOEDINGER, K., AND GROSS, M. 2014. Different parameters-same prediction: An analysis of learning curves. In Proceedings of 7th International Conference on Educational Data Mining. London, UK.

KHAJAH, M. M., HUANG, Y., GONZ´ALEZ-BRENES, J., MOZER, M. C., AND BRUSILOVSKY, P. 2014. Integrating knowledge traching and item response theory: A tale of two frameworks. In Fourth International Workshop on Personalization Approaches in Learning Environments (PALE 2014), M. Kravcik, O. C. Santos, and J. G. Boticario, Eds. 7–15.

KHAJAH, M. M., WING, R. M., LINDSEY, R. V., AND MOZER, M. C. 2014. Integrating latent-factor and knowledge-tracing models to predict individual differences in learning. In Proceedings of 7th International Conference on Educational Data Mining, J. Stamper, Z. Pardos, M. Mavrikis, and B. McLaren, Eds. London, England.

PAVLIK, P. I., CEN, H., AND KOEDINGER, K. 2009. Performance Factors Analysis–a new alternative to Knowledge Tracing. In Proceedings of 14th International Conference on Artificial Intelligence in Education. IOS Press, 531–538.

PAVLIK, P. I., YUDELSON, M., AND KOEDINGER, K. R. 2015. A measurement model of microgenetic transfer for improving instructional outcomes. International Journal of Artificial Intelligence in Education, 1–34.

RUPP, A., TEMPLIN, J., AND HENSON, R. 2010. Diagnostic Measurement: Theory, Methods and Applications. Guilford Press, New York, NY.

STONE, M. 1977. An asymptotic equivalence of choice of model by cross-validation and Akaike’s criterion. Journal of the Royal Statistical Society. Series B (Methodological) 39, 1, 44–47.

WASSERMAN, L. 2006. All of Nonparametric Statistics. Springer, New York, NY.

YUDELSON, M., HOSSEINI, R., VIHAVAINEN, A., AND BRUSILOVSKY, P. 2014. Investigating automated student modeling in a Java MOOC. In Proceedings of 7th International Conference on Educational Data Mining. London, UK.
Issue
Vol. 7 No. 2 (2015): EDM Journal Track Special Issue
Section
EDM 2015 Journal Track

Authors who publish with this journal agree to the following terms:

  1. The Author retains copyright in the Work, where the term “Work” shall include all digital objects that may result in subsequent electronic publication or distribution.
  2. Upon acceptance of the Work, the author shall grant to the Publisher the right of first publication of the Work.
  3. The Author shall grant to the Publisher and its agents the nonexclusive perpetual right and license to publish, archive, and make accessible the Work in whole or in part in all forms of media now or hereafter known under a Creative Commons 4.0 License (Attribution-Noncommercial-No Derivatives 4.0 International), or its equivalent, which, for the avoidance of doubt, allows others to copy, distribute, and transmit the Work under the following conditions:
    1. Attribution—other users must attribute the Work in the manner specified by the author as indicated on the journal Web site;
    2. Noncommercial—other users (including Publisher) may not use this Work for commercial purposes;
    3. No Derivative Works—other users (including Publisher) may not alter, transform, or build upon this Work,with the understanding that any of the above conditions can be waived with permission from the Author and that where the Work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.
  4. The Author is able to enter into separate, additional contractual arrangements for the nonexclusive distribution of the journal's published version of the Work (e.g., post it to an institutional repository or publish it in a book), as long as there is provided in the document an acknowledgement of its initial publication in this journal.
  5. Authors are permitted and encouraged to post online a pre-publication manuscript (but not the Publisher’s final formatted PDF version of the Work) in institutional repositories or on their Websites prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (see The Effect of Open Access). Any such posting made before acceptance and publication of the Work shall be updated upon publication to include a reference to the Publisher-assigned DOI (Digital Object Identifier) and a link to the online abstract for the final published Work in the Journal.
  6. Upon Publisher’s request, the Author agrees to furnish promptly to Publisher, at the Author’s own expense, written evidence of the permissions, licenses, and consents for use of third-party material included within the Work, except as determined by Publisher to be covered by the principles of Fair Use.
  7. The Author represents and warrants that:
    1. the Work is the Author’s original work;
    2. the Author has not transferred, and will not transfer, exclusive rights in the Work to any third party;
    3. the Work is not pending review or under consideration by another publisher;
    4. the Work has not previously been published;
    5. the Work contains no misrepresentation or infringement of the Work or property of other authors or third parties; and
    6. the Work contains no libel, invasion of privacy, or other unlawful matter.
  8. The Author agrees to indemnify and hold Publisher harmless from Author’s breach of the representations and warranties contained in Paragraph 6 above, as well as any claim or proceeding relating to Publisher’s use and publication of any content contained in the Work, including third-party content.