Effect of Gamification on Gamers: Evaluating Interventions for Students Who Game the System Evaluating Interventions for Students Who Gaming the System
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
Jun 27, 2024
Kirk P. Vanacore
Ashish Gurung
Adam Sales
Neil Heffernan
https://orcid.org/0000-0003-0673-5721
Abstract
Gaming the system is a persistent problem in Computer-Based Learning Platforms. While substantial
progress has been made in identifying and understanding such behaviors, effective interventions remain
scarce. This study uses a method of causal moderation known as Fully Latent Principal Stratification to
explore the impact of two types of interventions – gamification and manipulation of assistance access –
on the learning outcomes of students who tend to game the system. The results indicate that gamification
does not consistently mitigate these negative behaviors. One gamified condition had a consistently
positive effect on learning regardless of students’ propensity to game the system, whereas the other had a
negative effect on gamers. However, delaying access to hints and feedback may have a positive effect on
the learning outcomes of those gaming the system. This paper also illustrates the potential for integrating
detection and causal methodologies within educational data mining to evaluate effective responses to detected
behaviors.
How to Cite
Vanacore, K. P., Gurung, A., Sales, A., & Heffernan, N. (2024). Effect of Gamification on Gamers: Evaluating Interventions for Students Who Game the System: Evaluating Interventions for Students Who Gaming the System. Journal of Educational Data Mining, 16(1), 112–140. https://doi.org/10.5281/zenodo.11549799
##plugins.themes.bootstrap3.article.details##
Keywords
gamification, gaming the system, causal inference, computer-based learning platforms
References
ABRAHAMSON, D., NATHAN, M. J., WILLIAMS-PIERCE, C., WALKINGTON, C., OTTMAR, E. R., SOTO, H., AND ALIBALI, M. W. 2020. The future of embodied design for mathematics teaching and learning. Frontiers in Education 5, 1–29.
ADAMS, D. M., MCLAREN, B. M., DURKIN, K., MAYER, R. E., RITTLE-JOHNSON, B., ISOTANI, S., AND VAN VELSEN, M. 2014. Using erroneous examples to improve mathematics learning with a web-based tutoring system. Computers in Human Behavior 36, 401–411.
ADJEI, S. A., BAKER, R. S., AND BAHEL, V. 2021. Seven-year longitudinal implications of wheel spinning and productive persistence. In Artificial Intelligence in Education: 22nd International Conference (AIED 2021), I. Roll, D. McNamara, S. Sosnovsky, R. Luckin, and D. V., Eds. Springer International Publishing, 16–28.
ALEVEN, V. AND KOEDINGER, K. R. 2000. Limitations of student control: Do students know when they need help? In Intelligent Tutoring Systems, G. Gauthier, C. Frasson, and K. VanLehn, Eds. Lecture Notes in Computer Science. Springer, Berlin, Heidelberg, 292–303.
ALEVEN, V., MCLAREN, B., ROLL, I., AND KOEDINGER, K. 2006. Toward meta-cognitive tutoring: A model of helpseeking with a cognitive tutor. International Journal of Artificial Intelligence in Education 16, 101–130.
ALEVEN, V., ROLL, I., MCLAREN, B. M., AND KOEDINGER, K. R. 2016. Help helps, but only so much: Research on help seeking with intelligent tutoring systems. International Journal of Artificial Intelligence in Education 26, 1 (Mar), 205–223.
AREZOOJI, D. M. 2020. A markov chain monte-carlo approach to dose-response optimization using probabilistic programming (rstan). arXiv Preprint.
BAKER, R., CARVALHO, A., RASPAT, J., ALEVEN, V., AND KOEDINGER, K. R. 2009. Educational software features that encourage and discourage “gaming the system”. In Artificial Intelligence in Education: 14th International Conference (AIED 2009), S. D. Craig and D. Dicheva, Eds. Vol. 14. Springer International Publishing, 475–482.
BAKER, R., WALONOSKI, J., HEFFERNAN, N., ROLL, I., CORBETT, A., AND KOEDINGER, K. 2008. Why students engage in “gaming the system” behavior in interactive learning environments. Journal of Interactive Learning Research 19, 2, 185–224.
BAKER, R. S., CORBETT, A. T., KOEDINGER, K. R., AND WAGNER, A. Z. 2004. Off-task behavior in the cognitive tutor classroom: when students ”game the system”. In Proceedings of the Conference on Human Factors in Computing Systems (CHI 2004). Association for Computing Machinery, 383–390.
BAKER, R. S. J. D., CORBETT, A. T., KOEDINGER, K. R., EVENSON, S., ROLL, I., WAGNER, A. Z., NAIM, M., RASPAT, J., BAKER, D. J., AND BECK, J. E. 2006. Adapting to when students game an intelligent tutoring system. In Intelligent Tutoring Systems: 8th International Conference, M. Ikeda, K. D. Ashley, and T.-W. Chan, Eds. Springer, Jhongil, Taiwan, 392–401.
BAKER, R. S. J. D., CORBETT, A. T., KOEDINGER, K. R., AND ROLL, I. 2006. Generalizing detection of gaming the system across a tutoring curriculum. In 8th International Conference of Intelligent Tutoring Systems (ITS 2006), M. Ikeda, K. D. Ashley, and T.-W. Chan, Eds. Spinger, 402–411.
BAKER, R. S. J. D., CORBETT, A. T., ROLL, I., AND KOEDINGER, K. R. 2008. Developing a generalizable detector of when students game the system. User Modeling and User-Adapted Interaction 18, 3 (Aug), 287–314.
BAKER, R. S. J. D., MITROVÍC , A., AND MATHEWS, M. 2010. Detecting gaming the system in constraint-based tutors. In User Modeling, Adaptation, and Personalization, P. De Bra, A. Kobsa, and D. Chin, Eds. Lecture Notes in Computer Science. Springer, Berlin, Heidelberg, 267–278.
BUTLER, A. C. AND WOODWARD, N. R. 2018. Toward consilience in the use of task-level feedback to promote learning. Psychology of Learning and Motivation 69, 1–38.
CARPENTER, B., GELMAN, A., HOFFMAN, M. D., LEE, D., GOODRICH, B., BETANCOURT, M., BRUBAKER, M. A., GUO, J., LI, P., AND RIDDELL, A. 2017. Stan: A probabilistic programming language. Journal of statistical software 76, 1–32.
CAYTON-HODGES, G. A., FENG, G., AND PAN, X. 2015. Tablet-based math assessment: What can we learn from math apps? Journal of Educational Technology & Society 18, 2, 3–20.
CHIU, L.-H. AND HENRY, L. L. 1990. Development and validation of the mathematics anxiety scale for children. Measurement and evaluation in counseling and development 23, 3, 121–127.
CORBETT, A. T. AND ANDERSON, J. R. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction 4, 4 (Dec), 253–278.
DANG, S. AND KOEDINGER, K. 2019. Exploring the link between motivations and gaming. In Proceedings of The 12th International Conference on Educational Data Mining (EDM 2019), C. F. Lynch, A. Merceron, M. Desmarais, and N. R., Eds. International Educational Data Mining Society, 276– 281.
DECKER-WOODROW, L. E., MASON, C. A., LEE, J.-E., CHAN, J. Y.-C., SALES, A., LIU, A., AND TU, S. 2023. The impacts of three educational technologies on algebraic understanding in the context of covid-19. AERA Open 9, 23328584231165919.
DIETER, K. C., STUDWELL, J., AND VANACORE, K. P. 2020. Differential responses to personalized learning recommendations revealed by event-related analysis. In Proceedings of the 13th International Conference on Educational Data Mining (EDM 2020), A. N. Rafferty, J. Whitehill, C. Romero, and V. Cavalli-Sforza, Eds. Vol. 13. International Educational Data Mining Society, Online, 736–742.
DIHOFF, R. E., BROSVIC, G. M., AND EPSTEIN, M. L. 2003. The role of feedback during academic testing: The delay retention effect revisited. The Psychological Record 53, 4, 533–548.
DOLONEN, J. A. AND KLUGE, A. 2015. Algebra learning through digital gaming in school. In Exploring the Material Conditions of Learning: The Computer Supported Collaborative Learning (CSCL) Conference 2015. Vol. 1. International Society of the Learning Sciences, Inc. [ISLS]., 252–259.
FENG, M. AND HEFFERNAN, N. T. 2006. Informing teachers live about student learning: Reporting in the assistments system. Technology Instruction Cognition and Learning 3, 1–14.
FRANGAKIS, C. E. AND RUBIN, D. B. 2002. Principal stratification in causal inference. Biometrics 58, 1, 21–29.
GARRIS, R., AHLERS, R., AND DRISKELL, J. E. 2002. Games, motivation, and learning: A research and practice model. Simulation& Gaming 33, 4 (Dec.), 441–467.
GASTON, J. AND COOPER, S. 2017. To three or not to three: Improving human computation game onboarding with a three-star system. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI 2017), G. Mark and P. Fussell, Eds. ACM, Denver Colorado USA, 5034–5039.
GEE, J. P. 2005. Learning by design: Good video games as learning machines. E-Learning and Digital Media 2, 1.
GOLDSTONE, R. L., LANDY, D. H., AND SON, J. Y. 2010. The education of perception. Topics in Cognitive Science 2, 2, 265–284.
GOODRICH, B., GABRY, J., ALI, I., AND BRILLEMAN, S. 2020. rstanarm: Bayesian applied regression modeling via Stan. R Package Version 2.21.1.
GURUNG, A., BARAL, S., LEE, M. P., SALES, A. C., HAIM, A., VANACORE, K. P., MCREYNOLDS, A. A., KREISBERG, H., HEFFERNAN, C., AND HEFFERNAN, N. T. 2023. How common are common wrong answers? crowdsourcing remediation at scale. In Proceedings of the Tenth ACM Conference on Learning@ Scale (L@S2023), D. Spikol, A. P. Viberg, A. Martínez-Monés, and P. Guo, Eds. Association for Computing Machinery, 70–80.
GURUNG, A., BARAL, S., VANACORE, K. P., MCREYNOLDS, A. A., KREISBERG, H., BOTELHO, A. F., SHAW, S. T., AND HEFFERNA, N. T. 2023. Identification, exploration, and remediation: Can teachers predict common wrong answers? In LAK23: 13th International Learning Analytics and Knowledge Conference, Association for Computing Machinery, 399–410.
HEFFERNAN, N. T. AND HEFFERNAN, C. L. 2014. The assistments ecosystem: Building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching. International Journal of Artificial Intelligence in Education 24, 4 (Oct), 470–497.
JIN, H. AND RUBIN, D. B. 2008. Principal stratification for causal inference with extended partial compliance. Journal of the American Statistical Association 103, 481, 101–111.
JUUL, G. J. 2009. Routledge, Chapter Fear of Failing? The Many Meanings of Difficulty in Video, 237–252.
KARAGIORGAS, D. N. AND NIEMANN, S. 2017. Gamification and game-based learning. Journal of Educational Technology Systems 45, 4 (June), 499–519.
LANDERS, R. N. 2014. Developing a theory of gamified learning: Linking serious games and gamification of learning. Simulation& Gaming 45, 6 (Dec.), 752–768.
LEE, M., CROTEAU, E., GURUNG, A., BOTELHO, A., AND HEFFERNAN, N. 2023. Knowledge tracing over time: A longitudinal analysis. In The Proceedings of the 16th International Conference on Educational Data Mining (EDM 2023)., M. Feng, T. Kaser, and P. Talukdar, Eds. International Educational Data Mining Society, 296–301.
LEVIN, N., BAKER, R., NASIAR, N., STEPHEN, F., AND HUTT, S. 2022. Evaluating gaming detector model robustness over time. In Proceedings of the 15th International Conference on Educational Data Mining (EDM 2022), A. Mitrovic and N. Bosch, Eds. International Educational Data Mining Society, 398–405.
LU, X., SALES, A., AND HEFFERNAN, N. T. 2021. Immediate versus delayed feedback on learning: Do people’s instincts really conflict with reality? Journal of Higher Education Theory and Practice 21, 16 (Dec.).
LU, X., WANG, W., MOTZ, B. A., YE, W., AND HEFFERNAN, N. T. 2023. Immediate text-based feedback timing on foreign language online assignments: How immediate should immediate feedback be? Computers and Education Open 5, 1–12.
MALKIEWICH, L. J., LEE, A., SLATER, S., XING, C., AND CHASE, C. C. 2016. No lives left: How common game features could undermine persistence, challenge-seeking and learning to program. In Proceedings of The International Conference of the Learning Sciences (ICLS) 2016. International Society of the Learning Sciences, 186–193.
MCKERNAN, B., MARTEY, R. M., STROMER-GALLEY, J., KENSKI, K., CLEGG, B. A., FOLKESTAD, J. E., RHODES, M. G., SHAW, A., SAULNIER, E. T., AND STRZALKOWSKI, T. 2015. We don’t need no stinkin’ badges: The impact of reward features and feeling rewarded in educational games. Computers in Human Behavior 45, 299–306.
MCLAREN, B. M., VAN GOG, T., GANOE, C., KARABINOS, M., AND YARON, D. 2016. The efficiency of worked examples compared to erroneous examples, tutored problem solving, and problem solving in computer-based learning environments. Computers in Human Behavior 55, 87–99.
MIDGLEY, C., MAEHR, M. L., HRUDA, L. Z., ANDERMAN, E., ANDERMAN, L., FREEMAN, K. E., URDAN, T., ET AL. 2000. Manual for the patterns of adaptive learning scales. University of Michigan.
MIHAELA, C. AND HERSHKOVITZ, A. 2009. The impact of off-task and gaming behaviors on learning: Immediate or aggregate? In Frontiers in Artificial Intelligence and Applications: Building Learning Systems that Care: From Knowledge Representation to Affective Modelling, V. Dimitrova, R. Mizoguchi, B. du Boulay, and A. Graesser, Eds. Vol. 200. Ios Press, 507–514.
MURRAY, R. C. AND VANLEHN, K. 2005. Effects of dissuading unnecessary help requests while providing proactive help. In Artificial Intelligence in Education: 12th International Conference (AIED 2005), C.-K. Looi, G. McCalla, B. Bredeweg, and J. Breuker, Eds. Springer International Publishing, 887–889.
OTTMAR, E., LEE, J.-E., VANACORE, K., PRADHAN, S., DECKER-WOODROW, L., AND MASON, C. A. 2023. Data from the efficacy study of from here to there! a dynamic technology for improving algebraic understanding. Journal of Open Psychology Data 11, 1 (Apr), 1–15.
PAGE, L. C. 2012. Principal stratification as a framework for investigating mediational processes in experimental settings. Journal of Research on Educational Effectiveness 5, 3, 215–244.
PAGE, L. C., FELLER, A., GRINDAL, T., MIRATRIX, L., AND SOMERS, M.-A. 2015. Principal stratification: A tool for understanding variation in program effects across endogenous subgroups. American Journal of Evaluation 36, 4 (Dec), 514–531.
PAQUETTE, L. AND BAKER, R. S. 2017. Variations of gaming behaviors across populations of students and across learning environments. In Artificial Intelligence in Education: 17th International Conference (AIED 2017), E. André, R. Baker, X. Hu, M. M. T. Rodrigo, and B. du Boulay, Eds. Springer International Publishing, Cham, 274–286.
PAQUETTE, L. AND BAKER, R. S. 2019. Comparing machine learning to knowledge engineering for student behavior modeling: a case study in gaming the system. Interactive Learning Environments 27, 5–6 (Aug), 585–597.
PAQUETTE, L., BAKER, R. S., DE CARVALHO, A., AND OCUMPAUGH, J. 2015. Cross-system transfer of machine learned and knowledge engineered models of gaming the system. In User Modeling, Adaptation and Personalization, F. Ricci, K. Bontcheva, O. Conlan, and S. Lawless, Eds. Lecture Notes in Computer Science. Springer International Publishing, Cham, 183–194.
PAQUETTE, L., DE CARVALHO, AND A. M. J. A., & BAKER, R. S. 2014. Towards understanding expert coding of student disengagement in online learning. In Proceedings of the 36th Annual Cognitive Science Conference, P. Bello, M. Guarini, M. McShane, and B. Scassellati, Eds. CogSci, 1–6.
PARDOS, Z. A., BAKER, R. S. J. D., SAN PEDRO, M. O. C. Z., GOWDA, S. M., AND GOWDA, S. M. 2014. Affective states and state tests: Investigating how affect and engagement during the school year predict end-of-year learning outcomes. Journal of Learning Analytics 1, 1, 107–128.
PATIKORN, T. AND HEFFERNAN, N. T. 2020. Effectiveness of crowd-sourcing on-demand assistance from teachers in online learning platforms. In L@S 2020 - Proceedings of the 7th ACM Conference on Learning @ Scale. Association for Computing Machinery, 115–124.
PHYE, G. D. AND ANDRE, T. 1989. Delayed retention effect: Attention, perseveration, or both? Contemporary Educational Psychology 14, 2 (Apr), 173–185.
PRIHAR, E., PATIKORN, T., BOTELHO, A., SALES, A., AND HEFFERNAN, N. 2021. Toward personalizing students’ education with crowdsourced tutoring. In L@S 2021 - Proceedings of the 8th ACM Conference on Learning @ Scale. Association for Computing Machinery, 37–45.
R CORE TEAM. 2016. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria.
RAZZAQ, L., HEFFERNAN, N. T., AND LINDEMAN, R. W. 2007. What level of tutor interaction is best? Frontiers in Artificial Intelligence and Applications 158, 222–229.
RICHEY, J. E., ZHANG, J., DAS, R., ANDRES-BRAY, J. M., SCRUGGS, R., MOGESSIE, M., BAKER, R. S., AND MCLAREN, B. M. 2021. Gaming and confrustion explain learning advantages for a math digital learning game. In Artificial Intelligence in Education: 22nd International Conference, I. Roll, D. McNamara, S. Sosnovsky, R. Luckin, and V. Dimitrova, Eds. Lecture Notes in Computer Science. Springer International Publishing, Utrecht, The Netherlands, 342–355.
RODRIGO, M. M. T., BAKER, R. S. J. D., D’MELLO, S., GONZALEZ, M. C. T., LAGUD, M. C. V., LIM, S. A. L., MACAPANPAN, A. F., PASCUA, S. A. M. S., SANTILLANO, J. Q., SUGAY, J. O., TEP, S., AND VIEHLAND, N. J. B. 2008. Comparing learners’ affect while using an intelligent tutoring system and a simulation problem solving game. In Intelligent Tutoring Systems, B. P. Woolf, E. Aïmeur, R. Nkambou, and S. Lajoie, Eds. Lecture Notes in Computer Science. Springer, Berlin, Heidelberg, 40–49.
ROSCHELLE, J., FENG, M., MURPHY, R. F., AND MASON, C. A. 2016. Online mathematics homework increases student achievement. AERA Open 2, 4 (Oct.), 2332858416673968.
SALES, A. C. AND PANE, J. F. 2019. The role of mastery learning in an intelligent tutoring system: Principal stratification on a latent variable. The Annals of Applied Statistics 13, 1 (Mar), 420–443.
SHUTE, V. J. 2008. Focus on formative feedback. Review of Educational Research 78, 1 (Mar), 153–189.
SIEW, N. M., GEOFREY, J., AND LEE, B. N. 2016. Students’ algebraic thinking and attitudes towards algebra: The effects of game-based learning using dragonbox 12 + app. The Research Journal of Mathematics and Technology 5, 1, 66–79.
STAR, J. R., POLLACK, C., DURKIN, K., RITTLE-JOHNSON, B., LYNCH, K., NEWTON, K., AND GOGOLEN, C. 2015. Learning from comparison in algebra. Contemporary Educational Psychology 40, 41–54.
STEKHOVEN, D. J. AND BUEHLMANN, P. 2012. Missforest - non-parametric missing value imputation for mixed-type data. Bioinformatics 28, 1, 112–118.
TORRES, R., TOUPS, Z. O., WIBURG, K., CHAMBERLIN, B., GOMEZ, C., AND OZER, M. A. 2016. Initial design implications for early algebra games. In Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts. CHI PLAY Companion ’16. Association for Computing Machinery, New York, NY, USA, 325–333.
VANACORE, K., GURUNG, A., SALES, A., AND HEFFERNAN, N. T. 2024. The effect of assistance on gamers: Assessing the impact of on-demand hints & feedback availability on learning for students who game the system. In Proceedings of the 14th Learning Analytics and Knowledge Conference. Association for Computing Machinery, 462–472.
VANACORE, K., OTTMAR, E., LIU, A., AND SALES, A. 2024. Remote monitoring of implementation fidelity using log-file data from multiple online learning platforms. Journal of Research on Technology in Education, 1–21.
VANACORE, K., SALES, A., LIU, A., AND OTTMAR, E. 2023a. Benefit of gamification for persistent learners: Propensity to replay problems moderates algebra-game effectiveness. In Tenth ACM Conference on Learning @ Scale (L@S ’23), D. Spikol, O. Viberg, A. Mart´ınez-Mones, and P. Guo, Eds. ACM, Copenhagen, Denmark, 164–173.
VANACORE, K., SALES, A., LIU, A., AND OTTMAR, E. 2023b. Heterogeneous effects of game-based failure on student persistence in an online algebra game. In Society for Research Educational Effectiveness Conference (SREE 2023). SREE, 1–4.
VANACORE, K., SALES, A. C., HANSEN, B., LIU, A., AND OTTMAR, E. 2024. Effect of game-based failure on productive persistence: an application of regression discontinuity design for evaluating the impact of program features on learning-related behaviors. Available at Social Science Research Network 4789291.
VEHTARI, A., GELMAN, A., SIMPSON, D., CARPENTER, B., AND BÜRKNER, P.-C. 2021. Ranknormalization, folding, and localization: An improved r hat for assessing convergence of mcmc (with discussion). Bayesian analysis 16, 2, 667–718.
WALONOSKI, J. A. AND HEFFERNAN, N. T. 2006. Prevention of off-task gaming behavior in intelligent tutoring systems. In Intelligent Tutoring Systems: 8th International Conference, ITS 2006, Jhongli, Taiwan, June 26-30, 2006. Proceedings 8, T.-W. C. Mitsuru Ikeda, Kevin D. Ashley, Ed. Lecture Notes in Computer Science. Springer, 722–724.
WILLIAMS, J. J., KIM, J., RAFFERTY, A., MALDONADO, S., GAJOS, K. Z., LASECKI, W. S., AND HEFFERNAN, N. 2016. Axis: Generating explanations at scale with learnersourcing and machine learning. In Proceedings of the Third (2016) ACM Conference on Learning @ Scale. L@S ’16. Association for Computing Machinery, New York, NY, USA, 379–388.
XIA, M., ASANO, Y., WILLIAMS, J. J., QU, H., AND MA, X. 2020. Using information visualization to promote students’ reflection on “gaming the system” in online learning. In Proceedings of the Seventh ACM Conference on Learning@Scale. L@S ’20. Association for Computing Machinery, New York, NY, USA, 37–49.
ADAMS, D. M., MCLAREN, B. M., DURKIN, K., MAYER, R. E., RITTLE-JOHNSON, B., ISOTANI, S., AND VAN VELSEN, M. 2014. Using erroneous examples to improve mathematics learning with a web-based tutoring system. Computers in Human Behavior 36, 401–411.
ADJEI, S. A., BAKER, R. S., AND BAHEL, V. 2021. Seven-year longitudinal implications of wheel spinning and productive persistence. In Artificial Intelligence in Education: 22nd International Conference (AIED 2021), I. Roll, D. McNamara, S. Sosnovsky, R. Luckin, and D. V., Eds. Springer International Publishing, 16–28.
ALEVEN, V. AND KOEDINGER, K. R. 2000. Limitations of student control: Do students know when they need help? In Intelligent Tutoring Systems, G. Gauthier, C. Frasson, and K. VanLehn, Eds. Lecture Notes in Computer Science. Springer, Berlin, Heidelberg, 292–303.
ALEVEN, V., MCLAREN, B., ROLL, I., AND KOEDINGER, K. 2006. Toward meta-cognitive tutoring: A model of helpseeking with a cognitive tutor. International Journal of Artificial Intelligence in Education 16, 101–130.
ALEVEN, V., ROLL, I., MCLAREN, B. M., AND KOEDINGER, K. R. 2016. Help helps, but only so much: Research on help seeking with intelligent tutoring systems. International Journal of Artificial Intelligence in Education 26, 1 (Mar), 205–223.
AREZOOJI, D. M. 2020. A markov chain monte-carlo approach to dose-response optimization using probabilistic programming (rstan). arXiv Preprint.
BAKER, R., CARVALHO, A., RASPAT, J., ALEVEN, V., AND KOEDINGER, K. R. 2009. Educational software features that encourage and discourage “gaming the system”. In Artificial Intelligence in Education: 14th International Conference (AIED 2009), S. D. Craig and D. Dicheva, Eds. Vol. 14. Springer International Publishing, 475–482.
BAKER, R., WALONOSKI, J., HEFFERNAN, N., ROLL, I., CORBETT, A., AND KOEDINGER, K. 2008. Why students engage in “gaming the system” behavior in interactive learning environments. Journal of Interactive Learning Research 19, 2, 185–224.
BAKER, R. S., CORBETT, A. T., KOEDINGER, K. R., AND WAGNER, A. Z. 2004. Off-task behavior in the cognitive tutor classroom: when students ”game the system”. In Proceedings of the Conference on Human Factors in Computing Systems (CHI 2004). Association for Computing Machinery, 383–390.
BAKER, R. S. J. D., CORBETT, A. T., KOEDINGER, K. R., EVENSON, S., ROLL, I., WAGNER, A. Z., NAIM, M., RASPAT, J., BAKER, D. J., AND BECK, J. E. 2006. Adapting to when students game an intelligent tutoring system. In Intelligent Tutoring Systems: 8th International Conference, M. Ikeda, K. D. Ashley, and T.-W. Chan, Eds. Springer, Jhongil, Taiwan, 392–401.
BAKER, R. S. J. D., CORBETT, A. T., KOEDINGER, K. R., AND ROLL, I. 2006. Generalizing detection of gaming the system across a tutoring curriculum. In 8th International Conference of Intelligent Tutoring Systems (ITS 2006), M. Ikeda, K. D. Ashley, and T.-W. Chan, Eds. Spinger, 402–411.
BAKER, R. S. J. D., CORBETT, A. T., ROLL, I., AND KOEDINGER, K. R. 2008. Developing a generalizable detector of when students game the system. User Modeling and User-Adapted Interaction 18, 3 (Aug), 287–314.
BAKER, R. S. J. D., MITROVÍC , A., AND MATHEWS, M. 2010. Detecting gaming the system in constraint-based tutors. In User Modeling, Adaptation, and Personalization, P. De Bra, A. Kobsa, and D. Chin, Eds. Lecture Notes in Computer Science. Springer, Berlin, Heidelberg, 267–278.
BUTLER, A. C. AND WOODWARD, N. R. 2018. Toward consilience in the use of task-level feedback to promote learning. Psychology of Learning and Motivation 69, 1–38.
CARPENTER, B., GELMAN, A., HOFFMAN, M. D., LEE, D., GOODRICH, B., BETANCOURT, M., BRUBAKER, M. A., GUO, J., LI, P., AND RIDDELL, A. 2017. Stan: A probabilistic programming language. Journal of statistical software 76, 1–32.
CAYTON-HODGES, G. A., FENG, G., AND PAN, X. 2015. Tablet-based math assessment: What can we learn from math apps? Journal of Educational Technology & Society 18, 2, 3–20.
CHIU, L.-H. AND HENRY, L. L. 1990. Development and validation of the mathematics anxiety scale for children. Measurement and evaluation in counseling and development 23, 3, 121–127.
CORBETT, A. T. AND ANDERSON, J. R. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction 4, 4 (Dec), 253–278.
DANG, S. AND KOEDINGER, K. 2019. Exploring the link between motivations and gaming. In Proceedings of The 12th International Conference on Educational Data Mining (EDM 2019), C. F. Lynch, A. Merceron, M. Desmarais, and N. R., Eds. International Educational Data Mining Society, 276– 281.
DECKER-WOODROW, L. E., MASON, C. A., LEE, J.-E., CHAN, J. Y.-C., SALES, A., LIU, A., AND TU, S. 2023. The impacts of three educational technologies on algebraic understanding in the context of covid-19. AERA Open 9, 23328584231165919.
DIETER, K. C., STUDWELL, J., AND VANACORE, K. P. 2020. Differential responses to personalized learning recommendations revealed by event-related analysis. In Proceedings of the 13th International Conference on Educational Data Mining (EDM 2020), A. N. Rafferty, J. Whitehill, C. Romero, and V. Cavalli-Sforza, Eds. Vol. 13. International Educational Data Mining Society, Online, 736–742.
DIHOFF, R. E., BROSVIC, G. M., AND EPSTEIN, M. L. 2003. The role of feedback during academic testing: The delay retention effect revisited. The Psychological Record 53, 4, 533–548.
DOLONEN, J. A. AND KLUGE, A. 2015. Algebra learning through digital gaming in school. In Exploring the Material Conditions of Learning: The Computer Supported Collaborative Learning (CSCL) Conference 2015. Vol. 1. International Society of the Learning Sciences, Inc. [ISLS]., 252–259.
FENG, M. AND HEFFERNAN, N. T. 2006. Informing teachers live about student learning: Reporting in the assistments system. Technology Instruction Cognition and Learning 3, 1–14.
FRANGAKIS, C. E. AND RUBIN, D. B. 2002. Principal stratification in causal inference. Biometrics 58, 1, 21–29.
GARRIS, R., AHLERS, R., AND DRISKELL, J. E. 2002. Games, motivation, and learning: A research and practice model. Simulation& Gaming 33, 4 (Dec.), 441–467.
GASTON, J. AND COOPER, S. 2017. To three or not to three: Improving human computation game onboarding with a three-star system. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI 2017), G. Mark and P. Fussell, Eds. ACM, Denver Colorado USA, 5034–5039.
GEE, J. P. 2005. Learning by design: Good video games as learning machines. E-Learning and Digital Media 2, 1.
GOLDSTONE, R. L., LANDY, D. H., AND SON, J. Y. 2010. The education of perception. Topics in Cognitive Science 2, 2, 265–284.
GOODRICH, B., GABRY, J., ALI, I., AND BRILLEMAN, S. 2020. rstanarm: Bayesian applied regression modeling via Stan. R Package Version 2.21.1.
GURUNG, A., BARAL, S., LEE, M. P., SALES, A. C., HAIM, A., VANACORE, K. P., MCREYNOLDS, A. A., KREISBERG, H., HEFFERNAN, C., AND HEFFERNAN, N. T. 2023. How common are common wrong answers? crowdsourcing remediation at scale. In Proceedings of the Tenth ACM Conference on Learning@ Scale (L@S2023), D. Spikol, A. P. Viberg, A. Martínez-Monés, and P. Guo, Eds. Association for Computing Machinery, 70–80.
GURUNG, A., BARAL, S., VANACORE, K. P., MCREYNOLDS, A. A., KREISBERG, H., BOTELHO, A. F., SHAW, S. T., AND HEFFERNA, N. T. 2023. Identification, exploration, and remediation: Can teachers predict common wrong answers? In LAK23: 13th International Learning Analytics and Knowledge Conference, Association for Computing Machinery, 399–410.
HEFFERNAN, N. T. AND HEFFERNAN, C. L. 2014. The assistments ecosystem: Building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching. International Journal of Artificial Intelligence in Education 24, 4 (Oct), 470–497.
JIN, H. AND RUBIN, D. B. 2008. Principal stratification for causal inference with extended partial compliance. Journal of the American Statistical Association 103, 481, 101–111.
JUUL, G. J. 2009. Routledge, Chapter Fear of Failing? The Many Meanings of Difficulty in Video, 237–252.
KARAGIORGAS, D. N. AND NIEMANN, S. 2017. Gamification and game-based learning. Journal of Educational Technology Systems 45, 4 (June), 499–519.
LANDERS, R. N. 2014. Developing a theory of gamified learning: Linking serious games and gamification of learning. Simulation& Gaming 45, 6 (Dec.), 752–768.
LEE, M., CROTEAU, E., GURUNG, A., BOTELHO, A., AND HEFFERNAN, N. 2023. Knowledge tracing over time: A longitudinal analysis. In The Proceedings of the 16th International Conference on Educational Data Mining (EDM 2023)., M. Feng, T. Kaser, and P. Talukdar, Eds. International Educational Data Mining Society, 296–301.
LEVIN, N., BAKER, R., NASIAR, N., STEPHEN, F., AND HUTT, S. 2022. Evaluating gaming detector model robustness over time. In Proceedings of the 15th International Conference on Educational Data Mining (EDM 2022), A. Mitrovic and N. Bosch, Eds. International Educational Data Mining Society, 398–405.
LU, X., SALES, A., AND HEFFERNAN, N. T. 2021. Immediate versus delayed feedback on learning: Do people’s instincts really conflict with reality? Journal of Higher Education Theory and Practice 21, 16 (Dec.).
LU, X., WANG, W., MOTZ, B. A., YE, W., AND HEFFERNAN, N. T. 2023. Immediate text-based feedback timing on foreign language online assignments: How immediate should immediate feedback be? Computers and Education Open 5, 1–12.
MALKIEWICH, L. J., LEE, A., SLATER, S., XING, C., AND CHASE, C. C. 2016. No lives left: How common game features could undermine persistence, challenge-seeking and learning to program. In Proceedings of The International Conference of the Learning Sciences (ICLS) 2016. International Society of the Learning Sciences, 186–193.
MCKERNAN, B., MARTEY, R. M., STROMER-GALLEY, J., KENSKI, K., CLEGG, B. A., FOLKESTAD, J. E., RHODES, M. G., SHAW, A., SAULNIER, E. T., AND STRZALKOWSKI, T. 2015. We don’t need no stinkin’ badges: The impact of reward features and feeling rewarded in educational games. Computers in Human Behavior 45, 299–306.
MCLAREN, B. M., VAN GOG, T., GANOE, C., KARABINOS, M., AND YARON, D. 2016. The efficiency of worked examples compared to erroneous examples, tutored problem solving, and problem solving in computer-based learning environments. Computers in Human Behavior 55, 87–99.
MIDGLEY, C., MAEHR, M. L., HRUDA, L. Z., ANDERMAN, E., ANDERMAN, L., FREEMAN, K. E., URDAN, T., ET AL. 2000. Manual for the patterns of adaptive learning scales. University of Michigan.
MIHAELA, C. AND HERSHKOVITZ, A. 2009. The impact of off-task and gaming behaviors on learning: Immediate or aggregate? In Frontiers in Artificial Intelligence and Applications: Building Learning Systems that Care: From Knowledge Representation to Affective Modelling, V. Dimitrova, R. Mizoguchi, B. du Boulay, and A. Graesser, Eds. Vol. 200. Ios Press, 507–514.
MURRAY, R. C. AND VANLEHN, K. 2005. Effects of dissuading unnecessary help requests while providing proactive help. In Artificial Intelligence in Education: 12th International Conference (AIED 2005), C.-K. Looi, G. McCalla, B. Bredeweg, and J. Breuker, Eds. Springer International Publishing, 887–889.
OTTMAR, E., LEE, J.-E., VANACORE, K., PRADHAN, S., DECKER-WOODROW, L., AND MASON, C. A. 2023. Data from the efficacy study of from here to there! a dynamic technology for improving algebraic understanding. Journal of Open Psychology Data 11, 1 (Apr), 1–15.
PAGE, L. C. 2012. Principal stratification as a framework for investigating mediational processes in experimental settings. Journal of Research on Educational Effectiveness 5, 3, 215–244.
PAGE, L. C., FELLER, A., GRINDAL, T., MIRATRIX, L., AND SOMERS, M.-A. 2015. Principal stratification: A tool for understanding variation in program effects across endogenous subgroups. American Journal of Evaluation 36, 4 (Dec), 514–531.
PAQUETTE, L. AND BAKER, R. S. 2017. Variations of gaming behaviors across populations of students and across learning environments. In Artificial Intelligence in Education: 17th International Conference (AIED 2017), E. André, R. Baker, X. Hu, M. M. T. Rodrigo, and B. du Boulay, Eds. Springer International Publishing, Cham, 274–286.
PAQUETTE, L. AND BAKER, R. S. 2019. Comparing machine learning to knowledge engineering for student behavior modeling: a case study in gaming the system. Interactive Learning Environments 27, 5–6 (Aug), 585–597.
PAQUETTE, L., BAKER, R. S., DE CARVALHO, A., AND OCUMPAUGH, J. 2015. Cross-system transfer of machine learned and knowledge engineered models of gaming the system. In User Modeling, Adaptation and Personalization, F. Ricci, K. Bontcheva, O. Conlan, and S. Lawless, Eds. Lecture Notes in Computer Science. Springer International Publishing, Cham, 183–194.
PAQUETTE, L., DE CARVALHO, AND A. M. J. A., & BAKER, R. S. 2014. Towards understanding expert coding of student disengagement in online learning. In Proceedings of the 36th Annual Cognitive Science Conference, P. Bello, M. Guarini, M. McShane, and B. Scassellati, Eds. CogSci, 1–6.
PARDOS, Z. A., BAKER, R. S. J. D., SAN PEDRO, M. O. C. Z., GOWDA, S. M., AND GOWDA, S. M. 2014. Affective states and state tests: Investigating how affect and engagement during the school year predict end-of-year learning outcomes. Journal of Learning Analytics 1, 1, 107–128.
PATIKORN, T. AND HEFFERNAN, N. T. 2020. Effectiveness of crowd-sourcing on-demand assistance from teachers in online learning platforms. In L@S 2020 - Proceedings of the 7th ACM Conference on Learning @ Scale. Association for Computing Machinery, 115–124.
PHYE, G. D. AND ANDRE, T. 1989. Delayed retention effect: Attention, perseveration, or both? Contemporary Educational Psychology 14, 2 (Apr), 173–185.
PRIHAR, E., PATIKORN, T., BOTELHO, A., SALES, A., AND HEFFERNAN, N. 2021. Toward personalizing students’ education with crowdsourced tutoring. In L@S 2021 - Proceedings of the 8th ACM Conference on Learning @ Scale. Association for Computing Machinery, 37–45.
R CORE TEAM. 2016. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria.
RAZZAQ, L., HEFFERNAN, N. T., AND LINDEMAN, R. W. 2007. What level of tutor interaction is best? Frontiers in Artificial Intelligence and Applications 158, 222–229.
RICHEY, J. E., ZHANG, J., DAS, R., ANDRES-BRAY, J. M., SCRUGGS, R., MOGESSIE, M., BAKER, R. S., AND MCLAREN, B. M. 2021. Gaming and confrustion explain learning advantages for a math digital learning game. In Artificial Intelligence in Education: 22nd International Conference, I. Roll, D. McNamara, S. Sosnovsky, R. Luckin, and V. Dimitrova, Eds. Lecture Notes in Computer Science. Springer International Publishing, Utrecht, The Netherlands, 342–355.
RODRIGO, M. M. T., BAKER, R. S. J. D., D’MELLO, S., GONZALEZ, M. C. T., LAGUD, M. C. V., LIM, S. A. L., MACAPANPAN, A. F., PASCUA, S. A. M. S., SANTILLANO, J. Q., SUGAY, J. O., TEP, S., AND VIEHLAND, N. J. B. 2008. Comparing learners’ affect while using an intelligent tutoring system and a simulation problem solving game. In Intelligent Tutoring Systems, B. P. Woolf, E. Aïmeur, R. Nkambou, and S. Lajoie, Eds. Lecture Notes in Computer Science. Springer, Berlin, Heidelberg, 40–49.
ROSCHELLE, J., FENG, M., MURPHY, R. F., AND MASON, C. A. 2016. Online mathematics homework increases student achievement. AERA Open 2, 4 (Oct.), 2332858416673968.
SALES, A. C. AND PANE, J. F. 2019. The role of mastery learning in an intelligent tutoring system: Principal stratification on a latent variable. The Annals of Applied Statistics 13, 1 (Mar), 420–443.
SHUTE, V. J. 2008. Focus on formative feedback. Review of Educational Research 78, 1 (Mar), 153–189.
SIEW, N. M., GEOFREY, J., AND LEE, B. N. 2016. Students’ algebraic thinking and attitudes towards algebra: The effects of game-based learning using dragonbox 12 + app. The Research Journal of Mathematics and Technology 5, 1, 66–79.
STAR, J. R., POLLACK, C., DURKIN, K., RITTLE-JOHNSON, B., LYNCH, K., NEWTON, K., AND GOGOLEN, C. 2015. Learning from comparison in algebra. Contemporary Educational Psychology 40, 41–54.
STEKHOVEN, D. J. AND BUEHLMANN, P. 2012. Missforest - non-parametric missing value imputation for mixed-type data. Bioinformatics 28, 1, 112–118.
TORRES, R., TOUPS, Z. O., WIBURG, K., CHAMBERLIN, B., GOMEZ, C., AND OZER, M. A. 2016. Initial design implications for early algebra games. In Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts. CHI PLAY Companion ’16. Association for Computing Machinery, New York, NY, USA, 325–333.
VANACORE, K., GURUNG, A., SALES, A., AND HEFFERNAN, N. T. 2024. The effect of assistance on gamers: Assessing the impact of on-demand hints & feedback availability on learning for students who game the system. In Proceedings of the 14th Learning Analytics and Knowledge Conference. Association for Computing Machinery, 462–472.
VANACORE, K., OTTMAR, E., LIU, A., AND SALES, A. 2024. Remote monitoring of implementation fidelity using log-file data from multiple online learning platforms. Journal of Research on Technology in Education, 1–21.
VANACORE, K., SALES, A., LIU, A., AND OTTMAR, E. 2023a. Benefit of gamification for persistent learners: Propensity to replay problems moderates algebra-game effectiveness. In Tenth ACM Conference on Learning @ Scale (L@S ’23), D. Spikol, O. Viberg, A. Mart´ınez-Mones, and P. Guo, Eds. ACM, Copenhagen, Denmark, 164–173.
VANACORE, K., SALES, A., LIU, A., AND OTTMAR, E. 2023b. Heterogeneous effects of game-based failure on student persistence in an online algebra game. In Society for Research Educational Effectiveness Conference (SREE 2023). SREE, 1–4.
VANACORE, K., SALES, A. C., HANSEN, B., LIU, A., AND OTTMAR, E. 2024. Effect of game-based failure on productive persistence: an application of regression discontinuity design for evaluating the impact of program features on learning-related behaviors. Available at Social Science Research Network 4789291.
VEHTARI, A., GELMAN, A., SIMPSON, D., CARPENTER, B., AND BÜRKNER, P.-C. 2021. Ranknormalization, folding, and localization: An improved r hat for assessing convergence of mcmc (with discussion). Bayesian analysis 16, 2, 667–718.
WALONOSKI, J. A. AND HEFFERNAN, N. T. 2006. Prevention of off-task gaming behavior in intelligent tutoring systems. In Intelligent Tutoring Systems: 8th International Conference, ITS 2006, Jhongli, Taiwan, June 26-30, 2006. Proceedings 8, T.-W. C. Mitsuru Ikeda, Kevin D. Ashley, Ed. Lecture Notes in Computer Science. Springer, 722–724.
WILLIAMS, J. J., KIM, J., RAFFERTY, A., MALDONADO, S., GAJOS, K. Z., LASECKI, W. S., AND HEFFERNAN, N. 2016. Axis: Generating explanations at scale with learnersourcing and machine learning. In Proceedings of the Third (2016) ACM Conference on Learning @ Scale. L@S ’16. Association for Computing Machinery, New York, NY, USA, 379–388.
XIA, M., ASANO, Y., WILLIAMS, J. J., QU, H., AND MA, X. 2020. Using information visualization to promote students’ reflection on “gaming the system” in online learning. In Proceedings of the Seventh ACM Conference on Learning@Scale. L@S ’20. Association for Computing Machinery, New York, NY, USA, 37–49.
Issue
Section
EDM 2024 Journal Track
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors who publish with this journal agree to the following terms:
- The Author retains copyright in the Work, where the term “Work” shall include all digital objects that may result in subsequent electronic publication or distribution.
- Upon acceptance of the Work, the author shall grant to the Publisher the right of first publication of the Work.
- The Author shall grant to the Publisher and its agents the nonexclusive perpetual right and license to publish, archive, and make accessible the Work in whole or in part in all forms of media now or hereafter known under a Creative Commons 4.0 License (Attribution-Noncommercial-No Derivatives 4.0 International), or its equivalent, which, for the avoidance of doubt, allows others to copy, distribute, and transmit the Work under the following conditions:
- Attribution—other users must attribute the Work in the manner specified by the author as indicated on the journal Web site;
- Noncommercial—other users (including Publisher) may not use this Work for commercial purposes;
- No Derivative Works—other users (including Publisher) may not alter, transform, or build upon this Work,with the understanding that any of the above conditions can be waived with permission from the Author and that where the Work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.
- The Author is able to enter into separate, additional contractual arrangements for the nonexclusive distribution of the journal's published version of the Work (e.g., post it to an institutional repository or publish it in a book), as long as there is provided in the document an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post online a pre-publication manuscript (but not the Publisher’s final formatted PDF version of the Work) in institutional repositories or on their Websites prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (see The Effect of Open Access). Any such posting made before acceptance and publication of the Work shall be updated upon publication to include a reference to the Publisher-assigned DOI (Digital Object Identifier) and a link to the online abstract for the final published Work in the Journal.
- Upon Publisher’s request, the Author agrees to furnish promptly to Publisher, at the Author’s own expense, written evidence of the permissions, licenses, and consents for use of third-party material included within the Work, except as determined by Publisher to be covered by the principles of Fair Use.
- The Author represents and warrants that:
- the Work is the Author’s original work;
- the Author has not transferred, and will not transfer, exclusive rights in the Work to any third party;
- the Work is not pending review or under consideration by another publisher;
- the Work has not previously been published;
- the Work contains no misrepresentation or infringement of the Work or property of other authors or third parties; and
- the Work contains no libel, invasion of privacy, or other unlawful matter.
- The Author agrees to indemnify and hold Publisher harmless from Author’s breach of the representations and warranties contained in Paragraph 6 above, as well as any claim or proceeding relating to Publisher’s use and publication of any content contained in the Work, including third-party content.