Decision Tree Modeling of Wheel-Spinning and Productive Persistence in Skill Builders



Published Jun 30, 2018
Shimin Kai Ma. Victoria Almeda Ryan S. Baker Cristina Heffernan Neil Heffernan


Research on non-cognitive factors has shown that persistence in the face of challenges plays an important role in learning. However, recent work on wheel-spinning, a type of unproductive persistence where students spend too much time struggling without achieving mastery of skills, show that not all persistence is uniformly beneficial for learning. For this reason, it becomes increasingly pertinent to identify the key differences between unproductive and productive persistence toward informing interventions in computer-based learning environments. In this study, we use a classification model to distinguish between productive persistence and wheel-spinning in ASSISTments, an online math learning platform. Our results indicate that there are two types of students who wheel-spin: first, students who do not request any hints in at least one problem but request more than one bottom-out hint across any 8 problems in the problem set; second, students who never request two or more bottom out hints across any 8 problems, do not request any hints in at least one problem, but who engage in relatively short delays between solving problems of the same skill. These findings suggest that encouraging students to both engage in spaced practice and use bottom-out hints sparingly is likely helpful for reducing their wheel-spinning and improving learning. These findings also provide insight on when students are struggling and how to make students' persistence more productive.

How to Cite

Kai, S., Almeda, M. V., Baker, R. S., Heffernan, C., & Heffernan, N. (2018). Decision Tree Modeling of Wheel-Spinning and Productive Persistence in Skill Builders. Journal of Educational Data Mining, 10(1), 36–71.
Abstract 1950 | PDF Downloads 1294



predictive modeling, wheel-spinning, productive persistence, decision tree, intelligent tutoring system

ALEVEN, V., & KOEDINGER, K. R. (2000, June). Limitations of student control: Do students know when they need help? In G. Gauthier, C. Frasson, & K. VanLehn (Eds.), Proceedings of the 5th International Conference on Intelligent Tutoring Systems (pp.292-303). Springer Berlin Heidelberg.

ALEVEN, V. A., & KOEDINGER, K. R. (2002). An effective metacognitive strategy: Learning by doing and explaining with a computer-based Cognitive Tutor. Cognitive Science, 26(2), 147-179.

ALEVEN, V., MCLAREN, B., ROLL, I., & KOEDINGER, K. (2006). Toward meta-cognitive tutoring: A model of help-seeking with a Cognitive Tutor. International Journal of Artificial Intelligence in Education, 16(2), 101-128.

ARROYO, I., WOOLF, B.P., ROYER, J.M., TAI, M., & ENGLISH, S. (2010). Improving math learning through intelligent tutoring and basic skills training. In V. Alenven, J. Kay, & J. Mostow (Eds.), Intelligent Tutoring Systems (pp. 423–432). Berlin Heidelberg: Springer-Verlag.

BAKER, R.S. (2008). A' implementation [Sourcecode]. analytics/ryanbaker/

BAKER, R.S. (2015). Big Data and Education. 2 nd Edition. New York, NY: Teachers College, Columbia University.

BAKER, R.S.J.D., CORBETT, A.T., ALEVEN, V. (2008) Improving Contextual Models of Guessing and Slipping with a Truncated Training Set. In T. Barnes & J. Beck (Eds.), Proceedings of the 1st International Conference on Educational Data Mining, 67-76.

BAKER, R. S., CORBETT, A. T., KOEDINGER, K. R., & ROLL, I. (2006, June). Generalizing detection of gaming the system across a tutoring curriculum. In M. Ikeda, K. Ashlay, & T. Chan (Eds.), Proceedings of the 8t h International Conference on Intelligent Tutoring Systems (pp. 402-411). Springer Berlin Heidelberg.

BAKER, R., WALONOSKI, J., HEFFERNAN, N., ROLL, I., CORBETT, A., & KOEDINGER, K. (2008). Why students engage in “gaming the system” behavior in interactive learning environments. Journal of Interactive Learning Research, 19(2), 185.

BAKER, R. S., GOLDSTEIN, A. B., & HEFFERNAN, N. T. (2011). Detecting learning moment-bymoment. International Journal of Artificial Intelligence in Education, 21(1-2), 5-25.

BAKER, R.S.J.D., GOWDA, S.M., & CORBETT, A.T. (2011). Automatically detecting a student's preparation for future learning: Help use is key. In M. Penchenizkiy, T. Calders, C. Conati, S. Ventura, C. Romero, & J. Stamper (Eds.), Proceedings of the 4th International Conference on Educational Data Mining, 179–188.

BARRICK, M. R., & MOUNT, M. K. (1991). The Big Five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44, 1–26.

BECK, J. E., & GONG, Y. (2013). Wheel-spinning: Students who fail to master a skill. In H. Lane, K. Yacef, J. Mostow, & P. Pavlik (Eds.), Proceedings of the 16th International Conference on Artificial Intelligence in Education (pp. 431-440). Springer Berlin Heidelberg.

BECK, J., & RODRIGO, M. M. T. (2014). Understanding wheel-spinning in the context of affective factors. In S. Trausan-Matu, K. Boyer, K. Crosby, & K. Panourgia (Eds.), Proceedings of the 12th International Conference on Intelligent Tutoring Systems (pp. 162- 167). Springer International Publishing.

BORGHANS, L., MEIJERS, F. AND B. TER WEEL (2006). The Role of Noncognitive Skills in Explaining Cognitive Test Scores. Economic Inquiry 46 (1), 2-12.

BORGHANS, L., DUCKWORTH, A. L., HECKMAN, J. J., & TER WEEL, B. (2008). The economics and psychology of personality traits. Journal of Human Resources, 43(4), 972-1059.

BOWLES, S., & GINTIS, H. (1976). Schooling in capitalist America. New York: Basic Books.

BOWLES, S., GINTIS, H. AND M. OSBORNE (2001). The determinants of earnings: A behavioral approach. Journal of Economic Literature, 39 (4), 1137-1176.

BRUNELLO, G., & SCHLOTTER, M. (2011). Non-cognitive skills and personality traits: Labour market relevance and their development in education & training systems. Discussion Paper series // Forschungsinstitut zur Zukunft der Arbeit, No. 5743, Institute for the Study of Labor (IZA), Bonn.

CARNEIRO, P., CRAWFORD, C. AND A. GOODMAN (2007). The impact of early cognitive and non-cognitive skills on later outcomes. Center for the Economics of Education Discussion Paper 0092, London School of Economics, London.

CHEUNG, A. C., & SLAVIN, R. E. (2013). The effectiveness of educational technology applications for enhancing mathematics achievement in K-12 classrooms: A meta-analysis. Educational Research Review, 9, 88-113.

CHITEJI, N. (2010). Time-preference, non-cognitive skills and well-being across the life course: Do non-cognitive skills encourage healthy behavior? The American Economic Review, 100(2), 200.

CLONINGER, C. R., SVRAKIC, D. M., & PRZYBECK, T. R. (1993). A psychobiological model of temperament and character. Archives of General Psychiatry, 50(12), 975-990.

CEPEDA, N. J., PASHLER, H., VUL, E., WIXTED, J. T., & ROHRER, D. (2006). Distributed practice in verbal recall tasks: A review and quantitative synthesis. Psychological Bulletin, 132(3), 354-380.

CREDÉ, M., TYNAN, M. C., & HARMS, P. D. (2017). Much Ado About Grit: A Meta-Analytic Synthesis of the Grit Literature. Journal of Personality and Social Psychology., 1113(3), 492-511.

DAVIS, J., & GOADRICH, M. (2006). The relationship between Precision-Recall and ROC curves. In W. Cohen & A. Moore (Eds.), Proceedings of the 23rd International Conference on Machine Learning (pp. 233-240). ACM.

DILLON, J. T. (1988). Questioning and Teaching: A Manual of Practice, New York Teachers College.

DE FRUYT, F., VAN DE WIELE, L., & VAN HEERINGEN, C. (2000). Cloninger's psychobiological model of temperament and character and the five-factor model of personality. Personality and Individual Differences, 29(3), 441-452.

DEKE, J. AND HAIMSON, J. (2006). Valuing student competencies: Which ones predict postsecondary educational attainment and earnings, and for whom? Final Report. Mathematica Policy Research, Inc.

DIGMAN, JOHN M. (1990). Personality structure: Emergence of the five-factor model. Annual review of psychology, 41(1), 417-440.

DILLON, J. T. (1988). Questioning and teaching: A manual of practice, New York Teachers College.

DUCKWORTH, A. L., PETERSON, C., MATTHEWS, M. D., & KELLY, D. R. (2007). Grit: perseverance and passion for long-term goals. Journal of Personality and Social Psychology, 92(6), 1087-1101.

DUCKWORTH, A. L., & QUINN, P. D. (2009). Development and validation of the Short Grit Scale (GRIT–S). Journal of Personality Assessment, 91(2), 166-174.

EISENBERGER, R., & LEONARD, J. M. (1980). Effects of conceptual task difficulty on generalized persistence. American Journal of Psychology, 93, 285–298.

FENG, M., ROSCHELLE, J., HEFFERNAN, N., FAIRMAN, J., & MURPHY, R. (2014). Implementation of an intelligent tutoring system for online homework support in an efficacy trial. In S. Trausan-Matu, K. Boyer, K. Crosby, & K. Panourgia (Eds.), Proceedings of the 12th International Conference on Intelligent Tutoring Systems (pp. 561-566). Springer International Publishing.

GARCIA, E. (2014). The need to address noncognitive skills in the education policy agenda. Briefing Paper 386. Economic Policy Institute.

GOLDBERG, L. R. (1990). An alternative “description of personality”: The Big-Five factor structure. Journal of Personality and Social Psychology, 59, 1216–1229.

HANLEY, J.A. AND MCNEIL, B.J. (1982) The meaning and use of the Area Under a Receiver Operating Characteristic (ROC) Curve. Radiology, 143, 29-36.

HECKMAN, J., HSEE, J. AND Y. RUBINSTEIN (2001). The GED is a mixed signal: The effect of cognitive and noncognitive skills on human capital and labor market. Working Paper, University of Chicago.

HEFFERNAN, N., HEFFERNAN, C., DIETZ, K., SOFFER, D., PELLEGRINO, J. W., GOLDMAN, S. R., & DAILEY, M. (2012). Improving mathematical learning outcomes through automatic reassessment and relearning. In Annual Meeting of American Educational Research Association, Vancouver, British Columbia.

HEFFERNAN, N. T., & HEFFERNAN, C. L. (2014). The ASSISTments ecosystem: Building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching. International Journal of Artificial Intelligence in Education, 24(4), 470-497.

HUANG, X., CRAIG, S. D., XIE, J., GRAESSER, A., & HU, X. (2016). Intelligent tutoring systems work as a math gap reducer in 6th grade after-school program. Learning and Individual Differences, 47, 258-265.

KIM, Y. J., ALMOND, R. G., & SHUTE, V. J. (2016). Applying evidence-centered design for the development of game-based assessments in physics playground. International Journal of Testing, 16(2), 142-163.

KLEIN, R., SPADY, R., & WEISS, A. (1991). Factors affecting the output and quit propensities of production workers. The Review of Economic Studies, 58(5), 929-953.

KULIK, J. A., & FLETCHER, J. D. (2016). Effectiveness of intelligent tutoring systems: a metaanalytic review. Review of Educational Research, 86(1), 42-78.

LAURSEN, E. K. (2015). The power of grit, perseverance, and tenacity. Reclaiming Children and Youth, 23(4), 19-24.

JOHN, O. P., & SRIVASTAVA, S. (1999). The Big Five trait taxonomy: History, measurement, and theoretical perspectives. Handbook of personality: Theory and research, 2(1999), 102- 138.

MA, W., ADESOPE, O. O., NESBIT, J. C., & LIU, Q. (2014). Intelligent tutoring systems and learning outcomes: A meta-analysis. Journal of Educational Psychology, 106(4), 901-918.

MACCANN, C., & ROBERTS, R. D. (2010). Do time management, grit, and self-control relate to academic achievement independently of conscientiousness? In R. Hicks (Ed.), Personality and Individual Differences: Current Directions (pp. 79–90). Queensland, Australia: Australian Academic Press.

MCCRAE, R. R., & COSTA, P. T.,JR. (1987). Validation of the five-factor model of personality across instruments and observers. Journal of Personality and Social Psychology, 52, 81–90.

MATSUDA, N., CHANDRASEKARAN, S., & STAMPER, J. (2016). How quickly can wheel-spinning be detected? In T. Barnes, M. Chi, & M. Feng, (Eds.), Proceedings of the International Conference on Educational Data Mining, 607-608.

MIERSWA, I., WURST, M., KLINKENBERG, R., SCHOLZ, M., & EULER, T. (2006). Yale: Rapid prototyping for complex data mining tasks. In L. Ungar, M. Craven, D. Gunopulos, & T. Eliassi-Rad (Eds.), Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 935-940.

MONTESSORI, M. (1912). The Discovery of the Child. India: Kalakshetra Publications. NELSON-LE GALL, S. (1981). Help-seeking: An understudied problem-solving skill in children. Developmental Review, 1(3), 224-246.

NYHUS, E. AND PONS, E. (2005). The effect of personality on earnings. Journal of Economic Psychology, 26, 363–384.

PALAOAG, T. D., RODRIGO, M. M. T., ANDRES, J. M. L., ANDRES, J. M. A. L., & BECK, J. E. (2016). Wheel-spinning in a game-based learning environment for physics. In A. Micarelli, J. Stamper, & K. Panourgia (Eds.), Proceedings of the International Conference on Intelligent Tutoring Systems (pp. 234-239). Springer International Publishing.

PANE, J. F., GRIFFIN, B. A., MCCAFFREY, D. F., & KARAM, R. (2013). Effectiveness of Cognitive Tutor Algebra I at scale. Educational Evaluation and Policy Analysis, 36(2), 127-144.

PARDOS, Z.A., BAKER, R.S., SAN PEDRO, M.O.C.Z., GOWDA, S.M., GOWDA, S.M. (2014) Affective states and state tests: Investigating how affect and engagement during the school year predict end of year learning outcomes. Journal of Learning Analytics, 1 (1), 107-128.

PAUNONEN, S. V., & ASHTON, M. C. (2001). Big five predictors of academic achievement. Journal of Research in Personality, 35(1), 78–90.

POROPAT, A. E. (2009). A meta-analysis of the five-factor model of personality and academic performance. Psychological Bulletin, 135, 322–338.

PRABHU, V., SUTTON, C., & SAUSER, W. (2008). Creativity and certain personality traits: Understanding the mediating effect of intrinsic motivation. Creativity Research Journal, 20(1), 53-66.

QUINLAN, J. R. (1993). C4.5: Programs for Machine Learning. San Mateo, CA: Morgan Kaufmann.

RANK, J., PACE, V. L., & FRESE, M. (2004). Three avenues for future research on creativity, innovation, and initiative. Applied Psychology: An International Review, 53, 518–528.

REVELL, A. D., WANG, D., WOOD, R., MORROW, C., TEMPELMAN, H., HAMERS, R. L., ALVAREZ- URIA, G., STREINU-CERCEL, A., ENE, L., WENSING, A.M.J., DEWOLF, F., NELSON, M., MONTANER, J.S., LANE, H.C., LARDER, B.A. (2013). Computational models can predict response to HIV therapy without a genotype and may reduce treatment failure in different resource-limited settings. Journal of Antimicrobial Chemotherapy, 68 (6), 1406-1414.

RITTER, S., ANDERSON, J. R., KOEDINGER, K. R., & CORBETT, A. (2007). Cognitive Tutor: Applied research in mathematics education. Psychonomic Bulletin & Review, 14(2), 249- 255.

ROBERTS, B. W., CHERNYSHENKO, O. S., STARK, S., & GOLDBERG, L. R. (2005). The structure of conscientiousness: An empirical investigation based on seven major personality questionnaires. Personnel Psychology, 58(1), 103-139.

ROHRER, D., DEDRICK, R. F., & BURGESS, K. (2014). The benefit of interleaved mathematics practice is not limited to superficially similar kinds of problems. Psychonomic Bulletin & Review, 21(5), 1323-1330.

ROHRER, D., & TAYLOR, K. (2007). The shuffling of mathematics problems improves learning. Instructional Science, 35(6), 481-498.

ROSCHELLE, J., FENG, M., MURPHY, R. F., & MASON, C. A. (2016). Online mathematics homework increases student achievement. AERA Open, 2(4), 1-12.

ROSCHELLE, J., TATAR, D., SCHECHTMAN,N., HEGEDUS, S., HOPKINS, W., KNUDSON, J. & STROTER, A. (2007). Scaling up SimCalc project: Can a technology enhanced curriculum improve student learning of important mathematics? Menlo Park, CA: SRI international.

SAN PEDRO, M.O.Z., BAKER, R.S.J.D., BOWERS, A.J., HEFFERNAN, N.T. (2013) Predicting college enrollment from student interaction with an intelligent tutoring system in middle school. In S. D'Mello, R. Calvo, & A. Olney (Eds.), Proceedings of the 6th International Conference on Educational Data Mining, 177-184.

SEDEK, G., & KOFTA, M. (1990). When cognitive exertion does not yield cognitive gain: Toward an informational explanation of learned helplessness. Journal of Personality and Social Psychology, 58(4), 729-743.

SETTLES, B., & MEEDER, B. (2016). A trainable spaced repetition model for language learning. In K. Erk & N. Smith (Eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 1848–1858.

STEENBERGEN-HU, S., & COOPER, H. (2013). A meta-analysis of the effectiveness of intelligent tutoring systems on K–12 students' mathematical learning. Journal of Educational Psychology, 105(4), 970-987.

STRAYHORN, TERRELL L. (2014). What role does grit play in the academic success of black male collegians at predominantly White institutions? Journal of African American Studies, 18(1), 1-10.

TETT, R. P., JACKSON, D. N., & ROTHSTEIN, M. (1991). Personality measures as predictors of job performance: A meta-analytic review. Personnel Psychology, 44(4), 703–742.

VENTURA, M., & SHUTE, V. (2013). The validity of a game-based assessment of persistence. Computers in Human Behavior, 29(6), 2568-2572.

Most read articles by the same author(s)

1 2 > >>