Extending the Hint Factory for the Assistance Dilemma: A Novel, Data-driven HelpNeed Predictor for Proactive Problem-solving Help
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
Dec 29, 2020
Mehak Maniktala
Christa Cody
Amy Isvik
Nicholas Lytle
Min Chi
Tiffany Barnes
https://orcid.org/0000-0002-8779-6616
Abstract
Determining when and whether to provide personalized support is a well-known challenge called the assistance dilemma. A core problem in solving the assistance dilemma is the need to discover when students are unproductive so that the tutor can intervene. Such a task is particularly challenging for open-ended domains, even those that are well-structured with defined principles and goals. We present a set of data-driven methods to classify, predict, and prevent unproductive problem-solving steps in the well-structured open-ended domain of logic. This approach leverages and extends the Hint Factory, a set of methods that leverages prior student solution attempts to build data-driven intelligent tutors. We present a HelpNeed classification that uses prior student data to determine when students are likely to be unproductive and need help learning optimal problem-solving strategies. We present a controlled study to determine the impact of an Adaptive pedagogical policy that provides proactive hints at the start of each step based on the outcomes of our HelpNeed predictor: productive vs. unproductive. Our results show that the students in the Adaptive condition exhibited better training behaviors, with lower help avoidance, and higher help appropriateness (a higher chance of receiving help when it was likely to be needed), as measured using the HelpNeed classifier, when compared to the Control. Furthermore, the results show that the students who received Adaptive hints based on HelpNeed predictions during training significantly outperform their Control peers on the posttest, with the former producing shorter, more optimal solutions in less time. We conclude with suggestions on how these HelpNeed methods could be applied in other well-structured open-ended domains.
How to Cite
Maniktala, M., Cody, C., Isvik, A., Lytle, N., Chi, M., & Barnes, T. (2020). Extending the Hint Factory for the Assistance Dilemma: A Novel, Data-driven HelpNeed Predictor for Proactive Problem-solving Help. Journal of Educational Data Mining, 12(4), 24–65. https://doi.org/10.5281/zenodo.4399683
##plugins.themes.bootstrap3.article.details##
Keywords
adaptive support, propositional logic, assistance dilemma, problem solving, unproductivity, efficiency, unsolicited hints, data-driven tutoring, help appropriateness
References
ALEVEN, V. AND KOEDINGER, K. R. 2000. Limitations of student control: Do students know when they need help? In Proceedings of the 5th International Conference on Intelligent Tutoring Systems, G. Gauthier, C. Frasson, and K. VanLehn, Eds. Springer, Berlin, Heidelberg, 292–303.
ALEVEN, V., MCLAREN, B., ROLL, I., AND KOEDINGER, K. 2006. Toward meta-cognitive tutoring: A model of help seeking with a cognitive tutor. International Journal of Artificial Intelligence in Education 16, 2, 101–128.
ANOHINA, A. 2007. Advances in intelligent tutoring systems: problem-solving modes and model of hints. International Journal of Computers Communications & Control 2, 1, 48–55.
ARROYO, I., BECK, J. E., BEAL, C. R., WING, R., AND WOOLF, B. P. 2001. Analyzing students' response to help provision in an elementary mathematics intelligent tutoring system. In Papers of the Workshop on Help Provision and Help Seeking in Interactive Learning Environments at the Tenth International Conference on Artificial Intelligence in Education. 34–46.
ARVAI, K. 2018. Fine tuning a classifier in scikit-learn. https://towardsdatascience.com/fine-tuning-a-classifier-in-scikit-learn-66e048c21e65.
AZEVEDO, R. AND CROMLEY, J. G. 2004. Does training on self-regulated learning facilitate students' learning with hypermedia? Journal of Educational Psychology 96, 3, 523–535.
BAKER, R. S., CORBETT, A. T., KOEDINGER, K. R., EVENSON, S., ROLL, I., WAGNER, A. Z., NAIM, M., RASPAT, J., BAKER, D. J., AND BECK, J. E. 2006. Adapting to when students game an intelligent tutoring system. In Proceedings of the 8th International Conference on Intelligent Tutoring Systems, M. Ikeda, K. D. Ashley, and T.-W. Chan, Eds. Springer, 392–401.
BAKER, R. S., CORBETT, A. T., ROLL, I., AND KOEDINGER, K. R. 2008. Developing a generalizable detector of when students game the system. User Modeling and User-Adapted Interaction 18, 3, 287–314.
BAKER, R. S., D E CARVALHO, A., RASPAT, J., ALEVEN, V., CORBETT, A. T., AND KOEDINGER, K. R. 2009. Educational software features that encourage and discourage "gaming the system". In Proceedings of the 14th International Conference on Artificial Intelligence in Education, V. Aleven, J. Kay, and J. Mostow, Eds. 475–482.
BARNES, T. AND STAMPER, J. 2010. Automatic hint generation for logic proof tutoring using historical data. Journal of Educational Technology & Society 13, 1, 3–12.
BARNES, T., STAMPER, J., AND CROY, M. 2011. Using Markov decision processes for automatic hint generation. In Handbook of Educational Data Mining, C. Romero, S. Ventura, M. Pechenizkiy, and R. S. J. de Baker, Eds. Chapman & Hall/CRC Data Mining and Knowledge Discovery Series. CRC Press, Chapter 33, 467–480.
BARNES, T., STAMPER, J. C., LEHMANN, L., AND CROY, M. J. 2008. A pilot study on logic proof tutoring using hints generated from historical student data. In Proceedings of the 1st International Conference on Educational Data Mining, R. S. J. de Baker, T. Barnes, and J. E. Beck, Eds. 197–201.
BARTHOLOM É, T., STAHL, E., PIESCHL, S., AND BROMME, R. 2006. What matters in help-seeking? a study of help effectiveness and learner-related factors. Computers in Human Behavior 22, 1, 113–129.
BECK, J. E., CHANG, K.M ., MOSTOW, J., AND CORBETT, A. 2008. Does help help? Introducing the Bayesian evaluation and assessment methodology. In Proceedings of the 9th International Conference on Intelligent Tutoring Systems, B. P. Woolf, E. Aı̈meur, R. Nkambou, and S. Lajoie, Eds. Springer, Berlin, Heidelberg, 383–394.
BECK, J. E. AND GONG, Y. 2013. Wheel-spinning: Students who fail to master a skill. In Proceedings of the 16th International Conference on Artificial Intelligence in Education, H. C. Lane, K. Yacef, J. Mostow, and P. Pavlik, Eds. Springer, Berlin, Heidelberg, 431–440.
BOREK, A., M C LAREN, B. M., KARABINOS, M., AND YARON, D. 2009. How much assistance is helpful to students in discovery learning? In Proceedings of the 2009 European Conference on Technology Enhanced Learning, U. Cress, V. Dimitrova, and M. Specht, Eds. Springer, Berlin, Heidelberg, 391–404.
BOTELHO, A., VARATHARAJ, A., PATIKORN, T., DOHERTY, D., ADJEI, S., AND BECK, J. 2019. Developing early detectors of student attrition and wheel spinning using deep learning. IEEE Transactions on Learning Technologies 12, 2, 158–170.
BUNT, A. AND CONATI, C. 2003. Probabilistic student modelling to improve exploratory behaviour. User Modeling and User-Adapted Interaction 13, 3, 269–309.
BUNT, A., CONATI, C., AND MULDNER, K. 2004. Scaffolding self-explanation to improve learning in exploratory learning environments. In Proceedings of the 7th International Conference on Intelligent Tutoring Systems, J. C. Lester, R. M. Vicari, and F. Paraguaçu, Eds. Springer, Berlin, Heidelberg, 656–667.
CAPRARO, M. M., AN, S. A., MA, T., RANGEL-CHAVEZ, A. F., AND HARBAUGH, A. 2012. An investigation of preservice teachers' use of guess and check in solving a semi open-ended mathematics problem. The Journal of Mathematical Behavior 31, 1, 105–116.
CEN, H., KOEDINGER, K. R., AND JUNKER, B. 2007. Is over practice necessary? Improving learning efficiency with the cognitive tutor through educational data mining. In Proceedings of the 13th International Conference on Artificial Intelligence in Education, R. Luckin, K. R. Koedinger, and J. Greer, Eds. Frontiers in Artificial Intelligence and Applications, vol. 158. IOS Press, NLD, 511–518.
CHAUDHRY, R., SINGH, H., DOGGA, P., AND SAINI, S. K. 2018. Modeling hint-taking behavior and knowledge state of students with multi-task learning. In Proceedings of the 11th International Conference on Educational Data Mining, K. E. Boyer and M. Yudelson, Eds. International Educational Data Mining Society, 21–31.
CHEN, T. AND GUESTRIN, C. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACMSIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 785–794.
CODY, C. AND MOSTAFAVI, B. 2017. Investigating the impact of unsolicited next-step and subgoal hints on dropout in a logic proof tutor. In Proceedings of the 2017 ACMSIGCSE Technical Symposium on Computer Science Education. ACM, 705–705.
CONATI, C., GERTNER, A., AND VANLEHN, K. 2002. Using Bayesian networks to manage uncertainty in student modeling. User Modeling and User-Adapted Interaction 12, 4, 371–417.
CORBETT, A. T. AND ANDERSON, J. R. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction 4, 4, 253–278.
DONG, Y., MARWAN, S., CATETE, V., PRICE, T., AND BARNES, T. 2019. Defining tinkering behavior in open-ended block-based programming assignments. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education. 1204–1210.
EAGLE, M., HICKS, D., AND BARNES, T. 2015. Interaction network estimation: Predicting problemsolving diversity in interactive environments. In Proceedings of the 8th International Conference on Educational Data Mining, O. C. Santos, J. Boticario, C. Romero, M. Pechenizkiy, A. Merceron, P. Mitros, J. M. Luna, M. C. Mihaescu, P. Moreno, A. Hershkovitz, S. Ventura, and M. C. Desmarais, Eds. International Educational Data Mining Society, 342–349.
FOSSATI, D., DIEUGENIO, B., OHLSSON, S., BROWN, C., AND CHEN, L. 2015. Data driven automatic feedback generation in the iList intelligent tutoring system. Technology, Instruction, Cognition and Learning 10, 1, 5–26.
FRATAMICO, L., CONATI, C., KARDAN, S., AND ROLL, I. 2017. Applying a framework for student modeling in exploratory learning environments: Comparing data representation granularity to handle environment complexity. International Journal of Artificial Intelligence in Education 27, 2, 320–352.
III, B. P., HICKS, A., AND BARNES, T. 2014. Generating hints for programming problems using intermediate output. In Proceedings of the 7th International Conference on Educational Data Mining, J. C. Stamper, Z. A. Pardos, M. Mavrikis, and B. M. McLaren, Eds. International Educational Data Mining Society, 92–98.
KAI, S., ALMEDA, M. V., BAKER, R. S., HEFFERNAN, C., AND HEFFERNAN, N. 2018. Decision tree modeling of wheel-spinning and productive persistence in skill builders. Journal of Educational Data Mining 10, 1, 36–71.
KARDAN, S. AND CONATI, C. 2015. Providing adaptive support in an interactive simulation for learning: An experimental evaluation. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 3671–3680.
KINNEBREW, J. S., SEGEDY, J. R., AND BISWAS, G. 2014. Analyzing the temporal evolution of students' behaviors in open-ended learning environments. Metacognition and Learning 9, 2, 187–215.
KLAHR, D. 2009. "To every thing there is a season, and a time to every purpose under the heavens": What about direct instruction? In Constructivist Instruction: Success or failure?, S. Tobias and T. M. Duffy, Eds. Routledge, New York, Chapter 15, 291–310.
KOCK, M. AND PARAMYTHIS, A. 2010. Towards adaptive learning support on the basis of behavioural patterns in learning activity sequences. In 2010 International Conference on Intelligent Networking and Collaborative Systems. IEEE, 100–107.
KOEDINGER, K. R. AND ALEVEN, V. 2007. Exploring the assistance dilemma in experiments with cognitive tutors. Educational Psychology Review 19, 3, 239–264.
MANIKTALA, M., CODY, C., BARNES, T., AND CHI, M. 2020. Avoiding help avoidance: Using interface design changes to promote unsolicited hint usage in an intelligent tutor. International Journal of Artificial Intelligence in Education 30, 4, 637–667.
MARWAN, S., DOMBE, A., AND PRICE, T. W. 2020. Unproductive help-seeking in programming: What it is and how to address it. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education. 54–60.
MAYER, R. E. 1992. Thinking, problem solving, cognition. WH Freeman/Times Books/Henry Holt & Co.
MCLAREN, B., TIMMS, M., WEIHNACHT, D., BRENNER, D., LUTTGEN, K., GRILLO -HILL, A., AND H. BROWN, D. 2014. A web-based system to support inquiry learning. In Proceedings of the 6th International Conference on Computer Supported Education Volume 1. SciTePress, Setubal, PRT, 43–52.
MCLAREN, B. M., LIM, S.-J., AND KOEDINGER, K. R. 2008. When and how often should worked examples be given to students? New results and a summary of the current state of research. InProceedings of the 30th Annual Conference of the Cognitive Science Society, B. C. Love, K. McRae, and V. M. Sloutsky, Eds. Cognitive Science Society, Austin, TX, 2176–2181.
MERCERON, A. AND YACEF, K. 2005. Educational data mining: a case study. In Proceedings of the 12th International Conference on Artificial Intelligence in Education, C. Looi, G. I. McCalla, B. Bredeweg, and J. Breuker, Eds. Frontiers in Artificial Intelligence and Applications, vol. 125. IOS Press, 467–474.
MOSTAFAVI, B. AND BARNES, T. 2017. Evolution of an intelligent deductive logic tutor using datadriven elements. International Journal of Artificial Intelligence in Education 27, 1, 5–36.
MOSTAFAVI, B., LIU, Z., AND BARNES, T. 2015. Data-driven proficiency profiling. In Proceedings of the 8th International Conference on Educational Data Mining, O. C. Santos, J. Boticario, C. Romero, M. Pechenizkiy, A. Merceron, P. Mitros, J. M. Luna, M. C. Mihaescu, P. Moreno, A. Hershkovitz, S. Ventura, and M. C. Desmarais, Eds. International Educational Data Mining Society, 335–341.
MURRAY, R. C. AND VANLEHN, K. 2005. Effects of dissuading unnecessary help requests while providing proactive help. In Proceedings of the 12th International Conference on Artificial Intelligence in Education, C. Looi, G. I. McCalla, B. Bredeweg, and J. Breuker, Eds. Frontiers in Artificial Intelligence and Applications, vol. 125. IOS Press, 887–889.
MURRAY, R. C. AND VANLEHN, K. 2006. A comparison of decision-theoretic, fixed-policy and random tutorial action selection. In Proceedings of the 8th International Conference on Intelligent Tutoring Systems, M. Ikeda, K. D. Ashley, and T. Chan, Eds. Lecture Notes in Computer Science, vol. 4053. Springer, 114–123.
MURRAY, R. C., VANLEHN, K., AND MOSTOW, J. 2004. Looking ahead to select tutorial actions: A decision-theoretic approach. International Journal of Artificial Intelligence in Education 14, 3, 4, 235–278.
PARK, S. AND MATSUDA, N. 2018. Predicting students' unproductive failure on intelligent tutors in adaptive online courseware. In Proceedings of the Sixth Annual GIFT Users Symposium. Vol. 6. US Army Research Laboratory, 131–138.
PEDREGOSA, F., VAROQUAUX, G., GRAMFORT, A., MICHEL, V., THIRION, B., GRISEL, O., BLONDEL, M., PRETTENHOFER, P., WEISS, R., DUBOURG, V., ETAL . 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, Oct, 2825–2830.
PEÑA, A., KAYASHIMA, M., MIZOGUCHI, R., AND DOMINGUEZ, R. 2011. Improving students' metacognitive skills within intelligent educational systems: A review. In Proceedings of the 6th International Conference on Foundations of Augmented Cognition, D. D. Schmorrow and C. M. Fidopiastis, Eds. Springer, Berlin, Heidelberg, 442–451.
POLYA, G. 2004. How to solve it: A new aspect of mathematical method. Princeton University Press.
PRICE, T. W., DONG, Y., AND BARNES, T. 2016. Generating data-driven hints for open-ended programming. In Proceedings of the 9th International Conference on Educational Data Mining, T. Barnes, M. Chi, and M. Feng, Eds. International Educational Data Mining Society, 191–198.
PRICE, T. W., LIU, Z., CATET É, V., AND BARNES, T. 2017. Factors influencing students' help-seeking behavior while programming with human and computer tutors. In Proceedings of the 2017 ACM Conference on International Computing Education Research. ACM, 127–135.
PRICE, T. W., ZHI, R., AND BARNES, T. 2017. Hint generation under uncertainty: The effect of hint quality on help-seeking behavior. In Proceedings of the 18th International Conference on Artificial Intelligence in Education, E. André, R. S. Baker, X. Hu, M. M. T. Rodrigo, and B. du Boulay, Eds. Lecture Notes in Computer Science, vol. 10331. Springer, 311–322.
PUUSTINEN, M. 1998. Help-seeking behavior in a problem-solving situation: Development of selfregulation. European Journal of Psychology of Education 13, 2, 271–282.
RIVERS, K. AND KOEDINGER, K. R. 2013. Automatic generation of programming feedback: A datadriven approach. In Proceedings of the First Workshop on AI-supported Education for Computer Science at the 16th International Conference on Artificial Intelligence in Education, E. Walker and C. Looi, Eds. CEUR Workshop Proceedings, vol. 1009. 50–59.
RIVERS, K. AND KOEDINGER, K. R. 2017. Data-driven hint generation in vast solution spaces: a self-improving Python programming tutor. International Journal of Artificial Intelligence in Education 27, 1, 37–64.
ROLL, I., ALEVEN, V., M C LAREN, B. M., AND KOEDINGER, K. R. 2011. Improving students' helpseeking skills using metacognitive feedback in an intelligent tutoring system. Learning and Instruction 21, 2, 267–280.
RUS, V., BANJADE, R., NIRAULA, N., GIRE, E., AND FRANCESCHETTI, D. 2017. A study on two hint-level policies in conversational intelligent tutoring systems. In Innovations in Smart Learning, E. Popescu, Kinshuk, M. K. Khribi, R. Huang, M. Jemni, N.-S. Chen, and D. G. Sampson, Eds. Springer, Singapore, 175–184.
SMITH, M. U. 2012. Toward a Unified Theory of Problem Solving: Views From the Content Domains. Routledge, New York.
STAMPER, J. AND BARNES, T. 2009. An unsupervised, frequency-based metric for selecting hints in an MDP-based tutor. In Proceedings of the 2nd International Conference on Educational Data Mining, T. Barnes, M. C. Desmarais, C. Romero, and S. Ventura, Eds. International Educational Data Mining Society, 181–190.
STAMPER, J., BARNES, T., LEHMANN, L., AND CROY, M. 2008. The Hint Factory: Automatic generation of contextualized help for existing computer aided instruction. In Proceedings of the 9th International Conference on Intelligent Tutoring Systems Young Researchers Track. 71–78.
TCHÉTAGNI, J. M. AND NKAMBOU, R. 2002. Hierarchical representation and evaluation of the student in an intelligent tutoring system. In Proceedings of the 6th International Conference on Intelligent Tutoring Systems, S. A. Cerri, G. Gouardères, and F. Paraguaçu, Eds. Lecture Notes in Computer Science, vol. 2363. Springer, 708–717.
UENO, M. AND MIYAZAWA, Y. 2017. IRT-based adaptive hints to scaffold learning in programming. IEEE Transactions on Learning Technologies 11, 4, 415–428.
VANLEHN, K. 2011. The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist 46, 4, 197–221.
WOOD, H. AND WOOD, D. 1999. Help seeking, learning and contingent tutoring. Computers & Education 33, 2-3, 153–169.
YACEF, K. 2005. The Logic-ITA in the classroom: a medium scale experiment. International Journal of Artificial Intelligence in Education 15, 1, 41–62.
ZHOU, G., AZIZSOLTANI, H., AUSIN, M. S., BARNES, T., AND CHI, M. 2019. Hierarchical reinforcement learning for pedagogical policy induction. In Proceedings of the 20th International Conference on Artificial Intelligence in Education, S. Isotani, E. Millán, A. Ogan, P. M. Hastings, B. M. McLaren, and R. Luckin, Eds. Lecture Notes in Computer Science, vol. 11625. Springer, 544–556.
ALEVEN, V., MCLAREN, B., ROLL, I., AND KOEDINGER, K. 2006. Toward meta-cognitive tutoring: A model of help seeking with a cognitive tutor. International Journal of Artificial Intelligence in Education 16, 2, 101–128.
ANOHINA, A. 2007. Advances in intelligent tutoring systems: problem-solving modes and model of hints. International Journal of Computers Communications & Control 2, 1, 48–55.
ARROYO, I., BECK, J. E., BEAL, C. R., WING, R., AND WOOLF, B. P. 2001. Analyzing students' response to help provision in an elementary mathematics intelligent tutoring system. In Papers of the Workshop on Help Provision and Help Seeking in Interactive Learning Environments at the Tenth International Conference on Artificial Intelligence in Education. 34–46.
ARVAI, K. 2018. Fine tuning a classifier in scikit-learn. https://towardsdatascience.com/fine-tuning-a-classifier-in-scikit-learn-66e048c21e65.
AZEVEDO, R. AND CROMLEY, J. G. 2004. Does training on self-regulated learning facilitate students' learning with hypermedia? Journal of Educational Psychology 96, 3, 523–535.
BAKER, R. S., CORBETT, A. T., KOEDINGER, K. R., EVENSON, S., ROLL, I., WAGNER, A. Z., NAIM, M., RASPAT, J., BAKER, D. J., AND BECK, J. E. 2006. Adapting to when students game an intelligent tutoring system. In Proceedings of the 8th International Conference on Intelligent Tutoring Systems, M. Ikeda, K. D. Ashley, and T.-W. Chan, Eds. Springer, 392–401.
BAKER, R. S., CORBETT, A. T., ROLL, I., AND KOEDINGER, K. R. 2008. Developing a generalizable detector of when students game the system. User Modeling and User-Adapted Interaction 18, 3, 287–314.
BAKER, R. S., D E CARVALHO, A., RASPAT, J., ALEVEN, V., CORBETT, A. T., AND KOEDINGER, K. R. 2009. Educational software features that encourage and discourage "gaming the system". In Proceedings of the 14th International Conference on Artificial Intelligence in Education, V. Aleven, J. Kay, and J. Mostow, Eds. 475–482.
BARNES, T. AND STAMPER, J. 2010. Automatic hint generation for logic proof tutoring using historical data. Journal of Educational Technology & Society 13, 1, 3–12.
BARNES, T., STAMPER, J., AND CROY, M. 2011. Using Markov decision processes for automatic hint generation. In Handbook of Educational Data Mining, C. Romero, S. Ventura, M. Pechenizkiy, and R. S. J. de Baker, Eds. Chapman & Hall/CRC Data Mining and Knowledge Discovery Series. CRC Press, Chapter 33, 467–480.
BARNES, T., STAMPER, J. C., LEHMANN, L., AND CROY, M. J. 2008. A pilot study on logic proof tutoring using hints generated from historical student data. In Proceedings of the 1st International Conference on Educational Data Mining, R. S. J. de Baker, T. Barnes, and J. E. Beck, Eds. 197–201.
BARTHOLOM É, T., STAHL, E., PIESCHL, S., AND BROMME, R. 2006. What matters in help-seeking? a study of help effectiveness and learner-related factors. Computers in Human Behavior 22, 1, 113–129.
BECK, J. E., CHANG, K.M ., MOSTOW, J., AND CORBETT, A. 2008. Does help help? Introducing the Bayesian evaluation and assessment methodology. In Proceedings of the 9th International Conference on Intelligent Tutoring Systems, B. P. Woolf, E. Aı̈meur, R. Nkambou, and S. Lajoie, Eds. Springer, Berlin, Heidelberg, 383–394.
BECK, J. E. AND GONG, Y. 2013. Wheel-spinning: Students who fail to master a skill. In Proceedings of the 16th International Conference on Artificial Intelligence in Education, H. C. Lane, K. Yacef, J. Mostow, and P. Pavlik, Eds. Springer, Berlin, Heidelberg, 431–440.
BOREK, A., M C LAREN, B. M., KARABINOS, M., AND YARON, D. 2009. How much assistance is helpful to students in discovery learning? In Proceedings of the 2009 European Conference on Technology Enhanced Learning, U. Cress, V. Dimitrova, and M. Specht, Eds. Springer, Berlin, Heidelberg, 391–404.
BOTELHO, A., VARATHARAJ, A., PATIKORN, T., DOHERTY, D., ADJEI, S., AND BECK, J. 2019. Developing early detectors of student attrition and wheel spinning using deep learning. IEEE Transactions on Learning Technologies 12, 2, 158–170.
BUNT, A. AND CONATI, C. 2003. Probabilistic student modelling to improve exploratory behaviour. User Modeling and User-Adapted Interaction 13, 3, 269–309.
BUNT, A., CONATI, C., AND MULDNER, K. 2004. Scaffolding self-explanation to improve learning in exploratory learning environments. In Proceedings of the 7th International Conference on Intelligent Tutoring Systems, J. C. Lester, R. M. Vicari, and F. Paraguaçu, Eds. Springer, Berlin, Heidelberg, 656–667.
CAPRARO, M. M., AN, S. A., MA, T., RANGEL-CHAVEZ, A. F., AND HARBAUGH, A. 2012. An investigation of preservice teachers' use of guess and check in solving a semi open-ended mathematics problem. The Journal of Mathematical Behavior 31, 1, 105–116.
CEN, H., KOEDINGER, K. R., AND JUNKER, B. 2007. Is over practice necessary? Improving learning efficiency with the cognitive tutor through educational data mining. In Proceedings of the 13th International Conference on Artificial Intelligence in Education, R. Luckin, K. R. Koedinger, and J. Greer, Eds. Frontiers in Artificial Intelligence and Applications, vol. 158. IOS Press, NLD, 511–518.
CHAUDHRY, R., SINGH, H., DOGGA, P., AND SAINI, S. K. 2018. Modeling hint-taking behavior and knowledge state of students with multi-task learning. In Proceedings of the 11th International Conference on Educational Data Mining, K. E. Boyer and M. Yudelson, Eds. International Educational Data Mining Society, 21–31.
CHEN, T. AND GUESTRIN, C. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACMSIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 785–794.
CODY, C. AND MOSTAFAVI, B. 2017. Investigating the impact of unsolicited next-step and subgoal hints on dropout in a logic proof tutor. In Proceedings of the 2017 ACMSIGCSE Technical Symposium on Computer Science Education. ACM, 705–705.
CONATI, C., GERTNER, A., AND VANLEHN, K. 2002. Using Bayesian networks to manage uncertainty in student modeling. User Modeling and User-Adapted Interaction 12, 4, 371–417.
CORBETT, A. T. AND ANDERSON, J. R. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction 4, 4, 253–278.
DONG, Y., MARWAN, S., CATETE, V., PRICE, T., AND BARNES, T. 2019. Defining tinkering behavior in open-ended block-based programming assignments. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education. 1204–1210.
EAGLE, M., HICKS, D., AND BARNES, T. 2015. Interaction network estimation: Predicting problemsolving diversity in interactive environments. In Proceedings of the 8th International Conference on Educational Data Mining, O. C. Santos, J. Boticario, C. Romero, M. Pechenizkiy, A. Merceron, P. Mitros, J. M. Luna, M. C. Mihaescu, P. Moreno, A. Hershkovitz, S. Ventura, and M. C. Desmarais, Eds. International Educational Data Mining Society, 342–349.
FOSSATI, D., DIEUGENIO, B., OHLSSON, S., BROWN, C., AND CHEN, L. 2015. Data driven automatic feedback generation in the iList intelligent tutoring system. Technology, Instruction, Cognition and Learning 10, 1, 5–26.
FRATAMICO, L., CONATI, C., KARDAN, S., AND ROLL, I. 2017. Applying a framework for student modeling in exploratory learning environments: Comparing data representation granularity to handle environment complexity. International Journal of Artificial Intelligence in Education 27, 2, 320–352.
III, B. P., HICKS, A., AND BARNES, T. 2014. Generating hints for programming problems using intermediate output. In Proceedings of the 7th International Conference on Educational Data Mining, J. C. Stamper, Z. A. Pardos, M. Mavrikis, and B. M. McLaren, Eds. International Educational Data Mining Society, 92–98.
KAI, S., ALMEDA, M. V., BAKER, R. S., HEFFERNAN, C., AND HEFFERNAN, N. 2018. Decision tree modeling of wheel-spinning and productive persistence in skill builders. Journal of Educational Data Mining 10, 1, 36–71.
KARDAN, S. AND CONATI, C. 2015. Providing adaptive support in an interactive simulation for learning: An experimental evaluation. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 3671–3680.
KINNEBREW, J. S., SEGEDY, J. R., AND BISWAS, G. 2014. Analyzing the temporal evolution of students' behaviors in open-ended learning environments. Metacognition and Learning 9, 2, 187–215.
KLAHR, D. 2009. "To every thing there is a season, and a time to every purpose under the heavens": What about direct instruction? In Constructivist Instruction: Success or failure?, S. Tobias and T. M. Duffy, Eds. Routledge, New York, Chapter 15, 291–310.
KOCK, M. AND PARAMYTHIS, A. 2010. Towards adaptive learning support on the basis of behavioural patterns in learning activity sequences. In 2010 International Conference on Intelligent Networking and Collaborative Systems. IEEE, 100–107.
KOEDINGER, K. R. AND ALEVEN, V. 2007. Exploring the assistance dilemma in experiments with cognitive tutors. Educational Psychology Review 19, 3, 239–264.
MANIKTALA, M., CODY, C., BARNES, T., AND CHI, M. 2020. Avoiding help avoidance: Using interface design changes to promote unsolicited hint usage in an intelligent tutor. International Journal of Artificial Intelligence in Education 30, 4, 637–667.
MARWAN, S., DOMBE, A., AND PRICE, T. W. 2020. Unproductive help-seeking in programming: What it is and how to address it. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education. 54–60.
MAYER, R. E. 1992. Thinking, problem solving, cognition. WH Freeman/Times Books/Henry Holt & Co.
MCLAREN, B., TIMMS, M., WEIHNACHT, D., BRENNER, D., LUTTGEN, K., GRILLO -HILL, A., AND H. BROWN, D. 2014. A web-based system to support inquiry learning. In Proceedings of the 6th International Conference on Computer Supported Education Volume 1. SciTePress, Setubal, PRT, 43–52.
MCLAREN, B. M., LIM, S.-J., AND KOEDINGER, K. R. 2008. When and how often should worked examples be given to students? New results and a summary of the current state of research. InProceedings of the 30th Annual Conference of the Cognitive Science Society, B. C. Love, K. McRae, and V. M. Sloutsky, Eds. Cognitive Science Society, Austin, TX, 2176–2181.
MERCERON, A. AND YACEF, K. 2005. Educational data mining: a case study. In Proceedings of the 12th International Conference on Artificial Intelligence in Education, C. Looi, G. I. McCalla, B. Bredeweg, and J. Breuker, Eds. Frontiers in Artificial Intelligence and Applications, vol. 125. IOS Press, 467–474.
MOSTAFAVI, B. AND BARNES, T. 2017. Evolution of an intelligent deductive logic tutor using datadriven elements. International Journal of Artificial Intelligence in Education 27, 1, 5–36.
MOSTAFAVI, B., LIU, Z., AND BARNES, T. 2015. Data-driven proficiency profiling. In Proceedings of the 8th International Conference on Educational Data Mining, O. C. Santos, J. Boticario, C. Romero, M. Pechenizkiy, A. Merceron, P. Mitros, J. M. Luna, M. C. Mihaescu, P. Moreno, A. Hershkovitz, S. Ventura, and M. C. Desmarais, Eds. International Educational Data Mining Society, 335–341.
MURRAY, R. C. AND VANLEHN, K. 2005. Effects of dissuading unnecessary help requests while providing proactive help. In Proceedings of the 12th International Conference on Artificial Intelligence in Education, C. Looi, G. I. McCalla, B. Bredeweg, and J. Breuker, Eds. Frontiers in Artificial Intelligence and Applications, vol. 125. IOS Press, 887–889.
MURRAY, R. C. AND VANLEHN, K. 2006. A comparison of decision-theoretic, fixed-policy and random tutorial action selection. In Proceedings of the 8th International Conference on Intelligent Tutoring Systems, M. Ikeda, K. D. Ashley, and T. Chan, Eds. Lecture Notes in Computer Science, vol. 4053. Springer, 114–123.
MURRAY, R. C., VANLEHN, K., AND MOSTOW, J. 2004. Looking ahead to select tutorial actions: A decision-theoretic approach. International Journal of Artificial Intelligence in Education 14, 3, 4, 235–278.
PARK, S. AND MATSUDA, N. 2018. Predicting students' unproductive failure on intelligent tutors in adaptive online courseware. In Proceedings of the Sixth Annual GIFT Users Symposium. Vol. 6. US Army Research Laboratory, 131–138.
PEDREGOSA, F., VAROQUAUX, G., GRAMFORT, A., MICHEL, V., THIRION, B., GRISEL, O., BLONDEL, M., PRETTENHOFER, P., WEISS, R., DUBOURG, V., ETAL . 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, Oct, 2825–2830.
PEÑA, A., KAYASHIMA, M., MIZOGUCHI, R., AND DOMINGUEZ, R. 2011. Improving students' metacognitive skills within intelligent educational systems: A review. In Proceedings of the 6th International Conference on Foundations of Augmented Cognition, D. D. Schmorrow and C. M. Fidopiastis, Eds. Springer, Berlin, Heidelberg, 442–451.
POLYA, G. 2004. How to solve it: A new aspect of mathematical method. Princeton University Press.
PRICE, T. W., DONG, Y., AND BARNES, T. 2016. Generating data-driven hints for open-ended programming. In Proceedings of the 9th International Conference on Educational Data Mining, T. Barnes, M. Chi, and M. Feng, Eds. International Educational Data Mining Society, 191–198.
PRICE, T. W., LIU, Z., CATET É, V., AND BARNES, T. 2017. Factors influencing students' help-seeking behavior while programming with human and computer tutors. In Proceedings of the 2017 ACM Conference on International Computing Education Research. ACM, 127–135.
PRICE, T. W., ZHI, R., AND BARNES, T. 2017. Hint generation under uncertainty: The effect of hint quality on help-seeking behavior. In Proceedings of the 18th International Conference on Artificial Intelligence in Education, E. André, R. S. Baker, X. Hu, M. M. T. Rodrigo, and B. du Boulay, Eds. Lecture Notes in Computer Science, vol. 10331. Springer, 311–322.
PUUSTINEN, M. 1998. Help-seeking behavior in a problem-solving situation: Development of selfregulation. European Journal of Psychology of Education 13, 2, 271–282.
RIVERS, K. AND KOEDINGER, K. R. 2013. Automatic generation of programming feedback: A datadriven approach. In Proceedings of the First Workshop on AI-supported Education for Computer Science at the 16th International Conference on Artificial Intelligence in Education, E. Walker and C. Looi, Eds. CEUR Workshop Proceedings, vol. 1009. 50–59.
RIVERS, K. AND KOEDINGER, K. R. 2017. Data-driven hint generation in vast solution spaces: a self-improving Python programming tutor. International Journal of Artificial Intelligence in Education 27, 1, 37–64.
ROLL, I., ALEVEN, V., M C LAREN, B. M., AND KOEDINGER, K. R. 2011. Improving students' helpseeking skills using metacognitive feedback in an intelligent tutoring system. Learning and Instruction 21, 2, 267–280.
RUS, V., BANJADE, R., NIRAULA, N., GIRE, E., AND FRANCESCHETTI, D. 2017. A study on two hint-level policies in conversational intelligent tutoring systems. In Innovations in Smart Learning, E. Popescu, Kinshuk, M. K. Khribi, R. Huang, M. Jemni, N.-S. Chen, and D. G. Sampson, Eds. Springer, Singapore, 175–184.
SMITH, M. U. 2012. Toward a Unified Theory of Problem Solving: Views From the Content Domains. Routledge, New York.
STAMPER, J. AND BARNES, T. 2009. An unsupervised, frequency-based metric for selecting hints in an MDP-based tutor. In Proceedings of the 2nd International Conference on Educational Data Mining, T. Barnes, M. C. Desmarais, C. Romero, and S. Ventura, Eds. International Educational Data Mining Society, 181–190.
STAMPER, J., BARNES, T., LEHMANN, L., AND CROY, M. 2008. The Hint Factory: Automatic generation of contextualized help for existing computer aided instruction. In Proceedings of the 9th International Conference on Intelligent Tutoring Systems Young Researchers Track. 71–78.
TCHÉTAGNI, J. M. AND NKAMBOU, R. 2002. Hierarchical representation and evaluation of the student in an intelligent tutoring system. In Proceedings of the 6th International Conference on Intelligent Tutoring Systems, S. A. Cerri, G. Gouardères, and F. Paraguaçu, Eds. Lecture Notes in Computer Science, vol. 2363. Springer, 708–717.
UENO, M. AND MIYAZAWA, Y. 2017. IRT-based adaptive hints to scaffold learning in programming. IEEE Transactions on Learning Technologies 11, 4, 415–428.
VANLEHN, K. 2011. The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist 46, 4, 197–221.
WOOD, H. AND WOOD, D. 1999. Help seeking, learning and contingent tutoring. Computers & Education 33, 2-3, 153–169.
YACEF, K. 2005. The Logic-ITA in the classroom: a medium scale experiment. International Journal of Artificial Intelligence in Education 15, 1, 41–62.
ZHOU, G., AZIZSOLTANI, H., AUSIN, M. S., BARNES, T., AND CHI, M. 2019. Hierarchical reinforcement learning for pedagogical policy induction. In Proceedings of the 20th International Conference on Artificial Intelligence in Education, S. Isotani, E. Millán, A. Ogan, P. M. Hastings, B. M. McLaren, and R. Luckin, Eds. Lecture Notes in Computer Science, vol. 11625. Springer, 544–556.
Issue
Section
Articles
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors who publish with this journal agree to the following terms:
- The Author retains copyright in the Work, where the term “Work” shall include all digital objects that may result in subsequent electronic publication or distribution.
- Upon acceptance of the Work, the author shall grant to the Publisher the right of first publication of the Work.
- The Author shall grant to the Publisher and its agents the nonexclusive perpetual right and license to publish, archive, and make accessible the Work in whole or in part in all forms of media now or hereafter known under a Creative Commons 4.0 License (Attribution-Noncommercial-No Derivatives 4.0 International), or its equivalent, which, for the avoidance of doubt, allows others to copy, distribute, and transmit the Work under the following conditions:
- Attribution—other users must attribute the Work in the manner specified by the author as indicated on the journal Web site;
- Noncommercial—other users (including Publisher) may not use this Work for commercial purposes;
- No Derivative Works—other users (including Publisher) may not alter, transform, or build upon this Work,with the understanding that any of the above conditions can be waived with permission from the Author and that where the Work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.
- The Author is able to enter into separate, additional contractual arrangements for the nonexclusive distribution of the journal's published version of the Work (e.g., post it to an institutional repository or publish it in a book), as long as there is provided in the document an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post online a pre-publication manuscript (but not the Publisher’s final formatted PDF version of the Work) in institutional repositories or on their Websites prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (see The Effect of Open Access). Any such posting made before acceptance and publication of the Work shall be updated upon publication to include a reference to the Publisher-assigned DOI (Digital Object Identifier) and a link to the online abstract for the final published Work in the Journal.
- Upon Publisher’s request, the Author agrees to furnish promptly to Publisher, at the Author’s own expense, written evidence of the permissions, licenses, and consents for use of third-party material included within the Work, except as determined by Publisher to be covered by the principles of Fair Use.
- The Author represents and warrants that:
- the Work is the Author’s original work;
- the Author has not transferred, and will not transfer, exclusive rights in the Work to any third party;
- the Work is not pending review or under consideration by another publisher;
- the Work has not previously been published;
- the Work contains no misrepresentation or infringement of the Work or property of other authors or third parties; and
- the Work contains no libel, invasion of privacy, or other unlawful matter.
- The Author agrees to indemnify and hold Publisher harmless from Author’s breach of the representations and warranties contained in Paragraph 6 above, as well as any claim or proceeding relating to Publisher’s use and publication of any content contained in the Work, including third-party content.