Extending the Hint Factory for the Assistance Dilemma: A Novel, Data-driven HelpNeed Predictor for Proactive Problem-solving Help

##plugins.themes.bootstrap3.article.main##

##plugins.themes.bootstrap3.article.sidebar##

Published Dec 29, 2020
Mehak Maniktala Christa Cody Amy Isvik Nicholas Lytle Min Chi Tiffany Barnes

Abstract

Determining when and whether to provide personalized support is a well-known challenge called the assistance dilemma. A core problem in solving the assistance dilemma is the need to discover when students are unproductive so that the tutor can intervene. Such a task is particularly challenging for open-ended domains, even those that are well-structured with defined principles and goals. We present a set of data-driven methods to classify, predict, and prevent unproductive problem-solving steps in the well-structured open-ended domain of logic. This approach leverages and extends the Hint Factory, a set of methods that leverages prior student solution attempts to build data-driven intelligent tutors. We present a HelpNeed classification that uses prior student data to determine when students are likely to be unproductive and need help learning optimal problem-solving strategies. We present a controlled study to determine the impact of an Adaptive pedagogical policy that provides proactive hints at the start of each step based on the outcomes of our HelpNeed predictor: productive vs. unproductive. Our results show that the students in the Adaptive condition exhibited better training behaviors, with lower help avoidance, and higher help appropriateness (a higher chance of receiving help when it was likely to be needed), as measured using the HelpNeed classifier, when compared to the Control. Furthermore, the results show that the students who received Adaptive hints based on HelpNeed predictions during training significantly outperform their Control peers on the posttest, with the former producing shorter, more optimal solutions in less time. We conclude with suggestions on how these HelpNeed methods could be applied in other well-structured open-ended domains.

How to Cite

Maniktala, M., Cody, C., Isvik, A., Lytle, N., Chi, M., & Barnes, T. (2020). Extending the Hint Factory for the Assistance Dilemma: A Novel, Data-driven HelpNeed Predictor for Proactive Problem-solving Help. Journal of Educational Data Mining, 12(4), 24-65. https://doi.org/10.5281/zenodo.4399683
Abstract 328 | PDF Downloads 180

##plugins.themes.bootstrap3.article.details##

Keywords

adaptive support, propositional logic, assistance dilemma, problem solving, unproductivity, efficiency, unsolicited hints, data-driven tutoring, help appropriateness

References
ALEVEN, V. AND KOEDINGER, K. R. 2000. Limitations of student control: Do students know when they need help? In Proceedings of the 5th International Conference on Intelligent Tutoring Systems, G. Gauthier, C. Frasson, and K. VanLehn, Eds. Springer, Berlin, Heidelberg, 292–303.

ALEVEN, V., MCLAREN, B., ROLL, I., AND KOEDINGER, K. 2006. Toward meta-cognitive tutoring: A model of help seeking with a cognitive tutor. International Journal of Artificial Intelligence in Education 16, 2, 101–128.

ANOHINA, A. 2007. Advances in intelligent tutoring systems: problem-solving modes and model of hints. International Journal of Computers Communications & Control 2, 1, 48–55.

ARROYO, I., BECK, J. E., BEAL, C. R., WING, R., AND WOOLF, B. P. 2001. Analyzing students' response to help provision in an elementary mathematics intelligent tutoring system. In Papers of the Workshop on Help Provision and Help Seeking in Interactive Learning Environments at the Tenth International Conference on Artificial Intelligence in Education. 34–46.

ARVAI, K. 2018. Fine tuning a classifier in scikit-learn. https://towardsdatascience.com/fine-tuning-a-classifier-in-scikit-learn-66e048c21e65.

AZEVEDO, R. AND CROMLEY, J. G. 2004. Does training on self-regulated learning facilitate students' learning with hypermedia? Journal of Educational Psychology 96, 3, 523–535.

BAKER, R. S., CORBETT, A. T., KOEDINGER, K. R., EVENSON, S., ROLL, I., WAGNER, A. Z., NAIM, M., RASPAT, J., BAKER, D. J., AND BECK, J. E. 2006. Adapting to when students game an intelligent tutoring system. In Proceedings of the 8th International Conference on Intelligent Tutoring Systems, M. Ikeda, K. D. Ashley, and T.-W. Chan, Eds. Springer, 392–401.

BAKER, R. S., CORBETT, A. T., ROLL, I., AND KOEDINGER, K. R. 2008. Developing a generalizable detector of when students game the system. User Modeling and User-Adapted Interaction 18, 3, 287–314.

BAKER, R. S., D E CARVALHO, A., RASPAT, J., ALEVEN, V., CORBETT, A. T., AND KOEDINGER, K. R. 2009. Educational software features that encourage and discourage "gaming the system". In Proceedings of the 14th International Conference on Artificial Intelligence in Education, V. Aleven, J. Kay, and J. Mostow, Eds. 475–482.

BARNES, T. AND STAMPER, J. 2010. Automatic hint generation for logic proof tutoring using historical data. Journal of Educational Technology & Society 13, 1, 3–12.

BARNES, T., STAMPER, J., AND CROY, M. 2011. Using Markov decision processes for automatic hint generation. In Handbook of Educational Data Mining, C. Romero, S. Ventura, M. Pechenizkiy, and R. S. J. de Baker, Eds. Chapman & Hall/CRC Data Mining and Knowledge Discovery Series. CRC Press, Chapter 33, 467–480.

BARNES, T., STAMPER, J. C., LEHMANN, L., AND CROY, M. J. 2008. A pilot study on logic proof tutoring using hints generated from historical student data. In Proceedings of the 1st International Conference on Educational Data Mining, R. S. J. de Baker, T. Barnes, and J. E. Beck, Eds. 197–201.

BARTHOLOM É, T., STAHL, E., PIESCHL, S., AND BROMME, R. 2006. What matters in help-seeking? a study of help effectiveness and learner-related factors. Computers in Human Behavior 22, 1, 113–129.

BECK, J. E., CHANG, K.M ., MOSTOW, J., AND CORBETT, A. 2008. Does help help? Introducing the Bayesian evaluation and assessment methodology. In Proceedings of the 9th International Conference on Intelligent Tutoring Systems, B. P. Woolf, E. Aı̈meur, R. Nkambou, and S. Lajoie, Eds. Springer, Berlin, Heidelberg, 383–394.

BECK, J. E. AND GONG, Y. 2013. Wheel-spinning: Students who fail to master a skill. In Proceedings of the 16th International Conference on Artificial Intelligence in Education, H. C. Lane, K. Yacef, J. Mostow, and P. Pavlik, Eds. Springer, Berlin, Heidelberg, 431–440.

BOREK, A., M C LAREN, B. M., KARABINOS, M., AND YARON, D. 2009. How much assistance is helpful to students in discovery learning? In Proceedings of the 2009 European Conference on Technology Enhanced Learning, U. Cress, V. Dimitrova, and M. Specht, Eds. Springer, Berlin, Heidelberg, 391–404.

BOTELHO, A., VARATHARAJ, A., PATIKORN, T., DOHERTY, D., ADJEI, S., AND BECK, J. 2019. Developing early detectors of student attrition and wheel spinning using deep learning. IEEE Transactions on Learning Technologies 12, 2, 158–170.

BUNT, A. AND CONATI, C. 2003. Probabilistic student modelling to improve exploratory behaviour. User Modeling and User-Adapted Interaction 13, 3, 269–309.

BUNT, A., CONATI, C., AND MULDNER, K. 2004. Scaffolding self-explanation to improve learning in exploratory learning environments. In Proceedings of the 7th International Conference on Intelligent Tutoring Systems, J. C. Lester, R. M. Vicari, and F. Paraguaçu, Eds. Springer, Berlin, Heidelberg, 656–667.

CAPRARO, M. M., AN, S. A., MA, T., RANGEL-CHAVEZ, A. F., AND HARBAUGH, A. 2012. An investigation of preservice teachers' use of guess and check in solving a semi open-ended mathematics problem. The Journal of Mathematical Behavior 31, 1, 105–116.

CEN, H., KOEDINGER, K. R., AND JUNKER, B. 2007. Is over practice necessary? Improving learning efficiency with the cognitive tutor through educational data mining. In Proceedings of the 13th International Conference on Artificial Intelligence in Education, R. Luckin, K. R. Koedinger, and J. Greer, Eds. Frontiers in Artificial Intelligence and Applications, vol. 158. IOS Press, NLD, 511–518.

CHAUDHRY, R., SINGH, H., DOGGA, P., AND SAINI, S. K. 2018. Modeling hint-taking behavior and knowledge state of students with multi-task learning. In Proceedings of the 11th International Conference on Educational Data Mining, K. E. Boyer and M. Yudelson, Eds. International Educational Data Mining Society, 21–31.

CHEN, T. AND GUESTRIN, C. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACMSIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 785–794.

CODY, C. AND MOSTAFAVI, B. 2017. Investigating the impact of unsolicited next-step and subgoal hints on dropout in a logic proof tutor. In Proceedings of the 2017 ACMSIGCSE Technical Symposium on Computer Science Education. ACM, 705–705.

CONATI, C., GERTNER, A., AND VANLEHN, K. 2002. Using Bayesian networks to manage uncertainty in student modeling. User Modeling and User-Adapted Interaction 12, 4, 371–417.

CORBETT, A. T. AND ANDERSON, J. R. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction 4, 4, 253–278.

DONG, Y., MARWAN, S., CATETE, V., PRICE, T., AND BARNES, T. 2019. Defining tinkering behavior in open-ended block-based programming assignments. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education. 1204–1210.

EAGLE, M., HICKS, D., AND BARNES, T. 2015. Interaction network estimation: Predicting problemsolving diversity in interactive environments. In Proceedings of the 8th International Conference on Educational Data Mining, O. C. Santos, J. Boticario, C. Romero, M. Pechenizkiy, A. Merceron, P. Mitros, J. M. Luna, M. C. Mihaescu, P. Moreno, A. Hershkovitz, S. Ventura, and M. C. Desmarais, Eds. International Educational Data Mining Society, 342–349.

FOSSATI, D., DIEUGENIO, B., OHLSSON, S., BROWN, C., AND CHEN, L. 2015. Data driven automatic feedback generation in the iList intelligent tutoring system. Technology, Instruction, Cognition and Learning 10, 1, 5–26.

FRATAMICO, L., CONATI, C., KARDAN, S., AND ROLL, I. 2017. Applying a framework for student modeling in exploratory learning environments: Comparing data representation granularity to handle environment complexity. International Journal of Artificial Intelligence in Education 27, 2, 320–352.

III, B. P., HICKS, A., AND BARNES, T. 2014. Generating hints for programming problems using intermediate output. In Proceedings of the 7th International Conference on Educational Data Mining, J. C. Stamper, Z. A. Pardos, M. Mavrikis, and B. M. McLaren, Eds. International Educational Data Mining Society, 92–98.

KAI, S., ALMEDA, M. V., BAKER, R. S., HEFFERNAN, C., AND HEFFERNAN, N. 2018. Decision tree modeling of wheel-spinning and productive persistence in skill builders. Journal of Educational Data Mining 10, 1, 36–71.

KARDAN, S. AND CONATI, C. 2015. Providing adaptive support in an interactive simulation for learning: An experimental evaluation. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 3671–3680.

KINNEBREW, J. S., SEGEDY, J. R., AND BISWAS, G. 2014. Analyzing the temporal evolution of students' behaviors in open-ended learning environments. Metacognition and Learning 9, 2, 187–215.

KLAHR, D. 2009. "To every thing there is a season, and a time to every purpose under the heavens": What about direct instruction? In Constructivist Instruction: Success or failure?, S. Tobias and T. M. Duffy, Eds. Routledge, New York, Chapter 15, 291–310.

KOCK, M. AND PARAMYTHIS, A. 2010. Towards adaptive learning support on the basis of behavioural patterns in learning activity sequences. In 2010 International Conference on Intelligent Networking and Collaborative Systems. IEEE, 100–107.

KOEDINGER, K. R. AND ALEVEN, V. 2007. Exploring the assistance dilemma in experiments with cognitive tutors. Educational Psychology Review 19, 3, 239–264.

MANIKTALA, M., CODY, C., BARNES, T., AND CHI, M. 2020. Avoiding help avoidance: Using interface design changes to promote unsolicited hint usage in an intelligent tutor. International Journal of Artificial Intelligence in Education 30, 4, 637–667.

MARWAN, S., DOMBE, A., AND PRICE, T. W. 2020. Unproductive help-seeking in programming: What it is and how to address it. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education. 54–60.

MAYER, R. E. 1992. Thinking, problem solving, cognition. WH Freeman/Times Books/Henry Holt & Co.

MCLAREN, B., TIMMS, M., WEIHNACHT, D., BRENNER, D., LUTTGEN, K., GRILLO -HILL, A., AND H. BROWN, D. 2014. A web-based system to support inquiry learning. In Proceedings of the 6th International Conference on Computer Supported Education Volume 1. SciTePress, Setubal, PRT, 43–52.

MCLAREN, B. M., LIM, S.-J., AND KOEDINGER, K. R. 2008. When and how often should worked examples be given to students? New results and a summary of the current state of research. InProceedings of the 30th Annual Conference of the Cognitive Science Society, B. C. Love, K. McRae, and V. M. Sloutsky, Eds. Cognitive Science Society, Austin, TX, 2176–2181.

MERCERON, A. AND YACEF, K. 2005. Educational data mining: a case study. In Proceedings of the 12th International Conference on Artificial Intelligence in Education, C. Looi, G. I. McCalla, B. Bredeweg, and J. Breuker, Eds. Frontiers in Artificial Intelligence and Applications, vol. 125. IOS Press, 467–474.

MOSTAFAVI, B. AND BARNES, T. 2017. Evolution of an intelligent deductive logic tutor using datadriven elements. International Journal of Artificial Intelligence in Education 27, 1, 5–36.

MOSTAFAVI, B., LIU, Z., AND BARNES, T. 2015. Data-driven proficiency profiling. In Proceedings of the 8th International Conference on Educational Data Mining, O. C. Santos, J. Boticario, C. Romero, M. Pechenizkiy, A. Merceron, P. Mitros, J. M. Luna, M. C. Mihaescu, P. Moreno, A. Hershkovitz, S. Ventura, and M. C. Desmarais, Eds. International Educational Data Mining Society, 335–341.

MURRAY, R. C. AND VANLEHN, K. 2005. Effects of dissuading unnecessary help requests while providing proactive help. In Proceedings of the 12th International Conference on Artificial Intelligence in Education, C. Looi, G. I. McCalla, B. Bredeweg, and J. Breuker, Eds. Frontiers in Artificial Intelligence and Applications, vol. 125. IOS Press, 887–889.

MURRAY, R. C. AND VANLEHN, K. 2006. A comparison of decision-theoretic, fixed-policy and random tutorial action selection. In Proceedings of the 8th International Conference on Intelligent Tutoring Systems, M. Ikeda, K. D. Ashley, and T. Chan, Eds. Lecture Notes in Computer Science, vol. 4053. Springer, 114–123.

MURRAY, R. C., VANLEHN, K., AND MOSTOW, J. 2004. Looking ahead to select tutorial actions: A decision-theoretic approach. International Journal of Artificial Intelligence in Education 14, 3, 4, 235–278.

PARK, S. AND MATSUDA, N. 2018. Predicting students' unproductive failure on intelligent tutors in adaptive online courseware. In Proceedings of the Sixth Annual GIFT Users Symposium. Vol. 6. US Army Research Laboratory, 131–138.

PEDREGOSA, F., VAROQUAUX, G., GRAMFORT, A., MICHEL, V., THIRION, B., GRISEL, O., BLONDEL, M., PRETTENHOFER, P., WEISS, R., DUBOURG, V., ETAL . 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, Oct, 2825–2830.

PEÑA, A., KAYASHIMA, M., MIZOGUCHI, R., AND DOMINGUEZ, R. 2011. Improving students' metacognitive skills within intelligent educational systems: A review. In Proceedings of the 6th International Conference on Foundations of Augmented Cognition, D. D. Schmorrow and C. M. Fidopiastis, Eds. Springer, Berlin, Heidelberg, 442–451.

POLYA, G. 2004. How to solve it: A new aspect of mathematical method. Princeton University Press.

PRICE, T. W., DONG, Y., AND BARNES, T. 2016. Generating data-driven hints for open-ended programming. In Proceedings of the 9th International Conference on Educational Data Mining, T. Barnes, M. Chi, and M. Feng, Eds. International Educational Data Mining Society, 191–198.

PRICE, T. W., LIU, Z., CATET É, V., AND BARNES, T. 2017. Factors influencing students' help-seeking behavior while programming with human and computer tutors. In Proceedings of the 2017 ACM Conference on International Computing Education Research. ACM, 127–135.

PRICE, T. W., ZHI, R., AND BARNES, T. 2017. Hint generation under uncertainty: The effect of hint quality on help-seeking behavior. In Proceedings of the 18th International Conference on Artificial Intelligence in Education, E. André, R. S. Baker, X. Hu, M. M. T. Rodrigo, and B. du Boulay, Eds. Lecture Notes in Computer Science, vol. 10331. Springer, 311–322.

PUUSTINEN, M. 1998. Help-seeking behavior in a problem-solving situation: Development of selfregulation. European Journal of Psychology of Education 13, 2, 271–282.

RIVERS, K. AND KOEDINGER, K. R. 2013. Automatic generation of programming feedback: A datadriven approach. In Proceedings of the First Workshop on AI-supported Education for Computer Science at the 16th International Conference on Artificial Intelligence in Education, E. Walker and C. Looi, Eds. CEUR Workshop Proceedings, vol. 1009. 50–59.

RIVERS, K. AND KOEDINGER, K. R. 2017. Data-driven hint generation in vast solution spaces: a self-improving Python programming tutor. International Journal of Artificial Intelligence in Education 27, 1, 37–64.

ROLL, I., ALEVEN, V., M C LAREN, B. M., AND KOEDINGER, K. R. 2011. Improving students' helpseeking skills using metacognitive feedback in an intelligent tutoring system. Learning and Instruction 21, 2, 267–280.

RUS, V., BANJADE, R., NIRAULA, N., GIRE, E., AND FRANCESCHETTI, D. 2017. A study on two hint-level policies in conversational intelligent tutoring systems. In Innovations in Smart Learning, E. Popescu, Kinshuk, M. K. Khribi, R. Huang, M. Jemni, N.-S. Chen, and D. G. Sampson, Eds. Springer, Singapore, 175–184.

SMITH, M. U. 2012. Toward a Unified Theory of Problem Solving: Views From the Content Domains. Routledge, New York.

STAMPER, J. AND BARNES, T. 2009. An unsupervised, frequency-based metric for selecting hints in an MDP-based tutor. In Proceedings of the 2nd International Conference on Educational Data Mining, T. Barnes, M. C. Desmarais, C. Romero, and S. Ventura, Eds. International Educational Data Mining Society, 181–190.

STAMPER, J., BARNES, T., LEHMANN, L., AND CROY, M. 2008. The Hint Factory: Automatic generation of contextualized help for existing computer aided instruction. In Proceedings of the 9th International Conference on Intelligent Tutoring Systems Young Researchers Track. 71–78.

TCHÉTAGNI, J. M. AND NKAMBOU, R. 2002. Hierarchical representation and evaluation of the student in an intelligent tutoring system. In Proceedings of the 6th International Conference on Intelligent Tutoring Systems, S. A. Cerri, G. Gouardères, and F. Paraguaçu, Eds. Lecture Notes in Computer Science, vol. 2363. Springer, 708–717.

UENO, M. AND MIYAZAWA, Y. 2017. IRT-based adaptive hints to scaffold learning in programming. IEEE Transactions on Learning Technologies 11, 4, 415–428.

VANLEHN, K. 2011. The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist 46, 4, 197–221.

WOOD, H. AND WOOD, D. 1999. Help seeking, learning and contingent tutoring. Computers & Education 33, 2-3, 153–169.

YACEF, K. 2005. The Logic-ITA in the classroom: a medium scale experiment. International Journal of Artificial Intelligence in Education 15, 1, 41–62.

ZHOU, G., AZIZSOLTANI, H., AUSIN, M. S., BARNES, T., AND CHI, M. 2019. Hierarchical reinforcement learning for pedagogical policy induction. In Proceedings of the 20th International Conference on Artificial Intelligence in Education, S. Isotani, E. Millán, A. Ogan, P. M. Hastings, B. M. McLaren, and R. Luckin, Eds. Lecture Notes in Computer Science, vol. 11625. Springer, 544–556.
Section
Articles