Mining Collaborative Patterns in Tutorial Dialogues



Published Dec 1, 2010
Sidney D'Mello Andrew Olney Natalie Person


We present a method to automatically detect collaborative patterns of student and tutor dialogue moves. The method identifies significant two-step excitatory transitions between dialogue moves, integrates the transitions into a directed graph representation, and generates and tests data-driven hypotheses from the directed graph. The method was applied to a large corpus of student-tutor dialogue moves from expert tutoring sessions. An examination of the subset of the corpus consisting of tutor lectures revealed collaborative patterns consistent with information-transmission, information-elicitation, off topic-conversation, and student initiated questions. Sequences of dialogue moves within each of these patterns were also identified. Comparisons of the method to other approaches and applications towards the computational modeling of expert human tutors are discussed.

How to Cite

D’Mello, S., Olney, A., & Person, N. (2010). Mining Collaborative Patterns in Tutorial Dialogues. Journal of Educational Data Mining, 2(1), 2–37.
Abstract 745 | PDF Downloads 358



tutorial dialogue, pattern mining, expert tutors, lecutres, likelihood metric, information transmission, information elicitation, human tutoring, educational data mining

AUSUBEL, D. 1978. Educational Psychology: A Cognitive View. Holt, Rinehart and Winston, Inc., Austin, TX.

BAKEMAN, R. AND GOTTMAN, J.M. 1997. Observing interaction: An introduction to sequential analysis. Cambridge University Press., Cambridge, UK.

BEAL, C., MITRA, S. AND COHEN, P. 2007. Modeling Learning Patterns of Students with a Tutoring System Using Hidden Markov Models. In Proceedings of the 13th International Conference on Artificial Intelligence in Education, R. LUCKIN, K. KOEDINGER AND J. GREER Eds. IOS Press, Amsterdam, 238-245.

BISWAS, G., LEELAWONG, K., SCHWARTZ, D. AND VYE, N. 2005. Learning by teaching: A new agent paradigm for educational software. Applied Artificial Intelligence 19, 363-392.

BLOOM, B. 1984. The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher 13, 4-16.

BOYER, K., PHILLIPS, R., HA, E., WALLIS, M., VOUK, M. AND LESTER, J. 2009. Modeling Dialogue Structure with Adjacency Pair Analysis and Hidden Markov Models. In Proceedings of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies Conference (NAACL HLT), Boulder, CO, 49-52.

BOYER, K., YOUNG, E., WALLIS, M., PHILLIPS, R., VOUK, M. AND LESTER, J. 2009. Discovering Tutorial Dialogue Strategies with Hidden Markov Models. In Proceedings of the 14th International Conference on Artificial Intelligence in Education V. DIMITROVA, R. MIZOGUCHI, B. DU BOULAY AND A. GRAESSER Eds. IOS Press, Amsterdam, 141 - 148.

BROWN, J. AND BURTON, R. 1978. Diagnostic models for procedural bugs in basic mathematical skills. Cognitive Science 2, 155-192.

CADE, W., COPELAND, J., PERSON, N. AND D'MELLO, S. 2008. Dialogue modes in expert tutoring. In Proceedings of the 9th international conference on Intelligent Tutoring Systems, B. WOOLF, E. AIMEUR, R. NKAMBOU AND S. LAJOIE Eds. Springer-Verlag, Berlin, Heidelberg, 470-479.


BECKET, B. 1994. Animated conversation: Rule-based generation of facial expression, gesture and spoken information for multiple conversational agents. In SIGGRAPH 1994 ACM Press.

CHI, M. 1996. Constructing self-explanations and scaffolded explanations in tutoring. Applied Cognitive Psychology 10, S33-S49.

CHI, M., DELEEUW, N., CHIU, M. AND LAVANCHER, C. 1994. Eliciting SelfExplanations Improves Understanding. Cognitive Science 18, 439-477.

CHI, M., FELTOVICH, P. AND GLASER, R. 1981. Categorization and Representation of Physics Problems by Experts and Novices. Cognitive Science 5, 121-152.

CHI, M., ROY, M. AND HAUSMANN, R. 2008. Observing tutorial dialogues collaboratively: Insights about human tutoring effectiveness from vicarious learning. Cognitive Science 32, 301-341.

CHI, M., SILER, S., JEONG, H., YAMAUCHI, T. AND HAUSMANN, R. 2001. Learning from human tutoring. Cognitive Science 25, 471-533.

COHEN, J. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 20, 37-46.

COHEN, J. 1992. A power primer. Psychological Bulletin 112, 155-159.

COHEN, P., KULIK, J. AND KULIK, C. 1982. Educational outcomes of tutoring: A meta-analysis of findings. American Educational Research Journal 19, 237-248.

COOKE, N.J., NEVILLE, K.J. AND ROWE, A.L. 1996. Procedural network representations of sequential data. Human-Computer Interaction 11, 29-68.

CORBETT, A. 2001. Cognitive computer tutors: Solving the two-sigma problem. In Proceedings of 8th International Conference on User Modeling, M. BAUER, P. GMYTRASIEWICZ AND J. VASSILEVA Eds. Springer, Berlin / Heidelberg, 137- 147.

CORBETT, A. AND ANDERSON, J. 1992. LISP intelligent tutoring system: Research in skill acquisition. In Computer-assisted instruction and intelligent tutoring systems, J. LARKIN AND R. CHABAY Eds. Lawrence Erlbaum Associates, Inc., Hillsdale, NJ, 73-109.

CORBETT, A., ANDERSON, J., GRAESSER, A., KOEDINGER, K. AND VANLEHN, K. 1999. Third generation computer tutors: Learn from or ignore human tutors? In Proceedings of CHI Conference on Human Factors in Computing Systems ACM, New York, 85 - 86.

CORE, M., MOORE, J. AND ZINN, C. 2003. The role of initiative in tutorial dialogue. In 10th Conference of the European Chapter of the Association for Computational Linguistics Linguistic Data Consortium, Budapest, Hungary.

CROMLEY, J. AND AZEVEDO, R. 2005. What Do Reading Tutors Do?: A Naturalistic Study of More and Less Experienced Tutors in Reading. Discourse Processes 40, 83-113.

D'MELLO, S., HAYS, P., WILLIAMS, C., CADE, W., BROWN, J. AND OLNEY, A. in press. Collaborative Lecturing by Human and Computer Tutors In Proceedings of 10th International Conference on Intelligent Tutoring Systems, J. KAY AND V. ALEVEN Eds. Springer, Berlin / Heidelberg.

D'MELLO, S., TAYLOR, R. AND GRAESSER, A. 2007. Monitoring affective trajectories during complex learning. In Proceedings of the 29th Annual Cognitive Science Society D. MCNAMARA AND G. TRAFTON Eds. Cognitive Science Society, Austin, TX, 203-208. DEL SOLDATO, T. AND DU BOULAY, B. 1995. Implementation of motivational tactics in tutoring systems. International Journal of Intelligence in Education 6, 337- 378.

DERRY, S. AND POTTS, M. 1998. How tutors model students: A study of personal constructs in adaptive tutoring. American Educational Research Journal 35, 65-99.

DI BATTISTA, G., EADES, P., TAMASSIA, R. AND TOLLIS, I.G. 1994. Algorithms for Drawing Graphs - an Annotated-Bibliography. Computational Geometry-Theory and Applications 4, 235-282.

DI EUGENIO, B., FOSSATI, D., OHLSSON, S. AND COSEJO, D. 2009. Towards explaining effective tutorial dialogues. In Annual Meeting of the Cognitive Science Society Cognitive Science Society, Amsterdam, Netherlands, 1430-1435.

DU BOULAY, B. AND LUCKIN, R. 2001. Modeling human teaching tactics and strategies for tutoring systems. International Journal of Artificial Intelligence 12, 235-256.

EVENS, M., SPITKOVSKY, J., BOYLE, P., MICHAEL, J. AND ROVICK, A. 1993. Synthesizing tutorial dialogues. In Proceedings of the 15th Annual Conference of the Cognitive Science Society Lawrence Erlbaum Associates, Boulder.

FELTOVICH, P., COULSON, R. AND SPIRO, R. 2001. Learners' (mis)understanding of important and difficult concepts: A challenge to smart machines in education. In Smart machines in education, K. FORBUS AND P. FELTOVICH Eds. AAAI/MIT Press, Menlo Park, CA, 349-375.

FORBES-RILEY, K., LITMAN, D., HUETTNER, A. AND WARD, A. 2005. Dialogue-Learning Correlations in Spoken Dialogue Tutoring. In Artificial Intelligence in Education - Supporting Learning through Intelligent and Socially Informed Technology, C.K. LOOI, G. MCCALLA, B. BREDEWEG AND J. BREUKER Eds., 225-232.

FOX, B. 1991. Cognitive and interactional aspects of correction in tutoring. In Teaching knowledge and intelligent tutoring, P. GOODYEAR Ed. Ablex, Norwood, NJ.

FOX, B. 1993. The human tutorial dialogue project. Lawrence Erlbaum Associates., Hillsdale, NJ.

GLASS, M., KIM, J., EVENS, M., MICHAEL, J. AND ROVICK, A. 1999. Novice vs. expert tutors: A comparison of style. In Midwest Artificial Intelligence and Cognitive Science Conference, Bloomington, IN.

GRAESSER, A., JEON, M. AND DUFTY, D. 2008. Agent technologies designed to facilitate interactive knowledge construction. Discourse Processes 45, 298-322.

GRAESSER, A., LU, S.L., JACKSON, G., MITCHELL, H., VENTURA, M., OLNEY, A. AND LOUWERSE, M. 2004. AutoTutor: A tutor with dialogue in natural language. Behavioral Research Methods, Instruments, and Computers 36, 180-193.

GRAESSER, A. AND PERSON, N. 1994. Question Asking During Tutoring. American Education Research Journal 31, 104-137.

GRAESSER, A., PERSON, N., HARTER, D. AND TRG. 2000. Teaching tactics in Autotutor. In ITS 2000 Proceedings of the Workshop on Modeling Human Teaching Tactics and Strategies, Montreal, Canada, 49-57.

GRAESSER, A., PERSON, N. AND MAGLIANO, J. 1995. Collaborative dialogue patterns in naturalistic one-to-one tutoring. Applied Cognitive Psychology 9, 495- 522.

HAGEN, A., PELLOM, B. AND COLE, R. 2007. Highly accurate children’s speech recognition for interactive reading tutors using subword units. Speech Communication 49, 861-873.

HAN, J., PEI, J., MORTAZAVI-ASL, B., CHEN, Q., DAYAL, U. AND HSU, M. 2000. FreeSpan: frequent pattern-projected sequential pattern mining. In Proceedings of the Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining, Boston, Massachusetts, United States2000 ACM.

HEINZ, B. 2003. Backchannel responses as strategic responses in bilingual speakers' conversations. Journal of Pragmatics 35, 1113-1142.

HERMAN, I., MELANCON, G. AND MARSHALL, M.S. 2000. Graph visualization and navigation in information visualization: A survey. Ieee Transactions on Visualization and Computer Graphics 6, 24-43.

HOQUE, M.E., SOROWER, M.S., YEASIN, M. AND LOUWERSE, M.M. 2007. What Speech Tells us about Discourse: The Role of Prosodic and Discourse Features in Dialogue Act Classification. In Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN), Orlando, FL2007.

ISSROFF, K. AND DEL SOLDATO, T. 1996. Incorporating motivation into computersupported collaborative learning. In Proceedings of the Proceedings of European conference on artificial intelligence in education, Ficha Tecnica, Lisbon.

JEONG, H., GUPTA, A., ROSCOE, R., WAGSTER, J., BISWAS, G. AND SCHWARTZ, D. 2008. Using hidden Markov models to characterize student behaviors in learning-by-teaching environments. In 9th International Conference on Intelligent Tutoring Systems, B.P. WOOLF, E. AIMEUR, R. NKAMBOU AND S. LAJOIE Eds., Montreal, CANADA, 614-625.

JORDAN, P. AND SILER, S. 2002. Student initiative and questioning strategies in computer-mediated human tutoring dialogues. In Proceedings of the ITS Workshop on Empirical Methods for Tutorial Dialogue Systems, San Sabastian, Spain.

JURAFSKY, D. AND MARTIN, J. 2008. Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition. Prentice-Hall, Upper Saddle River, NJ.

KATO, K., NANJO, H. AND KAWAHARA, T. 2000. Automatic transcription of lecture speech using topic-independent language modeling. In 6th International Conference on Spoken Language Processing, Beijing, China, 162-165.

KUMAR, R., ROSE, C.P., WANG, Y.C., JOSHI, M. AND ROBINSON, A. 2007. Tutorial Dialogue as Adaptive Collaborative Learning Support. In Artificial Intelligence in Education - Building Technology Rich Learning Contexts That Work, R. LUCKIN, K.R. KOEDINGER AND J. GREER Eds., 383-390.

LAJOIE, S., FAREMO, S. AND WISEMAN, J. 2001. Tutoring strategies for effective instruction in internal medicine. International Journal of Artificial Intelligence and Education 12, 293-309.

LEEUWIS, E., FEDERICO, M. AND CETTOLO, M. 2003. Language modeling and transcription of the TED corpus lectures. In IEEE International Conference on Acoustics, Speech, and Signal Processing IEEE, 232-235.

LEPPER, M., ASPINWALL, L., MUMME, D. AND CHABAY, R. 1990. Selfperception and social perception processes in tutoring: Subtle social control strategies of expert tutors. In Self-inference and social inference: The Ontario symposium, J. OLSON AND M. ZANNA Eds. Erlbaum, Hillsdale, NJ, 217-237.

LEPPER, M., DRAKE, M. AND O'DONNELL-JOHNSON, T. 1997. Scaffolding techniques of expert human tutors. In Scaffolding student learning: Instructional approaches and issues, K. HOGAN AND M. PRESSLEY Eds. Brookline Books, New York, 108-144.

LEPPER, M. AND WOOLVERTON, M. 2002. The wisdom of practice: Lessons learned from the study of highly effective tutors. In Improving academic achievement: Impact of psychological factors on education, J. ARONSON Ed. Academic Press, Orlando, FL, 135-158.

LEPPER, M., WOOLVERTON, M., MUMME, D. AND GURTNER, J. 1993. Motivational techniques of expert human tutors: Lessons for the design of computer based tutors. In Computers as cognitive tools, S. LAJOIE AND S. DERRY Eds. Erlbaum, Hillsdale, NJ.

LITMAN, D. AND FORBES-RILEY, K. 2006. Correlations between dialogue acts and learning in spoken tutoring dialogues. Natural Language Engineering 12, 161-176.

LITMAN, D., ROSE, C., FORBES-RILEY, K., VANLEHN, K., BHEMBE, D. AND SILLIMAN, S. 2006. Spoken versus typed human and computer dialogue tutoring. International Journal of Artificial Intelligence In Education 16, 145-170.

LITTMAN, D., PINTO, J. AND SOLOWAY, E. 1990. The knowledge required for tutorial planning: An empirical analysis. Interactive Learning Environments 1, 124- 151.

LIU, B., HSU, W. AND MA, Y. 1998. Integrating classification and association rule mining. Knowledge Discovery and Data Mining, 80-86.

LU, X., DI EUGENIO, B., KERSHAW, T., OHLSSON, S. AND CORRIGANHALPERN, A. 2007. Expert vs. Non-expert Tutoring: Dialogue Moves, Interaction Patterns and Multi-Utterance Turns. In Proceedings of the Eighth International Conference on Intelligent Text Processing and Computational Linguistics, Mexico City, Mexico.

LU, X., DI EUGENIO, B., OHLSSON, S. AND FOSSATI, D. 2008. Simple but effective feedback generation to tutor abstract problem solving. In Proceedings of the International Natural Language Generation Conference, Salt Fork, Ohio.

MASSEGLIA, F., CATHALA, F. AND PONCELET, P. 1998. The PSP Approach for Mining Sequential Patterns. In Proceedings of the Proceedings of the Second European Symposium on Principles of Data Mining and Knowledge Discovery1998 Springer-Verlag.

MCARTHUR, D., STASZ, C. AND ZMUIDZINAS, M. 1990. Tutoring Techniques in Algebra. Cognition and Instruction 7, 197-244.

MCKENDREE, J. 1990. Effective feedback content for tutoring complex skills. HumanComputer Interaction 5, 381-413.

MEHAN, H. 1979. Learning lessons: Social organization in the classroom. Harvard University Press, Cambridge.

MERRILL, D., REISER, B., RANNEY, M. AND TRAFTON, J. 1992. Effective Tutoring Techniques: A Comparison of Human Tutors and Intelligent Tutoring Systems. The Journal of the Learning Sciences 2, 277-305.

MORENO, R., MAYER, R., SPIRES, H. AND LESTER, J. 2001. The case for social agency in computer-based teaching: Do students learn more deeply when they interact with animated pedagogical agents? Cognition and Instruction 19, 177-213.

MORTAZAVI-ASL, B., WANG, J., PINTO, H., CHEN, Q. AND HSU, M. 2004. Mining Sequential Patterns by Pattern-Growth: The PrefixSpan Approach 16, 1424-1440.

MOUNT, D.M. 2004. Bioinformatics: Sequence and Genome Analysis. Cold Spring Harbor Laboratory Press, Cold Spring Harbor, NY.

OHLSSON, S., DI EUGENIO, B., CHOW, B., FOSSATI, D., LU, X. AND KERSHAW, T. 2007. Beyond the code-and-count analysis of tutoring dialogues. In Proceeding of the 2007 conference on Artificial Intelligence in Education, R. LUCKIN, K. KOEDINGER AND J. GREER Eds. IOS Press, Amsterdam, 349-356.

OLNEY, A., LOUWERSE, M., MATHEWS, E., MARINEAU, J., HITE-MITCHELL, H. AND GRAESSER, A. 2003. Utterance classification in AutoTutor. In Human Language Technology - North American Chapter of the Association for Computational Linguistics Conference Association for Computational Linguistics, Edmonton, Canada.

OLSON, G.M., HERBSLEB, J.D. AND RUETER, H.H. 1994. Characterizing the Sequential Structure of Interactive Behaviors through Statistical and Grammatical Techniques. Human-Computer Interaction 9, 427-472.

PALINSCAR, A. AND BROWN, A. 1984. Reciprocal teaching and comprehension fostering and comprehension-monitoring activities. Cognition and Instruction 1, 117-175.

PELLOM, B. AND HACIOGLU, K. 2003. Recent improvements in the CU Sonic ASR system for noisy speech: the SPINE task. In IEEE International Conference on Acoustics, Speech, and Signal Processing IEEE, 4-7.

PERSON, N., GRAESSER, A., MAGLIANO, J. AND KREUZ, R. 1994. Inferring What the Student Knows in One-to-One Tutoring - the Role of Student Questions and Answers. Learning and Individual Differences 6, 205-229.

PERSON, N., LEHMAN, B. AND OZBUN, R. 2007. Pedagogical and Motivational Dialogue Moves Used by Expert Tutors. In 17th Annual Meeting of the Society for Text and Discourse, Glasgow, Scotland.

PERSON, N., MOSES, J. AND WILLSON, W. 2007. Question asking in expert and nonexpert tutoring sessions. In 2007 AERA Annual Meeting, Chicago, IL.

PUNTAMBEKAR, S. AND HUBSCHER, R. 2005. Tools for scaffolding students in a complex learning environment: What have we gained and what have we missed? Educational Psychologist 40, 1-12.

PUTNAM, R. 1987. Structuring and Adjusting Content for Students - a Study of Live and Simulated Tutoring of Addition. American Educational Research Journal 24, 13-48.

RABINER, L. 1989. A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE 77, 257-286.

ROBSON, C. 1993. Real word research: A resource for social scientist and practitioner researchers. Blackwell, Oxford.

ROGINA, I. AND SCHAAF, T. 2002. Lecture and presentation tracking in an intelligent meeting room. In Fourth IEEE International Conference on Multimodal Interfaces IEEE, 47-52.

ROGOFF, B. 1990. Apprenticeship in thinking. Oxford University Press, New York.

ROGOFF, B. AND GARDNER, W. 1984. Adult guidance of cognitive development. In Everyday cognition: Its development in social context, B. ROGOFF AND J. LAVE Eds. Harvard University Press, Cambridge, MA.

SANDERSON, P.M. AND FISHER, C. 1994. Exploratory Sequential Data-Analysis - Foundations. Human-Computer Interaction 9, 251-317.

SCHVANEVELDT, R.W. 1990. Pathfinder Associative Networks: Studies in Knowledge Organization Ablex, Norwood, NJ.

SCHVANEVELDT, R.W., DURSO, F.T. AND DEARHOLT, D.W. 1989. Network structures in proximity data. In The psychology of learning and motivation: Advances in research and theory, G. BOWER Ed. Academic Press, New York, 249– 284.

SHAFFER, J.P. 1995. Multiple Hypothesis-Testing. Annual Review of Psychology 46, 561-584.

SHAH, F., EVENS, M., MICHAEL, J. AND ROVICK, A. 2002. Classifying student initiatives and tutor responses in human keyboard-to-keyboard tutoring sessions. Discourse Processes 33, 23-52.

SHUTE, V. 2008. Focus on Formative Feedback. Review of Educational Research 78, 153-189.

SLEEMAN, D., KELLY, A., MARTINAK, R., WARD, R. AND MOORE, J. 1989. Studies of diagnosis and remediation with high school algebra students. Cognitive Science 13, 551-568.

SOLLER, A. AND STEVENS, R. 2007. Applications of stochastic analyses for collaborative learning and cognitive assessment. In Advances in Latent Variable Mixture Models G. HANCOCK AND K. SAMUELSEN Eds. Information Age Publishing, 217-253.

SRIKANT, R. AND AGRAWAL, R. 1996. Mining Sequential Patterns: Generalizations and Performance Improvements. In Proceedings of the Proceedings of the 5th International Conference on Extending Database Technology: Advances in Database Technology1996 Springer-Verlag.

TARJAN, R. 1972. Depth-first search and linear graph algorithms. SIAM Journal on Computing 1, 146-160.

TUCKER, A. 1995. Applied Combinatorics. John Wiley & Sons, Inc.

VANLEHN, K., GRAESSER, A., JACKSON, G., JORDAN, P., OLNEY, A. AND ROSE, C.P. 2007. When are tutorial dialogues more effective than reading? Cognitive Science 31, 3-62.

VANLEHN, K., SILER, S., MURRAY, C., YAMAUCHI, T. AND BAGGETT, W. 2003. Why do only some events cause learning during human tutoring? Cognition and Instruction 21, 209-249.

WALKER, E., MCLAREN, B.M., RUMMEL, N. AND KOEDINGER, K. 2007. Who Says Three's a Crowd? Using a Cognitive Tutor to Support Peer Tutoring. In Artificial Intelligence in Education - Building Technology Rich Learning Contexts That Work, R. LUCKIN, K.R. KOEDINGER AND J. GREER Eds., 399-406.

WARD, N. AND TSUKAHARA, W. 2000. Prosodic features which cue back-channel responses in English and Japanese. Journal of Pragmatics 32, 1177-1207.

WHEATLEY, G. 1991. Constructivist perspectives on science and mathematical learning. Science Education 75, 9-21.

WILEY, J. AND BAILEY, J. 2006. Effects of collaboration and argumentation on learning from web pages. In Collaborative learning, reasoning, and technology, A.M. O'DONNEL, C.E. HMELO-SILVER AND G. ERKENS Eds. Lawrence Earlbaum, Mahwah, New Jersey:, 297-321.

WILEY, J. AND JENSEN, M. 2006. When three heads are better than two. In 28th Annual Meeting of the Cognitive Science Society, Vancouver, British Columbia, 2375-2380.

WILLIAMS, C. AND D'MELLO, S. in press. Predicting student knowledge levels from domain-independent function and content words. In Proceedings of 10th International Conference on Intelligent Tutoring Systems, J. KAY AND V. ALEVEN Eds. Springer, Berlin / Heidelberg.

ZAKI, M.J., PARTHASARATHY, S., OGIHARA, M. AND LI, W. 1997. Parallel algorithms for discovery of association rules. Data Mining and Knowledge Discovery 1, 343-373.

ZOLNAY, A., KOCHAROV, D., SCHLUTER, R. AND NEY, H. 2007. Using multiple acoustic feature sets for speech recognition. Speech Communication 49, 514-525.