Analyzing Process Data from Game/Scenario-Based Tasks: An Edit Distance Approach



Published Jan 29, 2015
Jiangang Hao Zhan Shu Alina von Davier


Students’ activities in game/scenario-based tasks (G/SBTs) can be characterized by a sequence of time-stamped actions of different types with different attributes. For a subset of G/SBTs in which only the order of the actions is of great interest, the process data can be well characterized as a string of characters (i.e., action string) if we encode each action name as a single character. In this article, we report our work on evaluating students’ performances by comparing how far their action strings are from the action string that corresponds to the best performance, where the proximity is quantified by the edit distance between the strings. Specifically, we choose the Levenshtein distance, which is defined as the minimum number of insertions, deletions, and replacements needed to convert one character string into another. Our results show a strong correlation between the edit distances and the scores obtained from the scoring rubrics of the pump repair task from the National Assessment of Education Progress Technology and Engineering Literacy assessments, implying that the edit distance to the best performance sequence can be considered as a new feature variable that encodes information about students’ proficiency, which sheds light on the value of data-driven scoring rules for test and task development and for refining the scoring rubrics.

How to Cite

Hao, J., Shu, Z., & von Davier, A. (2015). Analyzing Process Data from Game/Scenario-Based Tasks: An Edit Distance Approach. Journal of Educational Data Mining, 7(1), 33–50.
Abstract 886 | PDF Downloads 438



game/scenario-based tasks, Levenshtein distance, action string, data-driven scoring rules

ALMOND, R., STEINBERG, L., AND MISLEVY, R. 2002. Enhancing the design and delivery of assessment systems: A four-process architecture. The Journal of Technology, Learning and Assessment 1, 5.

BERGNER, Y., SHU, Z., AND VON DAVIER, A. 2014. Visualizing and clustering sequence data from a simulation-based assessment task. Journal of Educational Data Mining.

BRYANT, V. 1985. Metric spaces: iteration and application. Cambridge University Press.

CHIEU, V. M., LUENGO, V., VADCARD, L., AND TONETTI, J. 2010. Student modeling in orthopedic surgery training: Exploiting symbiosis between temporal bayesian networks and fine-grained didactic analysis. International Journal of Artificial Intelligence in Education 20, 3, 269–301.

DESMARAIS, M. C. AND LEMIEUX, F. 2013. Clustering and visualizing study state sequences. In Proceedings of 6th International Conference on Educational Data Mining, pp. 224–227.

GABADINHO, A., RITSCHARD, G., STUDER, M., AND M¨ULLER, N. S. 2009. Mining sequence data in r with the traminer package: A users guide for version 1.2. Geneva: University of Geneva.

GEE, J. P. 2007. What video games have to teach us about learning and literacy.: Revised and Updated Edition. Macmillan.

GUTIERREZ-SANTOS, S., MAVRIKIS, M., AND MAGOULAS, G. 2010. Sequence detection for adaptive feedback generation in an exploratory environment for mathematical generalisation. In Artificial Intelligence: Methodology, Systems, and Applications, pp. 181–190. Springer.

JURAFSKY, D. AND MARTIN, J. H. 2000. Speech & Language Processing. Pearson Education India.

KLOPFER, E., OSTERWEIL, S., GROFF, J., AND HAAS, J. 2009. Using the technology of today, in the classroom today. The Education arcade.

K¨OCK, M. AND PARAMYTHIS, A. 2011. Activity sequence modelling and dynamic clustering for personalized e-learning. User Modeling and User-Adapted Interaction 21, 1-2, 51–97.

LEVENSHTEIN, V. I. 1966. Binary codes capable of correcting deletions, insertions and reversals. In Soviet physics doklady, Volume 10, pp. 707.

MISLEVY, R., ORANJE, A., BAUER, M. I., VON DAVIER, A. A., HAO, J., CORRIGAN, S., HOFFMAN, E., DICERBO, K., AND JOHN, M. 2014. Psychometric considerations in game based assessments. CreateSpace Independent Publishing Platform.

MISLEVY, R. J. AND RICONSCENTE, M. 2006. Evidence-centered assessment design. Handbook of test

development, 61–90. PUMPREPAIR 2013. Pump Repair Sample Task. tel/wells_item.aspx.

SAO PEDRO, M. A., DE BAKER, R. S., GOBERT, J. D., MONTALVO, O., AND NAKAMA, A. 2013. Leveraging machine-learned detectors of systematic inquiry behavior to estimate and predict transfer of inquiry skill. User Modeling and User-Adapted Interaction 23, 1, 1–39.

SCHMIT, M. J. AND RYAN, A. M. 1992. Test-taking dispositions: A missing link? Journal of Applied Psychology 77, 5, 629.

SUNDRE, D. L. AND WISE, S. L. 2003. Motivation filtering: An exploration of the impact of low examinee motivation on the psychometric quality of tests. In annual meeting of the National Council on Measurement in Education, Chicago, IL.

TEL 2013. Technology and Engineering Literacy Assessments. nationsreportcard/tel/.

USMLE 2014. United States Medical Licensure Examinations. step-3/2014content_step3.pdf/.

VINH, N. X., EPPS, J., AND BAILEY, J. 2009. Information theoretic measures for clusterings comparison: is a correction for chance necessary? In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 1073–1080. ACM.