Analyzing Process Data from Game/Scenario-Based Tasks: An Edit Distance Approach

##plugins.themes.bootstrap3.article.main##

##plugins.themes.bootstrap3.article.sidebar##

Published Jan 29, 2015
Jiangang Hao Zhan Shu Alina von Davier

Abstract

Students' activities in game/scenario-based tasks (G/SBTs, hereafter) can be characterized by a sequence of time-stamped actions of different types with different attributes. For a subset of the G/SBTs where only the order of the actions are of great interest, the process data can be well characterized as a string of characters (action string, hereafter) if we encode each action name as a single character. In this paper, we report our work on evaluating students' performances by comparing how far their action strings are from the action string that corresponds to the best performance, where the proximity is quantified by the edit distance between the strings. Specifically, we choose the Levenshtein distance, which is defined as the minimum number of insertion, deletion and replacement needed to convert one character string to another. Our results show a strong correlation between the edit distances and the scores obtained from the scoring rubrics of the WELL task from NAEP TEL, implying the edit distance to the best performance sequence can be considered as a new feature variable that encodes information about students' proficiency, which sheds light on the value of data driven scoring rules for test and task development and for refining the scoring rubrics.

How to Cite

Hao, J., Shu, Z., & von Davier, A. (2015). Analyzing Process Data from Game/Scenario-Based Tasks: An Edit Distance Approach. JEDM | Journal of Educational Data Mining, 7(1), 33-50. Retrieved from https://jedm.educationaldatamining.org/index.php/JEDM/article/view/JEDM072
Abstract 483 | PDF Downloads 182

##plugins.themes.bootstrap3.article.details##

References
ALMOND, R., STEINBERG, L., and MISLEVY, R. 2002. Enhancing the design and delivery of assessment systems: Afour-process architecture. The Journal of Technology, Learning and Assessment 1, 5.

BERGNER, Y., SHU, Z., and VONDAVIER, A. 2014. Visualizing and clustering sequence data from a
simulation-based assessment task. Journal of Educational Data Mining.

BRYANT, V. 1985. Metric spaces: iteration and application. Cambridge University Press.

CHIEU, V. M., LUENGO, V., VADCARD, L., and TONETTI, J. 2010. Student modeling in orthopedic
surgery training: Exploiting symbiosis between temporal bayesian networks and fine-grained didactic
analysis. International Journal of Artificial Intelligence in Education 20, 3, 269–301.

DESMARAIS, M. C. and LEMIEUX, F. 2013. Clustering and visualizing study state sequences. In Proceedings of 6th International Conference on Educational Data Mining, pp. 224–227.

GABADINHO, A., RITSCHARD, G., STUDER, M., and MULLER, N. S. 2009. Mining sequence data in
r with the traminer package: Ausers guide for version 1.2. Geneva: University of Geneva.

GEE, J. P. 2007. What video games have to teach us about learning and literacy.: Revised and Updated
Edition. Macmillan.

GUTIERREZ-SANTOS, S., MAVRIKIS, M., and MAGOULAS, G. 2010. Sequence detection for adaptive feedback generation in an exploratory environment for mathematical generalisation. In Artificial

Intelligence: Methodology, Systems, and Applications, pp. 181–190. Springer.

JURAFSKY, D. and MARTIN, J. H. 2000. Speech & Language Processing. Pearson Education India.

KLOPFER, E., OSTERWEIL, S., GROFF, J., and HAAS, J. 2009. Using the technology of today, in the
classroom today. The Education arcade.

KOCK, M. and PARAMYTHIS, A. 2011. Activity sequence modelling and dynamic clustering for perKOCK
sonalized e-learning. User Modeling and User-Adapted Interaction 21, 1-2, 51–97.

LEVENSHTEIN, V. I. 1966. Binary codes capable of correcting deletions, insertions and reversals. In

Soviet physics doklady, Volume 10, pp. 707.

MISLEVY, R., ORANJE, A., BAUER, M. I., VONDAVIER, A. A., HAO, J., CORRIGAN, S., HOFFMAN,

E., DICERBO, K., and JOHN, M. 2014. Psychometric considerations in game based assessments.

CreateSpace Independent Publishing Platform.

MISLEVY, R. J. and RICONSCENTE, M. 2006. Evidence-centered assessment design. Handbook of test
development, 61–90.

PUMPREPAIR2013. Pump Repair Sample Task. http://nces.ed.gov/nationsreportcard/
tel/wells_item.aspx.

SAOPEDRO, M. A., DEBAKER, R. S., GOBERT, J. D., MONTALVO, O., and NAKAMA, A. 2013.
Leveraging machine-learned detectors of systematic inquiry behavior to estimate and predict transfer
of inquiry skill. User Modeling and User-Adapted Interaction 23, 1, 1–39.

SCHMIT, M. J. and RYAN, A. M. 1992. Test-taking dispositions: Amissing link? Journal of Applied
Psychology 77, 5, 629.

SUNDRE, D. L. and WISE, S. L. 2003. Motivation filtering: An exploration of the impact of low examinee motivation on the psychometric quality of tests. In annual meeting of the National Council on Measurement in Education, Chicago, IL.

TEL2013. Technology and Engineering Literacy Assessments. https://nces.ed.gov/
nationsreportcard/tel/.

USMLE2014. United States Medical Licensure Examinations. http://www.usmle.org/pdfs/
step-3/2014content_step3.pdf/.

VINH, N. X., EPPS, J., and BAILEY, J. 2009. Information theoretic measures for clusterings comparison: is a correction for chance necessary? In Proceed
Section
Articles