Analyzing Process Data from Game/Scenario-Based Tasks: An Edit Distance Approach
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
Jan 29, 2015
Jiangang Hao
Zhan Shu
Alina von Davier
Abstract
Students’ activities in game/scenario-based tasks (G/SBTs) can be characterized by a sequence of time-stamped actions of different types with different attributes. For a subset of G/SBTs in which only the order of the actions is of great interest, the process data can be well characterized as a string of characters (i.e., action string) if we encode each action name as a single character. In this article, we report our work on evaluating students’ performances by comparing how far their action strings are from the action string that corresponds to the best performance, where the proximity is quantified by the edit distance between the strings. Specifically, we choose the Levenshtein distance, which is defined as the minimum number of insertions, deletions, and replacements needed to convert one character string into another. Our results show a strong correlation between the edit distances and the scores obtained from the scoring rubrics of the pump repair task from the National Assessment of Education Progress Technology and Engineering Literacy assessments, implying that the edit distance to the best performance sequence can be considered as a new feature variable that encodes information about students’ proficiency, which sheds light on the value of data-driven scoring rules for test and task development and for refining the scoring rubrics.
How to Cite
Hao, J., Shu, Z., & von Davier, A. (2015). Analyzing Process Data from Game/Scenario-Based Tasks: An Edit Distance Approach. Journal of Educational Data Mining, 7(1), 33–50. https://doi.org/10.5281/zenodo.3554705
##plugins.themes.bootstrap3.article.details##
Keywords
game/scenario-based tasks, Levenshtein distance, action string, data-driven scoring rules
References
ALMOND, R., STEINBERG, L., AND MISLEVY, R. 2002. Enhancing the design and delivery of assessment systems: A four-process architecture. The Journal of Technology, Learning and Assessment 1, 5.
BERGNER, Y., SHU, Z., AND VON DAVIER, A. 2014. Visualizing and clustering sequence data from a simulation-based assessment task. Journal of Educational Data Mining.
BRYANT, V. 1985. Metric spaces: iteration and application. Cambridge University Press.
CHIEU, V. M., LUENGO, V., VADCARD, L., AND TONETTI, J. 2010. Student modeling in orthopedic surgery training: Exploiting symbiosis between temporal bayesian networks and fine-grained didactic analysis. International Journal of Artificial Intelligence in Education 20, 3, 269–301.
DESMARAIS, M. C. AND LEMIEUX, F. 2013. Clustering and visualizing study state sequences. In Proceedings of 6th International Conference on Educational Data Mining, pp. 224–227.
GABADINHO, A., RITSCHARD, G., STUDER, M., AND M¨ULLER, N. S. 2009. Mining sequence data in r with the traminer package: A users guide for version 1.2. Geneva: University of Geneva.
GEE, J. P. 2007. What video games have to teach us about learning and literacy.: Revised and Updated Edition. Macmillan.
GUTIERREZ-SANTOS, S., MAVRIKIS, M., AND MAGOULAS, G. 2010. Sequence detection for adaptive feedback generation in an exploratory environment for mathematical generalisation. In Artificial Intelligence: Methodology, Systems, and Applications, pp. 181–190. Springer.
JURAFSKY, D. AND MARTIN, J. H. 2000. Speech & Language Processing. Pearson Education India.
KLOPFER, E., OSTERWEIL, S., GROFF, J., AND HAAS, J. 2009. Using the technology of today, in the classroom today. The Education arcade.
K¨OCK, M. AND PARAMYTHIS, A. 2011. Activity sequence modelling and dynamic clustering for personalized e-learning. User Modeling and User-Adapted Interaction 21, 1-2, 51–97.
LEVENSHTEIN, V. I. 1966. Binary codes capable of correcting deletions, insertions and reversals. In Soviet physics doklady, Volume 10, pp. 707.
MISLEVY, R., ORANJE, A., BAUER, M. I., VON DAVIER, A. A., HAO, J., CORRIGAN, S., HOFFMAN, E., DICERBO, K., AND JOHN, M. 2014. Psychometric considerations in game based assessments. CreateSpace Independent Publishing Platform.
MISLEVY, R. J. AND RICONSCENTE, M. 2006. Evidence-centered assessment design. Handbook of test
development, 61–90. PUMPREPAIR 2013. Pump Repair Sample Task. http://nces.ed.gov/nationsreportcard/ tel/wells_item.aspx.
SAO PEDRO, M. A., DE BAKER, R. S., GOBERT, J. D., MONTALVO, O., AND NAKAMA, A. 2013. Leveraging machine-learned detectors of systematic inquiry behavior to estimate and predict transfer of inquiry skill. User Modeling and User-Adapted Interaction 23, 1, 1–39.
SCHMIT, M. J. AND RYAN, A. M. 1992. Test-taking dispositions: A missing link? Journal of Applied Psychology 77, 5, 629.
SUNDRE, D. L. AND WISE, S. L. 2003. Motivation filtering: An exploration of the impact of low examinee motivation on the psychometric quality of tests. In annual meeting of the National Council on Measurement in Education, Chicago, IL.
TEL 2013. Technology and Engineering Literacy Assessments. https://nces.ed.gov/ nationsreportcard/tel/.
USMLE 2014. United States Medical Licensure Examinations. http://www.usmle.org/pdfs/ step-3/2014content_step3.pdf/.
VINH, N. X., EPPS, J., AND BAILEY, J. 2009. Information theoretic measures for clusterings comparison: is a correction for chance necessary? In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 1073–1080. ACM.
BERGNER, Y., SHU, Z., AND VON DAVIER, A. 2014. Visualizing and clustering sequence data from a simulation-based assessment task. Journal of Educational Data Mining.
BRYANT, V. 1985. Metric spaces: iteration and application. Cambridge University Press.
CHIEU, V. M., LUENGO, V., VADCARD, L., AND TONETTI, J. 2010. Student modeling in orthopedic surgery training: Exploiting symbiosis between temporal bayesian networks and fine-grained didactic analysis. International Journal of Artificial Intelligence in Education 20, 3, 269–301.
DESMARAIS, M. C. AND LEMIEUX, F. 2013. Clustering and visualizing study state sequences. In Proceedings of 6th International Conference on Educational Data Mining, pp. 224–227.
GABADINHO, A., RITSCHARD, G., STUDER, M., AND M¨ULLER, N. S. 2009. Mining sequence data in r with the traminer package: A users guide for version 1.2. Geneva: University of Geneva.
GEE, J. P. 2007. What video games have to teach us about learning and literacy.: Revised and Updated Edition. Macmillan.
GUTIERREZ-SANTOS, S., MAVRIKIS, M., AND MAGOULAS, G. 2010. Sequence detection for adaptive feedback generation in an exploratory environment for mathematical generalisation. In Artificial Intelligence: Methodology, Systems, and Applications, pp. 181–190. Springer.
JURAFSKY, D. AND MARTIN, J. H. 2000. Speech & Language Processing. Pearson Education India.
KLOPFER, E., OSTERWEIL, S., GROFF, J., AND HAAS, J. 2009. Using the technology of today, in the classroom today. The Education arcade.
K¨OCK, M. AND PARAMYTHIS, A. 2011. Activity sequence modelling and dynamic clustering for personalized e-learning. User Modeling and User-Adapted Interaction 21, 1-2, 51–97.
LEVENSHTEIN, V. I. 1966. Binary codes capable of correcting deletions, insertions and reversals. In Soviet physics doklady, Volume 10, pp. 707.
MISLEVY, R., ORANJE, A., BAUER, M. I., VON DAVIER, A. A., HAO, J., CORRIGAN, S., HOFFMAN, E., DICERBO, K., AND JOHN, M. 2014. Psychometric considerations in game based assessments. CreateSpace Independent Publishing Platform.
MISLEVY, R. J. AND RICONSCENTE, M. 2006. Evidence-centered assessment design. Handbook of test
development, 61–90. PUMPREPAIR 2013. Pump Repair Sample Task. http://nces.ed.gov/nationsreportcard/ tel/wells_item.aspx.
SAO PEDRO, M. A., DE BAKER, R. S., GOBERT, J. D., MONTALVO, O., AND NAKAMA, A. 2013. Leveraging machine-learned detectors of systematic inquiry behavior to estimate and predict transfer of inquiry skill. User Modeling and User-Adapted Interaction 23, 1, 1–39.
SCHMIT, M. J. AND RYAN, A. M. 1992. Test-taking dispositions: A missing link? Journal of Applied Psychology 77, 5, 629.
SUNDRE, D. L. AND WISE, S. L. 2003. Motivation filtering: An exploration of the impact of low examinee motivation on the psychometric quality of tests. In annual meeting of the National Council on Measurement in Education, Chicago, IL.
TEL 2013. Technology and Engineering Literacy Assessments. https://nces.ed.gov/ nationsreportcard/tel/.
USMLE 2014. United States Medical Licensure Examinations. http://www.usmle.org/pdfs/ step-3/2014content_step3.pdf/.
VINH, N. X., EPPS, J., AND BAILEY, J. 2009. Information theoretic measures for clusterings comparison: is a correction for chance necessary? In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 1073–1080. ACM.
Issue
Section
Articles
Authors who publish with this journal agree to the following terms:
- The Author retains copyright in the Work, where the term “Work” shall include all digital objects that may result in subsequent electronic publication or distribution.
- Upon acceptance of the Work, the author shall grant to the Publisher the right of first publication of the Work.
- The Author shall grant to the Publisher and its agents the nonexclusive perpetual right and license to publish, archive, and make accessible the Work in whole or in part in all forms of media now or hereafter known under a Creative Commons 4.0 License (Attribution-Noncommercial-No Derivatives 4.0 International), or its equivalent, which, for the avoidance of doubt, allows others to copy, distribute, and transmit the Work under the following conditions:
- Attribution—other users must attribute the Work in the manner specified by the author as indicated on the journal Web site;
- Noncommercial—other users (including Publisher) may not use this Work for commercial purposes;
- No Derivative Works—other users (including Publisher) may not alter, transform, or build upon this Work,with the understanding that any of the above conditions can be waived with permission from the Author and that where the Work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.
- The Author is able to enter into separate, additional contractual arrangements for the nonexclusive distribution of the journal's published version of the Work (e.g., post it to an institutional repository or publish it in a book), as long as there is provided in the document an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post online a pre-publication manuscript (but not the Publisher’s final formatted PDF version of the Work) in institutional repositories or on their Websites prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (see The Effect of Open Access). Any such posting made before acceptance and publication of the Work shall be updated upon publication to include a reference to the Publisher-assigned DOI (Digital Object Identifier) and a link to the online abstract for the final published Work in the Journal.
- Upon Publisher’s request, the Author agrees to furnish promptly to Publisher, at the Author’s own expense, written evidence of the permissions, licenses, and consents for use of third-party material included within the Work, except as determined by Publisher to be covered by the principles of Fair Use.
- The Author represents and warrants that:
- the Work is the Author’s original work;
- the Author has not transferred, and will not transfer, exclusive rights in the Work to any third party;
- the Work is not pending review or under consideration by another publisher;
- the Work has not previously been published;
- the Work contains no misrepresentation or infringement of the Work or property of other authors or third parties; and
- the Work contains no libel, invasion of privacy, or other unlawful matter.
- The Author agrees to indemnify and hold Publisher harmless from Author’s breach of the representations and warranties contained in Paragraph 6 above, as well as any claim or proceeding relating to Publisher’s use and publication of any content contained in the Work, including third-party content.