Identifying Key Features of Student Performance in Educational Video Games and Simulations through Cluster Analysis
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
Oct 1, 2012
Deirdre Kerr
Gregory K.W.K. Chung
Abstract
The assessment cycle of evidence-centered design (ECD) provides a framework for treating an educational video game or simulation as an assessment. One of the main steps in the assessment cycle of ECD is the identification of the key features of student performance. While this process is relatively simple for multiple choice tests, when applied to log data from educational video games or simulations it becomes one of the most serious bottlenecks facing researchers interested in implementing ECD. In this paper we examine the utility of cluster analysis as a method of identifying key features of student performance in log data stemming from educational video games or simulations. In our study, cluster analysis was able to consistently identify key features of student performance in the form of solution strategies and error patterns across levels, which contained few extraneous actions and explained a sufficient amount of the data.
How to Cite
Kerr, D., & Chung, G. K. (2012). Identifying Key Features of Student Performance in Educational Video Games and Simulations through Cluster Analysis. Journal of Educational Data Mining, 4(1), 144–182. https://doi.org/10.5281/zenodo.3554647
##plugins.themes.bootstrap3.article.details##
Keywords
evidence-centered design, cluster analysis, fuzzy cluster analysis, feature cluster analysis, log data, educational video games, key features of student performance, student strategies
References
BERKHIN, R. 2006. A survey of clustering data mining techniques. In Grouping Multidimensional Data, J. KOGAN, C. NICHOLAS, AND M. TEBOULLE, Eds. Springer, New York, NY, 25-72.
BONCHI, F., GIANNOTI, F., GOZZI, C., MANCO, G., NANNI, M., PEDRESCHI, D., RENSO, C., AND RUGGIERI, S. 2001. Web log data warehouses and mining for intelligent web caching. Data & Knowledge Engineering 39, 165-189.
CRAGAR, G. E., BERRY, D. T. R., SCHMITT, F. A., AND FAKHOURY, T. A. 2005. Cluster analysis of normal personality traits in patients with psychogenic nonepileptic seizures. Epilepsy & Behavior 6, 593-600.
CHUNG, G. K. W. K., BAKER, E. L ., VENDLINSKI, T. P., BUSCHANG, R. E., DELACRUZ, G. C., MICHIUYE, J. K., AND BITTICK, S. J. 2010. Testing instructional design variations in a prototype math game. In Current Perspectives from Three National R&D Centers Focused on Game-based Learning: Issues in Learning, Instruction, Assessment, and Game Design. Structured poster session at the annual meeting of the American Educational Research Association, Denver, CO, April, 2010, R. ATKINSON, Chair.
CHUNG, G. K. W. K., AND KERR, D. 2012. A Primer on Data Logging to Support Extraction of Meaningful Information from Educational Games: An Example from Save Patch. (CRESST Tech. Rep. No. 814). University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing, Los Angeles, CA.
FRAWLEY, W. J., PIATESKI-SHAPIRO, G., AND MATHEUS, C. J. 1992. Knowledge discovery in databases: An overview. AI Magazine 13, 57-70.
FREZZO, D. C., BEHRENS, J. T., MISLEVY, R. J., WEST, P., AND D I CERBO, K. E. 2009. Psychometric and evidentiary approaches to simulation assessment in packet tracer software. In Proceedings of the Fifth International Conference on Networking and Services (ICNS 2009), Valencia, Spain, April, 2009, 555-560.
HAND, D., MANNILA, H., AND SMYTH, P. 2001. Principles of data mining. MIT Press, Cambridge, MA.
HUANG, Z. 1998. Extensions to the k-means algorithm for clustering large data sets with categorical values. Data Mining and Knowledge Discovery 2, 283-304.
JAIN, A. K., MURTY, M. N., AND FLYNN, P. J. 1999. Data clustering: A review. ACM Computing Surveys, 31, 264-323.
JAMES, F., AND M C CULLOCH, C. 1990. Multivariate analysis in ecology and systematic: Panacea or Pandora’s box? Annual Review of Ecology and Systematics 21, 129-166.
KERR, D., AND CHUNG, G. K. W. K. 2012. Using Cluster Analysis to Extend Usability Testing to Instructional Content. (CRESST Tech. Rep. No. 816). University of
California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing, Los Angeles, CA.
KRIER, C., FRANCOIS, D., ROSSI, F., AND VERLEYSEN, M. 2007. Feature clustering and mutual information for the selection of variables in spectral data. In Proceedings of the 2007 European Symposium on Artificial Neural Networks (ESANN 2007), Bruges, Belgium, April, 2007, 157-162.
MADHYASTHA, T., AND HUNT, E. 2009. Mining diagnostic assessment data for concept similarity. Journal of Educational Data Mining 1, 72-91.
MALCOM, S. M., CHUBIN, D. E. AND JESSE, J. K. 2004. Standing our ground: A guidebook for STEM educators in the Post-Michigan Era. American Association for the Advancement of Science, Washington, D.C.
MASIP, D., MINGUILLON, J., AND MOR, E. 2 011. Capturing and analyzing student behavior in a virtual learning environment. In Handbook of Educational Data Mining, C. ROMERO, S. VENTURA, M. PECHENIZKIY, AND R. S.J. D . BAKER, Eds. CRC Press, Boca Raton, FL, 339-351.
MERCERON, A., AND YACEF, K. 2004. Mining student data captured from a web- based tutoring tool: Initial exploration and results. Journal of Interactive Learning Research 15, 319-346.
MISLEVY, R. J., ALMOND, R. G., AND LUKAS, J. F. 2004. A Brief Introduction to ECD. (CRESST Tech. Rep. No. 632). University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing, Los Angeles, CA.
MISLEVY, R.J., STEINBERG, L.S., ALMOND, R.G., AND LUKAS, J.F. 2006.
Concepts, terminology, and basic models of evidence-centered design. In Automated Scoring of Complex Tasks in Computer-based Testing, D. M. WILLIAMSON, I. I. BEJAR, AND R. J. MISLEVY, Eds. Erlbaum, Mahwah, NJ, 15-48.
NATIONAL MATHEMATICS ADVISORY PANEL (NMAP). 2008. Foundations for success: The final report of the National Mathematics Advisory Panel. U.S. Department of Education, Washington, DC. R DEVELOPMENT CORE TEAM. 2010. R: A Language and Environment for Statistical Computing [Computer software]. Retrieved from http://www.R- project.org
ROMERO, C., GONZALEZ, P., VENTURA, S., DEL JESUS, M. J., AND HERRERA, F. 2009. Evolutionary algorithms for subgroup discovery in e-learning: A practical application using Moodle data. Expert Systems with Applications 39, 1632-1644.
ROMERO, C., AND VENTURA, S. 2007. Educational data mining: A survey from 1995 to 2005. Expert Systems with Applications 35, 135-146.
ROUSSOS, L., STOUT, W., AND MARDEN, J. 1998. Using new proximity measures with hierarchical cluster analysis to detect multidimensionality. Journal of Educational Measurement 35, 1-30.
ROWLEY, S. 2000. Profiles of African-American college students’ educational utility and performance: A cluster analysis. Journal of Black Psychology 26, 3-26.
RUPP, A. A. in press. Clustering and classification. In The Oxford Handbook of Quantitative Methods, T. LITTLE, Ed. Oxford University Press, New York, NY.
RUPP, A. A., GUSHTA, M., MISLEVY, R. J., AND SHAFFER, D. W. 2010. ECD of epistemic games: Measurement principles for complex learning environments. The Journal of Technology, Learning, and Assessment 8, Retrieved from http://ejournals.bc.edu/ojs/index.php/jtla/article/view/1623/1467
RUSPINI, E. H. 1969. A new approach to clustering. Information and Control 15, 22-32.
STEINLEY, D. 2006. K-means clustering: A half-century synthesis. British Journal of Mathematical and Statistical Psychology 59, 1-34.
TRIGWELL, K., PROSSER, M., AND WATERHOUSE, F. 1999. Relations between teachers’ approaches to teaching and students’ approaches to learning. Higher Education 37, 57-70.
VOGT, W., AND NAGEL, D. 1992. Cluster analysis in diagnosis. Clinical Chemistry 38, 182-198.
BONCHI, F., GIANNOTI, F., GOZZI, C., MANCO, G., NANNI, M., PEDRESCHI, D., RENSO, C., AND RUGGIERI, S. 2001. Web log data warehouses and mining for intelligent web caching. Data & Knowledge Engineering 39, 165-189.
CRAGAR, G. E., BERRY, D. T. R., SCHMITT, F. A., AND FAKHOURY, T. A. 2005. Cluster analysis of normal personality traits in patients with psychogenic nonepileptic seizures. Epilepsy & Behavior 6, 593-600.
CHUNG, G. K. W. K., BAKER, E. L ., VENDLINSKI, T. P., BUSCHANG, R. E., DELACRUZ, G. C., MICHIUYE, J. K., AND BITTICK, S. J. 2010. Testing instructional design variations in a prototype math game. In Current Perspectives from Three National R&D Centers Focused on Game-based Learning: Issues in Learning, Instruction, Assessment, and Game Design. Structured poster session at the annual meeting of the American Educational Research Association, Denver, CO, April, 2010, R. ATKINSON, Chair.
CHUNG, G. K. W. K., AND KERR, D. 2012. A Primer on Data Logging to Support Extraction of Meaningful Information from Educational Games: An Example from Save Patch. (CRESST Tech. Rep. No. 814). University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing, Los Angeles, CA.
FRAWLEY, W. J., PIATESKI-SHAPIRO, G., AND MATHEUS, C. J. 1992. Knowledge discovery in databases: An overview. AI Magazine 13, 57-70.
FREZZO, D. C., BEHRENS, J. T., MISLEVY, R. J., WEST, P., AND D I CERBO, K. E. 2009. Psychometric and evidentiary approaches to simulation assessment in packet tracer software. In Proceedings of the Fifth International Conference on Networking and Services (ICNS 2009), Valencia, Spain, April, 2009, 555-560.
HAND, D., MANNILA, H., AND SMYTH, P. 2001. Principles of data mining. MIT Press, Cambridge, MA.
HUANG, Z. 1998. Extensions to the k-means algorithm for clustering large data sets with categorical values. Data Mining and Knowledge Discovery 2, 283-304.
JAIN, A. K., MURTY, M. N., AND FLYNN, P. J. 1999. Data clustering: A review. ACM Computing Surveys, 31, 264-323.
JAMES, F., AND M C CULLOCH, C. 1990. Multivariate analysis in ecology and systematic: Panacea or Pandora’s box? Annual Review of Ecology and Systematics 21, 129-166.
KERR, D., AND CHUNG, G. K. W. K. 2012. Using Cluster Analysis to Extend Usability Testing to Instructional Content. (CRESST Tech. Rep. No. 816). University of
California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing, Los Angeles, CA.
KRIER, C., FRANCOIS, D., ROSSI, F., AND VERLEYSEN, M. 2007. Feature clustering and mutual information for the selection of variables in spectral data. In Proceedings of the 2007 European Symposium on Artificial Neural Networks (ESANN 2007), Bruges, Belgium, April, 2007, 157-162.
MADHYASTHA, T., AND HUNT, E. 2009. Mining diagnostic assessment data for concept similarity. Journal of Educational Data Mining 1, 72-91.
MALCOM, S. M., CHUBIN, D. E. AND JESSE, J. K. 2004. Standing our ground: A guidebook for STEM educators in the Post-Michigan Era. American Association for the Advancement of Science, Washington, D.C.
MASIP, D., MINGUILLON, J., AND MOR, E. 2 011. Capturing and analyzing student behavior in a virtual learning environment. In Handbook of Educational Data Mining, C. ROMERO, S. VENTURA, M. PECHENIZKIY, AND R. S.J. D . BAKER, Eds. CRC Press, Boca Raton, FL, 339-351.
MERCERON, A., AND YACEF, K. 2004. Mining student data captured from a web- based tutoring tool: Initial exploration and results. Journal of Interactive Learning Research 15, 319-346.
MISLEVY, R. J., ALMOND, R. G., AND LUKAS, J. F. 2004. A Brief Introduction to ECD. (CRESST Tech. Rep. No. 632). University of California, Los Angeles, National Center for Research on Evaluation, Standards, and Student Testing, Los Angeles, CA.
MISLEVY, R.J., STEINBERG, L.S., ALMOND, R.G., AND LUKAS, J.F. 2006.
Concepts, terminology, and basic models of evidence-centered design. In Automated Scoring of Complex Tasks in Computer-based Testing, D. M. WILLIAMSON, I. I. BEJAR, AND R. J. MISLEVY, Eds. Erlbaum, Mahwah, NJ, 15-48.
NATIONAL MATHEMATICS ADVISORY PANEL (NMAP). 2008. Foundations for success: The final report of the National Mathematics Advisory Panel. U.S. Department of Education, Washington, DC. R DEVELOPMENT CORE TEAM. 2010. R: A Language and Environment for Statistical Computing [Computer software]. Retrieved from http://www.R- project.org
ROMERO, C., GONZALEZ, P., VENTURA, S., DEL JESUS, M. J., AND HERRERA, F. 2009. Evolutionary algorithms for subgroup discovery in e-learning: A practical application using Moodle data. Expert Systems with Applications 39, 1632-1644.
ROMERO, C., AND VENTURA, S. 2007. Educational data mining: A survey from 1995 to 2005. Expert Systems with Applications 35, 135-146.
ROUSSOS, L., STOUT, W., AND MARDEN, J. 1998. Using new proximity measures with hierarchical cluster analysis to detect multidimensionality. Journal of Educational Measurement 35, 1-30.
ROWLEY, S. 2000. Profiles of African-American college students’ educational utility and performance: A cluster analysis. Journal of Black Psychology 26, 3-26.
RUPP, A. A. in press. Clustering and classification. In The Oxford Handbook of Quantitative Methods, T. LITTLE, Ed. Oxford University Press, New York, NY.
RUPP, A. A., GUSHTA, M., MISLEVY, R. J., AND SHAFFER, D. W. 2010. ECD of epistemic games: Measurement principles for complex learning environments. The Journal of Technology, Learning, and Assessment 8, Retrieved from http://ejournals.bc.edu/ojs/index.php/jtla/article/view/1623/1467
RUSPINI, E. H. 1969. A new approach to clustering. Information and Control 15, 22-32.
STEINLEY, D. 2006. K-means clustering: A half-century synthesis. British Journal of Mathematical and Statistical Psychology 59, 1-34.
TRIGWELL, K., PROSSER, M., AND WATERHOUSE, F. 1999. Relations between teachers’ approaches to teaching and students’ approaches to learning. Higher Education 37, 57-70.
VOGT, W., AND NAGEL, D. 1992. Cluster analysis in diagnosis. Clinical Chemistry 38, 182-198.
Section
Articles
Authors who publish with this journal agree to the following terms:
- The Author retains copyright in the Work, where the term “Work” shall include all digital objects that may result in subsequent electronic publication or distribution.
- Upon acceptance of the Work, the author shall grant to the Publisher the right of first publication of the Work.
- The Author shall grant to the Publisher and its agents the nonexclusive perpetual right and license to publish, archive, and make accessible the Work in whole or in part in all forms of media now or hereafter known under a Creative Commons 4.0 License (Attribution-Noncommercial-No Derivatives 4.0 International), or its equivalent, which, for the avoidance of doubt, allows others to copy, distribute, and transmit the Work under the following conditions:
- Attribution—other users must attribute the Work in the manner specified by the author as indicated on the journal Web site;
- Noncommercial—other users (including Publisher) may not use this Work for commercial purposes;
- No Derivative Works—other users (including Publisher) may not alter, transform, or build upon this Work,with the understanding that any of the above conditions can be waived with permission from the Author and that where the Work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.
- The Author is able to enter into separate, additional contractual arrangements for the nonexclusive distribution of the journal's published version of the Work (e.g., post it to an institutional repository or publish it in a book), as long as there is provided in the document an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post online a pre-publication manuscript (but not the Publisher’s final formatted PDF version of the Work) in institutional repositories or on their Websites prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (see The Effect of Open Access). Any such posting made before acceptance and publication of the Work shall be updated upon publication to include a reference to the Publisher-assigned DOI (Digital Object Identifier) and a link to the online abstract for the final published Work in the Journal.
- Upon Publisher’s request, the Author agrees to furnish promptly to Publisher, at the Author’s own expense, written evidence of the permissions, licenses, and consents for use of third-party material included within the Work, except as determined by Publisher to be covered by the principles of Fair Use.
- The Author represents and warrants that:
- the Work is the Author’s original work;
- the Author has not transferred, and will not transfer, exclusive rights in the Work to any third party;
- the Work is not pending review or under consideration by another publisher;
- the Work has not previously been published;
- the Work contains no misrepresentation or infringement of the Work or property of other authors or third parties; and
- the Work contains no libel, invasion of privacy, or other unlawful matter.
- The Author agrees to indemnify and hold Publisher harmless from Author’s breach of the representations and warranties contained in Paragraph 6 above, as well as any claim or proceeding relating to Publisher’s use and publication of any content contained in the Work, including third-party content.