Analyzing Student Process Data in Game-Based Assessments with Bayesian Knowledge Tracing and Dynamic Bayesian Networks

##plugins.themes.bootstrap3.article.main##

##plugins.themes.bootstrap3.article.sidebar##

Published Jun 24, 2019
Ying Cui Man-Wai Chu Fu Chen

Abstract

Digital game-based assessments generate student process data that is much more difficult to analyze than traditional assessments. The formative nature of game-based assessments permits students, through applying and practicing the targeted knowledge and skills during gameplay, to gain experiences, receive immediate feedback, and as a result, improve their skill mastery. Both Bayesian Knowledge Tracing and Dynamic Bayesian Networks are capable of updating students’ mastery levels based on their observed responses during the assessment. This paper investigates the use of these two models for analyzing student response process data from an interactive game-based assessment, Raging Skies. The game measures a set of knowledge and skill-based learner outcomes listed in a Canadian Provincial Grade 5 science program-of-study under the Weather Watch unit. To evaluate and compare the performance of Bayesian Knowledge Tracing and Dynamic Bayesian Networks, the classification consistency and accuracy are examined.

How to Cite

Cui, Y., Chu, M.-W., & Chen, F. (2019). Analyzing Student Process Data in Game-Based Assessments with Bayesian Knowledge Tracing and Dynamic Bayesian Networks. JEDM | Journal of Educational Data Mining, 11(1), 80-100. Retrieved from https://jedm.educationaldatamining.org/index.php/JEDM/article/view/397
Abstract 376 | PDF Downloads 346

##plugins.themes.bootstrap3.article.details##

Keywords

game-based assessment, Evidence-Centered Design, Bayesian Knowledge Tracing, Dynamic Bayesian Networks, formative feedback, process data analysis

References
ALBERTA EDUCATION 1996. Science (elementary). Alberta Education. http://www.education.alberta.ca/media/654825/elemsci.pdf

ALMOND, R. G., MISLEVY, R. J., STEINBERG, L. S., WILLIAMSON, D. M., AND YAN, D. 2015. Bayesian Networks in Educational Assessment. Springer, New York.

AMERICAN EDUCATIONAL RESEARCH ASSOCIATION [AERA], AMERICAN PSYCHOLOGICAL ASSOCIATION [APA], AND NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION [NCME]. 2014. Standards for Educational and Psychological Testing. Author, Washington, DC.

BARAB, S. A., GRESALFI, M. S., AND INGRAM-GOBLE, A. 2010. Transformational play: Using games to position person, content, and context. Educ. Res. 39, 525–536.

BEHRENS, J. T., MISLEVY, R. J., DICERBO, K. E., AND LEVY, R. 2012. Evidence centered design for learning and assessment. In Technology-based Assessments for 21st Century Skills: Theoretical and Practical Implications from Modern Research, M. Mayrath, J. Clarke-Midura, D. Robinson, and G. Schraw, Eds. Information Age Publishers, Charlotte, NC, 371–386.

BENNETT, R. E., PERSKY, H., WEISS, A. R., AND JENKINS, F. 2007. Problem solving in technology-rich environments: A report from the NAEP technology-based assessment project (NCES 2007–466). U.S. Department of Education. National Center for Education Statistics, Washington, DC. http://nces.ed.gov/nationsreportcard/pubs/studies/2007466.asp

BERTLING, M., JACKSON, G., ORANJE, A., AND OWEN, V. 2015. Measuring argumentation skills with game-based assessments: Evidence for incremental validity and learning. In C. Conati, N. Heffernan, A. Mitrovic, & M. Verdejo (Eds.), Proceedings of the 17th International Conference on Artificial Intelligence in Education. Springer International, New York, NY, 545-549.

BLACK, P., AND WILIAM, D. 1998. Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 80, 139– 144.

CHU, M-W., AND CHIANG, A. 2018. Raging skies: Development of a digital game-based science assessment using evidence-centered game design. Alta. J. Sci. Educ. 45, 37–47.

CHU, M-W., ASTON, R., CUI, Y., SHOJAEE, N., AND BAWEL, B. 2018. Development of a digital gamebased assessment to measure science skill-based outcomes. Paper presented at the Annual Meeting of Canadian Society for the Study of Education, Regina, SK, Canada.

CORBETT, A. T., AND ANDERSON, J. R. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Model. User-adapt. Interact. 4, 253–278.

DICERBO, K. (2017). Building the evidentiary argument in game-based assessment. Journal of Applied Testing Technology, 18, 7-18.

ERCIKAN, K. AND PELLIGRINO, J.W. (2017). Validation of score meaning using examinee response processes for the next generation of assessments. Routledge, New York.

FORSYTH, C. M., JACKSON, G. T., HEBERT, D., LEHMAN, B., INGLESE, P. AND GRACE,L. (2017). Striking a balance: User-experience and performance in computerized game-based assessment. In R.Baker, E. Andre, X. Hu, T. Rodrigo, B. du Bouley. The Proceedings of the International Conference on Artificial Intelligence in Education, LNCS, Springer-Verlag, 502-505.

GEE, J. P. (2007). Good Videogames + Good Learning: Collected Essays on Videogames, Learning and Literacy. Peter Lang Publishing, New York.

HABGOOD, J. M.P., AND AINSWORTH, S. E. (2011). Motivating children to learn effectively: exploring the value of intrinsic integration in educational games. Journal of the Learning Sciences, 20, 2, 169-206.

HECKERMAN, D., GEIGER, D., AND CHICKERING, D. M. 1995. Learning Bayesian networks: The combination of knowledge and statistical data. Mach. Learn. 20, 197–243.

INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION 2007. The National Educational Technology Standards and Performance Indicators for Students. ISTE, Eugene, OR.

JENSEN, F. V. 1996. An Introduction to Bayesian networks. University College London Press, London.

LEPPER, M. R., AND MALONE, T. W. (1987). Intrinsic motivation and instructional effectiveness in computer-based education. In R. E. Snow & M. J. Farr (Eds.), Aptitude, Learning, and Instruction: Vol. 3. Cognitive and Affective Process Analysis. Erlbaum, Hillsdale, NJ, 255- 286.

MALONE, T. W., AND LEPPER, M. R. (1987). Making learning fun: A taxonomy of intrinsic motivations for learning. In R. E. Snow & M. J. Farr (Eds.), Aptitude, Learning, and Instruction: Vol. 3. Cognitive and Affective Process Analysis. Erlbaum, Hillsdale, NJ, 223- 253.

MIHAJLOVIC, V., AND PETKOVIC, M. 2001. Dynamic Bayesian Networkss: A State of the Art. https://ris.utwente.nl/ws/portalfiles/portal/27679465/0000006a.pdf

MILLIS, K., FORSYTH, C.M., WALLACE, P., GRAESSER, A.C., AND TIMMINS, G. 2016. The impact of game-like features on learning from an intelligent tutoring system. Technology, Knowledge and Learning, 22, 1-22.

MISLEVY, R. J. 2006. Cognitive psychology and educational assessment. In Educational Measurement (4th ed.), R. L. Brennan Ed. American Council on Education/Praeger, Westport, CT, 257–305.

MISLEVY, R. J., ALMOND, R. G., AND LUKAS, J. 2003. A brief introduction to Evidence- Centered Design (Research Report No. RR-03-16). Educational Testing Service. https://www.ets.org/Media/Research/pdf/RR-03-16.pdf

MISLEVY, R. J., ORANJE, A., BAUER, M. I., VON DAVIER, A., HAO, J., CORRIGAN, S., HOFFMAN, E., DICERBO, K., AND JOHN, M. 2014. Psychometric considerations in game-based assessment. GlassLab: Institute of play. http://www.instituteofplay.org/work/projects/glasslab-research/

MURPHY, K. P. 2002. Dynamic Bayesian Networks: Representation, Inference, and Learning. Ph.D. dissertation. University of California, Berkeley, CA.

NATIONAL RESEARCH COUNCIL. 2011. Learning Science through Computer Games and Simulations. National Academies Press, Washington, DC. OECD. 2010. PISA 2009 Results: What Students Know and Can Do – Student Performance in Reading, Mathematics and Science (Volume I). PISA, OECD Publishing. http://dx.doi.org/10.1787/9789264091450-en.

PELLEGRINO, J. W. AND QUELLMALZ, E. S. 2010. Perspectives on the integration of technology and assessment. J. Res. Tech. Educ. 43, 119–134.

QUELLMALZ, E. S., KREIKEMEIER, P., DEBARGER, A. H., AND HAERTEL, G. 2007. A study of the alignment of the NAEP, TIMSS, and New Standards Science Assessments with the inquiry abilities in the National Science Education Standards. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL.

QUELLMALZ, E. S., TIMMS, M. J., AND SCHNEIDER, S. A. 2009. Assessment of student learning in science simulations and games. National Research Council, Washington, D.C. R CORE TEAM. 2018. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria.

RIEBER, L. 1996. Seriously considering play: Designing interactive learning environments based on the blending of microworlds, simulations, and games. Education and Technology Research & Development, 44, 42-58. Doi: 10.1007/BF02300540

SHUTE, V. J., AND VENTURA, M. 2013. Measuring and supporting learning in games: Stealth assessment. Massachusetts Institute of Technology Press, Cambridge, MA. http://myweb.fsu.edu/vshute/pdf/white.pdf

STEVENS, R., BEAL, C. R., AND SPRANG, M. 2013. Assessing students’ problem solving ability and cognitive regulation with learning trajectories. In International Handbook of Metacognition and Learning Technologies, R. Azevedo and V. Aleven Eds. Springer, New York, 409–423.
Section
Articles