Analyzing Student Process Data in Game-Based Assessments with Bayesian Knowledge Tracing and Dynamic Bayesian Networks
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
Jun 24, 2019
Ying Cui
Man-Wai Chu
Fu Chen
Abstract
Digital game-based assessments generate student process data that is much more difficult to analyze than traditional assessments. The formative nature of game-based assessments permits students, through applying and practicing the targeted knowledge and skills during gameplay, to gain experiences, receive immediate feedback, and as a result, improve their skill mastery. Both Bayesian Knowledge Tracing and Dynamic Bayesian Networks are capable of updating students’ mastery levels based on their observed responses during the assessment. This paper investigates the use of these two models for analyzing student response process data from an interactive game-based assessment, Raging Skies. The game measures a set of knowledge and skill-based learner outcomes listed in a Canadian Provincial Grade 5 science program-of-study under the Weather Watch unit. To evaluate and compare the performance of Bayesian Knowledge Tracing and Dynamic Bayesian Networks, the classification consistency and accuracy are examined.
How to Cite
Cui, Y., Chu, M.-W., & Chen, F. (2019). Analyzing Student Process Data in Game-Based Assessments with Bayesian Knowledge Tracing and Dynamic Bayesian Networks. Journal of Educational Data Mining, 11(1), 80–100. https://doi.org/10.5281/zenodo.3554751
##plugins.themes.bootstrap3.article.details##
Keywords
game-based assessment, Evidence-Centered Design, Bayesian Knowledge Tracing, Dynamic Bayesian Networks, formative feedback, process data analysis
References
ALBERTA EDUCATION 1996. Science (elementary). Alberta Education. http://www.education.alberta.ca/media/654825/elemsci.pdf
ALMOND, R. G., MISLEVY, R. J., STEINBERG, L. S., WILLIAMSON, D. M., AND YAN, D. 2015. Bayesian Networks in Educational Assessment. Springer, New York.
AMERICAN EDUCATIONAL RESEARCH ASSOCIATION [AERA], AMERICAN PSYCHOLOGICAL ASSOCIATION [APA], AND NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION [NCME]. 2014. Standards for Educational and Psychological Testing. Author, Washington, DC.
BARAB, S. A., GRESALFI, M. S., AND INGRAM-GOBLE, A. 2010. Transformational play: Using games to position person, content, and context. Educ. Res. 39, 525–536.
BEHRENS, J. T., MISLEVY, R. J., DICERBO, K. E., AND LEVY, R. 2012. Evidence centered design for learning and assessment. In Technology-based Assessments for 21st Century Skills: Theoretical and Practical Implications from Modern Research, M. Mayrath, J. Clarke-Midura, D. Robinson, and G. Schraw, Eds. Information Age Publishers, Charlotte, NC, 371–386.
BENNETT, R. E., PERSKY, H., WEISS, A. R., AND JENKINS, F. 2007. Problem solving in technology-rich environments: A report from the NAEP technology-based assessment project (NCES 2007–466). U.S. Department of Education. National Center for Education Statistics, Washington, DC. http://nces.ed.gov/nationsreportcard/pubs/studies/2007466.asp
BERTLING, M., JACKSON, G., ORANJE, A., AND OWEN, V. 2015. Measuring argumentation skills with game-based assessments: Evidence for incremental validity and learning. In C. Conati, N. Heffernan, A. Mitrovic, & M. Verdejo (Eds.), Proceedings of the 17th International Conference on Artificial Intelligence in Education. Springer International, New York, NY, 545-549.
BLACK, P., AND WILIAM, D. 1998. Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 80, 139– 144.
CHU, M-W., AND CHIANG, A. 2018. Raging skies: Development of a digital game-based science assessment using evidence-centered game design. Alta. J. Sci. Educ. 45, 37–47.
CHU, M-W., ASTON, R., CUI, Y., SHOJAEE, N., AND BAWEL, B. 2018. Development of a digital gamebased assessment to measure science skill-based outcomes. Paper presented at the Annual Meeting of Canadian Society for the Study of Education, Regina, SK, Canada.
CORBETT, A. T., AND ANDERSON, J. R. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Model. User-adapt. Interact. 4, 253–278.
DICERBO, K. (2017). Building the evidentiary argument in game-based assessment. Journal of Applied Testing Technology, 18, 7-18.
ERCIKAN, K. AND PELLIGRINO, J.W. (2017). Validation of score meaning using examinee response processes for the next generation of assessments. Routledge, New York.
FORSYTH, C. M., JACKSON, G. T., HEBERT, D., LEHMAN, B., INGLESE, P. AND GRACE,L. (2017). Striking a balance: User-experience and performance in computerized game-based assessment. In R.Baker, E. Andre, X. Hu, T. Rodrigo, B. du Bouley. The Proceedings of the International Conference on Artificial Intelligence in Education, LNCS, Springer-Verlag, 502-505.
GEE, J. P. (2007). Good Videogames + Good Learning: Collected Essays on Videogames, Learning and Literacy. Peter Lang Publishing, New York.
HABGOOD, J. M.P., AND AINSWORTH, S. E. (2011). Motivating children to learn effectively: exploring the value of intrinsic integration in educational games. Journal of the Learning Sciences, 20, 2, 169-206.
HECKERMAN, D., GEIGER, D., AND CHICKERING, D. M. 1995. Learning Bayesian networks: The combination of knowledge and statistical data. Mach. Learn. 20, 197–243.
INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION 2007. The National Educational Technology Standards and Performance Indicators for Students. ISTE, Eugene, OR.
JENSEN, F. V. 1996. An Introduction to Bayesian networks. University College London Press, London.
LEPPER, M. R., AND MALONE, T. W. (1987). Intrinsic motivation and instructional effectiveness in computer-based education. In R. E. Snow & M. J. Farr (Eds.), Aptitude, Learning, and Instruction: Vol. 3. Cognitive and Affective Process Analysis. Erlbaum, Hillsdale, NJ, 255- 286.
MALONE, T. W., AND LEPPER, M. R. (1987). Making learning fun: A taxonomy of intrinsic motivations for learning. In R. E. Snow & M. J. Farr (Eds.), Aptitude, Learning, and Instruction: Vol. 3. Cognitive and Affective Process Analysis. Erlbaum, Hillsdale, NJ, 223- 253.
MIHAJLOVIC, V., AND PETKOVIC, M. 2001. Dynamic Bayesian Networkss: A State of the Art. https://ris.utwente.nl/ws/portalfiles/portal/27679465/0000006a.pdf
MILLIS, K., FORSYTH, C.M., WALLACE, P., GRAESSER, A.C., AND TIMMINS, G. 2016. The impact of game-like features on learning from an intelligent tutoring system. Technology, Knowledge and Learning, 22, 1-22.
MISLEVY, R. J. 2006. Cognitive psychology and educational assessment. In Educational Measurement (4th ed.), R. L. Brennan Ed. American Council on Education/Praeger, Westport, CT, 257–305.
MISLEVY, R. J., ALMOND, R. G., AND LUKAS, J. 2003. A brief introduction to Evidence- Centered Design (Research Report No. RR-03-16). Educational Testing Service. https://www.ets.org/Media/Research/pdf/RR-03-16.pdf
MISLEVY, R. J., ORANJE, A., BAUER, M. I., VON DAVIER, A., HAO, J., CORRIGAN, S., HOFFMAN, E., DICERBO, K., AND JOHN, M. 2014. Psychometric considerations in game-based assessment. GlassLab: Institute of play. http://www.instituteofplay.org/work/projects/glasslab-research/
MURPHY, K. P. 2002. Dynamic Bayesian Networks: Representation, Inference, and Learning. Ph.D. dissertation. University of California, Berkeley, CA.
NATIONAL RESEARCH COUNCIL. 2011. Learning Science through Computer Games and Simulations. National Academies Press, Washington, DC. OECD. 2010. PISA 2009 Results: What Students Know and Can Do – Student Performance in Reading, Mathematics and Science (Volume I). PISA, OECD Publishing. http://dx.doi.org/10.1787/9789264091450-en.
PELLEGRINO, J. W. AND QUELLMALZ, E. S. 2010. Perspectives on the integration of technology and assessment. J. Res. Tech. Educ. 43, 119–134.
QUELLMALZ, E. S., KREIKEMEIER, P., DEBARGER, A. H., AND HAERTEL, G. 2007. A study of the alignment of the NAEP, TIMSS, and New Standards Science Assessments with the inquiry abilities in the National Science Education Standards. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL.
QUELLMALZ, E. S., TIMMS, M. J., AND SCHNEIDER, S. A. 2009. Assessment of student learning in science simulations and games. National Research Council, Washington, D.C. R CORE TEAM. 2018. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria.
RIEBER, L. 1996. Seriously considering play: Designing interactive learning environments based on the blending of microworlds, simulations, and games. Education and Technology Research & Development, 44, 42-58. Doi: 10.1007/BF02300540
SHUTE, V. J., AND VENTURA, M. 2013. Measuring and supporting learning in games: Stealth assessment. Massachusetts Institute of Technology Press, Cambridge, MA. http://myweb.fsu.edu/vshute/pdf/white.pdf
STEVENS, R., BEAL, C. R., AND SPRANG, M. 2013. Assessing students’ problem solving ability and cognitive regulation with learning trajectories. In International Handbook of Metacognition and Learning Technologies, R. Azevedo and V. Aleven Eds. Springer, New York, 409–423.
ALMOND, R. G., MISLEVY, R. J., STEINBERG, L. S., WILLIAMSON, D. M., AND YAN, D. 2015. Bayesian Networks in Educational Assessment. Springer, New York.
AMERICAN EDUCATIONAL RESEARCH ASSOCIATION [AERA], AMERICAN PSYCHOLOGICAL ASSOCIATION [APA], AND NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION [NCME]. 2014. Standards for Educational and Psychological Testing. Author, Washington, DC.
BARAB, S. A., GRESALFI, M. S., AND INGRAM-GOBLE, A. 2010. Transformational play: Using games to position person, content, and context. Educ. Res. 39, 525–536.
BEHRENS, J. T., MISLEVY, R. J., DICERBO, K. E., AND LEVY, R. 2012. Evidence centered design for learning and assessment. In Technology-based Assessments for 21st Century Skills: Theoretical and Practical Implications from Modern Research, M. Mayrath, J. Clarke-Midura, D. Robinson, and G. Schraw, Eds. Information Age Publishers, Charlotte, NC, 371–386.
BENNETT, R. E., PERSKY, H., WEISS, A. R., AND JENKINS, F. 2007. Problem solving in technology-rich environments: A report from the NAEP technology-based assessment project (NCES 2007–466). U.S. Department of Education. National Center for Education Statistics, Washington, DC. http://nces.ed.gov/nationsreportcard/pubs/studies/2007466.asp
BERTLING, M., JACKSON, G., ORANJE, A., AND OWEN, V. 2015. Measuring argumentation skills with game-based assessments: Evidence for incremental validity and learning. In C. Conati, N. Heffernan, A. Mitrovic, & M. Verdejo (Eds.), Proceedings of the 17th International Conference on Artificial Intelligence in Education. Springer International, New York, NY, 545-549.
BLACK, P., AND WILIAM, D. 1998. Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 80, 139– 144.
CHU, M-W., AND CHIANG, A. 2018. Raging skies: Development of a digital game-based science assessment using evidence-centered game design. Alta. J. Sci. Educ. 45, 37–47.
CHU, M-W., ASTON, R., CUI, Y., SHOJAEE, N., AND BAWEL, B. 2018. Development of a digital gamebased assessment to measure science skill-based outcomes. Paper presented at the Annual Meeting of Canadian Society for the Study of Education, Regina, SK, Canada.
CORBETT, A. T., AND ANDERSON, J. R. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Model. User-adapt. Interact. 4, 253–278.
DICERBO, K. (2017). Building the evidentiary argument in game-based assessment. Journal of Applied Testing Technology, 18, 7-18.
ERCIKAN, K. AND PELLIGRINO, J.W. (2017). Validation of score meaning using examinee response processes for the next generation of assessments. Routledge, New York.
FORSYTH, C. M., JACKSON, G. T., HEBERT, D., LEHMAN, B., INGLESE, P. AND GRACE,L. (2017). Striking a balance: User-experience and performance in computerized game-based assessment. In R.Baker, E. Andre, X. Hu, T. Rodrigo, B. du Bouley. The Proceedings of the International Conference on Artificial Intelligence in Education, LNCS, Springer-Verlag, 502-505.
GEE, J. P. (2007). Good Videogames + Good Learning: Collected Essays on Videogames, Learning and Literacy. Peter Lang Publishing, New York.
HABGOOD, J. M.P., AND AINSWORTH, S. E. (2011). Motivating children to learn effectively: exploring the value of intrinsic integration in educational games. Journal of the Learning Sciences, 20, 2, 169-206.
HECKERMAN, D., GEIGER, D., AND CHICKERING, D. M. 1995. Learning Bayesian networks: The combination of knowledge and statistical data. Mach. Learn. 20, 197–243.
INTERNATIONAL SOCIETY FOR TECHNOLOGY IN EDUCATION 2007. The National Educational Technology Standards and Performance Indicators for Students. ISTE, Eugene, OR.
JENSEN, F. V. 1996. An Introduction to Bayesian networks. University College London Press, London.
LEPPER, M. R., AND MALONE, T. W. (1987). Intrinsic motivation and instructional effectiveness in computer-based education. In R. E. Snow & M. J. Farr (Eds.), Aptitude, Learning, and Instruction: Vol. 3. Cognitive and Affective Process Analysis. Erlbaum, Hillsdale, NJ, 255- 286.
MALONE, T. W., AND LEPPER, M. R. (1987). Making learning fun: A taxonomy of intrinsic motivations for learning. In R. E. Snow & M. J. Farr (Eds.), Aptitude, Learning, and Instruction: Vol. 3. Cognitive and Affective Process Analysis. Erlbaum, Hillsdale, NJ, 223- 253.
MIHAJLOVIC, V., AND PETKOVIC, M. 2001. Dynamic Bayesian Networkss: A State of the Art. https://ris.utwente.nl/ws/portalfiles/portal/27679465/0000006a.pdf
MILLIS, K., FORSYTH, C.M., WALLACE, P., GRAESSER, A.C., AND TIMMINS, G. 2016. The impact of game-like features on learning from an intelligent tutoring system. Technology, Knowledge and Learning, 22, 1-22.
MISLEVY, R. J. 2006. Cognitive psychology and educational assessment. In Educational Measurement (4th ed.), R. L. Brennan Ed. American Council on Education/Praeger, Westport, CT, 257–305.
MISLEVY, R. J., ALMOND, R. G., AND LUKAS, J. 2003. A brief introduction to Evidence- Centered Design (Research Report No. RR-03-16). Educational Testing Service. https://www.ets.org/Media/Research/pdf/RR-03-16.pdf
MISLEVY, R. J., ORANJE, A., BAUER, M. I., VON DAVIER, A., HAO, J., CORRIGAN, S., HOFFMAN, E., DICERBO, K., AND JOHN, M. 2014. Psychometric considerations in game-based assessment. GlassLab: Institute of play. http://www.instituteofplay.org/work/projects/glasslab-research/
MURPHY, K. P. 2002. Dynamic Bayesian Networks: Representation, Inference, and Learning. Ph.D. dissertation. University of California, Berkeley, CA.
NATIONAL RESEARCH COUNCIL. 2011. Learning Science through Computer Games and Simulations. National Academies Press, Washington, DC. OECD. 2010. PISA 2009 Results: What Students Know and Can Do – Student Performance in Reading, Mathematics and Science (Volume I). PISA, OECD Publishing. http://dx.doi.org/10.1787/9789264091450-en.
PELLEGRINO, J. W. AND QUELLMALZ, E. S. 2010. Perspectives on the integration of technology and assessment. J. Res. Tech. Educ. 43, 119–134.
QUELLMALZ, E. S., KREIKEMEIER, P., DEBARGER, A. H., AND HAERTEL, G. 2007. A study of the alignment of the NAEP, TIMSS, and New Standards Science Assessments with the inquiry abilities in the National Science Education Standards. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL.
QUELLMALZ, E. S., TIMMS, M. J., AND SCHNEIDER, S. A. 2009. Assessment of student learning in science simulations and games. National Research Council, Washington, D.C. R CORE TEAM. 2018. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria.
RIEBER, L. 1996. Seriously considering play: Designing interactive learning environments based on the blending of microworlds, simulations, and games. Education and Technology Research & Development, 44, 42-58. Doi: 10.1007/BF02300540
SHUTE, V. J., AND VENTURA, M. 2013. Measuring and supporting learning in games: Stealth assessment. Massachusetts Institute of Technology Press, Cambridge, MA. http://myweb.fsu.edu/vshute/pdf/white.pdf
STEVENS, R., BEAL, C. R., AND SPRANG, M. 2013. Assessing students’ problem solving ability and cognitive regulation with learning trajectories. In International Handbook of Metacognition and Learning Technologies, R. Azevedo and V. Aleven Eds. Springer, New York, 409–423.
Issue
Section
Articles
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors who publish with this journal agree to the following terms:
- The Author retains copyright in the Work, where the term “Work” shall include all digital objects that may result in subsequent electronic publication or distribution.
- Upon acceptance of the Work, the author shall grant to the Publisher the right of first publication of the Work.
- The Author shall grant to the Publisher and its agents the nonexclusive perpetual right and license to publish, archive, and make accessible the Work in whole or in part in all forms of media now or hereafter known under a Creative Commons 4.0 License (Attribution-Noncommercial-No Derivatives 4.0 International), or its equivalent, which, for the avoidance of doubt, allows others to copy, distribute, and transmit the Work under the following conditions:
- Attribution—other users must attribute the Work in the manner specified by the author as indicated on the journal Web site;
- Noncommercial—other users (including Publisher) may not use this Work for commercial purposes;
- No Derivative Works—other users (including Publisher) may not alter, transform, or build upon this Work,with the understanding that any of the above conditions can be waived with permission from the Author and that where the Work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.
- The Author is able to enter into separate, additional contractual arrangements for the nonexclusive distribution of the journal's published version of the Work (e.g., post it to an institutional repository or publish it in a book), as long as there is provided in the document an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post online a pre-publication manuscript (but not the Publisher’s final formatted PDF version of the Work) in institutional repositories or on their Websites prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (see The Effect of Open Access). Any such posting made before acceptance and publication of the Work shall be updated upon publication to include a reference to the Publisher-assigned DOI (Digital Object Identifier) and a link to the online abstract for the final published Work in the Journal.
- Upon Publisher’s request, the Author agrees to furnish promptly to Publisher, at the Author’s own expense, written evidence of the permissions, licenses, and consents for use of third-party material included within the Work, except as determined by Publisher to be covered by the principles of Fair Use.
- The Author represents and warrants that:
- the Work is the Author’s original work;
- the Author has not transferred, and will not transfer, exclusive rights in the Work to any third party;
- the Work is not pending review or under consideration by another publisher;
- the Work has not previously been published;
- the Work contains no misrepresentation or infringement of the Work or property of other authors or third parties; and
- the Work contains no libel, invasion of privacy, or other unlawful matter.
- The Author agrees to indemnify and hold Publisher harmless from Author’s breach of the representations and warranties contained in Paragraph 6 above, as well as any claim or proceeding relating to Publisher’s use and publication of any content contained in the Work, including third-party content.