Using the ECD Framework to Support Evidentiary Reasoning in the Context of a Simulation Study for Detecting Learner Differences in Epistemic Games



Published Oct 1, 2012
Shauna J. Sweet André A. Rupp


The evidence-centered design (ECD) framework is a powerful tool that supports careful and critical thinking about the identification and accumulation of evidence in assessment contexts. In this paper, we demonstrate how the ECD framework provides critical support for designing simulation studies to investigate statistical methods within an ill-defined methodological domain like games-based assessment. We discuss the design and selected findings of a complex simulation study to investigate the utility of statistics derived from a non-parametric method called epistemic network analysis; this method is used in practice for the analysis of real data from a suite of digital learning environments called epistemic games. We present findings that show that one marginal ENA statistic, the weighted density statistic from social network analysis, has some ability to detect differences among a variety of simulated learners when they play different types of simulated games. Detection strength ranged from weak to strong, depending jointly on game design characteristics and the types of learners being compared. Our work illustrates the complex challenges of how best to describe, justify, and evaluate design decisions for simulation studies in the context of games-based assessment.

How to Cite

Sweet, S. J., & Rupp, A. A. (2012). Using the ECD Framework to Support Evidentiary Reasoning in the Context of a Simulation Study for Detecting Learner Differences in Epistemic Games. Journal of Educational Data Mining, 4(1), 183–223.
Abstract 585 | PDF Downloads 297



evidence-centered design, simulation study, epistemic games, weighted density, social network analysis

BAGLEY, E., AND SHAFFER, D. W. 2009. When people get in the way: Promoting civic thinking through epistemic gameplay. International Journal of Gaming and Computer-mediated Simulations 1, 36-52.

BAGLEY, E., AND SHAFFER, D. W. 2010. Stop talking and type: Mentoring in a virtual and face-to-face environment. Manuscript under review.

BORG, I., AND GROENED, P. J. F. 2010. Modern multidimensional scaling: Theory and applications. Springer, New York, NY.

COOK, A. R., AND TEO, S. W. L. 2011. The communicability of graphical alternatives to tabular displays of statistical simulation studies. PLoS ONE 6, 1-7.

D'ANGELO, C., SHAFFER, D. W., AND HATFIELD, D. 2011. Design, implementation, and data analysis for an epistemic game using a novel variant of epistemic network analysis. Manuscript submitted for publication.

DE AYALA, R. J. 2009. Principles and Practice of Item Response Theory. Guilford Press, New York, NY.

MISLEVY, R. J., STEINBERG, L. S., ALMOND, R. G, AND LUKAS, J. F. 2006. Concepts, terminology and basic models of evidence-centered design. D. M. WILLIAMSON, R. J. MISLEVY, AND I. I. BEJAR. Eds. Automated Scoring of Complex Tasks in Computer-Based Testing, Lawrence Erlbaum Associates, Mahwah, NJ, 15–47.

NASH, P., AND SHAFFER, D. W. 2011. Mentor modelling: The internalization of modelled professional thinking in an epistemic game. Journal of Computer Assisted Learning 27, 173-189.

R DEVELOPMENT CORE TEAM. 2008. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Available online at

RECKASE, M. 2009. Multidimensional Item Response Theory. Springer, New York, NY.

RUPP, A. A., TEMPLIN, J., AND HENSON, R. J. (2010). Diagnostic Measurement: Theory, Methods, and Applications. Guilford Press, New York, NY.

RUPP, A. A., GUSHTA, M., MISLEVY, R. J., AND SHAFFER, D. W. 2010. Evidence-centered design of epistemic games: Measurement principles for complex learning environments. Journal of Technology, Learning, and Assessment 8(4). Available online at

SHAFFER, D. W. 2006a. How Computer Games Help Children Learn. Palgrave / Macmillan, New York, NY.

SHAFFER, D. W. 2006b. Epistemic frames for epistemic games. Computers and Education 46(3), 223-234.

SHAFFER, D. W., HATFIELD, D., SVAROVSKY, G. N., NASH, P., NULTY, A., BAGLEY, E., FRANKI, K., RUPP, A. A., AND MISLEVY, R. J. 2009. Epistemic network analysis: A prototype for 21st century assessment of learning. The International Journal of Learning and Media 1(2), 33-53.

TATSUOKA, K. K. 2009. Cognitive Assessment: An Introduction to the Rule-space Method. Routledge, Florence, KY