Evidence-centered Design for Diagnostic Assessment within Digital Learning Environments: Integrating Modern Psychometrics and Educational Data Mining



Published Oct 1, 2012
André A. Rupp Rebecca Nugent Brian Nelson


In recent years the educational community has increasingly embraced digital technologies for the purposes of developing alternative learning environments, providing diagnostic feedback, and fostering the development of socalled 21st -century skills. This special issue is dedicated to bridging recent work from the disciplines of educational and psychological assessment and educational data mining (EDM) via the assessment design and implementation framework of evidence-centered design (ECD). It consists of a series of five papers: one conceptual paper on ECD, three applied case studies that use ECD and EDM tools, and one simulation study that relies on ECD for its design and EDM for its implementation. In this introduction, we discuss the underlying rationales for the special issue in more detail, provide a short introduction to ECD, and describe the focus of the five selected papers.

How to Cite

Rupp, A. A., Nugent, R., & Nelson, B. (2012). Evidence-centered Design for Diagnostic Assessment within Digital Learning Environments: Integrating Modern Psychometrics and Educational Data Mining. Journal of Educational Data Mining, 4(1), 1–10. https://doi.org/10.5281/zenodo.3554639
Abstract 852 | PDF Downloads 675



educational data mining, psychometrics, digital learning environments, special issue

APPLIED MEASUREMENT IN EDUCATION. 2010. Evidence-centered assessment design in practice (special issue), 23(4).

ALMOND, R. G., STEINBERG, L. S., & MISLEVY, R. J. 2001. A Sample Assessment Using the Four Process Framework. (CSE Tech. Rep. 543). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).

BENNET, R. 2010. Cognitively based assessment as, of, and for learning (C-BAL): A preliminary theory of action for summative and formative assessment. Measurement: Interdisciplinary Research and Perspectives, 8(2-3), 70-91.

BRENNAN, R. J. 2001. Generalizability theory. New York, NY: Springer.

CROCKER, L., & ALGINA, J. 2006, 2nd ed.. Introduction to Classical and Modern Test Theory. Wadsworth, Belmont, CA.

DE AYALA, R. J. 2009. The Theory and Practice of Item Response Theory. Guilford Press, New York, NY.

DEBOECK, P., & WILSON, M. 2004. Explanatory Item Response Models: A Generalized Linear and Nonlinear Approach. Springer, New York, NY.

DICERBO, K. E. & BEHRENS, J. T. 2012. Implications of the digital ocean on current and future assessment. In R. Lissitz, Ed., Computers and their Impact on State Assessment: Recent History and Predictions for the Future (pp. 273-306). Information Age Publishing, Charlotte, NC.

FREZZO, D.C., BEHRENS, J.T., & MISLEVY, R.J. 2009. Design patterns for learning and assessment: facilitating the introduction of a complex simulation-based learning environment into a community of instructors. The Journal of Science Education and Technology. Springer Open Access http://www.springerlink.com/content/566p6g4307405346/

HUFF, K., EWING, M., HENDRICKSON, A., KALISKI, P., & PACKMAN, S. 2012, April. Applications of Evidence-centered Design (ECD) in Large-scale Assessment. Workshop given at the annual meeting of the National Council on Measurement in Education (NCME), Vancouver, BC, Canada.

LATTIN, J., CARROLL, D., & GREEN, P. 2002. Analyzing Multivariate Data. New York, NY: Duxbury Press.

LEIGHTON, J., & GIERL, M. 2007. Eds. Cognitive Diagnostic Assessment for Education: Theory and Applications. Cambridge University Press, New York, NY.

MISLEVY, R. J. 2011. Evidence-centered design for simulation based assessment (CRESST Report 800). University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST), Los Angeles, CA.

MISLEVY, R. J., STEINBERG, L. S., & ALMOND, R. G. 2003. On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3-62.

RAYKOV, T., & MARCOULIDES, G. 2011. Introduction to psychometric theory. Routledge, New York, NY.

ROMERO, C., VENTURA, S., PECHENIZKIY, M., & BAKER, R. S. J. D. 2010. Eds. Handbook of educational data mining. Chapman & Hall / CRC, New York, NY.

RUPP, A. A., TEMPLIN, J., & HENSON, R. A. 2010. Diagnostic measurement: Theory, methods, and applications. Guilford Press, New York, NY.

USER MODELING AND USER-ADAPTED INTERACTION. 2011, special issue. Data mining for personalised educational systems, 21(1-2).

VAN DER AALST, W. M. P. 2011. Process mining: Discovery, conformance and enhancement of business processes. Springer, New York, NY.

ZALLES, D., HAERTEL, G., & MISLEVY, R. 2010. Using Evidence-Centered Design to Support Assessment, Design and Validation of Learning Progressions (Large-Scale Assessment Technical Report 10). SRI International, Menlo Park, CA.

ZHOU, J. 2009. A review of assessment engineering principles with select applications to the Certified Public Accountant examination (Technical Report W0903). American Institute of Certified Public Accountants, Inc. Available online via http://www.aicpa.org/Pages/Default.aspx