Evidence-centered Design for Diagnostic Assessment within Digital Learning Environments: Integrating Modern Psychometrics and Educational Data Mining
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
Oct 1, 2012
André A. Rupp
Rebecca Nugent
Brian Nelson
Abstract
In recent years the educational community has increasingly embraced digital technologies for the purposes of developing alternative learning environments, providing diagnostic feedback, and fostering the development of socalled 21st -century skills. This special issue is dedicated to bridging recent work from the disciplines of educational and psychological assessment and educational data mining (EDM) via the assessment design and implementation framework of evidence-centered design (ECD). It consists of a series of five papers: one conceptual paper on ECD, three applied case studies that use ECD and EDM tools, and one simulation study that relies on ECD for its design and EDM for its implementation. In this introduction, we discuss the underlying rationales for the special issue in more detail, provide a short introduction to ECD, and describe the focus of the five selected papers.
How to Cite
Rupp, A. A., Nugent, R., & Nelson, B. (2012). Evidence-centered Design for Diagnostic Assessment within Digital Learning Environments: Integrating Modern Psychometrics and Educational Data Mining. Journal of Educational Data Mining, 4(1), 1–10. https://doi.org/10.5281/zenodo.3554639
##plugins.themes.bootstrap3.article.details##
Keywords
educational data mining, psychometrics, digital learning environments, special issue
References
APPLIED MEASUREMENT IN EDUCATION. 2010. Evidence-centered assessment design in practice (special issue), 23(4).
ALMOND, R. G., STEINBERG, L. S., & MISLEVY, R. J. 2001. A Sample Assessment Using the Four Process Framework. (CSE Tech. Rep. 543). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
BENNET, R. 2010. Cognitively based assessment as, of, and for learning (C-BAL): A preliminary theory of action for summative and formative assessment. Measurement: Interdisciplinary Research and Perspectives, 8(2-3), 70-91.
BRENNAN, R. J. 2001. Generalizability theory. New York, NY: Springer.
CROCKER, L., & ALGINA, J. 2006, 2nd ed.. Introduction to Classical and Modern Test Theory. Wadsworth, Belmont, CA.
DE AYALA, R. J. 2009. The Theory and Practice of Item Response Theory. Guilford Press, New York, NY.
DEBOECK, P., & WILSON, M. 2004. Explanatory Item Response Models: A Generalized Linear and Nonlinear Approach. Springer, New York, NY.
DICERBO, K. E. & BEHRENS, J. T. 2012. Implications of the digital ocean on current and future assessment. In R. Lissitz, Ed., Computers and their Impact on State Assessment: Recent History and Predictions for the Future (pp. 273-306). Information Age Publishing, Charlotte, NC.
FREZZO, D.C., BEHRENS, J.T., & MISLEVY, R.J. 2009. Design patterns for learning and assessment: facilitating the introduction of a complex simulation-based learning environment into a community of instructors. The Journal of Science Education and Technology. Springer Open Access http://www.springerlink.com/content/566p6g4307405346/
HUFF, K., EWING, M., HENDRICKSON, A., KALISKI, P., & PACKMAN, S. 2012, April. Applications of Evidence-centered Design (ECD) in Large-scale Assessment. Workshop given at the annual meeting of the National Council on Measurement in Education (NCME), Vancouver, BC, Canada.
LATTIN, J., CARROLL, D., & GREEN, P. 2002. Analyzing Multivariate Data. New York, NY: Duxbury Press.
LEIGHTON, J., & GIERL, M. 2007. Eds. Cognitive Diagnostic Assessment for Education: Theory and Applications. Cambridge University Press, New York, NY.
MISLEVY, R. J. 2011. Evidence-centered design for simulation based assessment (CRESST Report 800). University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST), Los Angeles, CA.
MISLEVY, R. J., STEINBERG, L. S., & ALMOND, R. G. 2003. On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3-62.
RAYKOV, T., & MARCOULIDES, G. 2011. Introduction to psychometric theory. Routledge, New York, NY.
ROMERO, C., VENTURA, S., PECHENIZKIY, M., & BAKER, R. S. J. D. 2010. Eds. Handbook of educational data mining. Chapman & Hall / CRC, New York, NY.
RUPP, A. A., TEMPLIN, J., & HENSON, R. A. 2010. Diagnostic measurement: Theory, methods, and applications. Guilford Press, New York, NY.
USER MODELING AND USER-ADAPTED INTERACTION. 2011, special issue. Data mining for personalised educational systems, 21(1-2).
VAN DER AALST, W. M. P. 2011. Process mining: Discovery, conformance and enhancement of business processes. Springer, New York, NY.
ZALLES, D., HAERTEL, G., & MISLEVY, R. 2010. Using Evidence-Centered Design to Support Assessment, Design and Validation of Learning Progressions (Large-Scale Assessment Technical Report 10). SRI International, Menlo Park, CA.
ZHOU, J. 2009. A review of assessment engineering principles with select applications to the Certified Public Accountant examination (Technical Report W0903). American Institute of Certified Public Accountants, Inc. Available online via http://www.aicpa.org/Pages/Default.aspx
ALMOND, R. G., STEINBERG, L. S., & MISLEVY, R. J. 2001. A Sample Assessment Using the Four Process Framework. (CSE Tech. Rep. 543). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
BENNET, R. 2010. Cognitively based assessment as, of, and for learning (C-BAL): A preliminary theory of action for summative and formative assessment. Measurement: Interdisciplinary Research and Perspectives, 8(2-3), 70-91.
BRENNAN, R. J. 2001. Generalizability theory. New York, NY: Springer.
CROCKER, L., & ALGINA, J. 2006, 2nd ed.. Introduction to Classical and Modern Test Theory. Wadsworth, Belmont, CA.
DE AYALA, R. J. 2009. The Theory and Practice of Item Response Theory. Guilford Press, New York, NY.
DEBOECK, P., & WILSON, M. 2004. Explanatory Item Response Models: A Generalized Linear and Nonlinear Approach. Springer, New York, NY.
DICERBO, K. E. & BEHRENS, J. T. 2012. Implications of the digital ocean on current and future assessment. In R. Lissitz, Ed., Computers and their Impact on State Assessment: Recent History and Predictions for the Future (pp. 273-306). Information Age Publishing, Charlotte, NC.
FREZZO, D.C., BEHRENS, J.T., & MISLEVY, R.J. 2009. Design patterns for learning and assessment: facilitating the introduction of a complex simulation-based learning environment into a community of instructors. The Journal of Science Education and Technology. Springer Open Access http://www.springerlink.com/content/566p6g4307405346/
HUFF, K., EWING, M., HENDRICKSON, A., KALISKI, P., & PACKMAN, S. 2012, April. Applications of Evidence-centered Design (ECD) in Large-scale Assessment. Workshop given at the annual meeting of the National Council on Measurement in Education (NCME), Vancouver, BC, Canada.
LATTIN, J., CARROLL, D., & GREEN, P. 2002. Analyzing Multivariate Data. New York, NY: Duxbury Press.
LEIGHTON, J., & GIERL, M. 2007. Eds. Cognitive Diagnostic Assessment for Education: Theory and Applications. Cambridge University Press, New York, NY.
MISLEVY, R. J. 2011. Evidence-centered design for simulation based assessment (CRESST Report 800). University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST), Los Angeles, CA.
MISLEVY, R. J., STEINBERG, L. S., & ALMOND, R. G. 2003. On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3-62.
RAYKOV, T., & MARCOULIDES, G. 2011. Introduction to psychometric theory. Routledge, New York, NY.
ROMERO, C., VENTURA, S., PECHENIZKIY, M., & BAKER, R. S. J. D. 2010. Eds. Handbook of educational data mining. Chapman & Hall / CRC, New York, NY.
RUPP, A. A., TEMPLIN, J., & HENSON, R. A. 2010. Diagnostic measurement: Theory, methods, and applications. Guilford Press, New York, NY.
USER MODELING AND USER-ADAPTED INTERACTION. 2011, special issue. Data mining for personalised educational systems, 21(1-2).
VAN DER AALST, W. M. P. 2011. Process mining: Discovery, conformance and enhancement of business processes. Springer, New York, NY.
ZALLES, D., HAERTEL, G., & MISLEVY, R. 2010. Using Evidence-Centered Design to Support Assessment, Design and Validation of Learning Progressions (Large-Scale Assessment Technical Report 10). SRI International, Menlo Park, CA.
ZHOU, J. 2009. A review of assessment engineering principles with select applications to the Certified Public Accountant examination (Technical Report W0903). American Institute of Certified Public Accountants, Inc. Available online via http://www.aicpa.org/Pages/Default.aspx
Section
Articles
Authors who publish with this journal agree to the following terms:
- The Author retains copyright in the Work, where the term “Work” shall include all digital objects that may result in subsequent electronic publication or distribution.
- Upon acceptance of the Work, the author shall grant to the Publisher the right of first publication of the Work.
- The Author shall grant to the Publisher and its agents the nonexclusive perpetual right and license to publish, archive, and make accessible the Work in whole or in part in all forms of media now or hereafter known under a Creative Commons 4.0 License (Attribution-Noncommercial-No Derivatives 4.0 International), or its equivalent, which, for the avoidance of doubt, allows others to copy, distribute, and transmit the Work under the following conditions:
- Attribution—other users must attribute the Work in the manner specified by the author as indicated on the journal Web site;
- Noncommercial—other users (including Publisher) may not use this Work for commercial purposes;
- No Derivative Works—other users (including Publisher) may not alter, transform, or build upon this Work,with the understanding that any of the above conditions can be waived with permission from the Author and that where the Work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.
- The Author is able to enter into separate, additional contractual arrangements for the nonexclusive distribution of the journal's published version of the Work (e.g., post it to an institutional repository or publish it in a book), as long as there is provided in the document an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post online a pre-publication manuscript (but not the Publisher’s final formatted PDF version of the Work) in institutional repositories or on their Websites prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (see The Effect of Open Access). Any such posting made before acceptance and publication of the Work shall be updated upon publication to include a reference to the Publisher-assigned DOI (Digital Object Identifier) and a link to the online abstract for the final published Work in the Journal.
- Upon Publisher’s request, the Author agrees to furnish promptly to Publisher, at the Author’s own expense, written evidence of the permissions, licenses, and consents for use of third-party material included within the Work, except as determined by Publisher to be covered by the principles of Fair Use.
- The Author represents and warrants that:
- the Work is the Author’s original work;
- the Author has not transferred, and will not transfer, exclusive rights in the Work to any third party;
- the Work is not pending review or under consideration by another publisher;
- the Work has not previously been published;
- the Work contains no misrepresentation or infringement of the Work or property of other authors or third parties; and
- the Work contains no libel, invasion of privacy, or other unlawful matter.
- The Author agrees to indemnify and hold Publisher harmless from Author’s breach of the representations and warranties contained in Paragraph 6 above, as well as any claim or proceeding relating to Publisher’s use and publication of any content contained in the Work, including third-party content.