Design and Discovery in Educational Assessment: Evidence-Centered Design, Psychometrics, and Educational Data Mining
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
Oct 1, 2012
Robert J. Mislevy
John T. Behrens
Kristen E. Dicerbo
Roy Levy
Abstract
Evidence-centered design (ECD) is a comprehensive framework for describing the conceptual, computational and inferential elements of educational assessment. It emphasizes the importance of articulating inferences one wants to make and the evidence needed to support those inferences. At first blush, ECD and educational data mining (EDM) might seem in conflict: structuring situations to evoke particular kinds of evidence, versus discovering meaningful patterns in available data. However, a dialectic between the two stances increases understanding and improves practice. We first introduce ECD and relate its elements to the broad range of digital inputs relevant to modern assessment. We then discuss the relation between EDM and psychometric activities in educational assessment. We illustrate points with examples from the Cisco Networking Academy, a global program in which information technology is taught through a blended program of face-to-face classroom instruction, an online curriculum, and online assessments.
How to Cite
Mislevy, R. J., Behrens, J. T., Dicerbo, K. E., & Levy, R. (2012). Design and Discovery in Educational Assessment: Evidence-Centered Design, Psychometrics, and Educational Data Mining. Journal of Educational Data Mining, 4(1), 11–48. https://doi.org/10.5281/zenodo.3554641
##plugins.themes.bootstrap3.article.details##
Keywords
evidence-centered design, educational data mining, psychometrics, games and simulations, Cisco Networking Academy
References
ALMOND, R. G., STEINBERG, L. S., and MISLEVY, R. J. 2001. A sample assessment using the four process framework. CSE Technical Report 543. The National Center for Research on Evaluation, Standards, Student Testing (CRESST), Center for Studies in Education, UCLA, Los Angeles, CA. Retrieved from http://www.cse.ucla.edu/products/reports/TECH543.pdf
ALMOND, R.G., STEINBERG, L.S., AND MISLEVY, R.J. 2002. Enhancing the design and delivery of assessment systems: A four-process architecture. Journal of Technology, Learning, and Assessment, 5. Retrieved from http://escholarship.bc.edu/ojs/index.php/jtla/article/viewFile/1671/1509
ALMOND, R.G., STEINBERG, L.S., AND MISLEVY, R.J. 2003. A framework for reusing assessment components. In New Developments in Psychometrics, H. YANAI, A. OKADA, K. SHIGEMASU, Y. KANO, AND J.J. MEULMAN, Eds. Springer, Tokyo, Japan, 281-288.
AMERSHI, S., AND CONATI, C. 2010. Automatic recognition of learner types in exploratory learning environments. In Handbook of Educational Data Mining, C. ROMERO, S. VENTURA, S. VIOLA, M. PECHENIZKIY, AND R. BAKER, Eds. Chapman and Hall/CRC, Virginia Beach, VA, 213-229.
ARROYO, I., COOPER, D. G., BURLESON, W., AND WOOLF, B. 2010. Bayesian networks and linear regression models of students’ goals, moods, and emotions. In Handbook of Educational Data Mining, C. ROMERO, S. VENTURA, S. VIOLA, M. PECHENIZKIY, AND R. BAKER, Eds. Chapman and Hall/CRC, Virginia Beach, VA, 323-338.
BECK, J. 2005 Engagement tracing: Using response times to model student disengagement. In Proceedings of the 12th International Conference on Artificial Intelligence in Education, IOS Press, Amsterdam, The Netherlands, 88-95.
BEHRENS, J. T. 1997. Principles and procedures of exploratory data analysis. Psychological Methods, 2, 131-160.
BEHRENS, J.T., COLLISON, T.A., AND DEMARK, S.F. 2005. The Seven Cs of comprehensive assessment: Lessons learned from 40 million classroom exams in the Cisco Networking Academy Program. In Online Assessment and Measurement: Case Studies in Higher Education, K-12 and Corporate, S. HOWELL AND M. HRICKO, Eds. Information Science Publishing, Hershey, PA, 229-245.
BEHRENS, J. T., DICERBO, K. E., YEM, N., LEVY, R. in press. Exploratory data analysis. In Handbook of Psychology, 2nd ed., Volume II: Research Methods in Psychology, W. F. Velicer and I. Winer Eds. Wiley and Sons, New York, NY.
BEHRENS, J.T., FREZZO, D.C., MISLEVY, R.J., KROOPNICK, M., AND WISE, D. 2008. Structural, Functional and Semiotic Symmetries in Simulation-Based Games and Assessments. In Assessment of Problem Solving Using Simulations, E.L. BAKER, J. DICKIESON, W. WULFECK, AND H.F. O'NEIL, Eds. Erlbaum, New York, NY, 59-80.
BEHRENS, J.T., MISLEVY, R.J., DICERBO, K.E., AND LEVY, R. 2012. Evidence centered design for learning and assessment in the digital world. In Technology-Based Assessments for 21st Century Skills: Theoretical and Practical Implications from Modern Research, M. MAYRATH, J. CLARKE-MIDURA, AND D. H. ROBINSON, Eds. Information Age Publishing, Charlotte, NC, 13-54.
BEHRENS, J. T., AND SMITH, M. L. 1996. Data and data analysis. In Handbook of Educational Psychology, D. C. BERLINER AND R. C. CALFEE, Eds. MacMillan, New York, NY, 945–989.
BEJAR, I.I. 2010. Can Speech Technology Improve Assessment and Learning? New Capabilities May Facilitate Assessment Innovations. RDC-15. Princeton: Educational Testing Service. Available online at http://www.ets.org/research/policy_research_reports/rdc-15 .
BERNSTEIN, J. 1999. PhonePass Testing: Structure and Construct. Ordinate Corporation, Menlo Park, CA.
BLEICHER, E., HAWORTH, G. M., AND VAN DER HEIJDEN, H. M. J. F. 2010. Data-mining chess databases. ICGA Journal, 33 (4), 212-214.
CHUNG, G. K. W. K., NAGASHIMA, S. O., DELACRUZ, G. C., LEE, J. J., WAINESS, R., AND BAKER, E. L. 2011. Review of Rifle Marksmanship Training Research. CRESST Research Report. Los Angeles: The National Center for Research on Evaluation, Standards, Student Testing (CRESST), Center for Studies in Education, UCLA. Retrieved from http://www.cse.ucla.edu/products/reports/R783.pdf
CRONBACH, L.J. 1980. Validity on parole: How can we go straight? New directions for testing and measurement: Measuring achievement over a decade. In Proceedings of the 1979 ETS Invitational Conference, Jossey-Bass, San Francisco, CA, 99-108.
CRONBACH, L.J. 1988. Five perspectives on validity argument. In Test Validity, H. Wainer and H. Braun, Eds. Lawrence Erlbaum, Hillsdale, NJ, 3-17.
CRONBACH, L.J., GLESER, G.C., NANDA, H., AND RAJARATNAM, N. 1972. The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. John Wiley, New York, NY.
DE AYALA, R.J. 2009. The Theory and Practice of Item Response Theory. Guilford Press, New York, NY.
DEMARK, S. F. AND BEHRENS, J. T. 2004. Using statistical natural language processing for understanding complex responses to free-response tasks. International Journal of Testing, 4, 371-390.
DICERBO, K. E. 2007. Knowledge structures of entering networking students and their instructors. Journal of Information Technology Education, 6, 263-277. Retrieved from http://www.jite.org/documents/Vol6/JITEv6p263-277DiCerbo252.pdf
DICERBO, K. E. 2009. Communicating with instructors about complex data analysis. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA. Retrieved from https://research.netacad.net/mod/data/view.php?d=1&rid=29
DICERBO, K. E. AND BEHRENS, J. T. 2012. Implications of the digital ocean on current and future assessment. In Computers and Their Impact on State Assessment: Recent History and Predictions for the Future, R. LISSITZ AND H. JIAO, Eds. Information Age Publishing, Charlotte, NC, 273-306.
EMBRETSON, S. 1983. Construct validity: Construct representation versus nomothetic span. Psychological Bulletin, 93, 179-197.
EMBRETSON, S.E. (Ed.) 1985. Test Design: Developments in Psychology and Psychometrics. Academic Press, Orlando, FL.
FREZZO, D.C., BEHRENS, J.T., AND MISLEVY, R.J. 2009. Design patterns for learning and assessment: facilitating the introduction of a complex simulation-based learning environment into a community of instructors. The Journal of Science Education and Technology. Springer Open Access http://www.springerlink.com/content/566p6g4307405346/
FREZZO, D.C., BEHRENS, J.T, MISLEVY, R.J., WEST, P., AND DICERBO, K.E. 2009. Psychometric and evidentiary approaches to simulation assessment in Packet Tracer software. In ICNS '09: Proceedings of the Fifth International Conference on Networking and Services, IEEE Computer Society, Washington, D.C., 555 – 560.
GARCIA, G., ROMERO, C., VENTURA, S, DE CASTRO, C., AND CALDERS, T. 2010. Association rule mining in learning management systems. In Handbook of Educational Data Mining, C. Romero, S. Ventura, S. Viola, M. Pechenizkiy, and R. Baker, Eds. Chapman and Hall/CRC, Virginia Beach, VA, 93-106.
GITOMER, D. H., AND YAMAMOTO, K. 1991. Performance modeling that integrates latent trait and class theory. Journal of Educational Measurement, 28, 173–189.
GULLIKSEN, H. 1961. Measurement of learning and mental abilities. Psychometrika, 26, 93-107.
HAMBLETON, R., AND SWAMINATHAN, H. 1985. Item Response Theory: Principles and Applications. Kluwer Nijhoff Publishing, Boston, MA.
HERSHKOVITZ, A., AND NACHMIAS, R. 2010. Log-based assessment of motivation in online learning. In Handbook of Educational Data Mining, C. ROMERO, S. VENTURA, S. VIOLA, M. PECHENIZKIY, AND R. BAKER, Eds. Chapman and Hall/CRC, Virginia Beach, VA, 287-297.
HURST, K., CASILLAS, A., AND STEVENS, R. 1997. Exploring the Dynamics of Complex Problem-solving with Artificial Neural Network-based Assessment Systems. (CRESST Report 444). National Center for Research on Evaluation, Standards and Student Testing (CRESST), Los Angeles, CA. Retrieved from http://www.cse.ucla.edu/products/reports/TECH444.pdf
ISELI, M.R., KOENIG, A.D., LEE, J.J., AND WAINESS, R. 2010. Automatic Assessment of Complex Task Performance in Games and Simulations. CRESST Technical Report 775. University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST), Los Angeles, CA. Retrieved from http://www.cse.ucla.edu/products/reports/R775.pdf
JENSEN, F.V. 1996. An Introduction to Bayesian Networks. Springer, New York, NY.
JUNKER, B. 2011. The role of nonparametric analysis in assessment modeling: Then and now. In Looking Back: Proceedings of a Conference in Honor of Paul W. Holland, N. J. Dorans and S. Sinharay Eds. Springer, New York, NY, 67-85.
KANE, M.T. 1992. An argument-based approach to validation. Psychological Bulletin, 112, 527-535.
KANE, M.T. 2006. Validation. In Educational Measurement, 4th ed., R. L. Brennan, Ed., American Council on Education and Praeger Publishers, Westport, CT, 17-64.
KERR, D., AND CHUNG, G. K. W. K. this issue. Using cluster analysis to identify key features of student performance in educational video games and simulations. Journal of Educational Data Mining.
LEIGHTON, J. P., AND GIERL, M. J. (Eds.) 2007. Cognitive Diagnostic Assessment for Education: Theory and Practices. Cambridge University Press, Cambridge, UK.
LEVY, R., AND MISLEVY, R.J. 2004. Specifying and refining a measurement model for a simulation-based assessment. International Journal of Testing, 4, 333-369.
LEVY, F., AND MURNANE, R.J. 2004. The New Division of Labor: How Computers are Creating the Next Job Market. Princeton University Press, Princeton, NJ.
LEWIS, C. 1986. Test theory and Psychometrika: The past twenty-five years. Psychometrika, 51, 11-22.
LIU, M., AND HAERTEL, G. 2011. Design Patterns: A Tool to Support Assessment Task Authoring. Large-Scale Assessment Technical Report 11. Menlo Park, CA: SRI International. Retrieved from http://ecd.sri.com/downloads/ECD_TR11_DP_Supporting_Task_Authoring.pdf
LORD, F.M. 1980. Applications of Item Response Theory to Practical Testing Problems. Erlbaum, Hillsdale, NJ.
MANNING, C.D. AND SCHÜTZE, H. 1999. Foundations of Statistical Natural Language Processing. MIT Press, Cambridge, MA.
MAYRATH, M., CLARKE-MIDURA, J. AND ROBINSON, D.H. 2012. TechnologyBased Assessments for 21st Century skills: Theoretical and Practical Implications from Modern Research. Information Age Publishing, Charlotte, NC.
MESSICK, S. 1989. Validity. In Educational Measurement, 3rd ed., R. Linn, Ed., American Council on Education, Washington, D.C., 13-103.
MESSICK, S. 1994. The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), 13-23.
MISLEVY, R.J. 1994. Evidence and inference in educational assessment. Psychometrika, 59, 439-483.
MISLEVY, R.J. 2006. Cognitive psychology and educational assessment. In Educational Measurement, 4th ed., R.L. Brennan, Ed. Greenwood, Phoenix, AZ, 257-305.
MISLEVY, R.J. 2011. Evidence-centered Design for Simulation-based Assessment. Military Medicine. CRESST Report 800. University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST), Los Angeles, CA. Retrieved from http://www.cse.ucla.edu/products/reports/R800.pdf
MISLEVY, R.J., BEHRENS, J.T., BENNETT, R.E., DEMARK, S.F., FREZZO, D.C., LEVY, R., ROBINSON, D. H., RUTSTEIN, D.W., SHUTE, V.J., STANLEY, K., AND WINTERS, F.I. 2010. On the roles of external knowledge representations in assessment design. The Journal of Technology, Learning, and Assessment, 8(2). Retrieved from http://ejournals.bc.edu/ojs/index.php/jtla/article/view/1621
MISLEVY, R.J. BEHRENS, J.T., DICERBO, K.E., FREZZO, D.C., AND WEST, P. in press. Three things game designers need to know about assessment. In Assessment in Game-Based Learning: Foundations, Innovations, and Perspectives, D. IFENTHALER, D. ESERYEL, AND X. GE, Eds. Springer, New York, NY.
MISLEVY, R.J., AND GITOMER, D.H. 1996. The role of probability-based inference in an intelligent tutoring system. User-Modeling and User-Adapted Interaction, 5, 253-282.
MISLEVY, R.J. AND RICONSCENTE, M.M. 2005. Evidence-Centered Assessment Design: Layers, Structures, and Terminology. PADI Technical Report. Retrieved from http://padi.sri.com/downloads/TR9_ECD.pdf
MISLEVY, R.J., AND RICONSCENTE, M.M. 2006. Evidence-centered assessment design: Layers, concepts, and terminology. In Handbook of Test Development, S. Downing and T. Haladyna, Eds. Erlbaum, Mahwah, NJ, 61-90.
MISLEVY, R.J., RICONSCENTE, M.M., AND RUTSTEIN, D.W. 2009. Design Patterns for Assessing Model Based Reasoning. PADI-Large Systems Technical Report 6. Menlo Park, CA: SRI International. Retrieved from http://ecd.sri.com/downloads/ECD_TR6_Model-Based_Reasoning.pdf
MISLEVY, R.J., STEINBERG, L.S., AND ALMOND, R.G. 1999. On the Roles of Task Model Variables in Assessment Design. CSE Technical Report 500. The National Center for Research on Evaluation, Standards, Student Testing (CRESST), Center for Studies in Education, UCLA, Los Angeles, CA. Retrieved from http://www.cse.ucla.edu/products/reports/TECH500.pdf
MISLEVY, R.J., STEINBERG, L.S., AND ALMOND, R.G. 2003. On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3-62.
MOSTELLER, F., AND TUKEY, J.W. 1977. Data Analysis and Regression: A Second course in Statistics, 1st ed., Addison Wesley, New York, NY.
MURNANE, R.J., SHARKEY, N.S., AND LEVY, F. 2002. A role for the internet in America education? Lessons from the Cisco Networking Academies. In The Knowledge Economy and Postsecondary Education, P. Graham and N. Stacey, Eds. National Academy Press , Washington, DC, 127-157.
PARDOS, Z. A., HEFFERNAN, N. T., ANDERSON, B. S., AND HEFFERNAN, C. L. (2010). Using fine-grained skill models to fit student performance with Bayesian networks. In Handbook of Educational Data Mining, C. ROMERO, S. VENTURA, S. VIOLA, M. PECHENIZKIY, AND R. BAKER, Eds. Chapman and Hall/CRC, Virginia Beach, VA, 417-426.
PEARL, J. 1988. Probabilistic Reasoning in Intelligent Systems. Kaufmann, San Mateo, CA.
PELLEGRINO, J., CHUDOWSKY, N., AND GLASER, R. (Eds.). 2001. Knowing What Students Know: The Science and Design of Educational Assessment. National Research Council’s Committee on the Foundations of Assessment. National Academy Press, Washington, DC.
PIRRONE, R., COSSENTINO, M., PILATO, G., AND RIZZO, R. 2003. Concept maps and course ontology: A multi-level approach to e-learning. In Proceedings of the 8th AI*IA Workshop on Artificial Intelligence and E-learning, Pisa, Italy.
RAMSAY, J.O. 1982. When the data are functions. Psychometrika, 47, 379-396.
RECKASE, M.D. 2009. Multidimensional Item Response Theory. Springer, New York, NY.
ROMERO, C., VENTURA, S., PECHENIZKIY, M., AND BAKER, R.S.J.D. (Eds.). 2011. Handbook of Educational Data Mining. CRC Press, Boca Raton, FL.
RUMBAUGH, J., BLAHA, M., PREMERLANI, W., EDDY, F., AND LORENSEN, W. 1991. Object-Oriented Modeling and Design. Prentice-Hall, Englewood Cliffs, NJ.
GINSBERG, M. 1987. Readings in Nonmonotonic Reasoning. Morgan Kaufmann, Los Altos, CA.
RICONSCENTE, M., MISLEVY, R.J., AND HAMEL, L. 2005. An Introduction to PADI Task Templates. PADI Technical Report 3. SRI International, Menlo Park, CA. Retrieved from http://padi.sri.com/downloads/TR3_Templates.pdf
RUPP, A.A. 2002. Feature selection for choosing and assembling measurement models: A building-block-based organization. International Journal of Testing, 2, 311-360.
RUPP, A.A., DICERBO, K.E., LEVY, R., BENSON, M., SWEET, S., CRAWFORD, A.V., FAY, D., KUNZE, K. L., CALIÇO, T., AND BEHRENS, J.T. this issue. Putting ECD into practice: The interplay of theory and data in evidence identification and accumulation within a digital learning environment. Journal of Educational Data Mining.
RUPP, A.A., GUSHTA, M., MISLEVY, R.J., AND SHAFFER, D.W. (2010). Evidencecentered design of epistemic games: Measurement principles for complex learning environments. The Journal of Technology, Learning, and Assessment, 8(4). Retrieved from http://escholarship.bc.edu/jtla/vol8/4
RUPP, A.A., TEMPLIN, J., AND HENSON, R.J. (2010). Diagnostic Measurement: Theory, Methods, and Applications. Guilford Press, New York, NY.
SHAFFER, D.W. 2006. Epistemic frames for epistemic games. Computers & Education, 46, 223-234.
SHUTE, V. J. 2011. Stealth assessment in computer-based games to support learning. In Computer Games and Instruction, S. TOBIAS AND J.D. FLETCHER, Eds. Information Age Publishers, Charlotte, NC, 503-523.
STEINBERG, L.S., MISLEVY, R.J., AND ALMOND R.G. 2005. Portal Assessment Design System for Educational Testing. U.S. Patent 434350000, August 4, 2005.
TANNENBAUM, A.S. 2003. Computer Networks. Pearson Education, Upper Saddle River, NJ.
TATSUOKA, K.K. 1983. Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345-354.
THEODORIDIS, S. AND KOUTROUMBAS, K. 1999. Pattern Recognition. Academic Press, San Diego, CA.
TUKEY, J.W. 1977. Exploratory Data Analysis. Adison Wesley, Reading, MA.
TRCKA, N., PECHENIZKIY, M., AND VAN DER AALST, W. 2010. Process mining from educational data. In Handbook of Educational Data Mining, C. ROMERO, S. VENTURA, S. VIOLA, M. PECHENIZKIY, AND R. BAKER, Eds. Chapman and Hall/CRC, Virginia Beach, VA, 123-142.
VAN DER LINDEN, W.J., AND GLAS, C.A.W. (Eds.) 2010. Elements of Adaptive Testing. Springer, New York, NY.
VANLEHN, K. 2008. Intelligent tutoring systems for continuous, embedded assessment. In The Future of Assessment: Shaping Teaching and Learning, C.A. Dwyer, Ed. Erlbaum, New York, NY, 113-138.
WAINER, H., DORANS, N.J., FLAUGHER, R., GREEN, B.F., AND MISLEVY, R.J. 2000. Computerized Adaptive Testing: A Primer. Routledge, New York, NY.
WEST, P., RUTSTEIN, D.W., MISLEVY, R.J., LIU, J., LEVY, R., DICERBO, K.E., CRAWFORD, A., CHOI, Y., CHAPPEL, K., AND BEHRENS, J.T. 2010. A Bayesian Network Approach to Modeling Learning Progressions. CRESST Research Report. The National Center for Research on Evaluation, Standards, Student Testing (CRESST), Center for Studies in Education, UCLA, Los Angeles, CA. Retrieved from http://www.cse.ucla.edu/products/download_report.asp?r=776
WILLIAMSON, D. M., BAUER, M., STEINBERG, L. S., MISLEVY, R. J., BEHRENS, J. T., AND DEMARK, S. 2004. Design rationale for a complex performance assessment. International Journal of Measurement, 4, 333–369.
WILLIAMSON, D.M., MISLEVY, R.J., AND BEJAR, I.I. (Eds.). 2006. Automated Scoring of Complex Performances in Computer Based Testing. Erlbaum Associates, Mahwah, NJ.
WITTEN, I.H., AND FRANK, E. 2005. Data Mining: Practical Machine Learning Tools and Techniques, 2nd ed. Morgan Kaufmann, San Francisco, CA.
ZHOU, M., XU, Y., NESBIT, J.C., AND WINNE, P.H. 2010. Sequential pattern analysis of learning logs: Methodology and applications. In Handbook of Educational Data Mining, C. ROMERO, S. VENTURA, S. VIOLA, M. PECHENIZKIY, AND R. BAKER, Eds. Chapman and Hall/CRC, Virginia Beach, VA, 107-121.
ALMOND, R.G., STEINBERG, L.S., AND MISLEVY, R.J. 2002. Enhancing the design and delivery of assessment systems: A four-process architecture. Journal of Technology, Learning, and Assessment, 5. Retrieved from http://escholarship.bc.edu/ojs/index.php/jtla/article/viewFile/1671/1509
ALMOND, R.G., STEINBERG, L.S., AND MISLEVY, R.J. 2003. A framework for reusing assessment components. In New Developments in Psychometrics, H. YANAI, A. OKADA, K. SHIGEMASU, Y. KANO, AND J.J. MEULMAN, Eds. Springer, Tokyo, Japan, 281-288.
AMERSHI, S., AND CONATI, C. 2010. Automatic recognition of learner types in exploratory learning environments. In Handbook of Educational Data Mining, C. ROMERO, S. VENTURA, S. VIOLA, M. PECHENIZKIY, AND R. BAKER, Eds. Chapman and Hall/CRC, Virginia Beach, VA, 213-229.
ARROYO, I., COOPER, D. G., BURLESON, W., AND WOOLF, B. 2010. Bayesian networks and linear regression models of students’ goals, moods, and emotions. In Handbook of Educational Data Mining, C. ROMERO, S. VENTURA, S. VIOLA, M. PECHENIZKIY, AND R. BAKER, Eds. Chapman and Hall/CRC, Virginia Beach, VA, 323-338.
BECK, J. 2005 Engagement tracing: Using response times to model student disengagement. In Proceedings of the 12th International Conference on Artificial Intelligence in Education, IOS Press, Amsterdam, The Netherlands, 88-95.
BEHRENS, J. T. 1997. Principles and procedures of exploratory data analysis. Psychological Methods, 2, 131-160.
BEHRENS, J.T., COLLISON, T.A., AND DEMARK, S.F. 2005. The Seven Cs of comprehensive assessment: Lessons learned from 40 million classroom exams in the Cisco Networking Academy Program. In Online Assessment and Measurement: Case Studies in Higher Education, K-12 and Corporate, S. HOWELL AND M. HRICKO, Eds. Information Science Publishing, Hershey, PA, 229-245.
BEHRENS, J. T., DICERBO, K. E., YEM, N., LEVY, R. in press. Exploratory data analysis. In Handbook of Psychology, 2nd ed., Volume II: Research Methods in Psychology, W. F. Velicer and I. Winer Eds. Wiley and Sons, New York, NY.
BEHRENS, J.T., FREZZO, D.C., MISLEVY, R.J., KROOPNICK, M., AND WISE, D. 2008. Structural, Functional and Semiotic Symmetries in Simulation-Based Games and Assessments. In Assessment of Problem Solving Using Simulations, E.L. BAKER, J. DICKIESON, W. WULFECK, AND H.F. O'NEIL, Eds. Erlbaum, New York, NY, 59-80.
BEHRENS, J.T., MISLEVY, R.J., DICERBO, K.E., AND LEVY, R. 2012. Evidence centered design for learning and assessment in the digital world. In Technology-Based Assessments for 21st Century Skills: Theoretical and Practical Implications from Modern Research, M. MAYRATH, J. CLARKE-MIDURA, AND D. H. ROBINSON, Eds. Information Age Publishing, Charlotte, NC, 13-54.
BEHRENS, J. T., AND SMITH, M. L. 1996. Data and data analysis. In Handbook of Educational Psychology, D. C. BERLINER AND R. C. CALFEE, Eds. MacMillan, New York, NY, 945–989.
BEJAR, I.I. 2010. Can Speech Technology Improve Assessment and Learning? New Capabilities May Facilitate Assessment Innovations. RDC-15. Princeton: Educational Testing Service. Available online at http://www.ets.org/research/policy_research_reports/rdc-15 .
BERNSTEIN, J. 1999. PhonePass Testing: Structure and Construct. Ordinate Corporation, Menlo Park, CA.
BLEICHER, E., HAWORTH, G. M., AND VAN DER HEIJDEN, H. M. J. F. 2010. Data-mining chess databases. ICGA Journal, 33 (4), 212-214.
CHUNG, G. K. W. K., NAGASHIMA, S. O., DELACRUZ, G. C., LEE, J. J., WAINESS, R., AND BAKER, E. L. 2011. Review of Rifle Marksmanship Training Research. CRESST Research Report. Los Angeles: The National Center for Research on Evaluation, Standards, Student Testing (CRESST), Center for Studies in Education, UCLA. Retrieved from http://www.cse.ucla.edu/products/reports/R783.pdf
CRONBACH, L.J. 1980. Validity on parole: How can we go straight? New directions for testing and measurement: Measuring achievement over a decade. In Proceedings of the 1979 ETS Invitational Conference, Jossey-Bass, San Francisco, CA, 99-108.
CRONBACH, L.J. 1988. Five perspectives on validity argument. In Test Validity, H. Wainer and H. Braun, Eds. Lawrence Erlbaum, Hillsdale, NJ, 3-17.
CRONBACH, L.J., GLESER, G.C., NANDA, H., AND RAJARATNAM, N. 1972. The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. John Wiley, New York, NY.
DE AYALA, R.J. 2009. The Theory and Practice of Item Response Theory. Guilford Press, New York, NY.
DEMARK, S. F. AND BEHRENS, J. T. 2004. Using statistical natural language processing for understanding complex responses to free-response tasks. International Journal of Testing, 4, 371-390.
DICERBO, K. E. 2007. Knowledge structures of entering networking students and their instructors. Journal of Information Technology Education, 6, 263-277. Retrieved from http://www.jite.org/documents/Vol6/JITEv6p263-277DiCerbo252.pdf
DICERBO, K. E. 2009. Communicating with instructors about complex data analysis. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA. Retrieved from https://research.netacad.net/mod/data/view.php?d=1&rid=29
DICERBO, K. E. AND BEHRENS, J. T. 2012. Implications of the digital ocean on current and future assessment. In Computers and Their Impact on State Assessment: Recent History and Predictions for the Future, R. LISSITZ AND H. JIAO, Eds. Information Age Publishing, Charlotte, NC, 273-306.
EMBRETSON, S. 1983. Construct validity: Construct representation versus nomothetic span. Psychological Bulletin, 93, 179-197.
EMBRETSON, S.E. (Ed.) 1985. Test Design: Developments in Psychology and Psychometrics. Academic Press, Orlando, FL.
FREZZO, D.C., BEHRENS, J.T., AND MISLEVY, R.J. 2009. Design patterns for learning and assessment: facilitating the introduction of a complex simulation-based learning environment into a community of instructors. The Journal of Science Education and Technology. Springer Open Access http://www.springerlink.com/content/566p6g4307405346/
FREZZO, D.C., BEHRENS, J.T, MISLEVY, R.J., WEST, P., AND DICERBO, K.E. 2009. Psychometric and evidentiary approaches to simulation assessment in Packet Tracer software. In ICNS '09: Proceedings of the Fifth International Conference on Networking and Services, IEEE Computer Society, Washington, D.C., 555 – 560.
GARCIA, G., ROMERO, C., VENTURA, S, DE CASTRO, C., AND CALDERS, T. 2010. Association rule mining in learning management systems. In Handbook of Educational Data Mining, C. Romero, S. Ventura, S. Viola, M. Pechenizkiy, and R. Baker, Eds. Chapman and Hall/CRC, Virginia Beach, VA, 93-106.
GITOMER, D. H., AND YAMAMOTO, K. 1991. Performance modeling that integrates latent trait and class theory. Journal of Educational Measurement, 28, 173–189.
GULLIKSEN, H. 1961. Measurement of learning and mental abilities. Psychometrika, 26, 93-107.
HAMBLETON, R., AND SWAMINATHAN, H. 1985. Item Response Theory: Principles and Applications. Kluwer Nijhoff Publishing, Boston, MA.
HERSHKOVITZ, A., AND NACHMIAS, R. 2010. Log-based assessment of motivation in online learning. In Handbook of Educational Data Mining, C. ROMERO, S. VENTURA, S. VIOLA, M. PECHENIZKIY, AND R. BAKER, Eds. Chapman and Hall/CRC, Virginia Beach, VA, 287-297.
HURST, K., CASILLAS, A., AND STEVENS, R. 1997. Exploring the Dynamics of Complex Problem-solving with Artificial Neural Network-based Assessment Systems. (CRESST Report 444). National Center for Research on Evaluation, Standards and Student Testing (CRESST), Los Angeles, CA. Retrieved from http://www.cse.ucla.edu/products/reports/TECH444.pdf
ISELI, M.R., KOENIG, A.D., LEE, J.J., AND WAINESS, R. 2010. Automatic Assessment of Complex Task Performance in Games and Simulations. CRESST Technical Report 775. University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST), Los Angeles, CA. Retrieved from http://www.cse.ucla.edu/products/reports/R775.pdf
JENSEN, F.V. 1996. An Introduction to Bayesian Networks. Springer, New York, NY.
JUNKER, B. 2011. The role of nonparametric analysis in assessment modeling: Then and now. In Looking Back: Proceedings of a Conference in Honor of Paul W. Holland, N. J. Dorans and S. Sinharay Eds. Springer, New York, NY, 67-85.
KANE, M.T. 1992. An argument-based approach to validation. Psychological Bulletin, 112, 527-535.
KANE, M.T. 2006. Validation. In Educational Measurement, 4th ed., R. L. Brennan, Ed., American Council on Education and Praeger Publishers, Westport, CT, 17-64.
KERR, D., AND CHUNG, G. K. W. K. this issue. Using cluster analysis to identify key features of student performance in educational video games and simulations. Journal of Educational Data Mining.
LEIGHTON, J. P., AND GIERL, M. J. (Eds.) 2007. Cognitive Diagnostic Assessment for Education: Theory and Practices. Cambridge University Press, Cambridge, UK.
LEVY, R., AND MISLEVY, R.J. 2004. Specifying and refining a measurement model for a simulation-based assessment. International Journal of Testing, 4, 333-369.
LEVY, F., AND MURNANE, R.J. 2004. The New Division of Labor: How Computers are Creating the Next Job Market. Princeton University Press, Princeton, NJ.
LEWIS, C. 1986. Test theory and Psychometrika: The past twenty-five years. Psychometrika, 51, 11-22.
LIU, M., AND HAERTEL, G. 2011. Design Patterns: A Tool to Support Assessment Task Authoring. Large-Scale Assessment Technical Report 11. Menlo Park, CA: SRI International. Retrieved from http://ecd.sri.com/downloads/ECD_TR11_DP_Supporting_Task_Authoring.pdf
LORD, F.M. 1980. Applications of Item Response Theory to Practical Testing Problems. Erlbaum, Hillsdale, NJ.
MANNING, C.D. AND SCHÜTZE, H. 1999. Foundations of Statistical Natural Language Processing. MIT Press, Cambridge, MA.
MAYRATH, M., CLARKE-MIDURA, J. AND ROBINSON, D.H. 2012. TechnologyBased Assessments for 21st Century skills: Theoretical and Practical Implications from Modern Research. Information Age Publishing, Charlotte, NC.
MESSICK, S. 1989. Validity. In Educational Measurement, 3rd ed., R. Linn, Ed., American Council on Education, Washington, D.C., 13-103.
MESSICK, S. 1994. The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), 13-23.
MISLEVY, R.J. 1994. Evidence and inference in educational assessment. Psychometrika, 59, 439-483.
MISLEVY, R.J. 2006. Cognitive psychology and educational assessment. In Educational Measurement, 4th ed., R.L. Brennan, Ed. Greenwood, Phoenix, AZ, 257-305.
MISLEVY, R.J. 2011. Evidence-centered Design for Simulation-based Assessment. Military Medicine. CRESST Report 800. University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST), Los Angeles, CA. Retrieved from http://www.cse.ucla.edu/products/reports/R800.pdf
MISLEVY, R.J., BEHRENS, J.T., BENNETT, R.E., DEMARK, S.F., FREZZO, D.C., LEVY, R., ROBINSON, D. H., RUTSTEIN, D.W., SHUTE, V.J., STANLEY, K., AND WINTERS, F.I. 2010. On the roles of external knowledge representations in assessment design. The Journal of Technology, Learning, and Assessment, 8(2). Retrieved from http://ejournals.bc.edu/ojs/index.php/jtla/article/view/1621
MISLEVY, R.J. BEHRENS, J.T., DICERBO, K.E., FREZZO, D.C., AND WEST, P. in press. Three things game designers need to know about assessment. In Assessment in Game-Based Learning: Foundations, Innovations, and Perspectives, D. IFENTHALER, D. ESERYEL, AND X. GE, Eds. Springer, New York, NY.
MISLEVY, R.J., AND GITOMER, D.H. 1996. The role of probability-based inference in an intelligent tutoring system. User-Modeling and User-Adapted Interaction, 5, 253-282.
MISLEVY, R.J. AND RICONSCENTE, M.M. 2005. Evidence-Centered Assessment Design: Layers, Structures, and Terminology. PADI Technical Report. Retrieved from http://padi.sri.com/downloads/TR9_ECD.pdf
MISLEVY, R.J., AND RICONSCENTE, M.M. 2006. Evidence-centered assessment design: Layers, concepts, and terminology. In Handbook of Test Development, S. Downing and T. Haladyna, Eds. Erlbaum, Mahwah, NJ, 61-90.
MISLEVY, R.J., RICONSCENTE, M.M., AND RUTSTEIN, D.W. 2009. Design Patterns for Assessing Model Based Reasoning. PADI-Large Systems Technical Report 6. Menlo Park, CA: SRI International. Retrieved from http://ecd.sri.com/downloads/ECD_TR6_Model-Based_Reasoning.pdf
MISLEVY, R.J., STEINBERG, L.S., AND ALMOND, R.G. 1999. On the Roles of Task Model Variables in Assessment Design. CSE Technical Report 500. The National Center for Research on Evaluation, Standards, Student Testing (CRESST), Center for Studies in Education, UCLA, Los Angeles, CA. Retrieved from http://www.cse.ucla.edu/products/reports/TECH500.pdf
MISLEVY, R.J., STEINBERG, L.S., AND ALMOND, R.G. 2003. On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3-62.
MOSTELLER, F., AND TUKEY, J.W. 1977. Data Analysis and Regression: A Second course in Statistics, 1st ed., Addison Wesley, New York, NY.
MURNANE, R.J., SHARKEY, N.S., AND LEVY, F. 2002. A role for the internet in America education? Lessons from the Cisco Networking Academies. In The Knowledge Economy and Postsecondary Education, P. Graham and N. Stacey, Eds. National Academy Press , Washington, DC, 127-157.
PARDOS, Z. A., HEFFERNAN, N. T., ANDERSON, B. S., AND HEFFERNAN, C. L. (2010). Using fine-grained skill models to fit student performance with Bayesian networks. In Handbook of Educational Data Mining, C. ROMERO, S. VENTURA, S. VIOLA, M. PECHENIZKIY, AND R. BAKER, Eds. Chapman and Hall/CRC, Virginia Beach, VA, 417-426.
PEARL, J. 1988. Probabilistic Reasoning in Intelligent Systems. Kaufmann, San Mateo, CA.
PELLEGRINO, J., CHUDOWSKY, N., AND GLASER, R. (Eds.). 2001. Knowing What Students Know: The Science and Design of Educational Assessment. National Research Council’s Committee on the Foundations of Assessment. National Academy Press, Washington, DC.
PIRRONE, R., COSSENTINO, M., PILATO, G., AND RIZZO, R. 2003. Concept maps and course ontology: A multi-level approach to e-learning. In Proceedings of the 8th AI*IA Workshop on Artificial Intelligence and E-learning, Pisa, Italy.
RAMSAY, J.O. 1982. When the data are functions. Psychometrika, 47, 379-396.
RECKASE, M.D. 2009. Multidimensional Item Response Theory. Springer, New York, NY.
ROMERO, C., VENTURA, S., PECHENIZKIY, M., AND BAKER, R.S.J.D. (Eds.). 2011. Handbook of Educational Data Mining. CRC Press, Boca Raton, FL.
RUMBAUGH, J., BLAHA, M., PREMERLANI, W., EDDY, F., AND LORENSEN, W. 1991. Object-Oriented Modeling and Design. Prentice-Hall, Englewood Cliffs, NJ.
GINSBERG, M. 1987. Readings in Nonmonotonic Reasoning. Morgan Kaufmann, Los Altos, CA.
RICONSCENTE, M., MISLEVY, R.J., AND HAMEL, L. 2005. An Introduction to PADI Task Templates. PADI Technical Report 3. SRI International, Menlo Park, CA. Retrieved from http://padi.sri.com/downloads/TR3_Templates.pdf
RUPP, A.A. 2002. Feature selection for choosing and assembling measurement models: A building-block-based organization. International Journal of Testing, 2, 311-360.
RUPP, A.A., DICERBO, K.E., LEVY, R., BENSON, M., SWEET, S., CRAWFORD, A.V., FAY, D., KUNZE, K. L., CALIÇO, T., AND BEHRENS, J.T. this issue. Putting ECD into practice: The interplay of theory and data in evidence identification and accumulation within a digital learning environment. Journal of Educational Data Mining.
RUPP, A.A., GUSHTA, M., MISLEVY, R.J., AND SHAFFER, D.W. (2010). Evidencecentered design of epistemic games: Measurement principles for complex learning environments. The Journal of Technology, Learning, and Assessment, 8(4). Retrieved from http://escholarship.bc.edu/jtla/vol8/4
RUPP, A.A., TEMPLIN, J., AND HENSON, R.J. (2010). Diagnostic Measurement: Theory, Methods, and Applications. Guilford Press, New York, NY.
SHAFFER, D.W. 2006. Epistemic frames for epistemic games. Computers & Education, 46, 223-234.
SHUTE, V. J. 2011. Stealth assessment in computer-based games to support learning. In Computer Games and Instruction, S. TOBIAS AND J.D. FLETCHER, Eds. Information Age Publishers, Charlotte, NC, 503-523.
STEINBERG, L.S., MISLEVY, R.J., AND ALMOND R.G. 2005. Portal Assessment Design System for Educational Testing. U.S. Patent 434350000, August 4, 2005.
TANNENBAUM, A.S. 2003. Computer Networks. Pearson Education, Upper Saddle River, NJ.
TATSUOKA, K.K. 1983. Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345-354.
THEODORIDIS, S. AND KOUTROUMBAS, K. 1999. Pattern Recognition. Academic Press, San Diego, CA.
TUKEY, J.W. 1977. Exploratory Data Analysis. Adison Wesley, Reading, MA.
TRCKA, N., PECHENIZKIY, M., AND VAN DER AALST, W. 2010. Process mining from educational data. In Handbook of Educational Data Mining, C. ROMERO, S. VENTURA, S. VIOLA, M. PECHENIZKIY, AND R. BAKER, Eds. Chapman and Hall/CRC, Virginia Beach, VA, 123-142.
VAN DER LINDEN, W.J., AND GLAS, C.A.W. (Eds.) 2010. Elements of Adaptive Testing. Springer, New York, NY.
VANLEHN, K. 2008. Intelligent tutoring systems for continuous, embedded assessment. In The Future of Assessment: Shaping Teaching and Learning, C.A. Dwyer, Ed. Erlbaum, New York, NY, 113-138.
WAINER, H., DORANS, N.J., FLAUGHER, R., GREEN, B.F., AND MISLEVY, R.J. 2000. Computerized Adaptive Testing: A Primer. Routledge, New York, NY.
WEST, P., RUTSTEIN, D.W., MISLEVY, R.J., LIU, J., LEVY, R., DICERBO, K.E., CRAWFORD, A., CHOI, Y., CHAPPEL, K., AND BEHRENS, J.T. 2010. A Bayesian Network Approach to Modeling Learning Progressions. CRESST Research Report. The National Center for Research on Evaluation, Standards, Student Testing (CRESST), Center for Studies in Education, UCLA, Los Angeles, CA. Retrieved from http://www.cse.ucla.edu/products/download_report.asp?r=776
WILLIAMSON, D. M., BAUER, M., STEINBERG, L. S., MISLEVY, R. J., BEHRENS, J. T., AND DEMARK, S. 2004. Design rationale for a complex performance assessment. International Journal of Measurement, 4, 333–369.
WILLIAMSON, D.M., MISLEVY, R.J., AND BEJAR, I.I. (Eds.). 2006. Automated Scoring of Complex Performances in Computer Based Testing. Erlbaum Associates, Mahwah, NJ.
WITTEN, I.H., AND FRANK, E. 2005. Data Mining: Practical Machine Learning Tools and Techniques, 2nd ed. Morgan Kaufmann, San Francisco, CA.
ZHOU, M., XU, Y., NESBIT, J.C., AND WINNE, P.H. 2010. Sequential pattern analysis of learning logs: Methodology and applications. In Handbook of Educational Data Mining, C. ROMERO, S. VENTURA, S. VIOLA, M. PECHENIZKIY, AND R. BAKER, Eds. Chapman and Hall/CRC, Virginia Beach, VA, 107-121.
Section
Articles
Authors who publish with this journal agree to the following terms:
- The Author retains copyright in the Work, where the term “Work” shall include all digital objects that may result in subsequent electronic publication or distribution.
- Upon acceptance of the Work, the author shall grant to the Publisher the right of first publication of the Work.
- The Author shall grant to the Publisher and its agents the nonexclusive perpetual right and license to publish, archive, and make accessible the Work in whole or in part in all forms of media now or hereafter known under a Creative Commons 4.0 License (Attribution-Noncommercial-No Derivatives 4.0 International), or its equivalent, which, for the avoidance of doubt, allows others to copy, distribute, and transmit the Work under the following conditions:
- Attribution—other users must attribute the Work in the manner specified by the author as indicated on the journal Web site;
- Noncommercial—other users (including Publisher) may not use this Work for commercial purposes;
- No Derivative Works—other users (including Publisher) may not alter, transform, or build upon this Work,with the understanding that any of the above conditions can be waived with permission from the Author and that where the Work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.
- The Author is able to enter into separate, additional contractual arrangements for the nonexclusive distribution of the journal's published version of the Work (e.g., post it to an institutional repository or publish it in a book), as long as there is provided in the document an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post online a pre-publication manuscript (but not the Publisher’s final formatted PDF version of the Work) in institutional repositories or on their Websites prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (see The Effect of Open Access). Any such posting made before acceptance and publication of the Work shall be updated upon publication to include a reference to the Publisher-assigned DOI (Digital Object Identifier) and a link to the online abstract for the final published Work in the Journal.
- Upon Publisher’s request, the Author agrees to furnish promptly to Publisher, at the Author’s own expense, written evidence of the permissions, licenses, and consents for use of third-party material included within the Work, except as determined by Publisher to be covered by the principles of Fair Use.
- The Author represents and warrants that:
- the Work is the Author’s original work;
- the Author has not transferred, and will not transfer, exclusive rights in the Work to any third party;
- the Work is not pending review or under consideration by another publisher;
- the Work has not previously been published;
- the Work contains no misrepresentation or infringement of the Work or property of other authors or third parties; and
- the Work contains no libel, invasion of privacy, or other unlawful matter.
- The Author agrees to indemnify and hold Publisher harmless from Author’s breach of the representations and warranties contained in Paragraph 6 above, as well as any claim or proceeding relating to Publisher’s use and publication of any content contained in the Work, including third-party content.