Developing Computational Methods to Measure and Track Learners’ Spatial Reasoning in an Open-Ended Simulation



Published Aug 12, 2015
Aditi Mallavarapu Leilah Lyons Tia Shelley Emily Minor Brian Slattery Moira Zellner


Interactive learning environments can provide learners with opportunities to explore rich, real-world problem spaces, but the nature of these problem spaces can make assessing learner progress difficult. Such assessment can be useful for providing formative and summative feedback to the learners, to educators, and to the designers of the environments. This work adds to a growing body of research that is applying EDM techniques to more open-ended problem spaces. The open-ended problem space under study here was an environmental science simulation. Learners were confronted with the real-world challenge of effectively placing green infrastructure in an urban neighborhood to reduce surface flooding. Learners could try out different 2D spatial arrangements of green infrastructure and use the simulation to test each solution’s impact on flooding. The learners’ solutions and the solutions’ performances were logged during a controlled experiment with different user interface designs for the simulation. As with many open-problem spaces, analyzing this data was difficult due to the large state space, many good solutions, and many alternate paths to those good solutions. This work proposes a procedure for reducing the state space of solutions defined by 2D spatial patterns while maintaining their critical spatial properties. Spatial reasoning problems are a problem class not extensively examined by EDM, so this work sets the stage for further research in this area. This work also details a procedure for discovering effective 2D spatial strategies and solution paths, demonstrates how this information can be used to give formative feedback to the designers of the interactive learning environment, and speculates about how it could be used to provide formative feedback to learners.

How to Cite

Mallavarapu, A., Lyons, L., Shelley, T., Minor, E., Slattery, B., & Zellner, M. (2015). Developing Computational Methods to Measure and Track Learners’ Spatial Reasoning in an Open-Ended Simulation. Journal of Educational Data Mining, 7(2), 49–82.
Abstract 729 | PDF Downloads 362



environmental science simulation, open-ended problem spaces, effective 2D spatial strategies, solution paths, feedback to designers

AMERSHI, S., & CONATI, C. (2009). Combining Unsupervised and Supervised Classification to build user models for Exploratory Learning Environments. Journal of Educational Data Mining, 1 (1), 18-71.

ANDERSEN, E., YUN-EN LIU, APTER, E., BOUCHER-GENESSE, F., & POPOVIC´, Z. (2010). Gameplay Analysis through State Projection. International Conference on the Foundations of Digital Games. Monterey, California.

ANTLE, A. N., DROUMEVA, M., & HA, D. (2009). Hands on what? Comparing children's mousebased and tangible-based interaction. Proceedings of the 8th International Conference on Interaction Design and Children (pp. 80-88). ACM.

BAKER, R. S., & YACEF, K. (2009). The State of Educational Data Mining in 2009: A Review and Future Visions. Journal of Educational Data Mining , 1 (1), 3-17.

BERLAND, M., BAKER, R. S., & BLIKSTEIN, P. (2014). Educational data mining and learning analytics: Applications to constructionist research. Technology,Knowledge, and Learning , 19 (1-2), 205-220.

BERLAND, M., MARTIN, T., BENTON, T., SMITH, C., & DAVIS, D. (2013). Using Learning Analytics to Understand the Learing Pathwaysof Novice Programmers. Journal of Learing Sciences , 564-599.

BISWAS, G., LORETZ, K. M., & SEGEDY, J. R. (2013). Model-Driven Assessment of Learners in an Open-Ended Learning Environment. Third International Conference on Learning Analytics and Knowledge. New York, NY.

BLIKSTEIN, P. (2011). Using learning analytics to assess students’ behavior in open-ended Programming tasks. Learning Analytics and Knowledge. New York.

BRAVO, M., HERNANDEZ, J., SAORIN, J., & CONTERO, M. (2010). A 3D Educational Mobile Game to Enhance Student's Spatial Skills. IEEE 10th International Conference on Advanced Learning Technologies (ICALT). Sousse,Tunisia: IEEE Computer Society.

CONNELL, M., & STEVENS, D. (2002). A computer-based tutoring system for visual-spatial skills: dynamically adapting to the user's developmental range. Proceedings of the The 2nd International Conference on Development and Learning. Cambridge: IEEE COmputer Society.

DALE, M. (1999). Spatial pattern analysis in plant ecology. Cambridge, UK: Cambridge University Press.

DESMARAIS, M., & LEMIEUX, F. (2013). Clustering and Visualizing Study State Sequences. 6th International Conference on Educational Data Mining. Memphis, Tennessee.

DICERBO, K. E., & KIDWAI, K. (2013). Detecting Player Goals from Game Log Files. 6th International Conference on Educational Data Mining. Memphis, Tennessee.

DIXON, P. M. (1995). Ripley's K function. In Encyclopedia Environmetrics (pp. 1796-1803). NJ: Wiley.

EAGLE, M., & BARNES, T. (2014). Exploring Differences in Problem solving with Data-Driven Approach Maps. Educational Data Mining. Indianapolis.

EKSTROM, R. B., FRENCH, J. W., & HARMON, H. H. (1976). Manual for the Kit of Factor- Referenced Cognitive Tests. ETS.

FALAKMASIR, M. H., PARDOS, Z. A., GORDON, G. J., & BRUSILOVSKY, P. (2013). A Spectral Learning Approach to Knowledge Tracing. 6th International Conference on Educational Data Mining. Memphis, Tennessee.

FALCÃO, T., & PRICE, S. (2009). What have you done! The role of ‘interference’ in tangible environments for supporting collabarative learning. 8th International Conference on Computer Supported Collaborative Learning. Rhodes, Greece.

FORTIN, M. J., DALE, M. R., & HOEF, J. V. (2002). Spatial Analysis in Ecology. In Encyclopedia of Environmetrics (pp. 2051-2058). NJ: Wiley.

FOURNIER-VIGER, P., NKAMBOU, R., NGUIFO, E. M., MAYERS, A., & FAGHIHI, U. (2013). A multiparadigm intelligent tutoring system for robotic arm training. IEEE Transactions on Learning Technologies , 6 (4), 364-377.

FOURNIER-VIGER, P., NKAMBOU, R., NGUIFO, E., MAYERS, A., & FAGHIHI, U. (2013). A Mutilparadigm intelligent tutoring system for robotic arm training. Learning Technologies, IEEE Trasactions , 364-377.

GOBERT, J., SAO PEDRO, M., RAZIUDDIN, J., & BAKER, R. (2013). From Log Files to Assessment Metrics for Science Inquiry Using Educational Data Mining. Journal of the Learning Sciences, 22 (4), 521-563.

HALPERN, D., & COLLEAR, M. (2005). Sex Differences in Visuospatial Abilities. In P. Shah, & A. Miyake, Cambridge Handbook of Visuospatial Thinking (pp. 170-212). New York: Cambridge University Press.

HARPSTEAD, E., MACLELLAN, C. J., KOEDINGER, K. R., ALEVEN, V., DOW, S. P., & MYERS, B. A. (2013). Investigating the Solution Space of an Open-Ended Educational Game Using Conceptual Feature Extraction. 6th International Conference on Educational Data Mining. Memphis, Tennessee.

HEGARTY, M., & WALLER, D. (2005). Individual and age related differences in visuospatial abilities. In P. Shah, & A. Miyake, Cambridge handbook of Visuospatial Thinking (pp. 121- 169). New York: Cambridge University Press.

HUBBARD, C., MENGSHOEL, O., MOON, C., & YONG, S. (1996). Multimedia instructional software for visual reasoning: Visual Reasoning Tutor (VRT). Proceedings of the Third IEEE International Conference on Multimedia Computing and Systems,. Hiroshima, Japan: IEEE Computer Society.

JARUSEK, P., KLUSACEK, M., & PELANEK, R. (2013). Modeling Students’ Learning and Variability of Performance in Problem Solving. 6th International Conference on Educational Data Mining. Memphis, Tennessee.

JOHNSON, M. W., EAGLE, M., & BARNES, T. (2013). InVis: An Interactive Visualization Tool for Exploring Interaction Networks. Educational Data Mining. Memphis.

KARDAN, S., & CONATI, C. (2013). Evaluation of a Data Mining Approach to Providing Adaptive Support in an Open-Ended Learning Environment: A Pilot Study. Artificial Intelligence In Education , 2, 41-48.

KIM, M. J., & MAHER, M. (2008). The impact of tangible user interfaces on spatial cognition during collaborative design. Design Studies , 29 (3), 222-253.

KUTNER, M., NACHTSHEIM, C., & NETER, J. (2004). Applied Linear Regression Models. McGraw Hill.

LAKOFF, G., & JOHNSON, M. (1980). Metaphors We Live By. Chicago: University of Chicago Press.

LEE, S. J., YUN-EN, L., & POPOVIC, Z. (2014). Learning Individual Behavior in an Educational Game: A Data- Driven Approach. Educational data Mining. Indianapolis.

LEVY, S., & WILENSKY, U. (2007). How do I get there...straight, oscillate or inch? High-school students' exploration patterns of Connected Chemistry. 2007 meeting of the American Educational Research Association. Chicago: AERA.

LIU, Y.-E., MANDEL, T., BUTLER, E., ANDERSON, E., O'ROURKE, E., EMMA, B., ZORAN, P. (2013). Predicting Player moves in Educational Game: A Hybrid Approach. Educational Data Mining. Memphis.

LYNCH, C., ASHLEY, K., PINKWART, N., & ALEVEN, V. (2008). Argument graph classification with Genetic programming. Educational Data Mining. Montreal, Quebec.

LYONS, L., DASGUPTA, C., SHELLEY, T., SLATTERY, B., MINOR, E., & ZELLNER, M. (2012). Parsing Patterns: Developing Metrics to Characterize Spatial Problem Solving Strategies within an Environmental Science Simulation. AREA, (p. 19).

MAIMON, O., & ROKACH, L. (2010). Data Mining and Knowledge Discovery Handbook. Springer.

MARSHALL, P. (2007). Do tangible interfaces enhance learning? Proceedings of the 1st international conference on Tangible and embedded interaction (TEI '07) (pp. 163-170). ACM.

MARTINEZ-MALDONADO, R., YACEF, K., & KAY, J. (2013). Data Mining in the Classroom: Discovering Groups Strategies at a Multi-tabletop Environment . 6th International Conference on Educational Data Mining. Memphis,Tennessee.

MASSEY, D., ZELLNER, M. L., COTNER, L., MINOR, E., & GONZALEZ-MELER, M. (2010). Landscape Green Infrastructure Design Model (L-GrID) User’s Manual. Report and software to the Illinois Environmental Protection Agency. Chicago, IL: University of Illinois at Chicago. Chicago.

MENGSHOEL, O., CHAUHAN, S., & YONG, S. (1996). Intelligent critiquing and tutoring of spatial reasoning skills. Artificial Intelligence for Engineering, Design, Analysis and Manufacturing , 235-249.

MINOR, E. S., & URBAN, D. L. (2008). A GraphıTheory Framework for Evaluating Landscape Connectivity and Conservation Planning. Conservation Biology , 22 (2), 297-307.

MULLER, J., KRETZSCHMAR, A., & GREIFF, S. (2013). Exploring Exploration: Inquiries into Exploration Behavior in Complex Problem Solving. 6th International Conference on Educational Data Mining. Memphis, Tennessee.

NESBITT, K., SUTTON, K., WILSON, J., & HOOKHAM, G. (2009). Improving player spatial abilities for 3D challenges. The 6th Australasian Conference on Interactive Entertainment.

Sidney,Australia. PROTÁZIO, J., PEREIRA, W., & ELAYNE JESUS DE CASTRO, F. (1999). Explicit Formulas for an Area Based Edge Effect Correction Method and their Application to Ripley’s K-Function. Journal of Vegetation Science , 10 (3), 433-438.

RAFFERTY, A. N., DAVENPORT, J., & BRUNSKILL, E. (2013). Estimating Student Knowledge from Paired Interaction Data. 6th International Conference on Educational Data Mining, (pp. 260- 263). Memphis, Tennessee.

ROCK, I., & BROSGOLE, L. (1964). Grouping based on phenomenal proximity. Journal of Experimental Psychology , 67 (6), 531-538.

ROMERO, C., & VENTURA, S. (2010). Educational data mining: a review of the state of the art.

Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on , 40 (6), 601-618.

SCHNEIDER, B., JERMANN, P., ZUFFEREY, G., & DILLENBOURG, P. (2011). Benefits of a Tangible Interface for Collaborative Learning and Interaction. IEEE Transactions on Learning Technologies , 4 (3), 222-232.

SCHWEINGRUBER, H., KELLER, T., & QUINN, H. (2012). A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. National Academies Press.

SHELLEY, T., LYONS, L., MINOR, E., & ZELLNER, M. (2011). Evaluating the embodiment benefits of a paper- based TUI for educational simulations. 29th International Conference on Human Factors in Computing Systems. New York, NY.

SHELLEY, T., LYONS, L., SHI, J., MINOR, E., & ZELLNER, M. (2010). Paper to parameters: designing tangible simulation input. 12th ACM International Conference adjunct papers on Ubiquitous computing. New York, NY.

SHUTE, V. J. (2011). Stealth Assessment in computer based games to support learning. In Computer Games and Instruction (pp. 503-524). Charlotte, NC: Information Age Publishers.

SINGER, M., RADINSKY, J., & GOLDMAN, S. R. (2008). The Role of Gesture in Meaning Construction. Discourse Processes: A Multidisciplinary Journal , 45 (4-5), 365-386.

SISWONO, T. Y. (2008). Promoting creativity in learning mathematics using open-ended problems. The 3rd International Conference on Mathematics and Statistics (ICoMS-3). Indonesia.

SMITH, A., WIEBE, E., MOTT, B., & LESTER, J. (2014). SKETCHMINER: Mining Learner- Generated Science Drawings with Topological Abstraction. Educational Data Mining. Indianapolis.

STIEFF, M., DIXON, B., KUMI, B., & HEGARTY, M. (2014). Strategy Training Eliminates Sex Differences in Spatial Problem Solving in a STEM Domain. Journal of Educational Psychology, 106 (2), 390-402.

UTTAL, D., MEADOW, N., TIPTON, E., HAND , L., ALDEN, A., WARREN, C., NEWCOMBE, N. S (2013). The malleability of spatial skills: A meta-analysis of training studies. Psychological Bulletin , 352-402. VAN LEHN, K. (2011). The relative effectiveness of Human Tutoring, Intelligent Tutoring Systems and other Tutoring System. Educational Psychologist , 197-221.

VYGOTSKY, L. (1978). Mind in Society: The development of higher mental processes. Cambridge, MA: Harvard University Press.

WAI, J., LUBINSKI, D., & BENBOW, C. (2009). Spatial ability for STEMdomains: Aligning over 50 years of cumulative psychological knowledge solidifies its importance. Journal of Educational Psychology , 817-835.

WANG, E., & KIM, Y. (2005). Intelligent Visual Reasoning Tutor. Proceedings of the Fifth IEEE International Conference on Advanced Learning Technologies (pp. 511-515). Washington,D.C: IEEE Computer Society.

WERTHEIMER, M. (1923). Untersuchungen zur Lehre von der Gestalt II. Psycologische Forschung, 4, 301-350.

WIEDERRECHT, M. A., & ULINSKI, A. C. (2012, January). Developmentally appropriate intelligent spatial tutoring for mobile devices. Intelligent Tutoring Systems , 594-596.

WILENSKY, U. J. (1993). Connected Mathematics-Building Concrete Relationship with Mathematical Knowledge. Dissertation of Doctor of Philosophy, Massachusetts Institute of Technology.

ZELLNER, M. L. (2008). Embracing Complexity and Uncertainty: The Potential of Agent-Based Modeling for Environmental Planning and Policy. Planning Theory and Practice , 9 (4), 237-457.
EDM 2015 Journal Track