A First Step in Learning Analytics: Pre-processing Low-Level Alice Logging Data of Middle School Students
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
Jul 25, 2013
Linda Werner
Charlie McDowell
Jill Denner
Abstract
Educational data mining can miss or misidentify key findings about student learning without a transparent process of analyzing the data. This paper describes the first steps in the process of using low-level logging data to understand how middle school students used Alice, an initial programming environment. We describe the steps that were required and the decisions that were made in building a tool to translate the low-level logging data into a form that can be used to investigate educational questions about problem solving strategies for a range of different programming tasks. This work contributes to efforts to analyze educational data, and is important for researchers and tool builders involved with the design of logging systems for other programming environments and software tools.
How to Cite
Werner, L., McDowell, C., & Denner, J. (2013). A First Step in Learning Analytics: Pre-processing Low-Level Alice Logging Data of Middle School Students. Journal of Educational Data Mining, 5(2), 11–37. https://doi.org/10.5281/zenodo.3554631
##plugins.themes.bootstrap3.article.details##
Keywords
Alice, initial programming environment, problem solving strategies, analysing low-level logging data
References
ALEXANDER, P. 2003. The Development of Expertise: The Journey from Acclimation to Proficiency. Educational Researcher, 32, 10-14.
AMERSHI, S., and CONATI, C. 2009. Combining Unsupervised and Supervised Classification to Build User Models for Exploratory Learning Environments. The Journal of Educational Data Mining, 1(1), 18-71.
BAKER, R.S.J.D. 2007. Modeling and Understanding Students' Off-task Behavior in Intelligent Tutoring Systems. Proceedings of the SIGCHI conference on Human factors in computing systems (CHI '07). New York, NY: ACM, 1059-1068.
BAKER, R.S.J.D., and YACEF, K. 2009. The State of Educational Data Mining in 2009: A Review and Future Visions. Journal of Educational Data Mining, 1(1), 3-17.
BRANSFORD, J.D., BARRON, B., PEA, R.D., MELTZOFF, A., KUHL, P., BELL, P.,
STEVENS, R., SCHWARTZ, D.L., VYE, N., REEVES, B., ROSCHELLE, J., and
SABELLI, N. 2006. Foundations and Opportunities for an Interdisciplinary Science of Learning. In R.K. Sawyer (Ed.), The Cambridge Handbook of the Learning Sciences. 19- 34. New York: Cambridge University Press.
BRUCE, C., BUCKINGHAM, L., HYND, J., MCMAHON, C., ROGGENKAMP, M., and STOODLEY, I. 2004. Ways of Experiencing the Act of Learning to Program: A Phenomenographic Study of Introductory Programming Students at University. Journal of Information Technology Education, 3, 143-160.
BURNETT, M., COOK, C., PENDSE, O., ROTHERMEL, G., SUMMET, J., and WALLACE, C. 2003. End-user Software Engineering with Assertions in the Spreadsheet Paradigm. In Proceedings of the 25th International Conference on Software Engineering, 93-103. IEEE Computer Society.
CHOQUET, C., and IKSAL, S. 2007. Modeling Tracks for the Model Driven Reengineering of a TEL System. Journal of Interactive Learning Research, 18(2), 161-184.
CONWAY, M.J. 1997. Alice: Easy-to-Learn 3D Scripting for Novices (Doctoral dissertation, University of Virginia).
DANN, W.P., COOPER, S.P., and ERICSON, B. 2009. Exploring Wonderland: Java Programming Using Alice and Media Computation (1st ed.). Upper Saddle River, NJ: Prentice Hall Press.
DIMITRACOPOULOU, A. 2005. Designing Collaborative Learning Systems: Current Trends & Future Research Agenda. In Proceedings of the 2005 Conference on Computer Support for Collaborative Learning: Learning 2005: The next 10 years! 115-124. International Society of the Learning Sciences.
DISESSA, A.A. 2006. A History of Conceptual Change Research: Threads and Fault Lines. In R.K. SAWYER (Ed.), The Cambridge Handbook of the Learning Sciences, 265-281. New York: Cambridge University Press.
FINCHER, S., and UTTING, I. 2010. Machines for Thinking. Transactions on Computing Education, 10(4), Article 13, 1-7.
HATANO, G. 1988. Social and motivational bases for mathematical understanding. In G.B. SAXE and M. GEARHART (Eds.), Children’s Mathematics, 55–70. San Francisco, CA: Jossey-Bass.
JADUD, M.C. 2005. A 1st Look at Novice Compilation Behavior. Computer Science Education, 15(1), 25-40.
JADUD, M.C., and HENRIKSEN, P. 2009. Flexible, Reusable Tools for Studying Novice Programmers. In Proceedings of the Fifth International Computing Education Research Workshop (ICER '09). New York, NY: ACM, 37-42.
JONASSEN, D. H. 1999. Designing Constructivist Learning Environments. Instructional Design Theories and Models: A New Paradigm of Instructional Theory, 2, 215-239.
KELLEHER, C. 2006. Motivating Programming: Using Storytelling to Make Computer Programming Attractive to More Middle School Girls (Doctoral dissertation). Retrieved from reference http://www.dtic.mil/dtic/ number ADA492489.
KOEDINGER, K. R., BAKER, R.S.J.D., CUNNINGHAM, K., SKOGSHOLM, A.,
LEBER, B., and STAMPER, J. 2010. A Data Repository for the EDM Community: The PSLC DataShop. in C. ROMERO, S. VENTURA, M. PECHENIZKIY, R.S.J.D. BAKER (Eds.) Handbook of Educational Data Mining, 46-53. Boca Raton, FL: CRC Press.
MACFADYEN, L.P., and DAWSON, S. 2010. Mining LMS Data to Develop an “Early Warning System” for Educators: A Proof of Concept. Computers & Education, 54, 588- 599.
MCDOWELL, C., WERNER, L., BULLOCK, H.E, and FERNALD, J. 2006. Pair Programming Improves Student Retention, Confidence, and Program Quality. Communications of the ACM, 49(8), 90-95.
MERTEN, C., and CONATI, C. 2006. Eye-tracking to Model and Adapt to User Metacognition in Intelligent Learning Environments. In Proceedings of the 11th International Conference on Intelligent User Interfaces (IUI '06). New York: NY, 39-46.
PIECH, C., COOPER, S., SAHAMI, M., KOLLER, D., and BLIKSTEIN, P. 2012. Modeling How Students Learn to Program. In Proceedings of the 43st ACM Technical Symposium on Computer Science Education (SIGCSE). ACM, New York, NY, USA.
ROBINS, A., ROUNTREE, J., and ROUNTREE, N. 2003. Learning and teaching programming: A review and discussion. Computer Science Education, 13(2), 137-172.
SMITH, B. 2010, June. Effect of Pair Programming on Middle Schoolers Using Storytelling Alice. Poster session presented at the 13th Annual UCSC Undergraduate Research Symposium, Santa Cruz, CA, USA.
SOLOWAY, E., EHRLICH, K., BONAR, J., and GREENSPAN, J. 1983. What do Novices Know about Programming? In B. SHNEIDERMAN and A. BADRE (Eds.), Directions in Human–Computer Interactions, 27-54 . Norwood, NJ: Ablex.
SOLOWAY, E., and SPOHRER, J. C. (Eds.). 1989. Studying the Novice Programmer. Hillsdale N J: Lawrence Erlbaum.
UTTING, I., COOPER, S., KÖLLING, M., MALONEY, J., and RESNICK, M. 2010.
Alice, Greenfoot, and Scratch -- A Discussion. Transactions on Computing Education, 10(4), 1-11.
WERNER, L., DENNER, J., CAMPE, S., and KAWAMOTO, D.C. 2012. The Fairy Performance Assessment: Measuring Computational Thinking in Middle School. Proceedings of the 43st ACM Technical Symposium on Computer Science Education (SIGCSE). ACM, New York, NY, USA.
WING, J.M. 2011, Spring. Computational Thinking: What and Why? In Link Magazine, 6.0. Retrieved from http://link.cs.cmu.edu/article.php?a=600.
AMERSHI, S., and CONATI, C. 2009. Combining Unsupervised and Supervised Classification to Build User Models for Exploratory Learning Environments. The Journal of Educational Data Mining, 1(1), 18-71.
BAKER, R.S.J.D. 2007. Modeling and Understanding Students' Off-task Behavior in Intelligent Tutoring Systems. Proceedings of the SIGCHI conference on Human factors in computing systems (CHI '07). New York, NY: ACM, 1059-1068.
BAKER, R.S.J.D., and YACEF, K. 2009. The State of Educational Data Mining in 2009: A Review and Future Visions. Journal of Educational Data Mining, 1(1), 3-17.
BRANSFORD, J.D., BARRON, B., PEA, R.D., MELTZOFF, A., KUHL, P., BELL, P.,
STEVENS, R., SCHWARTZ, D.L., VYE, N., REEVES, B., ROSCHELLE, J., and
SABELLI, N. 2006. Foundations and Opportunities for an Interdisciplinary Science of Learning. In R.K. Sawyer (Ed.), The Cambridge Handbook of the Learning Sciences. 19- 34. New York: Cambridge University Press.
BRUCE, C., BUCKINGHAM, L., HYND, J., MCMAHON, C., ROGGENKAMP, M., and STOODLEY, I. 2004. Ways of Experiencing the Act of Learning to Program: A Phenomenographic Study of Introductory Programming Students at University. Journal of Information Technology Education, 3, 143-160.
BURNETT, M., COOK, C., PENDSE, O., ROTHERMEL, G., SUMMET, J., and WALLACE, C. 2003. End-user Software Engineering with Assertions in the Spreadsheet Paradigm. In Proceedings of the 25th International Conference on Software Engineering, 93-103. IEEE Computer Society.
CHOQUET, C., and IKSAL, S. 2007. Modeling Tracks for the Model Driven Reengineering of a TEL System. Journal of Interactive Learning Research, 18(2), 161-184.
CONWAY, M.J. 1997. Alice: Easy-to-Learn 3D Scripting for Novices (Doctoral dissertation, University of Virginia).
DANN, W.P., COOPER, S.P., and ERICSON, B. 2009. Exploring Wonderland: Java Programming Using Alice and Media Computation (1st ed.). Upper Saddle River, NJ: Prentice Hall Press.
DIMITRACOPOULOU, A. 2005. Designing Collaborative Learning Systems: Current Trends & Future Research Agenda. In Proceedings of the 2005 Conference on Computer Support for Collaborative Learning: Learning 2005: The next 10 years! 115-124. International Society of the Learning Sciences.
DISESSA, A.A. 2006. A History of Conceptual Change Research: Threads and Fault Lines. In R.K. SAWYER (Ed.), The Cambridge Handbook of the Learning Sciences, 265-281. New York: Cambridge University Press.
FINCHER, S., and UTTING, I. 2010. Machines for Thinking. Transactions on Computing Education, 10(4), Article 13, 1-7.
HATANO, G. 1988. Social and motivational bases for mathematical understanding. In G.B. SAXE and M. GEARHART (Eds.), Children’s Mathematics, 55–70. San Francisco, CA: Jossey-Bass.
JADUD, M.C. 2005. A 1st Look at Novice Compilation Behavior. Computer Science Education, 15(1), 25-40.
JADUD, M.C., and HENRIKSEN, P. 2009. Flexible, Reusable Tools for Studying Novice Programmers. In Proceedings of the Fifth International Computing Education Research Workshop (ICER '09). New York, NY: ACM, 37-42.
JONASSEN, D. H. 1999. Designing Constructivist Learning Environments. Instructional Design Theories and Models: A New Paradigm of Instructional Theory, 2, 215-239.
KELLEHER, C. 2006. Motivating Programming: Using Storytelling to Make Computer Programming Attractive to More Middle School Girls (Doctoral dissertation). Retrieved from reference http://www.dtic.mil/dtic/ number ADA492489.
KOEDINGER, K. R., BAKER, R.S.J.D., CUNNINGHAM, K., SKOGSHOLM, A.,
LEBER, B., and STAMPER, J. 2010. A Data Repository for the EDM Community: The PSLC DataShop. in C. ROMERO, S. VENTURA, M. PECHENIZKIY, R.S.J.D. BAKER (Eds.) Handbook of Educational Data Mining, 46-53. Boca Raton, FL: CRC Press.
MACFADYEN, L.P., and DAWSON, S. 2010. Mining LMS Data to Develop an “Early Warning System” for Educators: A Proof of Concept. Computers & Education, 54, 588- 599.
MCDOWELL, C., WERNER, L., BULLOCK, H.E, and FERNALD, J. 2006. Pair Programming Improves Student Retention, Confidence, and Program Quality. Communications of the ACM, 49(8), 90-95.
MERTEN, C., and CONATI, C. 2006. Eye-tracking to Model and Adapt to User Metacognition in Intelligent Learning Environments. In Proceedings of the 11th International Conference on Intelligent User Interfaces (IUI '06). New York: NY, 39-46.
PIECH, C., COOPER, S., SAHAMI, M., KOLLER, D., and BLIKSTEIN, P. 2012. Modeling How Students Learn to Program. In Proceedings of the 43st ACM Technical Symposium on Computer Science Education (SIGCSE). ACM, New York, NY, USA.
ROBINS, A., ROUNTREE, J., and ROUNTREE, N. 2003. Learning and teaching programming: A review and discussion. Computer Science Education, 13(2), 137-172.
SMITH, B. 2010, June. Effect of Pair Programming on Middle Schoolers Using Storytelling Alice. Poster session presented at the 13th Annual UCSC Undergraduate Research Symposium, Santa Cruz, CA, USA.
SOLOWAY, E., EHRLICH, K., BONAR, J., and GREENSPAN, J. 1983. What do Novices Know about Programming? In B. SHNEIDERMAN and A. BADRE (Eds.), Directions in Human–Computer Interactions, 27-54 . Norwood, NJ: Ablex.
SOLOWAY, E., and SPOHRER, J. C. (Eds.). 1989. Studying the Novice Programmer. Hillsdale N J: Lawrence Erlbaum.
UTTING, I., COOPER, S., KÖLLING, M., MALONEY, J., and RESNICK, M. 2010.
Alice, Greenfoot, and Scratch -- A Discussion. Transactions on Computing Education, 10(4), 1-11.
WERNER, L., DENNER, J., CAMPE, S., and KAWAMOTO, D.C. 2012. The Fairy Performance Assessment: Measuring Computational Thinking in Middle School. Proceedings of the 43st ACM Technical Symposium on Computer Science Education (SIGCSE). ACM, New York, NY, USA.
WING, J.M. 2011, Spring. Computational Thinking: What and Why? In Link Magazine, 6.0. Retrieved from http://link.cs.cmu.edu/article.php?a=600.
Issue
Section
Articles
Authors who publish with this journal agree to the following terms:
- The Author retains copyright in the Work, where the term “Work” shall include all digital objects that may result in subsequent electronic publication or distribution.
- Upon acceptance of the Work, the author shall grant to the Publisher the right of first publication of the Work.
- The Author shall grant to the Publisher and its agents the nonexclusive perpetual right and license to publish, archive, and make accessible the Work in whole or in part in all forms of media now or hereafter known under a Creative Commons 4.0 License (Attribution-Noncommercial-No Derivatives 4.0 International), or its equivalent, which, for the avoidance of doubt, allows others to copy, distribute, and transmit the Work under the following conditions:
- Attribution—other users must attribute the Work in the manner specified by the author as indicated on the journal Web site;
- Noncommercial—other users (including Publisher) may not use this Work for commercial purposes;
- No Derivative Works—other users (including Publisher) may not alter, transform, or build upon this Work,with the understanding that any of the above conditions can be waived with permission from the Author and that where the Work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.
- The Author is able to enter into separate, additional contractual arrangements for the nonexclusive distribution of the journal's published version of the Work (e.g., post it to an institutional repository or publish it in a book), as long as there is provided in the document an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post online a pre-publication manuscript (but not the Publisher’s final formatted PDF version of the Work) in institutional repositories or on their Websites prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (see The Effect of Open Access). Any such posting made before acceptance and publication of the Work shall be updated upon publication to include a reference to the Publisher-assigned DOI (Digital Object Identifier) and a link to the online abstract for the final published Work in the Journal.
- Upon Publisher’s request, the Author agrees to furnish promptly to Publisher, at the Author’s own expense, written evidence of the permissions, licenses, and consents for use of third-party material included within the Work, except as determined by Publisher to be covered by the principles of Fair Use.
- The Author represents and warrants that:
- the Work is the Author’s original work;
- the Author has not transferred, and will not transfer, exclusive rights in the Work to any third party;
- the Work is not pending review or under consideration by another publisher;
- the Work has not previously been published;
- the Work contains no misrepresentation or infringement of the Work or property of other authors or third parties; and
- the Work contains no libel, invasion of privacy, or other unlawful matter.
- The Author agrees to indemnify and hold Publisher harmless from Author’s breach of the representations and warranties contained in Paragraph 6 above, as well as any claim or proceeding relating to Publisher’s use and publication of any content contained in the Work, including third-party content.