Multimodal Learning Analytics

PROJECT DATES: 2009 – present


Politicians, educators, business leaders, and researchers are unanimous in stating that we need to redesign schools to teach “21st century skills”: creativity, innovation, critical thinking, problem solving, communication, and collaboration. None of those skills are easily measured using current assessment techniques, such as multiple choice tests or even portfolios. As a result, our schools are paralyzed by the push to teach new skills and the lack of reliable ways to assess those skills, or provide students with formative feedback. One of the difficulties is that current assessment instruments are based on products (an exam, a project, a portfolio), and not on processes (the actual cognitive and intellectual development while performing a learning activity), due to the intrinsic difficulties in capturing detailed process data for large numbers of students. However, new sensing and data mining technologies could make it possible to capture and analyze massive amounts of process data of classroom activities. We are conducting research on the use of biosensing, signal- and image-processing, text-mining, and machine learning to explore multimodal process-based student assessments.


Thus far, we have been able to show that using multimodal analysis techniques provide a powerful way to study complex learning environments. In our ongoing research we are developing open-source tools for capturing and analyzing multimodal data in hands-on learning environments. We are also continuing to conduct experimental studies that can be use to examine the efficacy of different learning strategies and learning environments.

Multimodal Learning Analytics (MMLA) was an idea we had in 2009 that grew from our lab into the world: there are now tens of researchers doing this work, Special Interest Groups at different academic organizations, and research labs with a strong focus on MMLA led by Marcelo Worsley (Northwestern University) and Bertrand Schneider (Harvard University).

(complete PDFs in the Publication page)

Blikstein, P. & Worsley, M. (2018). Multimodal Learning Analytics and Assessment of Open-Ended Artifacts. In Neimi, D., Pea, R., Saxberg, B., & Clark, R. (Eds), Learning Analytics in Education.

Worsley, M. & Blikstein, P. (2017). Reasoning Strategies in the Context of Engineering Design with Everyday Materials. The Journal of Pre-College Engineering Education Research, 6(2), 57-74.

Worsley, M.* & Blikstein, P. (2017). A Multimodal Analysis of Making. International Journal of Artificial Intelligence in Education, pp. 1-35. doi: 10.1007/s40593-017-0160-1.

Blikstein, P. & Worsley, M. (2016). Multimodal Learning Analytics and Education Data Mining: using computational technologies to measure complex learning tasks. Journal of Learning Analytics, 3(2), 220-238. PDF

Merceron, A., Blikstein, P., & Siemens, G. (2016). Learning analytics: From big data to meaningful data. Journal of Learning Analytics, 2 (3), pp. 4-8.

Worsley, M., Abrahamson, D., Blikstein, P., Grover, S., Schneider, B., & Tissenbaum, M. (2016). Situating multimodal learning analytics. In C.-K. Looi, J. L. Polman, U. Cress, & P. Reimann (Eds.), “Transforming learning, empowering learners,” Proceedings of the International Conference of the Learning Sciences (ICLS 2016) (Vol. 2, pp. 1346-1349). Singapore: International Society of the Learning Sciences. PDF

Schneider, B.* & Blikstein, P. (2015). Unraveling students’ interaction around a tangible interface using multimodal learning analytics. jEDM-Journal of Educational Data Mining, 7(3), 89-116.

Worsley, M. Scherer, S., Morency, L.P., & Blikstein, P. (2015). Exploring Behavior Representation for Learning Analytics. In Proceedings of the 2015 International Conference on Multimodal Interaction. ACM, New York, USA. pp. 251-258. PDF

Worsley, M. & Blikstein, P. (2015). Using Learning Analytics to Study Cognitive Disequilibrium in a Complex Learning Environments. In Proceedings of the 5th Annual Conference on Learning Analytics and Knowledge, ACM, New York, USA. pp. 426-247.

Worsley, M. & Blikstein, P. (2014). Deciphering the Practices and Affordances of Different Reasoning Strategies through Multimodal Learning Analytics. In Proceedings of the 2014 ACM workshop on Multimodal Learning Analytics Workshop and Grand Challenge (MLA ’14). ACM, New York, NY, USA. pp. 21-27. PDF

Blikstein, P., Worsley, M., Piech, C., Gibbons, A., Sahami, M., & Cooper, S. (2014). Programming Pluralism: Using Learning Analytics to Detect Patterns in Novices’ Learning of Computer Programming. International Journal of the Learning Sciences. Vol. 23, Iss. 4. pp. 561-599. PDF

Blikstein, P.**, Worsley, M.*, Piech, C.*, Sahami, M., Cooper, S., & Koller, D. (2014). Programming pluralism: Using learning analytics to detect patterns in novices’ learning of computer programming. Journal of the Learning Sciences, 23 (4), pp. 561-599. [45 citations on Google Scholar] PDF

Berland, M., Baker, R., & Blikstein, P. (2014) Learning analytics in constructivist, inquiry-based learning environments. Technology, Knowledge, and Learning, 19 (1-2), pp. 205-220. [60 citations on Google Scholar]

Worsley, M. & Blikstein, P. (2014). Using Multimodal Learning Analytics to Study Learning Mechanisms. In Proceedings of the 2014 Educational Data Mining Conference. pp. pp 431-432. PDF

Worsley, M. & Blikstein, P. (2014). Analyzing Engineering Design through the Lens of Computation. Journal of Learning Analytics. PDF

Worsley, M., &; Blikstein, P. (2013). Programming Pathways: A Technique for Analyzing Novice Programmers’ Learning Trajectories. In Artificial Intelligence in Education. Springer Berlin Heidelberg. 844-847. PDF

Gomes, J. Yassine, M., Worsley, M., & Blikstein, P. (2013) Analysing Engineering Expertise of High School Students Using Eye Tracking and Multimodal Learning Analytics. In Proceedings of the Educational Data Mining 2013 (EDM ’13). Memphis, TN, USA. 375-377. PDF

Worsley, M. & Blikstein, P. (2013). Toward the Development of Mulitmodal Action Based Assessment. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (LAK ’13). ACM, New York, NY, USA, 94-101. PDF

Worsley, M. & Blikstein, P. (2012). An Eye For Detail: Techniques For Using Eye Tracker Data to Explore Learning in Computer-Mediated Environments. In the Proceedings of the 2012 International Conference of the Learning Sciences (ICLS ’12). Sydney, Australia. 561-562. PDF

Worsley, M., Johnston, M. & Blikstein P. (2011). OpenGesture: a low cost authoring framework for gesture and speech based application development and learning analytics. In Proceedings of the 10th International Conference on Interaction Design and Children (IDC ’11). ACM, New York, NY, USA. 254-256. PDF

Worsley, M. & Blikstein P. (2011). What’s an Expert? Using learning analytics to identify emergent markers of expertise through automated speech, sentiment and sketch analysis. In Proceedings for the 4th Annual Conference on Educational Data Mining. Eindhoven, Netherlands. 235-240. PDF


Paulo Blikstein
Yipu Zheng
Engin Bumbacher


Marcelo Worsley (Northwestern University)
Bertrand Schneider (Harvard Graduate School of Education)
Mehran Sahami (Stanford University)
Steve Cooper (University of Nebraska–Lincoln)

Jesus Guzman

Cooper Lindsay


Google Faculty Research Award: “Using Learning Analytics to Detect Patterns in the Learning of Computer Programming”


For more information, please contact Yipu Zheng (