Multimodal Learning Analytics

PROJECT DATES: 2009 – present

Assessments for 21st-century learning

Politicians, educators, business leaders, and researchers agree that we need to redesign schools to teach “21st century skills” like creativity, innovation, critical thinking, problem solving, communication, and collaboration. The current assessment methods, like multiple choice tests and portfolios, don’t work well for these skills. New technologies could help us assess students better by looking at how they perform these activities or provide students with formative feedback. One of the difficulties is that current assessment instruments are based on products (an exam, a project, a portfolio), and not on processes (the actual cognitive and intellectual development while performing a learning activity), due to the intrinsic difficulties in capturing detailed process data for large numbers of students. However, new sensing and data mining technologies could make it possible to capture and analyze massive amounts of process data of classroom activities.

The TLTL pioneered research on the use of biosensing, signal- and image-processing, text-mining, and machine learning to explore multimodal process-based student assessments (see some of our foundational papers from 2012 and 2013)

A Field Pioneered by the TLT Lab

Thus far, we have been able to show that using multimodal analysis techniques provide a powerful way to study complex learning environments. We continue to conduct experimental studies that can be use to examine the efficacy of different learning strategies and learning environments.

Multimodal Learning Analytics (MMLA) was an idea that started in the TLTL lab in 2009 with Paulo Blikstein and Marcelo Worsley (now a professor at Northwestern University), and soon brought together a large team–in particular Bertrand Schneider (now a professor at Harvard University), Richard Davis, and Engin Bumbacher. It grew from our lab into the world: there are now tens of researchers doing this work, Special Interest Groups at different academic organizations, a handbook, etc. Worsley and Schneider continue lead research labs with a strong focus on MMLA.

(complete PDFs in the Publication page)

Blikstein, P. & Worsley, M. (2018). Multimodal Learning Analytics and Assessment of Open-Ended Artifacts. In Neimi, D., Pea, R., Saxberg, B., & Clark, R. (Eds), Learning Analytics in Education.

Worsley, M. & Blikstein, P. (2017). Reasoning Strategies in the Context of Engineering Design with Everyday Materials. The Journal of Pre-College Engineering Education Research, 6(2), 57-74.

Worsley, M.* & Blikstein, P. (2017). A Multimodal Analysis of Making. International Journal of Artificial Intelligence in Education, pp. 1-35. doi: 10.1007/s40593-017-0160-1.

Blikstein, P. & Worsley, M. (2016). Multimodal Learning Analytics and Education Data Mining: using computational technologies to measure complex learning tasks. Journal of Learning Analytics, 3(2), 220-238. PDF

Merceron, A., Blikstein, P., & Siemens, G. (2016). Learning analytics: From big data to meaningful data. Journal of Learning Analytics, 2 (3), pp. 4-8.

Worsley, M., Abrahamson, D., Blikstein, P., Grover, S., Schneider, B., & Tissenbaum, M. (2016). Situating multimodal learning analytics. In C.-K. Looi, J. L. Polman, U. Cress, & P. Reimann (Eds.), “Transforming learning, empowering learners,” Proceedings of the International Conference of the Learning Sciences (ICLS 2016) (Vol. 2, pp. 1346-1349). Singapore: International Society of the Learning Sciences. PDF

Schneider, B.* & Blikstein, P. (2015). Unraveling students’ interaction around a tangible interface using multimodal learning analytics. jEDM-Journal of Educational Data Mining, 7(3), 89-116.

Worsley, M. Scherer, S., Morency, L.P., & Blikstein, P. (2015). Exploring Behavior Representation for Learning Analytics. In Proceedings of the 2015 International Conference on Multimodal Interaction. ACM, New York, USA. pp. 251-258. PDF

Worsley, M. & Blikstein, P. (2015). Using Learning Analytics to Study Cognitive Disequilibrium in a Complex Learning Environments. In Proceedings of the 5th Annual Conference on Learning Analytics and Knowledge, ACM, New York, USA. pp. 426-247.

Worsley, M. & Blikstein, P. (2014). Deciphering the Practices and Affordances of Different Reasoning Strategies through Multimodal Learning Analytics. In Proceedings of the 2014 ACM workshop on Multimodal Learning Analytics Workshop and Grand Challenge (MLA ’14). ACM, New York, NY, USA. pp. 21-27. PDF

Blikstein, P., Worsley, M., Piech, C., Gibbons, A., Sahami, M., & Cooper, S. (2014). Programming Pluralism: Using Learning Analytics to Detect Patterns in Novices’ Learning of Computer Programming. International Journal of the Learning Sciences. Vol. 23, Iss. 4. pp. 561-599. PDF

Blikstein, P.**, Worsley, M.*, Piech, C.*, Sahami, M., Cooper, S., & Koller, D. (2014). Programming pluralism: Using learning analytics to detect patterns in novices’ learning of computer programming. Journal of the Learning Sciences, 23 (4), pp. 561-599. [45 citations on Google Scholar] PDF

Berland, M., Baker, R., & Blikstein, P. (2014) Learning analytics in constructivist, inquiry-based learning environments. Technology, Knowledge, and Learning, 19 (1-2), pp. 205-220. [60 citations on Google Scholar]

Worsley, M. & Blikstein, P. (2014). Using Multimodal Learning Analytics to Study Learning Mechanisms. In Proceedings of the 2014 Educational Data Mining Conference. pp. pp 431-432. PDF

Worsley, M. & Blikstein, P. (2014). Analyzing Engineering Design through the Lens of Computation. Journal of Learning Analytics. PDF

Worsley, M., &; Blikstein, P. (2013). Programming Pathways: A Technique for Analyzing Novice Programmers’ Learning Trajectories. In Artificial Intelligence in Education. Springer Berlin Heidelberg. 844-847. PDF

Gomes, J. Yassine, M., Worsley, M., & Blikstein, P. (2013) Analysing Engineering Expertise of High School Students Using Eye Tracking and Multimodal Learning Analytics. In Proceedings of the Educational Data Mining 2013 (EDM ’13). Memphis, TN, USA. 375-377. PDF

Worsley, M. & Blikstein, P. (2013). Toward the Development of Mulitmodal Action Based Assessment. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (LAK ’13). ACM, New York, NY, USA, 94-101. PDF

Worsley, M. & Blikstein, P. (2012). An Eye For Detail: Techniques For Using Eye Tracker Data to Explore Learning in Computer-Mediated Environments. In the Proceedings of the 2012 International Conference of the Learning Sciences (ICLS ’12). Sydney, Australia. 561-562. PDF

Worsley, M., Johnston, M. & Blikstein P. (2011). OpenGesture: a low cost authoring framework for gesture and speech based application development and learning analytics. In Proceedings of the 10th International Conference on Interaction Design and Children (IDC ’11). ACM, New York, NY, USA. 254-256. PDF

Worsley, M. & Blikstein P. (2011). What’s an Expert? Using learning analytics to identify emergent markers of expertise through automated speech, sentiment and sketch analysis. In Proceedings for the 4th Annual Conference on Educational Data Mining. Eindhoven, Netherlands. 235-240. PDF


Paulo Blikstein
Yipu Zheng
Engin Bumbacher


Marcelo Worsley (Northwestern University)
Bertrand Schneider (Harvard Graduate School of Education)
Mehran Sahami (Stanford University)
Steve Cooper (University of Nebraska–Lincoln)

Jesus Guzman

Cooper Lindsay


Google Faculty Research Award: “Using Learning Analytics to Detect Patterns in the Learning of Computer Programming”


For more information, please contact Yipu Zheng (