Building an evidence-base for teaching and learning design using learning analytics data
About the project
Learning analytics research uses large, anonymous sets of passively collected system data as an objective source of feedback on student interactions with online learning activities. Access to this kind of hard evidence allows teachers to understand the influence of different activities and therefore to design more effective and timely learning tasks.
The data is collected by e-learning systems as a matter of routine. However, extracting useful information from it requires a level of data literacy that many teachers do not have. Our project will use participatory action research case studies to develop ways to translate learning analytics data into useful information for tertiary teachers and learning designers.
This will enhance outcomes for learners through better design, and contribute to the knowledge base for effective tertiary teaching and learning in Aotearoa.
This project is in the process of producing a number of downloadable and shareable resources that you are welcome to use or adapt for your organisation. We look forward to bringing you these resources in the near future.
The six case studies presented in this report reflect the authentic experiences of early adopters of learning analytics practice in New Zealand tertiary institutions. They focus on key issues such as links between learning analytics and learning design, the need for policies and guidelines on ethical use of student data, user-friendly data access and reporting tools, and professional development for teachers. The case studies may be informative for anyone seeking to promote the benefits of learning analytics for teaching and learning design.
The Case Studies Report is an addition to Building an evidence-base for teaching and learning design using learning analytics: Project report by Cathy Gunn, Jenny McDonald, Claire Donald, John Milne & Marion Blumenstein (2017).
The aim of the research was to identify teaching and learning design questions that can be answered by the learning analytics data available through common e-learning systems. The case studies allows us to explore how the raw data required to answer these questions can be presented as accessible information for teachers.
The two-phase research project:
- Identifies the learning analytics data currently available through common learning management systems such as Moodle or BlackBoard, and e-learning tools such as peer and online assessment and tutorial dialogue systems.
- Generates and disseminates a taxonomy of analytics data to guide educators towards selection of data appropriate to the questions they want to answer or explore.
- Distinguishes which analytics data can illuminate the relationship between learning design, i.e. a teaching plan with intentions and assumptions about what students will learn; intermediate learning outcomes, i.e. learning strategies, engagement in activities and construction of new knowledge; and final learning outcomes, i.e. what students can demonstrate they have learned.
- Initiates sustainable changes in practice within the academic institutions represented by our research team and promote similar changes in others.
Design / Methodology
In the scoping phase of the study, we identified and categorised the learning analytics data collected by each type of system and used this as the basis to develop a taxonomy representing the range of data types collected, ways to extract meaning, and the permissions or restrictions that may apply to their use.
The project team liaised with senior institutional staff to negotiate access to data that is centrally hosted or managed. This initial contact was designed to encourage institutional buy-in to new practices involving the use of analytics data when later stages of the project produced evidence of benefits for learners and teachers.
In phase 2, we used case studies to explore how teachers interpret learning analytics data as feedback, and used the insights provided to develop or modify learning designs. Up to eight participatory action research case studies were selected from the participating institutions.
Teaching staff who use the e-learning tools identified in Phase 1 became co-investigators in Phase 2. This collaborative research approach supported the aim to drive sustainable change in teaching practice within participating institutions.
- A taxonomy of learning analytics data collected by common e-learning systems with guidelines for use of this information in teaching and learning design.
- Design frameworks, reusable digital resources including online tutorials and templates, and case study examples for the use of learning analytics data as an input to learning design.
- A series of interactive workshops on the extraction and use of learning analytics data for tertiary teachers and learning designers.
- Journal articles and conference papers describing the contribution of learning analytics data to teaching and course design, and to the field of learning technology research.
- Communication and dissemination strategies designed to encourage extensive use of the taxonomy and design information provided by learning analytics data.
- Policy recommendations to promote the use of learning analytics data in teaching and learning design.
Unlocking the Potential of Learning Analytics
A programme of seminars, workshops and panel discussions that were held during October 2016:
- University of Auckland on Tuesday 11th October
- Waikato University on Thursday 13th October
- Victoria University of Wellington on Friday 14th
- University of Canterbury on Tuesday 18th October
- Otago Polytechnic on Wednesday 19th October
For further details please visit:
- Download a workshop flyer
Pictured: Learning Analytics project team; John Milne (Massey University), Cathy Gunn, Claire Donald and Jenny McDonald (The University of Auckland) pictured with Ako Aotearoa Project Funds Manager, Rhonda Thomson, (far left) and Ako Aotearoa Director, Stanley Frielick (second from right), at the Victoria University of Wellington roadshow event.
Ako Aotearoa $150,000 (excl. GST)
The University of Auckland, Massey University, and The Open Polytechnic $154,931 (excl. GST)
Project commenced: January 2015
Project completion: October 2017
- Associate Professor Cathy Gunn (project co-leader), The University of Auckland
- Dr Claire Donald, The University of Auckland
- Dr Marion Blumenstein, The University of Auckland
- Dr Jenny McDonald (project co-leader), The University of Auckland
- John Milne, Massey University
- Dr Mark Nichols, The Open Polytechnic of New Zealand
- Gunn, C., Donald, M., Blumenstein, M., McDonald, J., & Milne, J. (2016). Learning Analytics: An Evidence Base for Teaching and Learning Design. Paper presented at the National Tertiary Learning and Teaching Conference, Rotorua.
- Gunn, C., Donald, C., Blumenstein, M., McDonald, J., & Milne, J. (2016). Workshop: Collect, Analyze, Act, Reflect: A method for using learning analytics in learning design. Presented at the National Tertiary Learning and Teaching Conference, Rotorua.
- McDonald, J., Blumenstein, M., Liu, D., Moskal, A., Pardo, A., Gunn, C., Milne, J. (2016). Cross-institutional Collaboration to Support Student Engagement: SRES Version 2. Paper presented at Ascilite 2016: Innovation, Practice and Research in the Use of Educational Technologies in Tertiary Education, Adelaide.
- Gunn, C., McDonald, J., & Donald, C. (2016). The Missing Link for Learning from Analytics. Paper presented at Ascilite 2016: Innovation, Practice and Research in the Use of Educational Technologies in Tertiary Education, Adelaide.
- Donald, C., Gunn, C., McDonald, J., Blumenstein, M., & Milne, J. (2016). Matching the rhythms of teaching to learning analytics. Paper presented at the International Consortium of Educational Developers University of Cape Town.
- Gunn, C., Donald, C., Blumenstein, M., & McDonald, J. (2016). Collect, Analyze, Act, Reflect: A framework for learning analytics professional development. Paper presented at the Australasian Learning Analytics Summer Institute, Adelaide.
- Blumenstein, M., McDonald, J., Moskal, A., Liu, D., & Leichtweis, S. (2016). Driving student engagement: SRES, two versions one purpose. Workshop presented at the Australasian Learning Analytics Summer Institute, Adelaide.
- Gunn, C. (2016). Using Scenarios to Promote Learning Analytics Practice for Teachers. Paper presented at ACODE 70: Analytics and Adaptive Learning and Teaching, Orange, NSW.
- McDonald, J. (2016). Learning analytics to explore teaching and learning discourse. Paper presented at ACODE 70: Analytics and Adaptive Learning and Teaching, Orange, NSW.
- Gunn, C., & Donald, C. (2015). Mapping analytics data to the learning design cycle to promote evidence-informed teaching practice. Paper presented at the Australasian Learning Analytics Summer Institute, Sydney University.
- Gunn, C., McDonald, J., Donald, C., Milne, J., Nichols, M., & Heinrich, E. (2015). A Practitioner's Guide to Learning Analytics. Paper presented at Ascilite 2015: Globally connected, digitally enabled, Perth, WA.
- Gunn, C., McDonald, J., Donald, C., & Milne, J. (2015). Are we ready to learn from learning analytics? Paper presented at the ALT-C, Manchester, UK.
The report from this project is published under the Creative Commons 3.0 New Zealand Attribution Non-commercial Share Alike Licence (BY-NC-SA/3.0-NZ). Under this licence you are free to copy, distribute, display and perform the work as well as to remix, tweak, and build upon this work non-commercially, as long as you credit the author/s and license your new creations under the identical terms.