A tertiary practitioner's guide to collecting evidence of learner benefit
This report provides an introduction to the process of collecting evidence of learner benefit, this report is designed to support professional, reflective practice.
Report prepared by Anne Alkema for Ako Aotearoa
You can download this publication as a pdf or purchase the print version from the Ako Aotearoa shop.
A complementary publication, Creating sustainable change to improve outcomes for tertiary learners, published in 2012 supports project teams to plan and implement tertiary teaching and learning projects.
Ako Aotearoa’s vision is the best possible educational outcomes for all learners. Achieving this vision has many dimensions, but a critical part is to develop more effective teaching and learning. This is an increasing challenge when, on the one hand, we are operating in times of inevitable resource constraints and on the other we are continually being urged to engage with new social and learning technologies to extend our practice.
Immediately this raises a question: “how do we know what is effective practice?” How do we test our assumptions about what is working for our learners and what is not? On the surface this is an obvious and fundamental question, but across the sector we have been surprised at the relatively small number of practitioners who systematically gather and use evidence of how learners do or do not benefit from different approaches to teaching.
We have therefore commissioned this guide as an introduction to the process of collecting evidence of learner benefit. This publication is not about researching tertiary teaching and learning: it is about supporting professional, reflective practice. Of course, the line between these two is somewhat blurred, and practitioners often need to think like researchers to ensure their practices are based on meaningful information. The body of this publication is firmly focussed on data gathering (and deriving information from that data) as part of the educative process. In the Appendix to this publication some strengths and weaknesses of
possible data sources are then explored in a way that is more ‘research-focussed’ than the rest of the material.
At the heart this is our view that being a professional – a core part of the everyday work of being an education practitioner in the tertiary sector – means having a commitment to using evidence to understand the learners we teach, train, or supervise, and using that evidence to ensure they achieve the best possible outcomes. Successful tertiary teaching can no longer be just about high student satisfaction scores and the experienced teacher gaining positive reinforcement from a class that seems to be accepting of their presence and their delivery. It is about maximising opportunities for learning and ensuring those opportunities are taken advantage of by as many learners as possible.
Having evidence allows us to inform and adapt teaching practices. But in addition, the evidence collection process itself can have benefits for both practitioners and learners. Reviewing one’s practice in the terms set out here more often than not involves discussion with students about their learning preferences and how they are responding to one’s teaching. These discussions break down some of the inevitable power relationships between teacher and learner and, handled professionally, foster engagement and build relationships of confidence and trust: mutual learning that embodies the concept of ako.
I would like to thank Anne Alkema for her preparation of this guide, and the reviewers who commented on earlier drafts. This document is by no means the last word in collecting evidence of learner benefit, and we aim to produce further publications exploring specific aspects of the topic in more depth. We do hope, however, that this guide provides a valuable starting point for a growing discussion on the importance of evidence-based teaching, and that it supports practitioners to start integrating such evidence collection into their day-to-day practice.
Dr Peter Coolbear
Director, Ako Aotearoa
This publication is not about researching tertiary teaching and learning: it is about supporting professional, reflective practice.
Why does it matter whether or not teachers use evidence to inform their practice?1 Simply because research shows that teaching has a substantial impact on students’ retention and engagement in tertiary organisations (Zepke, Leach & Butler, 2010). Therefore, one of the most fundamental questions practitioners can ask themselves is:
How do I know that what I am doing is working and making a difference to my students’ learning?
To provide evidence that will help to answer this question, you need to collect a range of data2 from different sources and/or in different ways. This is a high-level ‘how to’ guide intended to help you start to gather and use data to help determine how well what you are doing is supporting and improving your students’ learning. It is designed to provide some initial advice
on how you can gather and use data to reflect on and examine the practices that happen every day in tertiary settings.
The guide shows examples of the types of data you may have available or can collect, and provides suggestions about types of data to collect, how to collect them, and how to think about the evidence you’ve collected. It discusses how you can use the conclusions drawn from the data to find out more about the impact of what you are doing, and then use this information to refine and improve your practice.
The guide does not provide information on how to analyse data. Nor does it discuss in detail specific research methodologies, data analysis, or principles of research design.
The purpose of this guide is to introduce readers to the idea of collecting data that can be used to inform teaching practices. It is not a ‘how to’ research guide – there are already many excellent examples of these available, and we expect that many readers will be experienced researchers themselves.
We believe it is important to recognise the context in which tertiary teachers, trainers, lecturers and supervisors work, and the many demands on their time. In this environment, some may feel there is not enough time in which to collect data and use them to inform practice. This is why the report focuses on discussing the ‘data collection’ side of using evidence to inform practice. What we want to stress is that collecting evidence to support good practice does not require that practitioners conduct an extensive, detailed research project. It is of course important to gather evidence in a robust and systematic way, but the processes for doing so can be integrated into your practice. Rather, using evidence to think critically about how you approach teaching actually has the potential to reduce your own workload, as well as ensuring good outcomes for your students.
Data collection should be part of a professional’s everyday practice. Example One below illustrates how a tutor brings data together systematically to provide greater confidence that a newly implemented approach is impacting positively on outcomes for the students on their course. Throughout this report there are similar brief examples or ‘vignettes’ that illustrate other aspects of using data to inform tertiary teaching practice.
Example One: Using evidence to inform practice
Alex has been teaching his workplace horticulture students in a combination of face-to-face and distance learning settings for three years. During this time his students have not been completing the formative assessment aspects of the course. Alex decides he needs to trial a new way of getting students to do this. To meet course requirements Alex sets formative assessment tasks that require his students to complete five e-postings about the practical work they are doing that is related to their study.
In order to find out whether this exercise has made a difference to his students’ knowledge and skills, Alex includes questions about the e-postings in his post-course evaluation survey. He includes a question about whether and how they felt the postings helped the learning they had done and the practical horticulture work they were doing. As this was the only new intervention Alex had introduced to the course, he compared this cohort’s final results with
the results of cohorts from the previous three years.
Alex uses the information from the survey to refine the exercise for his next group of students, and the information from the comparative data to give him some confidence that the intervention could be making a difference to the outcomes for his learners.
- ‘Tertiary practitioners’ include all those people involved with teaching, learning and support processes in public and private tertiary organisations. The term includes practitioners involved with higher education, vocational education, workplace learning, and community-based learning.
- For the purposes of this guide, the term ‘data’ refers to any systematically collected information.
This work is published under the Creative Commons 3.0 New Zealand Attribution Non-commercial Share Alike Licence (BY-NC-SA). Under this licence you are free to copy, distribute, display and perform the work as well as to remix, tweak, and build upon this work noncommercially, as long as you credit the author/s and license your new creations under the identical terms.