A tertiary practitioner's guide to collecting evidence of learner benefit

Report prepared by Anne Alkema for Ako Aotearoa

January 2011

Preface

Ako Aotearoa’s vision is the best possible educational outcomes for all learners. Achieving this vision has many dimensions, but a critical part is to develop more effective teaching and learning. This is an increasing challenge when, on the one hand, we are operating in times of inevitable resource constraints and on the other we are continually being urged to engage with new social and learning technologies to extend our practice.

Immediately this raises a question: “how do we know what is effective practice?” How do we test our assumptions about what is working for our learners and what is not? On the surface this is an obvious and fundamental question, but across the sector we have been surprised at the relatively small number of practitioners who systematically gather and use evidence of how learners do or do not benefit from different approaches to teaching.

We have therefore commissioned this guide as an introduction to the process of collecting evidence of learner benefit. This publication is not about researching tertiary teaching and learning: it is about supporting professional, reflective practice. Of course, the line between these two is somewhat blurred, and practitioners often need to think like researchers to ensure their practices are based on meaningful information. The body of this publication is firmly focussed on data gathering (and deriving information from that data) as part of the educative process. In the Appendix to this publication some strengths and weaknesses of
possible data sources are then explored in a way that is more ‘research-focussed’ than the rest of the material.

At the heart this is our view that being a professional – a core part of the everyday work of  being an education practitioner in the tertiary sector – means having a commitment to using evidence to understand the learners we teach, train, or supervise, and using that evidence to ensure they achieve the best possible outcomes. Successful tertiary teaching can no longer be just about high student satisfaction scores and the experienced teacher gaining positive reinforcement from a class that seems to be accepting of their presence and their delivery. It is about maximising opportunities for learning and ensuring those opportunities are taken advantage of by as many learners as possible.

Having evidence allows us to inform and adapt teaching practices. But in addition, the evidence collection process itself can have benefits for both practitioners and learners. Reviewing one’s practice in the terms set out here more often than not involves discussion with students about their learning preferences and how they are responding to one’s teaching. These discussions break down some of the inevitable power relationships between teacher and learner and, handled professionally, foster engagement and build relationships of confidence and trust: mutual learning that embodies the concept of ako.

I would like to thank Anne Alkema for her preparation of this guide, and the reviewers who commented on earlier drafts. This document is by no means the last word in collecting evidence of learner benefit, and we aim to produce further publications exploring specific aspects of the topic in more depth. We do hope, however, that this guide provides a valuable starting point for a growing discussion on the importance of evidence-based teaching, and that it supports practitioners to start integrating such evidence collection into their day-to-day practice.

Dr Peter Coolbear
Director, Ako Aotearoa

Introduction

Why does it matter whether or not teachers use evidence to inform their practice?¹ Simply because research shows that teaching has a substantial impact on students’ retention and engagement in tertiary organisations (Zepke, Leach & Butler, 2010). Therefore, one of the most fundamental questions practitioners can ask themselves is:

How do I know that what I am doing is working and making a difference to my students’ learning?

To provide evidence that will help to answer this question, you need to collect a range of data² from different sources and/or in different ways. This is a high-level ‘how to’ guide intended to help you start to gather and use data to help determine how well what you are doing is supporting and improving your students’ learning. It is designed to provide some initial advice
on how you can gather and use data to reflect on and examine the practices that happen every day in tertiary settings.

The guide shows examples of the types of data you may have available or can collect, and provides suggestions about types of data to collect, how to collect them, and how to think about the evidence you’ve collected. It discusses how you can use the conclusions drawn from the data to find out more about the impact of what you are doing, and then use this information to refine and improve your practice.

The guide does not provide information on how to analyse data. Nor does it discuss in detail specific research methodologies, data analysis, or principles of research design.

The purpose of this guide is to introduce readers to the idea of collecting data that can be used to inform teaching practices. It is not a ‘how to’ research guide – there are already many excellent examples of these available, and we expect that many readers will be experienced researchers themselves.

We believe it is important to recognise the context in which tertiary teachers, trainers, lecturers and supervisors work, and the many demands on their time. In this environment, some may feel there is not enough time in which to collect data and use them to inform practice. This is why the report focuses on discussing the ‘data collection’ side of using evidence to inform practice. What we want to stress is that collecting evidence to support good practice does not require that practitioners conduct an extensive, detailed research project. It is of course important to gather evidence in a robust and systematic way, but the processes for doing so can be integrated into your practice. Rather, using evidence to think critically about how you approach teaching actually has the potential to reduce your own workload, as well as ensuring good outcomes for your students.

Data collection should be part of a professional’s everyday practice. Example One below illustrates how a tutor brings data together systematically to provide greater confidence that a newly implemented approach is impacting positively on outcomes for the students on their course. Throughout this report there are similar brief examples or ‘vignettes’ that illustrate other aspects of using data to inform tertiary teaching practice.

1. ‘Tertiary practitioners’ include all those people involved with teaching, learning and support processes in public and private tertiary organisations. The term includes practitioners involved with higher education, vocational education, workplace learning, and community-based learning.
2. For the purposes of this guide, the term ‘data’ refers to any systematically collected information.

  This work is published under the Creative Commons 3.0 New Zealand Attribution Non-commercial Share Alike Licence (BY-NC-SA). Under this licence you are free to copy, distribute, display and perform the work as well as to remix, tweak, and build upon this work noncommercially, as long as you credit the author/s and license your new creations under the identical terms.