Ako Aotearoa Regional Hub Project Fund Evaluation
A summary of the Formative Evaluation of the Ako Aotearoa Regional Hub Project Fund and our responses to the formative recommendations.
Introduction
The consultancy firm Networkers was contracted to undertake an independent evaluation of the operation and impact of the Ako Aotearoa Regional Hub Project Fund.
Presented on this page are an abridged version of the Networkers’ executive summary of the evaluation and their recommendations in full, plus our response to them. The full report is also available below.
Executive Summary
The fund is increasingly popular and well-used. The range of projects, and reasons for applying, is very broad. Projects range from small-scale interventions in a single course, to large networking projects
Applicants rated the process highly, and felt strongly supported by Ako Aotearoa throughout the project and beyond. The funding was seen as relatively simple to apply for and quite accessible to beginning researchers. Publication of project reports on the website was seen as a particular strength. The $10,000 limit was seen as a strength by some and a limitation by others, but did tend to provide focus.
The outcomes analysis was at the heart of this study. With such a range of different projects, it is difficult to quantify at this point the size and nature of both intended and unintended effects and impacts. A series of case studies is provided to illustrate aspects of impact. Many of the projects have led directly to changes in the status and work of individuals and project teams, as well as permanent developments in the process, delivery and/or content of tertiary courses.
As a result of these projects, some teachers are now better prepared and acculturated for the classroom, a range of mentors and supervisors assist students with course or field work, online and information technology systems are hugely improved, there is more inter-organisational collaboration and a research culture is being established for the first time in some organisations. Evidence of such impacts is provided.
This evaluation found it difficult to understand expectations around the fund in relation to Ako Aotearoa’s role as a treaty-based organisation. There are several projects with a Māori and/or Pacific basis, but little collaboration or partnership between organisations around culture. Ako Aotearoa should consider what kind of collaborations might build capacity in this area and encourage them through the fund.
The view of the evaluation team is that the hub project fund is effective. In funding terms, the projects are relatively small, but the commitment of Ako Aotearoa staff, the dedication and enthusiasm of fund recipients and research partners, and the determination of all parties to improve tertiary teaching and learning means there is a demonstrable short-term impact that has the potential to grow over time.
Formative recommendations
1. That Ako Aotearoa investigate ways, other than hub co-ordinators, to support new and emerging researchers.
Ako Aotearoa response: This will be dealt with on a case-by-case basis. Project teams with less experience are required to include someone who takes on a mentoring role. Hubs also provide this role in some aspects of projects. It was noted that some identified mentors have, in reality, little input into projects. Hubs need to be aware of this as they undertake their monitoring function and take remedial action as necessary. Hubs will collaborate to review the Frequently Asked Question (FAQ) page on the web-site regularly and collectively add to this resource.
2. That Ako Aotearoa ensure that hub funding is adequate to support each project that is approved, possibly by the development of a good funding ‘checklist’ to accompany or replace the budget calculator, and detailed advice from hub co-ordinators or project ‘buddies’ on likely costs.
This recommendation is covered by current documentation and advice available from the Hubs. It is noted that, as a result of organisational sign-off of project proposals, there is an expectation internal budget decisions within organisations reflect the proposed budget.
3. That Ako Aotearoa ensures that the guidelines for funding and reporting are clear and comprehensive. Areas noted in case studies as unclear include what staffing costs are allowable and additional information on the final reporting process (the FAQ section could also be used).
Hubs will monitor this and deal with it on a case-by-case basis. It is noted that, in general, the monitoring function by the hubs may need to be strengthened to avoid too many time slippages in projects.
4. That each hub run a one-day research colloquium for current (and possibly former) fund recipients each year, to encourage cross-sectoral sharing of information and skills.
It was noted that informal events have already been run. Hubs will explore this option further, canvassing the views of project teams and Hub advisory groups.
5. Ako Aotearoa note the wide range of impacts described by organisations, and consider further ways, as necessary, to support project teams to produce such impacts.
The examples of impacts of project work identified in the report are encouraging, but information about impact clearly needs to be more systematically collected within an organisation. The hubs will work in collaboration with national office to develop this framework, apply it as soon as possible to new projects and look at ways of retrofitting the data collection strategy to completed projects. We agree that this should be a priority.
6. In order to assist further evaluations, including self-evaluation of projects, current and future fund recipients should be asked to produce a document which charts potential impacts at the beginning of a project, actual at the end and then subsequent effects over an agreed period of (say) two years. The diary would discuss intended and unintended impacts of the project, subsequent project work and any transformative effects, and would attempt to gauge the scope of the impact.
Recommendations here will be taken into consideration in work planned under recommendation 5. The work currently being undertaken through national office, to provide a summary guide to practitioners about collecting evidence of learner benefits, will also be useful in this context.
7. The hub project fund policy and guidelines should clearly outline the organisation’s intention to support treaty-based projects and partnerships, and the implications of this for individual projects supported by the fund.
Hubs will work on developing policy and guidelines in this area, using those of the Māori Initiative Project stream of theNational Project Fund as a guide.
8. Ako Aotearoa should examine how to ensure that a focus on Māori and Pacific students, teachers, knowledge systems, organisations and partnerships can be more effectively integrated into all relevant projects.
This will be picked up alongside actions planned for recommendation 7 also utilising approaches being developed for our Pacific people's Project Funding stream in the National Project Fund.
9. That selection criteria be developed, agreed and published for the hub project fund, that are available to be utilised should demand for funding begin to exceed the number of eligible project applications.
Current projections for new projects for this year do not reflect the concerns in the report with respect to this recommendation. We will work on this if the demand for project support necessitates making the Regional Hub Project Fund a competitive fund in the future. It is our intention to keep this fund as an open fund for the foreseeable future.
10. A summative evaluation should be undertaken in about two years' time, which revisits a sample of projects and considers the quantitative impact and effects at that time via an in-depth survey.
Agreed. We plan to commission a summative evaluation on this fund in the second quarter of 2012.