ITO Workplace Assessment Structures and Systems: Survey and Focus Group Findings

Report prepared for the Industry Training Federation Research Network

Authors - Karen Vaughan and Marie Cameron, New Zealand Council for Educational Research

March 2010

 Also see the Background Paper

Executive summary

The ITO Workplace Assessment Structures and Systems project

This is a report on findings from a survey of 33 Industry Training Organisations (ITOs) and five focus groups with 19 ITO representatives – the second phase of a three-phase project entitled ITO Workplace Assessment Structures and Systems.

The overall purpose of the project is to explore the different models of workplace assessment used around the world and by ITOs in New Zealand, and to consider the kinds of arrangements that are, and could be, used to support workplace assessors. We first produced a paper based on a targeted review of the most relevant literature on workplace assessment structures and systems in the context of understanding the roles and purposes of assessment and workplace learning (Vaughan & Cameron, 2009). That paper informed the design of the survey and focus groups. The findings from these activities will in turn inform the final phase—focus groups with ITOs and the production of a guide to assist ITOs to think about how they could make their workplace assessment structures more robust, more effective and promoting of high-quality learning.

Key findings

Our key findings are based on an analysis of the perspectives gathered through the survey in which 33 ITOs participated, and the focus groups in which 16 ITO staff and three assessors participated. The findings highlight that ITOs have different understandings or practices related to apparently common terms across ITOs. They also highlight the way in which systems for assessor recruiting, training and managing impact upon the actual practice (and quality) of assessment. Our final suggestions for improving structures and systems are based on this point: that systems exist to support practices, and that systems issues (such as supporting assessor development) cannot be sorted out separately from practice issues (such as whether assessment is a compliance activity or a learning investment).

Assessor recruitment

ITOs have trouble finding enough assessors with the right attributes and then supporting them. It seems that assessors have a range of different motivations for wanting to become assessors—from being nominated by supervisors, to wanting the status of being an assessor, to seeking the position as a form of professional and career development. These motivations impact on how well suited assessors are to the job and on how well they can be supported to do a good job, as do ITO recruitment and management practices which range from training managers hand-picking assessor candidates to training all-comers.

Assessor training

ITO training policies and practices also impact upon assessors’ ability to do a good job and ITO capacity to support them. Most ITOs accredit assessors after successful completion of training to Unit Standard 4098 (Use Standards to Assess Candidate Performance). Few ITOs have any requirements around further professional development for assessors, except participation in moderation activities which is sometimes considered a professional development activity. Some ITOs have questioned the adequacy and relevance of Unit Standard 4098 for their particular industry and prefer to arrange their own assessor training. ITOs also appear to have few policies in place to manage the scope of assessor accreditation. In some cases this has resulted in a general overabundance of assessors, or too many assessors accredited in particular fields or accredited to assess a far wider range of standards than they regularly have cause to assess in reality. This can leave ITOs scrambling to address issues of quality at the moderation stage. Some ITOs are attempting to address this with more support materials for assessors and policies to limit assessor numbers and accreditation scope. This would allow them to better manage their own resources, improve some aspects of quality and better support the assessors.

Moderation

ITOs reported several approaches to, and understandings of, moderation. In the main, moderation involved pre-moderation of assessment tasks to ensure that assessments were assessing the right things and were connecting training goals to assessment, and postmoderation of assessments. Some ITOs considered that more work was required to enhance the “fit” between unit standards and their assessment. In some cases moderation was understood to be about ensuring that specific standards have been consistently and reliably assessed. In other cases it was understood to be about checking the assessment process.

Verification

There were mixed views about the use of workplace verifiers in checking performances that could not be observed in a single assessor visit or when it was not practical for an external person to do so. Some unit standards can only be observed when an opportunity presents itself, so verifiers are needed to document the achievement of these standards. Verifiers are often in a position to provide corrective and supportive feedback to trainees. People considered that quality assurance is stronger when verifiers also are trained to understand how the separate unit standards contribute to the qualification and are given the time and opportunity to assess. Without some investment in verifiers, the quality of their judgements was seen to be variable, and frequently based on a cursory tick-off approach. Although some ITOs were not in favour of verifiers, others believed that they were a valuable source of evidence if they were trained and supported in the workplace. However, the relationship between assessors and verifiers seems to vary widely and sometimes seemed unclear, even to ITO staff.

Some general suggestions for improving assessment structures and systems

All structures and systems are a means to an end. In this case structures and systems need to be understood as being for the purposes of supporting assessment, which in turn should support learning in the workplace. The scope of this project—focusing only on structures and systems and not assessment practice—means
that we cannot know what actually happens with assessment, and therefore how well it is actually supported by the structures and systems. It does seem, however, that there is an overall tussle between ITO staff who take a technical view of assessment as being about compliance and something that anyone (who knows the industry knowledge content) can do, and ITO staff who take an educational view of assessment and see it as deeply connected to learning and requiring a high-quality investment.

Although the research is limited to assessment structures and systems (and leaves out practices), we make several suggestions for ITOs to consider under the next four sub-headings.

Target investment in assessors more tightly

While we do not know ITOs’ reasons for having a lot of workplace-based assessors, it is likely to be a result of legacy systems that have supported a proliferation of assessors and widening of accreditation scope. This is unmanageable for many ITOs and the ways they deal with this are discussed in the report. We suggest that reducing the assessor pool to manageable levels makes it more feasible to provide the training and ongoing professional development required or desired to develop and maintain assessor competency levels. Investing in assessors is investing in the skills base of the industry. A wise investment in assessors means that they will have more to offer the industry as a whole.

Develop career pathways for assessors

It is worth considering the development of a career pathway within the industry for assessors. Assessors have or will have specialised content knowledge as well as higher level skills that they learn through well-designed and implemented assessor training. This combination of industry knowledge and skill, together with  assessment knowledge and skill, could be more formally recognised. Perhaps it should be harder to become an assessor, and once this investment has been made, assessors should be valued and supported because they have a role to play in building a high-capability workforce. But there needs to be an understanding of the ways in which learning and assessment and teaching are all interlinked. A focus on assessment only is attending to only one part of building workplace capability.

See assessment as part of an infrastructure for workplace development

This report shows that attending to the selection and training of assessors, while important, is only part of the challenge of developing workplace capability. Assessment and moderation provide information such as whether trainees are achieving standards, and which standards are proving more difficult to assess. They
can also be used as part of the evidence base for how workplaces are building capability over time. Ensuring that workplace structures and processes support trainees, assessors and moderators to get better at what they do appears to be a route to enhancing longer term organisational goals, including quality control, worker motivation and retention and organisational success.

Address gaps in knowledge about assessment in the workplace

While this research has given us a picture of current ITO workplace assessment structures and issues, it has also revealed what we do not know, and what we need to know, in order to improve those structures and systems. This research is the beginning of mapping the knowledge about how assessment works in ITOs. It is a useful start, and it has generated areas for further research that are needed before a comprehensive picture of assessment will be generated. It is based on the perceptions of key industry stakeholders, and has revealed the need to include perspectives of learners and a wider range of assessors before we could have confidence in the trustworthiness of our findings. It would seem essential to focus more deeply on a sample of workplaces—taking a careful look at how assessment structures and practices support or constrain the completion of qualifications would be a valuable first step. This sort of research would require that researchers spend time on site observing, analysing documentation and talking to trainees, trainers, assessors and anyone else who contributes to assessment  decisions. It might involve several visits over the course of a trainee’s progression through a qualification to obtain a deeper understanding of the impact of training and assessment on their achievement and motivation to learn and contribute to their industry. When there is a clearer picture of what works for learners, the implications for workplace assessment structures and what assessors need to do their jobs well will also be clearer.