Integrated assessments – Engaging ways to enhance learner outcomes

Author

Helen McPhun, McZoom Ltd Consultancy and LEARNPLUS™ PTE.

Whether assessments are based on NZQA unit standards or on other academic topics, this paper will discuss the merits and demonstrated successes of using integrated assessments. Throughout the process learners have opportunities of relevance and are actively engaged.

This approach, which integrates the assessment of multiple topics and processes in realistic scenarios provides a bridge connecting the learner, higher education, and the workplace (Brown, cited in Mandell & Michelson, 2004, p. 2).

Introduction

Integrated assessments provide an engaging and creative learning platform that closely links to the realities graduates will experience in the work force: a process that combines and blends the learning outcomes from multiple topics into a series of streamlined, realistic, employment focused activities. These assessments are conducted over a period of time with numerous formative and summative components. They demonstrate effective ways to synthesize topics into a coherent and contextualised framework using complementary skill and knowledge sets. Assessments no longer take on the feared final exams but instead naturally occur throughout the programme, allowing learners to operate their new found knowledge and skill. This assessment model tips the traditional model of “topic– teach–assess–graduate” on its head. The normal emphasis on the assessment of declarative knowledge and “synthetic” skills is replaced by measures of declarative and procedural knowledge blended into seamless assessment components that occur naturally. This creates experienced graduates ready for the workplace.

The Impetus for an Integrated Assessment Model

I had noticed top graduates, as new employees, often demonstrated difficulties when faced with multiple demands. They struggled in real situations where, in addition to the task at hand, they had to consider timelines, human relations, fiscal restraint, and business productivity. Without the security of a text-book answer or a straightforward process chart they lacked the experience and flexibility to juggle multiple skill and knowledge requirements in efficient and synthesised ways. Around the world this has been written about, with employers commenting on new graduates they have hired – “It’s as if they’re going into school again. They are not prepared at all” (Beirut-Daily-Star, 2007).

Many countries, including Switzerland, Lebanon, and Canada, put emphasis on internships or cooperative education placements to allow students to apply their knowledge in real workplaces as a critical learning component of their academic programme. In my teaching I needed to make the learning real, to bring life to it, to allow the learners not just to know the content of the course or recite the list of skills needed in the field but to experience the feelings, frustrations, successes, and processes that occur when knowledge and skills are merged into the most realistic processes possible. After all, what do graduate results made up of high exam scores really tell us about a student’s readiness to put knowledge into practice in creative ways (Lombardi, 2008)?

It seemed graduates could excel if they had to replicate textbook processes and recite textbook theory but when those items were reconfigured to adapt to the unpredictabilities of the workplace they could not cope well. They appeared operationally challenged. “Society and students are demanding more from higher education: achieving competence, moving from autonomy to independence, establishing identity, purpose and integrity and mature interpersonal relations” (Hattie, 2009).

Integrated assessments aid the movement from static shallow learning to a rich and compelling environment for living learning (Lombardi 2008). Through this process learners can master the rhythm of learning, applying, reflecting, and growth, a rhythm that goes on through life. “Learners are given the opportunity to see larger systems and forces at play and have to construct public, testable ways to express their learning” (Kleiner, Roberts et al., 1994 p. 59).

Overview of the Design Process

The design of integrated assessments, I believe, requires the assessment writer/s to combine algorithmic, routine-based thinking with heuristic thinking that allows conceptual freedom (Malnarich & Decker-Lardner, 2003). Reversing the traditional assessment design sequence (Figure 1) from a competency framework that Grzeda (2005) says is fraught with conceptual ambiguity to an integrated model that allows a springboard for qualitative change and open ended development? Learners work on multiple objectives linked to real tasks and construct and perform qualitatively different work than traditional testing. “Through this cognitive reconstruction learning takes place” (Friedrich & Lantz, 2003).

Figure 1 – Traditional Assessment Design Sequence

The design model for integrated assessments (Figure 2) starts by considering the types of jobs the learners will get with the qualification they are pursuing. Once these are identified, questions to consider include: 

• What skills will they have to perform?
• What contexts will they work in?
• What knowledge will they have to demonstrate?
• How will they have to demonstrate it?
• What attributes will employers be seeking?

With a firm understanding of real-world graduate requirements the topics or Unit Standards in the programme need to be reviewed. Identifying common themes or skills, complementary knowledge or performance outcomes and synergistic opportunities is important. Understanding how the topics relate and the natural sequence of events in the workplace will help guide the cumulative knowledge and skills required by graduates.

The next stage in the design process is to examine and combine topics and tasks to typify workplace realities. This will provide a deeper and more meaningful learning experience.

“The validity of simulations depends very much on how accurately the simulation mirrors real life situations” (Booth, Clayton, House & Roy, 2002).

Creativity and ‘outside the box thinking’ need to be implemented to assess as much as possible through integrating the common needs of a variety of topics. It is prudent to involve others to get as many ideas as possible, and assessment designers need to acknowledge their own limitations and strengths. Traditional assessment design models are often clung to as it is easier to stay within a comfort zone where growth and opportunity rarely occur. “Faculty resistance is present even where extraordinary levels of faculty support are made possible” (Lombardi, 2008).

By tying together multiple formative and summative assessment activities in a synthesized, coherent and fluid assessment package the learners will be exposed to a more credible experience. The assessments need a well-planned set of documents with easy instructions and straightforward language. Mechanisms should be included that allow learners to know their progress, as the final assessment result should not be a surprise.

Assessment activities should be as realistic as possible, broken into logical and straightforward steps, inclusive of measurable results along the way and, most all, the assessment process should be fun. Learning is best demonstrated in natural and contextualised situations.

Once the assessment tools are designed and pass through a quality assurance process they are ready to be piloted.

This integrated assessment design model focuses on graduate success (Figure 2).

Figure 2: Integrated Assessment Design Sequence

Transferring learning from the tertiary arena to practical application at work is not always easy but integrated assessments can help the process considerably.

Implementation of Assessment Practices

In 2004, I trialled this approach for one term where I transformed the business class of learners into a company of employees charged with a project mission that underpinned every activity in the term.

The Unit Standards included were:
7452 – Plan, organise, and allocate work to achieve
9675 – Facilitate a group and/or team to make decisions
9678 – Conduct formal meetings
9692 – Present information orally to an audience
11098 – Listen and respond to information received.

Two days each week the learners used the classroom as the boardroom/office while teams worked to develop and manage an innovative project. Formal meetings were held with documented minutes and a rotating role of chairperson. Benchmarks and timelines were set.

Roles were delegated. Gantt charts and critical path analysis tasks were completed both in groups and individually. Formal presentations were made to sell each group’s concept, and the winning project was selected. That winning team subcontracted the other teams to complete the planning stages. The project goal was then reached and all learners participated in the day away project outcome. A comprehensive post project evaluation process was used in the last week.

Assessment of learning was conducted throughout the term from multiple sources including lecturer observation, documentary evidence, panel feedback, and peer- and self-assessment. With on-going assessment of theoretical and procedural knowledge learners saw their topics realistically represented. Assessment results had validity due to the broad source of results and – perhaps the most insightful source – the learners themselves.

The results from this mini-pilot project were gathered from learner feedback, post-course surveys, and weekly peer and self assessments from the 10 students in the project. They demonstrated clearly that 90% of the learners were both motivated and fully engaged throughout the term. With one exception, all learners were successful and the group work served as practice and fuelled success for subsequent individual tasks.

More recently, in 2009, 8 learners pursuing a National Certificate in Adult Education and Training were similarly assessed. Their classes were modularised to include blended topics and interactive group work related to real training issues in their workplace. The sessions were delivered in-house in a corporate environment and their assessment was a portfolio of integrated activities to solve or enhance real work practices.

The Unit Standards covered in the assessment included:
7103 – Determine the adult training requirements of individuals
7093 – Design learning sessions for adult education and training
7115 – Create and maintain a positive learning environment for adult learners
19444 – Deliver group training sessions to adults
7091 – Establish a culturally safe and inclusive learning environment for adults in NZ’s cultural setting

The requirements of the Unit Standards were merged throughout the assessment activities. For example, rather than doing the Unit Standard on cultural diversity as a standalone option those principles were applied to the stages of analysis, design and delivery, with sources of evidence available at all points.

Learners were not assessed on each topic in isolation from the other. Instead, they designed and delivered sessions to meet specific learner requirements and cultural needs. Formative and summative assessment tasks were intertwined through the work the learners did.

I conducted post-training evaluations by an online anonymous survey – Survey Monkey.

Post-Programme Analysis and Learner Evaluation

One of the learners in 2004 gave these valuable insights “The way we did this whole term made it all make sense. I’m ready now and can’t wait”. Learners in 2009 commented, “I took away at lot of valuable tools that I feel I can implement immediately in my business”, and “I applied this approach on the opening workshop of our 2009 leadership development program, and seemingly got a very high engagement and an excited group of course participants (20) who are now all keen to attend the next workshop in Aug 09”.

In their feedback, all 18 learners from the two cohorts emphasised how they valued the practicality of the course design as it allowed the assessments to flow from activities that were purposeful, engaging, and fun. They had learned a lot about working with others, about handling multiple responsibilities, and about the impact of their performance on others. Learners often regard those insights or skills that were not part of the declared educational agenda as some of the most significant gains or outcomes of their learning (Brookfield, 1990).

One learner in the first cohort was not deemed competent, and his self-assessment identified he would not pass. Incomplete tasks that had been delegated to him by his team and the peer assessments he received confirmed his lack of performance. He had proved to himself that “some amount of opportunity, motivation and ability must be present for performance to be achieved” (Bramley, 2003).

Feedback from Graduates and Employees

Recent research allowed me to connect with 60% of the 2004 attendees by email, phone, and face to face; and in a quick conversational survey they each indicated their learning was meaningful, memorable and of tremendous value.

Their supporting examples showed:

• they felt prepared for multiple tasks and deadlines and knew they had successfully achieved these before in class,
• they felt experienced in team dynamics and entered into group work projects not expecting all things to go smoothly,
• they understood that some team members might not pull their weight or meet deadlines, and
• they knew their own weaknesses in these situations and felt aware of their challenges.

I was able to speak by phone to 60% of employers of those who responded. Each indicated their satisfaction with the job readiness of the graduates and commented on the graduates’ abilities to quickly understand and apply some of the workplace processes. One comment was. “I don’t know how they learned in their programme but he seemed able to cope well. He knew he had to juggle lots of things and that was a bonus as we are busy here. He understood that. He also wasn’t afraid to ask if he got stuck on the really difficult things.”

One employer had filled two employment vacancies in the time-share accommodation business with two graduates from the 2004 cohort. He was able to see they had worked out how to complement each other’s skills at work – he could assign a specific multi-faceted task to them and know they would work out between them ‘who would do what’ and that the deadlines and targets would be met. While the graduates did not really possess job maturity at that stage, I believe they did demonstrate a level of professional maturity that was above that seen in usual graduates.

The manager of the learners from the 2009 cohort was encouraged by the approach taken by 7 out of 8 learners (one learner moved to another job). It had been her experience that for some staff applying learning after the course was usually optional and training had often been viewed as a day off work. The assessments the staff members had done showed her the team embedded their new learning into their current projects with enthusiasm and skill. She could see a switch in energy and commitment from the new graduates that she attributed to the practical application of the theory in assessments that focused on real workplace processes.

Conclusion

Perhaps the most important skill set needed for assessment designers is common sense and practicality (Linda Suskie, Trudy W. Banta, 2009). It is worthy to note, that neither of these appear as prerequisites for those designing assessments. Any tertiary provider can adopt a process to design integrated assessments and, in my view, it requires a combination of skills and abilities that include:

• outside the box thinking
• inside the box compliance
• creative and imaginative ideas
• systematic and logical processes
• comprehensive understanding of graduate career opportunities
• real understanding of the true context, feel and immediacy of the world in which graduates will work

Those raised inside academia and outside the corporate/industrial world tend to find the levels of creativity and pragmatism necessary to make integrated assessments truly mirror reality more difficult than do those without such backgrounds.

Solid understanding of the principles of adult learning and an informed and experienced view on workplace realities, opportunities and consequences are needed. Left brain sequential thinking has to be combined with right brain creative designing and this combination in turn must be linked to skilled and practiced workplace specialists.

Integrated assessments can be applied in any educational institution that truly wants to prepare learners for the realities they may face in their future careers. Businesses want graduates with efficient and effective skills who can hit the ground – not necessarily running but, certainly, far beyond crawling. Rather than developing assessments that focus on the learning outcomes for narrowly defined topics, providers can develop realistic contextualised integrated assessments to meet workplace realities. This provides a climate for transfer, according to Bramley (2003).

From these assessments multiple unit standards or topic requirements and learning outcomes can be validly and robustly assessed. These provide authentic and valuable evidence through numerous realistic and relevant assessment activities. Learners are engaged and “a more holistic learning process is achieved that draws on rational, emotional, relational, physical and intuitive capabilities” (Griffin, 1993).

Integrated assessments provide engaging ways to enhance learner outcomes.

References

• Barer-Stein, T. a. (1993). The craft of teaching adults. Toronto, Canada: Concept Capers.
• Beirut-Daily-Star. (2007, September 10). Graduates entering the workforce are just getting started. Retrieved March 23, 2008, from http://www.zawya.com/marketing.cfm?zp&p=/story.cfm/sidDS100907_dsart32/G...
• Booth, R., Clayton, B., House, R., & Roy, S. (2002). Maximising confidence in assessment decision making. Australia: National Centre for Vocation Education Research.
• Bramley, P. (2003). Evaluating training from personal insight to organisational performance. London: The Chartered Institute of Personnel and Development.
• Brinkerhoff, R. O., & Gill, S. J. (1994). The learning alliance: Systems thinking in HRD. San Francisco, CA.: Jossey-Bass.
• Brookfield, S. D. (1990). The skillful teacher. San Francisco, CA: Jossey-Bass.
• Friedrich, P., & Lantz, A. (2003). Learning in the workplace - An instrument for competence assessment. The Learning Organisation, 10 (3), 185–194.
• Griffin, V. (1993). Holistic learning/teaching in adult education: Would you play a one string guitar? Toronto, Canada: Concept Capers.
• Grzeda, M. M. (2005). In competence we trust: Addressing conceptual ambiguity. Journal of Management Development, 24 (6), 530–545.
• Hattie, J. (2009). The black box of tertiary assessment. In L. H. Meyer, S. Davidson, H. Anderson, R. Fletcher, P.M. Johnston, & M. Rees (Eds.), Tertiary assessment & higher education student outcomes: Policy, practice & research (pp. 259–275). Wellington, New Zealand: Ako Aotearoa.
• Kemp, J. E., Morrison, G. R., & Ross, S. M. (1994). Designing effective Instruction. Toronto: Maxwell Macmillan Canada.
• Kleiner, A., Roberts, C., Ross, R., Senge, P., & Smith, S. (1994). The fifth discipline field book: Strategies and tools for builidng a learning organisation. New York: Doubleday.
• Lombardi, M. (2008). Making the grade: The role of assessment in authentic learning. Educause – Learning Initiatve , 1.
• Malnarich, G., & Decker-Lardner, E. (2003). Designing integrated learning for students: A heuristic for teaching, assessment and curriculum design. Washington Centre Occasional Paper , Vol. 1.
• Mandell, A., Michelson, E., & contributors. (2004). Portfolio development and the assessment of prior learning. Sterling, VA: Stylus Publishing.
• Suskie, L., & Banta, T. W. (2009). Assessing student learning: A common sense approach. San Francisco, CA: A. Wiley Imprint.
• Whitaker, U. (1989). Assessing learning: Standards, principles and procedures. Philadelphia, PA: Council of Adult and Experiential Learning.