Accuracy and confidence of objective structured clinical examination pass-fail decisions

Project leader: Mike Tweed, School of Medicine and Health Science, University of Otago

Date: 2009

Funded by the Ako Aotearoa Southern Hub Regional Hub Funding Scheme

Summary

Introduction

Assessment of the consultation skills of undergraduates is important. Information may be acquired from different consultations, such as an OSCE, and decisions made on aggregated information. The effort to improve OSCEs, including blueprinting, scoresheet development and examiner training, is wasted if the decisions from these data are inappropriate. The aim of this study was to investigate the accuracy and confidence for decisions made by staff assessors given increasing information on students’ performances.

Methods

Medical students at University of Otago sit a 10 station OSCE at the end of the 5^th year of six. Staff assessors were shown authentic anonymised student scores for an increasing number of OSCE stations and asked to make pass-fail decisions and give a degree of confidence in this decision. The scores chosen were used to demonstrate good performance and also variable performance with different degrees of under-performance. The student results given included several parameters including examiner scores, passmark scores and running totals. Subsequently the staff assessors were given a fictional anecdote from a single observation made previously. This information was made to be deliberately discordant with the staff assessors' views to that point and again they gave decisions and confidence.

Following completion of forms, the staff assessors were interviewed regarding the rationale for their decisions.

Confidence was given on a 0-100% Likert scale. Accuracy was defined as the comparison with a gold standard, which was determined in two ways: the actual decision of the Board of Examiners (compensatory method on total scores) and also from consensus of decisions made by staff assessors during this study.

Results

35 staff assessors made 11 pass-fail and confidence decisions for a mean of 5.9 candidates each.

Across the 10 stations for the candidate who was above pass threshold for all stations the mean level of confidence in a pass increased from 80 to 90%. For the students that failed the most stations the level of confidence in fail varied between 70 and 80%. The confidence-accuracy difference, a measure of overconfidence, was greatest for students whose performance was closest to the pass-fail threshold.

Despite provision of progressively poor performances the staff assessors were not as confident in assigning fail. Internal and external factors affecting decision making process tend to contribute to their doubts. The anecdotal information changed 12% of the decisions to the extent that pass was altered to fail or vice versa.

Discussion

Making decisions on students whose performance is close to decision thresholds is difficult and associated with overconfidence. As a majority of students perform above threshold, staff assessors are more comfortable assigning a pass rather than fail. In the presence of uncertainty the staff assessors will tend to pass the student. This has implications for standard setting and decision-making processes.

To some assessors the plausible but unreliable anecdote is given an equal or greater importance than the evidence from the OSCE, and so leads to a decision change.

---

  This work is published under the Creative Commons 3.0 New Zealand Attribution Non-commercial Share Alike Licence (BY-NC-SA). Under this licence you are free to copy, distribute, display and perform the work as well as to remix, tweak, and build upon this work noncommercially, as long as you credit the author/s and license your new creations under the identical terms.