Background

The objective structured clinical examination (OSCE) is a valid and reliable method of assessing practitioner competence. This has been used in the South East NHS region to test the competence of preregistration pharmacists (PRPs) for some 8 years although challenges remain to determine a passing score which is indicative of minimal competence. The passmark for the PRP OSCEs has been set at 60% which requires students to pass at least 9 out of 15 workstations. However recent research suggests that this method does not allow candidates to demonstrate their abilities over a range of competences. An alternative approach is to sum the criteria associated with the 15 workstations and set an overall number which students have to pass in order to be deemed competent. This figure would be independent of the number of workstations passed. This study describes a method by which "expert judges" reach a consensus view on the number of criteria PRPs should pass to be considered competent.

Method

In this study the Angoff methodology¹ was applied to an OSCE used to test the clinical competence of preregistration pharmacists at the end of their training year. Eleven judges independently determined which of the 133 criteria included in 15 workstations a minimally competent PRP would be expected to achieve. The judges then attended a meeting at which their results were collectively presented and compared to the actual performance of PRPs who took the examination in June 2000. They were then invited to discuss their original judgements and review their recommendations.

Results

At the end of session 2 the passing score was set as 104/133 (78%) criteria. There was no significant difference between judges in their overall scoring (ANOVA p=0.21). If this new passing score was applied to the results of PRPs sitting the June 2000 OSCE, only 19% would have been deemed competent. This is in contrast to the 73% who succeeded using the passmark of 9/15 workstations. All candidates who were considered not competent using the Angoff methodology passed less than 9 workstations and not one candidate who failed to register with the RPSGB in July 2000 passed the OSCE using the new passing score.

Discussion

These results suggest that the new passing score is effective at identifying candidates unlikely to register at the end of their preregistration training year, however the results also suggest a significant gulf between employers' expectations of the level of clinical competence of newly qualified pharmacists and their actual ability. The application of a passing score based on expert judges' expectations of the clinical abilities of preregistration trainees would deem the majority of preregistration pharmacists not competent to practise at the end of their training year. These findings expose the need for a robust clinical competency framework for the development of pharmacy staff through their undergraduate, preregistration and early postgraduate years through to specialisation. Such a framework would track the development from novice through to expert and would assist employers to ensure staff were being used to the best of their ability.

Reference

1. Angoff W.H. "Scales, norms and equivalent scores" in Educational Measurement, American Council on Education, Washington (1971)