How much is enough? : Involving occupational experts in setting standards on a specific-purpose language test for health professionals

Pill, John and McNamara, Tim (2016) How much is enough? : Involving occupational experts in setting standards on a specific-purpose language test for health professionals. Language Testing, 33 (2). pp. 217-234. ISSN 0265-5322

Abstract

This paper considers how to establish the minimum required level of professionally relevant oral communication ability in the medium of English for health practitioners with English as an additional language (EAL) to gain admission to practice in jurisdictions where English is the dominant language. A theoretical concern is the construct of clinical communicative competence and its separability (or not) from other aspects of professional competence, while a methodological question examines the technical difficulty of determining a defensible minimum standard. The paper reports on a standard-setting study to set a minimum standard of professionally relevant oral competence for three health professions – medicine, nursing, and physiotherapy – as measured by the speaking sub-test of the Occupational English Test, a profession-specific test of clinically related communicative competence. While clinical educators determined the standard, it is to be implemented by raters trained as teachers of EAL; therefore, the commensurability of the views of each group is a central issue. This also relates to where the limits of authenticity lie in the context of testing language for specific purposes: to represent the views of domain experts, a sufficient alignment of their views with scores given by the raters of test performances is vital. The paper considers the construct of clinical communicative competence and describes the standard-setting study, which used the analytical judgement method. The method proved successful in capturing sufficiently consistent judgements to define defensible standards. Findings also indicate that raters can act as proxies for occupational experts, although it remains unclear whether the views of performances held by these two groups are directly comparable. The new minimum standards represented by the cut scores were found to be somewhat harsher than those in current use, particularly in medicine.