Using a logic model to evaluate rater training for EAP writing assessment

O'Connell, J. (2022) Using a logic model to evaluate rater training for EAP writing assessment. Journal of English for Academic Purposes, 60: 101160. ISSN 1475-1585

Abstract

Assessment by written exams and coursework is common practice in pre-sessional and preliminary year EAP programmes, but the allocation of marks for written assessment is complex, as is training raters to apply specified assessment standards. This practitioner research uses a Logic Model, a visual diagram commonly used in programme evaluation, to evaluate the rater training procedure for writing assessment in an English-medium university department. This study integrates data from surveys, interviews and workshops with the stakeholders involved in the rater training procedure to develop a Logic Model as part of an ongoing theory of change evaluation. The final product is a Model that reveals the guiding principles of rater training in the department, text that describes the evaluation process, and a measurement plan. This paper showcases how practitioner research can enhance EAP practice by demonstrating how an essential component of EAP assessment, rater training, and the rationale behind it, can be made cogent to the various stakeholders involved in the procedure. This paper offers considerations for EAP practitioners, managers, and testing staff when developing or working with rater training, bridging the gap between EAP and language testing and assessment communities.