Towards social justice for item writers : Empowering item writers through language assessment literacy training

Rossi, Olena and Brunfaut, Tineke (2020) Towards social justice for item writers : Empowering item writers through language assessment literacy training. In: 41st LTRC (Language Testing Research Colloquium), 2019-03-04 - 2019-03-08. (Unpublished)

Abstract

Item writers play a key role in the language test cycle, as they essentially need to operationalise the construct into actual tasks. Often, however, these assessment professionals receive a rather narrowly-focused training in writing items to a particular set of specifications. Usually, this training is limited to ‘item writing guidelines’ or instruction in mechanical aspects of item writing. In this presentation, we argue that mechanical training is not sufficient for item writers to consistently produce high-quality items. Item writers need to be empowered through more comprehensive assessment literacy training so that they acquire a deeper understanding of not only ‘how’ but also ‘why’ item specifications contain specific requirements and how their work contributes to test validity. Our viewpoint is based on a study in which we explored the effectiveness of a three-month online item writer course that included instruction in writing specific item types as well as in broader language assessment principles. In our talk, we will provide an overview of the course’s content, and explain how we evaluated the training through a ‘pretest-posttest’ design. More specifically, the 25 novice item writers participating in the study completed three item writing tasks prior to the course as well as after having finished it. The quality of the items they produced were evaluated by expert item reviewers. In addition, pre- and post-course stimulated recall interviews were conducted with the item writers, course feedback was collected at various points during the training, and online group discussions were analysed. In our presentation, we will primarily focus on the interviews, which aimed to gain information on the approach, procedures, and techniques the trainees used while writing items, the difficulties they encountered, and their post-course reflections on the training’s effect on their item writing skills. Analysis of the interview data revealed that improvements in item quality pre- to post-course were associated with an increased awareness post-course – on behalf of the item writer – of fundamental assessment principles such as authenticity and fairness, a deeper understanding of the test construct, and conscious attempts to improve the validity of items by avoiding bias, construct underrepresentation, and construct-irrelevant variance. The findings suggest that bringing item writers to stage 4 of Pill & Harding’s (2013) language assessment literacy continuum, i.e. procedural and conceptual literacy, empowers item writers in their work by enabling them to develop higher-quality items, and by making them more interested and more confident in writing language test items. Our study has practical implications for item writer training and, more fundamentally, for the empowerment of item writers and the development of good-quality tests.