Item writing skills and their development : Insights from an induction item writer training course

Rossi, Olena and Brunfaut, Tineke (2021) Item writing skills and their development : Insights from an induction item writer training course. PhD thesis, Lancaster University.

Text (2021rossiphd)
2021rossiphd.pdf - Published Version
Restricted to Repository staff only until 31 March 2025.
Download (3MB)

Abstract

Item quality makes a significant contribution to test validity, thus rendering the work of item writers critically important for assessment. However, little empirical research has so far been done into item writing, including item-writing training. This thesis therefore aimed to investigate an online induction item-writing training course in order to gain insights into the nature of item-writing skills and their development. This research project, which is based on an existing item-writing training course, adopted a mixed-methods approach consisting of a pretest-posttest quasi-experimental study and a course feedback study. To investigate how the quality of items produced by participants changed from before to after the training (RQ1), 25 trainees produced grammar MC items, writing prompts, and listening tasks for the pre- and post-training assignments. The quality of items was evaluated by expert item reviewers against an evaluation scale; the evaluations were then analysed statistically to identify changes in item quality and individual item-writer variation. To investigate how the participants’ item-writing skills developed through the training (RQ2), interviews were conducted with willing participants upon completion of each assignment and analysed using the Grounded Theory approach. Finally, to identify what role the training played in the participants’ item-writing skill development (RQ3), participants’ reactions to the course were collected via four feedback questionnaires administered throughout the course and analysed using quantitative and qualitative methods. It was found that the total post-training scores for the grammar items and for the listening tasks were statistically significantly higher compared to the pre-training ones, largely due to an improvement in quality on objectively-scored criteria. Three main participant profiles were identified: (a) those whose item quality was low prior to the training but who produced better quality items following it; (b) those who produced good quality items before the training and whose post-training items were of even better quality; (c) those whose pre-training items were of reasonably good quality but whose post-training items scored one or several points lower. The analysis of interview transcripts showed that awareness of objective requirements and the ability to use item-writing tools were generally sufficient in complying with these requirements. For subjective requirements, however, the analysis revealed different approaches to item writing by participants in different profile groups. The course features that the participants reported as most useful for developing their item-writing skills included: input in language assessment principles, balance of theory and practice, variety of activities, extensive item-writing practice, and detailed feedback on items. The findings for the three research questions were then triangulated to provide rich insights into the nature of item-writing skills and their development. The findings were interpreted with reference to two learning theories – cognitive ACT-R theory (Anderson, 1993) and social Communities of Practice (CoP) theory (Love & Wenger, 1991; Wenger, 1998). It was found that item-writing skills are item-type and proficiency-level specific and consist of multiple components acquired at different rates. It was further found that, while item-writing skill acquisition follows the ‘typical’ process of complex cognitive skills’ acquisition as described in ACT-R (Anderson, 1993), the trajectories of acquisition for individual trainees might vary, with three main trainee profiles described. Finally, this study’s findings confirmed that item writers are a CoP, and elements of legitimate peripheral participation (Love & Wenger, 1991) in an item-writers’ community make item-writing training more effective. This study contributes to understanding the nature of item-writing skills and their development through induction training. The study also advances the methodology of research into item-writing training effects. From a practical perspective, this study provides a range of recommendations concerning operational item writing, item-writing training, and item-writer recruitment.