Operationalizing the reading-into-writing construct in analytic rating scales : Effects of different approaches on rating

Lestari, Santi B. and Brunfaut, Tineke (2023) Operationalizing the reading-into-writing construct in analytic rating scales : Effects of different approaches on rating. Language Testing, 40 (3). pp. 684-722. ISSN 0265-5322

[thumbnail of Lestari & Brunfaut 2023]
Text (Lestari & Brunfaut 2023)
02655322231155561.pdf - Published Version
Available under License Creative Commons Attribution.

Download (2MB)

Abstract

Assessing integrated reading-into-writing task performances is known to be challenging, and analytic rating scales have been found to better facilitate the scoring of these performances than other common types of rating scales. However, little is known about how specific operationalizations of the reading-into-writing construct in analytic rating scales may affect rating quality, and by extension score inferences and uses. Using two different analytic rating scales as proxies for two approaches to reading-into-writing construct operationalization, this study investigated the extent to which these approaches affect rating reliability and consistency. Twenty raters rated a set of reading-into-writing performances twice, each time using a different analytic rating scale, and completed post-rating questionnaires. The findings resulting from our convergent explanatory mixed-method research design show that both analytic rating scales functioned well, further supporting the use of analytic rating scales for scoring reading-into-writing. Raters reported that either type of analytic rating scale prompted them to attend to the reading-related aspects of reading-into-writing, although rating these aspects remained more challenging than judging writing-related aspects. The two scales differed, however, in the extent to which they led raters to uniform interpretations of performance difficulty levels. This study has implications for reading-into-writing scale design and rater training.

Item Type:
Journal Article
Journal or Publication Title:
Language Testing
Uncontrolled Keywords:
Research Output Funding/no_not_funded
Subjects:
?? language testingintegrated tasksreading-into-writingreading-to-writerating scaleno - not fundedlinguistics and languagesocial sciences (miscellaneous) ??
ID Code:
189600
Deposited By:
Deposited On:
24 Mar 2023 11:30
Refereed?:
Yes
Published?:
Published
Last Modified:
24 May 2024 02:23