Using Multi-Faceted Rasch Analysis to Examine Raters, Prompts, and Rubrics in an Office-Hour Role-Play Task

Ren, Haoshan (Sally) and Looney, Stephen and Cushing, Sara (2023) Using Multi-Faceted Rasch Analysis to Examine Raters, Prompts, and Rubrics in an Office-Hour Role-Play Task. In: Local Language Testing : Practice across Contexts. Educational Linguistics . Springer, Cham, pp. 13-33. ISBN 9783031335402

Abstract

Locally-administered placement tests for International Teaching Assistants (ITAs) impact the quality of ITAs’ graduate studies and the quality of undergraduate education. However, in-house testing programs face challenges in terms of funding, time, and expertise to ensure the reliability and address the validity of their ITA tests. This chapter examines comparability of prompts used in an office-hour roleplay task on an in-house ITA test. The role-play task requires test-takers to respond to one of four office-hour scenarios where the test-taker plays the role of the teacher and one of the two raters plays the role of the undergraduate student. We used Many-Faceted Rasch Measurement (MFRM) to investigate prompt difficulty, rubric consistency, and selected interactions. All raters were self-consistent, but we found significant differences in raters’ relative severities. Further examination reveals that, although raters were instructed to select one of the four prompts randomly, some raters selected certain prompts to the exclusion of others. The rater x prompt interaction shows that raters’ internal consistency is related to selection, while variation in rater severity is an independent effect. This chapter provides insights into practices that connect test analysis, test quality, and test administration in a local context.