Double play in listening assessment

Holzknecht, Franz and Harding, Luke (2019) Double play in listening assessment. PhD thesis, Lancaster University.

Text (2019holzknechtphd)
2019holzknechtphd.pdf - Published Version
Available under License Creative Commons Attribution-NonCommercial-NoDerivs.
Download (4MB)

Abstract

All listening test developers are faced with a choice: whether to play the listening text once (single play), or twice (double play). Although practices of international high-stakes test providers vary widely in this regard, only a relatively small number of studies have investigated the effects of single and double play. The central focus of most studies has been whether double play had an effect on test takers’ scores. Establishing effects on test scores is important, however an equally relevant question is whether double play impacts construct validity as conceptualised by Messick (1989, 1995). It has not been fully established in what ways double play influences test takers’ response processes compared to single play and thus how it alters the listening construct. The research in this thesis investigated the effects of double play on item parameters and test takers’ response processes, their anxiety levels, and perceptions. In Study 1, 306 candidates responded to four listening tasks in a complex counter-balanced research design involving two conditions (single and double play), two task formats (multiple-choice and open format), and two questionnaires targeting listening strategies, test-taking strategies, anxiety levels, and test takers’ perceptions. In Study 2, 16 candidates completed the same tasks in both conditions on an eye-tracker and performed verbal recalls, which were stimulated by their eye-movements while they had been solving the items. The verbal recalls were analysed in terms of candidates’ cognitive processing, their use of listening strategies and test-taking strategies, and their anxiety levels. The results from Study 1 confirm the common finding of previous research in that double play increases test scores. However, the results from Study 1 and Study 2 also agree in showing that double play is beneficial in terms of reducing construct-irrelevant variance and enhancing construct representation. Candidates displayed more higher-order cognitive processing and used a greater variety and a greater proportion of listening strategies in double play versus single play. Candidates also relied less on test-taking strategies and were markedly less anxious. In addition, items from both task formats were more reliable and showed better discrimination in the double play condition. The findings are discussed in light of the many competing priorities in listening assessment, including test purpose, authenticity, practicality, and construct validity.