Standard setting anchor statements : a double cross-over trial of two different methods

Burr, Steven and Martin, Theresa and Edwards, James and Ferguson, Colin and Gilbert, Kerry and Gray, Christian and Hill, Adele and Hosking, Joanne and Jonhstone, Karen and Kisielewska, Jolanta and Milsom, Chloe and Moyes, Siobhan and Rigby-Jones, Ann and Robinson, Iain M and Toms, Nick and Watson, Helen and Zahra, Daniel (2021) Standard setting anchor statements : a double cross-over trial of two different methods. MedEDPublish, 10: 32.

Full text not available from this repository.

Abstract

Context: We challenge the philosophical acceptability of the Angoff method, and propose an alternative method of standard setting based on how important it is for candidates to know the material each test item assesses, and not how difficult it is for a subgroup of candidates to answer each item. Methods: The practicalities of an alternative method of standard setting are evaluated here, for the first time, with direct comparison to an Angoff method. To negate bias due to any leading effects, a prospective cross-over design was adopted involving two groups of judges (n=7 and n=8), both of which set the standards for the same two 100 item multiple choice question tests, by the two different methods. Results: Overall, we found that the two methods took a similar amount of time to complete. The alternative method produced a higher cut-score (by 12-14%), and had a higher degree of variability between judges' cut-scores (by 5%). When using the alternative method, judges reported a small, but statistically significant, increase in their confidence to decide accurately the standard (by 3%). Conclusion: This is a new approach to standard setting where the quantitative differences are slight, but there are clear qualitative advantages associated with use of the alternative method.

Item Type:
Journal Article
Journal or Publication Title:
MedEDPublish
Uncontrolled Keywords:
Research Output Funding/no_not_funded
Subjects:
?? no - not funded ??
ID Code:
204260
Deposited By:
Deposited On:
18 Sep 2023 09:15
Refereed?:
Yes
Published?:
Published
Last Modified:
26 Sep 2024 12:55