Burr, Steven and Martin, Theresa and Edwards, James and Ferguson, Colin and Gilbert, Kerry and Gray, Christian and Hill, Adele and Hosking, Joanne and Jonhstone, Karen and Kisielewska, Jolanta and Milsom, Chloe and Moyes, Siobhan and Rigby-Jones, Ann and Robinson, Iain M and Toms, Nick and Watson, Helen and Zahra, Daniel (2021) Standard setting anchor statements:a double cross-over trial of two different methods. MedEDPublish, 10.
Full text not available from this repository.Abstract
Context: We challenge the philosophical acceptability of the Angoff method, and propose an alternative method of standard setting based on how important it is for candidates to know the material each test item assesses, and not how difficult it is for a subgroup of candidates to answer each item. Methods: The practicalities of an alternative method of standard setting are evaluated here, for the first time, with direct comparison to an Angoff method. To negate bias due to any leading effects, a prospective cross-over design was adopted involving two groups of judges (n=7 and n=8), both of which set the standards for the same two 100 item multiple choice question tests, by the two different methods. Results: Overall, we found that the two methods took a similar amount of time to complete. The alternative method produced a higher cut-score (by 12-14%), and had a higher degree of variability between judges' cut-scores (by 5%). When using the alternative method, judges reported a small, but statistically significant, increase in their confidence to decide accurately the standard (by 3%). Conclusion: This is a new approach to standard setting where the quantitative differences are slight, but there are clear qualitative advantages associated with use of the alternative method.