Dickson, Rumona and Tight, Malcolm and Saunders, Murray (2012) Conduct of a meta review of programme evaluations : a case study of the SEARCH Program. PhD thesis, Lancaster University.
Abstract
This thesis presents a retrospective case study that critically examines the evaluations that were undertaken as part of a continuing professional development (CPD) programme for health care professionals. The case is the SEARCH Program, an innovative CPD programme, which was designed to promote the implementation of evidence based practice (EBP) within the existing health care system in Alberta, Canada. Two approaches from the ‘using’ branch of Alkin and Chrisite’s evaluation theory tree are used in this research. The first employs a quantitative metaevaluation tool to retrospectively assess the quality of evaluations that were conducted from 2000 to 2005. The second is qualitative and explores the use of evaluations to inform programme development. The results of the quantitative analysis demonstrate that the evaluations scored poorly. In fact all evaluations failed to meet basic pass/fail criteria in three of the four standard categories. Reasons for this are explored and include the interdependence of criteria in the metaevaluation tool, the poor or incomplete quality of the reports and the retrospective nature of the process that did not allow for additional data collection. The apparent precision offered by the metaevaluation tool is questionable, as there is a lack of explanation regarding the weighting of the various items, the quantitative formulae used, and the criteria for classifying an evaluation as a failure. The tool is also limited by its focus on evaluation process with no consideration given to the results of the programme evaluations. The application of qualitative method was also time consuming but more fruitful. The results of the qualitative analysis demonstrate that the SEARCH Program was a complex, innovative and evolving programme functioning in a complex and changing health care system. Evaluation processes used within the programme were developmental in nature and informed substantive programme changes. The extent of the changes extend beyond what would be expected with standard formative or summative evaluation and fit with the concepts and use of developmental evaluation as articulated by Patton. The development of CPD programmes for health care professionals who are required to implement EBP is complex and requires collaboration between networks of professionals from institutions within health and higher education. Such programmes need to be reflective, innovative and flexible in nature due to the complex environments in which they are established and the complex outcomes that they wish to implement. This complexity and need for consistent re-evaluation of the goals of the programmes means that developmental evaluation may be an appropriate approach. It is acknowledged that developmental evaluation is difficult and requires both expertise and commitment of those involved. It is also acknowledged that such evaluation may be able to demonstrate immediate outcomes of the CPD programme for the participants and even the faculty but is much less likely to be able to demonstrate impact on the health care system in which it is used.