|
Session Title: Improving External Validity: A Realist Understanding of the Role of Context in Theories of Change
|
|
Panel Session 542 to be held in Panzacola Section G2 on Friday, Nov 13, 1:40 PM to 3:10 PM
|
|
Sponsored by the Program Theory and Theory-driven Evaluation TIG
|
| Chair(s): |
| Patricia Rogers, RMIT University, patricia.rogers@rmit.edu.au
|
| Abstract:
Realist evaluation theory is based on the understanding that programs work differently in different contexts by generating different change mechanisms; therefore it can’t be assumed that applying the same intervention in a different context will result in the same outcomes. Understanding the mechanisms triggered by interventions in differing contexts, commonly expressed as ‘knowledge about what works for whom in what circumstances’, improves the external validity of the findings of a single evaluation or a synthesis of multiple evidence sources. This session presents the key features and processes of realist evaluation and realist synthesis with examples of each of these – a realist evaluation of a program to improve the health of farming families and a realist synthesis of early intervention programs. It suggests ways to respond to the challenges in both undertaking realist evaluations and supporting uptake of more complex findings.
|
|
Key Features and Processes of Realist Evaluation and Realist Synthesis
|
| Patricia Rogers, RMIT University, patricia.rogers@rmit.edu.au
|
|
Realist approaches ask "What is it about this program that works for whom in what circumstances?" The essential elements of the logic of the realist approach; understanding what is meant by context, mechanisms and outcomes, and the relationships between them, are outlined.
Realist evaluation and realist synthesis processes draw on quantitative and qualitative evidence, successes and failures and similarities and differences in context and outcomes. Techniques for conducting realist program evaluations and for generating theories of change through a realist synthesis of evaluations and research from multiple sites and sources are discussed including examples of how to develop initial context-mechanism-outcome configurations.
Realist approaches generate knowledge of the if/then or yes/but kind rather than all encompassing judgements and recommendations about what works. The external validity and benefits and challenges for utilising the lessons learnt about theories of change from realist evaluations and synthesis are explored.
|
|
|
Using Realist Synthesis to Explore Context and Inform Policy
|
| Kaye Stevens, RMIT University, kaye.stevens@rmit.edu.au
|
|
The Sustainable Farming Families program, a three year intervention to improve the health and wellbeing of farming families includes annual health checks, educational workshops and individual action planning. The program has been implemented with varied farming sectors and was rolled out to dairy farmers at eleven sites in Victoria, Australia. The presentation discusses how we sought to make sense of puzzling patterns in the data by retrospectively exploring: why the pattern of results differed for dairy and broad acre farmers; differential effects for participants with different levels of risk at the beginning of the program, that were hidden in the analysis of average effects; the impact of differences in implementation contexts across sites. The evaluation findings raise the possibility that different mechanisms may be operating in different contexts and identify questions that future evaluations could explore to deepen understanding of what works for who, when and why.
The evaluation explored the influence of context by:
1) assessing the external validity of findings from previous evaluations of bushfire case management services
2) reflecting on the development and implementation of the service in different contexts - different communities and pre-existing service systems and different individual needs.
3) identifying lessons learnt about VBCMS interventions in different contexts to inform future programs
| |
|
Using Realist Synthesis to Explore Context and Inform Policy
|
| Gill Westhorpe, Community Matters, gill.westhorp@communitymatters.com.au
|
|
Realist synthesis (Pawson, 2006) extends the insights of realist evaluation into the realm of literature review. This paper will demonstrate how realist synthesis can be used to understand how and why programs generate different outcomes in different contexts. It will explain how realist synthesis differs from other review methodologies and demonstrate particular analytic processes.
The example is early intervention programs. Many early intervention programs that "work" on average do not work for their most disadvantaged participants - and some appear to make things worse (including Early Head Start). The paper will examine how various features of context - at personal, family, program, organizational and social levels - operate to influence the outcomes that programs generate for different sub-groups. It concludes by suggesting ways that the findings from realist evaluation and realist synthesis can be translated back to policy and funding bodies and to service providers.
| |