|
Session Title: Valuing Systems Thinking in Designing an Evaluation of a Complex System Intervention
|
|
Panel Session 403 to be held in California A on Thursday, Nov 3, 2:50 PM to 4:20 PM
|
|
Sponsored by the Systems in Evaluation TIG
|
| Chair(s): |
| Beverly Parsons, InSites, bparsons@insites.org
|
| Discussant(s):
|
| Melissa Brodowski, Administration for Children and Families, melissa.brodowski@acf.hhs.gov
|
| Abstract:
This session focuses on two tools and the overall framework used in the cross site evaluation of a complex multi-site national research project. Each site has a unique intervention designed to build protective factors among caregivers to prevent child abuse and neglect of children 0-5 year old. Each site focus on multiple levels of systems (e.g., direct intervention with caregivers/children plus community or state policy levels). The evaluation was designed to generate new tools and methodologies for evaluating complex systems as well as to generate results about how differences in the valuing and expression of protective factors affect the outcomes for caregivers and children. The presentation includes reflections on (a) how our values regarding complex systems and methodology shape our evaluation and (b) how our evaluation approaches shape our capacity to understand how differences in values within a complex system influence the changes that occur in that system.
|
|
The Challenge of Valuing Both Evidence-based Practice and Practice-based Evidence
|
| Charlyn Harper Browne, Center for the Study of Social Policy, charlyn.harperbrowne@cssp.org
|
|
The Quality Improvement Center for Early Childhood (QIC-EC) is unique in its efforts to use a complex systems orientation to the cross-site evaluation of the four 40-month research projects it funded in 2010. Each project uses an experimental or quasi-experimental design. The QIC-EC, with the encouragement and support of its federal program officer, selected research sites which each had designed an intervention that focuses on at least two levels of the social ecology (that is, operates at the caregiver/child level and also at community, program services, social norms, and/or policy levels). This presentation addresses the QIC-EC's overall research question and how a common set of outcome measures of three desired results- optimal child development, increased family strengths, and decreased likelihood of child maltreatment-were developed/selected. The presentation addresses the design challenges of valuing both evidence-based practice and practice-based evidence in complex systems.
|
|
|
Cross-Site Evaluation Design for Interventions Addressing Protective Factors to Prevent Child Maltreatment
|
| Beverly Parsons, InSites, bparsons@insites.org
|
|
This presentation describes the initial theory of change for the cross-site evaluation design; the analytical framework for the quantitative data gathered about the intervention and the outcomes for caregivers and children; and the strategy for bringing together the quantitative and qualitative data gathered through data collection tools described in later presentations. The change theory addresses the patterns of change over time across multiple levels of the social ecology as people shift their attention from risk to protective factors as their primary focus. It involves reaching a tipping point and moving to a new system dynamic of a sustainable adaptive balance across levels of the social ecology. A key feature of the change theory and thus the cross site evaluation is the role of partnerships in changing complex systems. Understanding the value of partnerships in complex systems change is emphasized.
| |
|
Instruments to Assess Support for Building Protective Factors to Prevent Child Maltreatment
|
| Patricia Jessup, InSites, pjessup@insites.org
|
|
This presentation describes the development and use of two instruments to assess the relationship of partnerships' interventions to support the building of protective factors at multiple system levels to prevent child maltreatment. Results of the instruments' initial use in sites visits will be available for discussion at the presentation. Each of these instruments provides data that are synthesized and used in multiple ways-to develop a quantitative measure that can be used in the quantitative analysis; to gather qualitative data to identify examples of how the partnerships support the building of protective factors; and to contribute to a systems-oriented analysis in which we look at patterns of action and system structures involved in building protective factors. Our valuing of mixed methods is seen in the development of tools that integrate quantitative and qualitative methods, provide data for both types of analyses, and break down the artificial distinction between these methods.
| |
|
The Cross-site Evaluation Site Visit: A Moment in Time to Understand the Unfolding of Change Over Time
|
| Marah Moore, i2i Institute, marah@i2i-institute.com
|
|
Conducting site visits for a cross-site evaluation is fairly straight-forward-or is it? This presentation will explore the design and implementation of site visits when viewed through a complexity lens. Issues addressed will include: using site visits to surface patterns of change across multiple levels of the social ecology; acknowledging and leveraging the site visit as a point of influence in the system; and, following up on the previous presentation, the role of the site visit in a mixed-methods approach. The first of three yearly site visits will occur prior to this AEA presentation. Of particular interest will be how methodological values shape the site visit approach, and how differences in values across stakeholders are given evaluative "voice" through the site visit.
| |