|
Session Title: Evaluating Structural Changes in Residential Children's Homes: Challenges, Strategies and Lessons Learned
|
|
Panel Session 929 to be held in Mineral Hall Section E on Saturday, Nov 8, 4:00 PM to 5:30 PM
|
|
Sponsored by the Non-profit and Foundations Evaluation TIG
|
| Chair(s): |
| Toni Freeman,
The Duke Endowment,
tfreeman@tde.org
|
| Abstract:
The ENRICH (Environmental Interventions in Children's Homes) evaluation plan included a group randomized design, along with comprehensive process evaluation and contextual assessments, to evaluate a structural intervention designed to promote and support physical activity and healthful nutrition (eating fruits and vegetables) among children and adolescents residing in approximately 30 residential children's homes (RCHs) in North and South Carolina. ENRICH was designed to be specific to the RCH setting; however, we believe that this evaluation approach and framework are applicable to interventions in other organizational settings, including schools, worksites, churches, and other community organizations. The three sessions included in this panel will provide detailed description of the organizational settings for the intervention, including strategies used to identify key organizational characteristics important to the design of the intervention and evaluation; an overview of the intervention, evaluation framework and methodology, including evaluation challenges; and, finally, outcome evaluation results and lessons learned.
|
|
Residential Children's Homes: The Context for the ENRICH Project
|
| Toni Freeman,
The Duke Endowment,
tfreeman@tde.org
|
|
Children are most commonly placed in residential care due to abuse, neglect, behavioral acting out, status offenses, pregnancy, family crisis, and substance abuse Thus, Residential Children's Homes (RCHs) play the role of 'family/home' for these underserved children. The RCHs involved in the ENRICH project vary widely on numerous characteristics including size, setting, populations served, location (two-state area) and staffing models. The Duke Endowment (funding agency) and ENRICH staff collaborated to gather information and enhance buy-in by holding a one day meeting for the CEOs of all potential participating organizations. During the meeting, representatives from both organizations provided an overview of the project, but more importantly explained the project design in detail and held small discussion groups with participants to gather information needed. This enhanced stakeholder buy-in to the design. In addition, each organization completed the ENRICH Organizational Assessment annually, and ENRICH staff closely monitored and documented organizational transitions experiences by RCHs during the project. Included in the presentation will be an overview of the organizational data used for selection, matching, and randomization to reduce heterogeneity among participating RCHs and reduce variability that would threaten statistical validity.
|
|
|
ENRICH Outcome Evaluation: Design and Methodology to Address Common Evaluation Challenges
|
| Ruth Saunders,
University of South Carolina,
rsaunder@gwm.sc.edu
|
|
The ENRICH intervention was designed to create environmental change by partnering with the RCHs and providing training, technical assistance, and support to wellness teams (adult staff working in the RCHs) that served as organizational change agents over 2 years. The researchers used a well-defined conceptual framework to define the 'healthy home' environment. This framework guided the intervention (the change target for the wellness teams) as well as development of outcome measures to assess the environment. We will share the instrument development process designed to establish construct validity and the instruments. We used a group randomized design with two groups and pre- and post-tests to assess program effectiveness. Prior to random assignment, RCHs were matched based on RCH characteristics. This design reduces threats to internal validity. Through process evaluation we also monitored project-related activities in all RCHs as well as contextual influences that could impact project outcomes.
| |
|
Outcome Evaluation Results and Lessons Learned from ENRICH Outcome Evaluation
|
| Kelli Kenison,
University of South Carolina,
kenison@gwm.sc.edu
|
|
Mixed methods were used to assess the effectiveness of the ENRICH intervention with Residential Children's Homes (RCHs). The intervention had a positive impact on the physical activity environment including characteristics of physical activity provided, the social environment related to physical activity, staff wellness support, and limited changes in policies related to physical activity. Nutrition environment changes included organizational support for improved nutrition and collaboration with nutrition-related community organizations outside the RCHs. In addition to sharing results, this session will focus on the numerous lessons learned related to various design challenges inherent in structural interventions. Included will be discussion of strategies to address the need for enhancing stakeholder support for evaluation design, quickly and creatively acquiring organizational information required for matching and random assignment to condition, monitoring setting and contextual issues that may affect project implementation, outcome or design integrity, and data collection methodology for structural interventions.
| |