|
Session Title: Health Evaluation TIG Business Meeting and Multipaper Session: Accounting for Differential Implementation in Evaluations of Multi-Site Community Initiatives
|
|
Business Meeting with Panel Session 419 to be held in Room 102 in the Convention Center on Thursday, Nov 6, 4:30 PM to 6:00 PM
|
|
Sponsored by the Health Evaluation TIG
|
| TIG Leader(s):
|
|
Ann Zukoski,
Oregon State University,
ann.zukoski@oregonstate.edu
|
|
Jeannette Oshitoye,
Nemours Health and Prevention Services,
joshitoy@nemours.org
|
|
Robert LaChausse,
California State University at San Bernardino,
rlachaus@csusb.edu
|
|
Shyanika Rose,
Battelle,
rosesw@battelle.org
|
| Chair(s): |
| Doug Easterling,
Wake Forest University School of Medicine,
dveaster@wfubmc.edu
|
| Abstract:
This panel brings together evaluators from three community-change initiatives who have developed innovative strategies for measuring and modeling 'site-level dose.' The session begins with a review of why it is important to account for differential implementation in evaluating multi-site initiatives, along with an overview of the challenges associated measuring implementation in the context of comprehensive community initiatives. The remaining papers describe the specific approaches to measurement and statistical modeling employed in the three evaluations. Each evaluation faced the challenge that the intervention was defined according to general principles rather than specific services or activities. However, each evaluation was able to develop reliable methods of assessing how fully the intervention was implemented, and was able to incorporate these assessments as moderator variables in analyses of impact. Site-level dose analyses provided a more complete and nuanced appraisal of the intervention's effectiveness than was available through simple Treatment by Time analyses.
|
|
Assessing Site-Level Dose in Multi-Site Community Initiatives: Rationale, Challenges, and General Strategies
|
| Doug Easterling,
Wake Forest University School of Medicine,
dveaster@wfubmc.edu
|
| Todd Rogers,
Public Health Institute,
txrogers@pacbell.net
|
| Mark Wolfson,
Wake Forest University,
mwolfson@wfubmc.edu
|
|
This paper provides background on the use of 'site-level dose' (SLD) as an adjunct to evaluations of a multi-site initiatives. The basic logic is that when a program model is fully implemented in only a few sites, including SLD as a moderator variable may allow for the detection of program effects that do not attain significance in an overall analysis of the program. However, this approach proves challenging in the case of comprehensive community initiatives. This paper describes the measurement challenges that arise when the initiative has a complex and/or 'fuzzy' program model, or when sites are encouraged to tailor the model to local conditions. We also consider the threats to validity that occur when SLD is used to predict change in participant behavior. These challenges set the stage for the remaining three papers which focus on successful strategies for measuring SLD and for incorporating SLD into impact analysis.
|
|
|
Measuring and Modeling Site-Level Dose in the Evaluation of Free to Grow
|
| Todd Rogers,
Public Health Institute,
txrogers@pacbell.net
|
| Doug Easterling,
Wake Forest University School of Medicine,
dveaster@wfubmc.edu
|
| Rebecca Neiberg,
Wake Forest University,
rneiberg@wfubmc.edu
|
| Mark Wolfson,
Wake Forest University,
mwolfson@wfubmc.edu
|
|
Free to Grow (FTG) was a national initiative in which Head Start (HS) programs partnered with community organizations to implement a comprehensive approach to the prevention of substance abuse and child abuse. The impact evaluation of FTG used a quasi-experimental design in which 14 FTG communities were compared with 14 comparison sites. Changes in parenting and neighborhood conditions were assessed through surveys of caregivers of young children, some of whom were enrolled in HS (cohort study) and some from the larger community (repeated cross-sectional). The primary analysis found no evidence that the intervention had an effect. To account for possible implementation failure, the FTG sites were classified into high versus low implementers. The paper describes how implementation was measured and modeled within the impact analyses. No consistent effect was detected within the two Cohort Samples. However, within the Community Sample, high-implementers showed positive improvements on two measures of neighborhood conditions.
| |
|
Measuring and Modeling Site-Level Dose in the Evaluation of the Study to Prevent Alcohol Related Consequences (SPARC)
|
| Kimberly Wagoner,
Wake Forest University,
kwagoner@wfubmc.edu
|
| Scott D Rhodes,
Wake Forest University,
srhodes@wfubmc.edu
|
| Thomas P McCoy,
Wake Forest University,
tmccoy@wfubmc.edu
|
| Heather Champion,
Wake Forest University,
hchampio@wfubmc.edu
|
| Ashley Wagoner,
Wake Forest University,
aswagone@wfubmc.edu
|
| Mark Wolfson,
Wake Forest University,
mwolfson@wfubmc.edu
|
|
SPARC was a five-year randomized community-trial using a community-organizing approach to implement environmental strategies in and around North Carolina college campuses. The Environmental Strategy and Implementation Survey (ESIS), completed by 'point-people' at the 10 schools (5 Intervention, 5 Comparison), measured each site's Inputs and Processes in implementing the intervention. These measures were used as covariates and effect modifiers to assess if schools implementing a SPARC-like intervention showed greater improvements in outcomes. Overall, Intervention schools had higher ESIS scores than Comparison schools, indicating that Intervention schools implemented higher levels of the intervention. ESIS by Time interactions were comparable to Treatment by Time interactions across measures that evaluated student drinking behavior and associated consequences. ESIS provided confirmation of the study's main effects findings and indicated that implementation occurred as anticipated.
| |
|
Measuring and Modeling Site-Level Dose in the Evaluation of Enforcing Underage Drinking Laws (EUDL)
|
| Rebecca Neiberg,
Wake Forest University,
rneiberg@wfubmc.edu
|
| Kimberly Wagoner,
Wake Forest University,
kwagoner@wfubmc.edu
|
| Jill Newman,
Wake Forest University,
jinewman@wfubmc.edu,
|
| Eun-Young Song,
Wake Forest University,
esong@wfubmc.edu
|
| Barbara A Martin,
Wake Forest University,
bmartin@wfubmc.edu
|
| Debbie Pleasants,
Wake Forest University,
dpleasan@wfubmc.edu
|
| Mark Wolfson,
Wake Forest University,
mwolfson@wfubmc.edu
|
|
The Enforcing Underage Drinking Laws Randomized Community Trial (EUDL-CT) was funded by the Office of Juvenile Justice and Delinquency Prevention to determine the effects of a local coalition-based approach to implement most promising strategies for increasing enforcement of laws related to underage drinking and reducing underage drinking. Assessments of quality, quantity, support and sustainability for the core strategies of compliance checks, enforcement focused on social availability of alcohol, DWI law enforcement, and policy initiatives were used to characterize the strength of dose in each community. Data from multiple sources, including Law Enforcement and Local Coalition surveys, Activity Tracking Data and site visit records, were employed. Combination of data reduction algorithms and consensus meetings of evaluation team members produced dose measures for the core strategies. These measures are being used for secondary analyses that will examine the relationship between degree of implementation and outcomes.
| |