|
Session Title: An Examination of Evaluation Challenges with the Nebraska Strategic Prevention Framework State Incentive Grant (SPF-SIG)
|
|
Panel Session 973 to be held in La Jolla on Saturday, Nov 5, 2:20 PM to 3:50 PM
|
|
Sponsored by the Cluster, Multi-site and Multi-level Evaluation TIG
|
| Chair(s): |
| Phillip Graham, RTI International, pgraham@rti.org
|
| Abstract:
In 2005, the Center for Substance Abuse Prevention (CSAP) made awards to a first cohort of states implementing the agency's flagship substance abuse prevention initiative, the Strategic Prevention Framework-State Incentive Grant (SPF-SIG) program, which is a 5-step public health approach to reduce substance use. CSAP has funded approximately 77 states, territories, and tribal nations since 2005, but evaluating SPF-SIG's effectiveness poses methodological and practical challenges. Presenters will highlight the evaluation challenges and share solutions devised in the evaluation of the Nebraska SPF-SIG, in which the Nebraska Department of Public Health funded sixteen (16) local coalitions to implement evidence-based prevention strategies using the SPF model. The local coalitions targeted three outcomes: underage drinking, binge drinking, and impaired driving. The first paper examines the use of archival data; the second examines challenges with implementation fidelity of environmental strategies; and, the third paper examines challenges and solutions associated with measures of effectiveness.
|
|
Developing a Rigorous Evaluation Design with Archival Data
|
| Monique Clinton-Sherrod, RTI International, mclinton@rti.org
|
| Phillip Graham, RTI International, pgraham@rti.org
|
| Lori Palen, RTI International, lpalen@rti.org
|
| Jason Williams, RTI International, jawilliams@rti.org
|
| Mindy Anderson-Knott, University of Nebraska-Lincoln, mandersonknott2@unl.edu
|
| Dave Palm, State of Nebraska, david.palm@nebraska.gov
|
|
A comprehensive yet feasible evaluation design is essential in capturing the impact of Nebraska's community-level efforts to address substance use through its SPF-SIG. For Nebraska, the outcome evaluation assesses the effectiveness of SPF activities in modifying targeted behaviors, root causes, and contributing factors at the state level through archival data. In addition, outcome data is combined with process data to answer questions about how and why SPF-SIG efforts achieved (or failed to achieve) the state's goals. However, we faced unique challenges in the development of a rigorous design that was viable given NE's unique geographical characteristics. While our design incorporates process and outcome evaluation at the state and local levels, challenges in developing and implementing this design stemmed largely from the rural nature of the state and addressing limited comparison sites given large intervention service areas. This presentation describes these and other challenges and steps taken to address them.
|
|
|
Measuring Implementation Fidelity for Environmental Strategies
|
| Lori Palen, RTI International, lpalen@rti.org
|
| Phillip Graham, RTI International, pgraham@rti.org
|
| Monique Clinton-Sherrod, RTI International, mclinton@rti.org
|
| Mindy Anderson-Knott, University of Nebraska-Lincoln, mandersonknott2@unl.edu
|
| Dave Palm, State of Nebraska, david.palm@nebraska.gov
|
|
As part of Nebraska's SPF SIG evaluation, local evaluators worked with program coordination/implementation staff to complete implementation fidelity rubrics. These rubrics assessed adherence to best practices for specific alcohol-related environmental interventions (e.g., social marketing, compliance checks). There were two challenges to analyzing fidelity rubric data. First, the rubrics confounded strategy progress with fidelity, such that an in-progress strategy might get the same score as a completed strategy with poor fidelity. Second, there was a ceiling effect that complicated comparisons across communities, strategies, and time points: Of environmental rubrics completed in the first year of implementation, average fidelity was 88% of the maximum possible score, and 56% of rubrics received a perfect score. There was also an unanticipated benefit of administering fidelity rubrics. For implementers who were less knowledgeable about strategies, the rubric items served as a checklist of actions that would need to be taken to ensure fidelity moving forward.
| |
|
Measuring Effectiveness in SPF-SIG Evaluation: Challenges and Solutions
|
| Jason Williams, RTI International, jawilliams@rti.org
|
| Phillip Graham, RTI International, pgraham@rti.org
|
| Lori Palen, RTI International, lpalen@rti.org
|
| Monique Clinton-Sherrod, RTI International, mclinton@rti.org
|
| Dave Palm, State of Nebraska, david.palm@nebraska.gov
|
|
Evaluation of SPF-SIG efforts in Nebraska faces challenges across multiple points of implementation. Relevant data originate from multiple levels, including state, county, and individual respondents. Models of interest may rely on data from more than one source which has implications for both the method of aggregating disparate data sources as well as for which level of analysis is most appropriate (e.g., individual vs. local community). Geographic and population characteristics of Nebraska pose challenges to finding appropriate comparison communities for those not exposed to SPF-SIG programming. Within SPF-SIG communities there is not uniformity on the primary outcomes, root causes, and contributing factors targeted by coalitions, which further complicates selection of relevant samples and comparisons for evaluation models of the initiative. This presentation describes these challenges and will offer proposed solutions or methods to ameliorate the impact of these challenges on drawing valid inferences about the impact of the Nebraska SPF-SIG.
| |