|
Session Title: Meeting Needs of Multiple Stakeholders in a High-Scrutiny Multi-site Evaluation: Evaluation of the Communities Putting Prevention to Work (CPPW) Initiative
|
|
Panel Session 867 to be held in Lone Star F on Saturday, Nov 13, 2:50 PM to 4:20 PM
|
|
Sponsored by the Cluster, Multi-site and Multi-level Evaluation TIG
|
| Abstract:
The Centers for Disease Control and Prevention (CDC) has dedicated $650 million in American Recovery and Investment Act (ARRA) funds to a large-scale initiative, Communities Putting Prevention to Work (CPPW). The goal is to implement supportive policies, systems, and environments in states and communities that will drive changes in behavior to reduce risk factors, and prevent/delay chronic disease. Funded recipients—all states and 44 selected communities—have 24 months to implement these strategies and to accomplish the intended policy and environmental change outcomes related to the strategy; in addition, states received funds to improve their tobacco quitlines and enhance media related to tobacco cessation. In this session, presenters will discuss the multi-faceted evaluation approach, the stakeholders, and how different stakeholder needs have been reconciled in the design. Other presentations will discuss the challenges and implementation of two facets of the CPPW effort—the policy/environmental change component and the quitline component.
|
|
On Rowing in the Right Direction: Creating an Evaluation Design for the CPPW Initiative
|
| Tom Chapel, Centers for Disease Control and Prevention, tchapel@cdc.gov
|
|
The evaluation of CPPW has many components, employing both quantitative and qualitative evaluation methods and recipient-, as well as aggregate-level, analysis. Core to the evaluation are performance measures to monitor state and community success in implementing strategies, the status of their intended policy or environmental changes, and, in the case of the quitline, increases in calls and callers. In addition, selected case studies will be conducted to determine the context of successful implementation and the factors that affect differential outcomes. Like all ARRA efforts, CPPW is a high-scrutiny initiative, with multiple government and public stakeholders. The many stakeholders bring diverse opinions about the intent of CPPW efforts, the outcomes that constitute “success”, and the timeframe within which success should be achieved. This session will discuss the challenges in developing an evaluation approach, and the ways in which the different needs of these stakeholders have been reconciled in the evaluation design.
|
|
|
Sense and Chaos in Multi-site Evaluations
|
| Rene Lavinghouze, Centers for Disease Control and Prevention, shl3@cdc.gov
|
|
Programs are often simultaneously encouraged to develop innovative, context-specific strategies while providing information that can be synthesized across sites. With this juxtaposition in mind, how do we evaluate program implementation and results toward improved practice while simultaneously meeting the needs of stakeholders and funded sites? A methodology that focuses on generating information for accountability and program improvement in widely varied settings is an ideal approach because variation is not only permitted, but celebrated. In this approach, the focus is on documenting and assessing a continuum of change and the linkage of program implementation to desired outcomes. It is through collaboration that successful designs can be co-created to allow for the generation of information that promotes cross-site analysis as well as furthers the exploration. This approach is an ideal methodology for identifying promising practices and encouraging adaptation to program context. It facilitated sense from what can often be chaos.
| |
|
Navigating the Complexities of Evaluating Quitlines: Design and Methodology
|
| Lei Zhang, Centers for Disease Control and Prevention, fpv4@cdc.gov
|
|
Because tobacco cessation quitlines vary in capacity, services, and contexts, evaluating quitlines in multiple states can be challenging. CDC’s Office on Smoking and Health is in the process of developing a National Quitline Data Warehouse (NQDW) that provides data for ongoing quitline evaluation and the evaluation of CDC’s ARRA expenditure on quitlines. The NQDW will standardize data collected by state quitlines using three questionnaires (based on the North American Quitline Consortium’s Minimum Data Set): an intake questionnaire, a 7-month follow-up questionnaire, and a quitline services questionnaire. The Intake and Follow-up questionnaires are administered to quitline callers and collect information on their tobacco use behavior and quit status. The Services questionnaire provides contextual information about the quitlines (e.g., type and availability of services, eligibility criteria, etc) that is critical to the proper understanding and interpretation of Intake and Follow-up data. This presentation focuses on the design and methodology of the NQDW, and how it addresses the challenges in multi-site evaluations.
| |
|
The Role of Technical Assistance and Training in Ensuring Evaluation Quality in a Multi-site, Multi-level Evaluation
|
| Marti Engstrom, Centers for Disease Control and Prevention, cpu5@cdc.gov
|
|
The evaluation of CDC’s CPPW initiative is complex and challenging. Some challenges include the following: a) it is a large multi-site, multi-level initiative: all states and 44 selected communities were funded, b) funded sites are implementing selected policy and environmental change strategies that vary across sites, c) evaluation of policy and environmental change is inherently complex due to differences in context, d) an expectation that important outcomes will be observed within a very short amount of time, and e) a need for integration of utilization-focused (and bottom-up) evaluation into an accountability (and top-down) evaluation framework. In this challenging environment, technical assistance and training related to programmatic reporting, monitoring, and evaluation is critical to ensuring a high-quality and useful evaluation. This presentation will use the CPPW to provide an example of how technical assistance and training can be used to promote a high quality evaluation of a diverse, multi-site, multi-level initiative.
| |