| Session Title: Challenges in Responding to Stakeholder Values: An Example From a Large-scale, Multi-site/Multi-level Effort to Create a National Warehouse of State and Territorial Tobacco Cessation Quitline Data |
| Multipaper Session 122 to be held in La Jolla on Wednesday, Nov 2, 4:30 PM to 6:00 PM |
| Sponsored by the Cluster, Multi-site and Multi-level Evaluation TIG |
| Chair(s): |
| Martha Engstrom, Centers for Disease Control and Prevention, mengstrom@cdc.gov |
| Abstract: This session will provide a real-world example that highlights the challenges evaluators face in their efforts to respond to the values of multiple stakeholders while planning and implementing a multi-site, multi-level evaluation. The example discussed is the National Quitline Data Warehouse (NQDW), a large-scale initiative by the Centers for Disease Control and Prevention (CDC) to create the first national warehouse of data from state and territorial tobacco cessation quitlines. The primary goal of the NQDW is to serve as a continuing national resource for data on utilization, success, and services of quitlines to support evaluation for accountability and program improvement at both federal and state levels. Stakeholders for this evaluation include: federal, state and territorial programs, funders of tobacco cessation quitlines, service providers, and cessation researchers. The panelists will describe challenges and lessons learned in their efforts to respond to stakeholder values throughout the planning and implementation of this evaluation. |
| Overview of the Challenges in Engaging Stakeholders and Responding to Their Values |
| Martha Engstrom, Centers for Disease Control and Prevention, mengstrom@cdc.gov |
| A multi-site, multi-level evaluation cannot be successful if it does not include a detailed and thoughtful plan to obtain and respond to stakeholder values. Challenges arise when it is time to consider whether or not (and how) to incorporate the values of multiple stakeholders into a complex evaluation effort. These challenges include: 1) engaging stakeholders when they have concerns about the evaluation and the use of the evaluation results; 2) responding to the values of multiple stakeholders when their values are in conflict; 3) responding to the values of multiple external stakeholders while staying true to the original goals and needs of the evaluation; and 4) responding to the values of multiple stakeholders while still obtaining reliable and valid data that answer the key evaluation questions in a credible way. These challenges will be presented with an eye to lessons learned for future efforts. |
| Project Management in Response to Stakeholder Values |
| Henraya McGruder, Centers for Disease Control and Prevention, hmcgruder@cdc.gov |
| The importance of obtaining stakeholder input and responding to stakeholder values is clear in many phases of an evaluation project such as evaluation planning and use. However, what may not always be clear is the importance and utility of engaging stakeholders and responding to their values throughout the project management and implementation process. Challenges, lessons learned, and successful examples of this will be discussed. For example, the original data collection plan for the National Quitline Data Warehouse included a specific data format that all sites were asked to follow when submitting their data files to us; however, we soon learned that requiring this format would be burdensome to many sites in terms of both cost and effort. Therefore, we changed the data collection plan so that sites are allowed to submit their data in the format that is easiest for them. |
| Provision of Technical Assistance in Response to Stakeholder Values |
| Adriane Niare, Centers for Disease Control and Prevention, aze2@cdc.gov |
| The creation of CDC's National Quitline Data Warehouse involves providing a high level of technical assistance to diverse stakeholders-all of whom may vary in their information needs and values. Successful technical assistance will facilitate "buy-in" and increase the likelihood that the data collection effort, and therefore, the evaluation, are successful. The panelist will discuss how an effective technical assistance (TA) team is created, and also provide examples of how to provide evidence and data-based technical assistance to stakeholders including: 1) the importance of training technical assistance providers on how to interact with stakeholders to respond appropriately to stakeholder values, 2) the utility of an effective, dynamic database for tracking interactions with stakeholders, 3) the importance of developing detailed guidance documents, sharing them with stakeholders and revising them based on stakeholder input, and 4) the importance of developing and maintaining positive rapport between the technical assistance team and the stakeholders. |
| Evaluation Planning, Data Analysis, and Dissemination in Response to Stakeholder Values |
| Lei Zhang, Centers for Disease Control and Prevention, lzhang2@cdc.gov |
| It is standard evaluation practice to engage multiple stakeholders and obtain their input throughout the evaluation process, from the initial planning to the analysis and dissemination of the results. However, there are significant challenges to accomplishing this objective successfully, especially when attempting to do this in a multi-site, multi-level evaluation. For every given level or site, a stakeholder may have differing needs or agendas. This presentation will describe how the National Quitline Data Warehouse project team accomplished this objective by convening an external Evaluation Advisory Workgroup comprised of state and territorial tobacco control programs, funders of tobacco cessation quitlines, tobacco cessation quitline service providers, and tobacco cessation researchers. The panelist will discuss challenges, lessons learned, and successes related to our experiences in convening this group of stakeholders, obtaining their input, and responding to their often conflicting values in the evaluation planning, data analysis, and dissemination phases. |