| Session Title: Insights From Rapid Evaluations: Improving School Programs for Better Results |
| Multipaper Session 120 to be held in Suwannee 11 on Wednesday, Nov 11, 4:30 PM to 6:00 PM |
| Sponsored by the Health Evaluation TIG |
| Chair(s): |
| Doryn Chervin, ICF Macro, doryn.d.chervin@macrointernational.com |
| Discussant(s): |
| Leah Robin, Centers for Disease Control and Prevention, ler7@cdc.gov |
| Abstract: The Division of Adolescent and School Health (DASH) of the Centers for Disease Control and Prevention (CDC) contracted with Macro International Inc. to conduct rapid evaluations of the effectiveness of innovative programs addressing child and adolescent health issues. Rapid evaluations use a mixed methods approach together with rapid, iterative, and team-based methods. DASH's use of the method has been particularly valuable for quickly gauging whether an initiative is effective and where program improvements are worthwhile. This session will describe one such program and the methods and measures used to evaluate it, how unexpected results were understood, and lessons learned for future rapid evaluations. Discussants will reflect on the project to date. |
| Value of the Rapid Evaluation Method |
| Leah Robin, Centers for Disease Control and Prevention, ler7@cdc.gov |
| Marian Huhman, University of Illinois at Urbana-Champaign, mhuhman@illinois.edu |
| Catherine Rasberry, Centers for Disease Control and Prevention, catherine.rasberry@cdc.hhs.gov |
| Rapid evaluations are designed to be completed within a relatively short time frame to provide information about the impact of programs, policies, and initiatives. In a rapid evaluation, appropriate stakeholders are engaged in the process to identify salient evaluation questions, develop and implement evaluation plans, and collect and analyze data. This method aims to describe program activities, short-term and intermediate outcomes, and impacts. Further, rapid evaluations can increase accountability by identifying why program components are not being implemented as planned. The value of rapid evaluations is the timely provision of data to encourage action. Results from rapid evaluations can also have implications for other organizations aiming to adopt and implement similar programs and can provide insights for program improvement and quality information for making decisions. |
| Conducting a Rapid Evaluation in a Local Education Agency in the Southeast |
| Karen Cheung, ICF Macro, karen.cheung@macrointernational.com |
| Pamela Lunca, ICF Macro, pamela.j.lunca@macrointernational.com |
| Sarah Merkle, Centers for Disease Control and Prevention, smerkle@cdc.gov |
| In 2006, CDC/DASH launched an initiative to provide local education agencies (LEAs) with evaluation technical assistance on asthma management programs using the rapid evaluation method. First, an evaluability assessment was conducted to determine the program's readiness and capacity for evaluation. Then, using participatory, team-based, and iterative processes, the authors assisted a large LEA in the Southeast to develop evaluation questions and an evaluation plan. Individual interviews, focus groups, and questionnaires were used to collect information from various stakeholders, including elementary school students and high school students; principals, nurses, and other program staff; key district staff members; and parents/guardians of elementary school students. The authors will describe how the rapid evaluation method provided the LEA with important feedback to strengthen the overall management of their asthma program by facilitating uniform implementation of program components across all sites and maintaining detailed records of asthma program services to allow for future evaluation activities. |
| Interpretation and Communication of Unexpected Findings of a School-Based Asthma Program in a Local Education Agency in the Southeast |
| Dana Keener, ICF Macro, dana.c.keener@macrointernational.com |
| Karen Cheung, ICF Macro, karen.cheung@macrointernational.com |
| Catherine Rasberry, Centers for Disease Control and Prevention, catherine.rasberry@cdc.hhs.gov |
| Just like any other approach to evaluation, rapid evaluations can reveal unexpected findings. This presentation will describe unanticipated results that emerged from a rapid evaluation of a school-based asthma program in a large LEA in the southeast. In addition, the presentation will describe efforts to explore possible confounding variables that might explain the findings; mixed-method techniques used to understand and interpret the findings; and the process for communicating the findings back to the school district. Finally, recommendations for program improvement and future evaluation that stemmed from the unexpected findings will be shared. |
| Insights From Conducting Rapid Evaluation in School Settings |
| Doryn Chervin, ICF Macro, doryn.d.chervin@macrointernational.com |
| Dana Keener, ICF Macro, dana.c.keener@macrointernational.com |
| Karen Cheung, ICF Macro, karen.cheung@macrointernational.com |
| Leah Robin, Centers for Disease Control and Prevention, ler7@cdc.gov |
| This presentation will share key lessons and insights gained from conducting rapid evaluations in school settings. Although some lessons may be specific to school settings, others apply across other settings as well. Some of the lessons that will be discussed include: (1) form strong working relationships with evaluation stakeholders; (2) identify and address gaps in implementation data early in the process; (3) develop formal and ongoing opportunities for sharing evaluation results as they become available; and (4) engage stakeholders in the interpretation of the results. In addition, key questions that emerged from the evaluations will be raised for group discussion. For example, would rapid evaluations benefit from a more robust evaluability assessment prior to the evaluation? Finally, tensions associated with making mid-course corrections and improvements while the evaluation is still ongoing will also be discussed. |