| Session Title: Reviving the Evaluability Assessment (EA) Method: EAs of Interventions to Prevent Childhood Obesity |
| Multipaper Session 535 to be held in Room 102 in the Convention Center on Friday, Nov 7, 9:15 AM to 10:45 AM |
| Sponsored by the Health Evaluation TIG |
| Chair(s): |
| David Cotton, Macro International Inc, david.a.cotton@macrointernational.com |
| Discussant(s): |
| Debra Rog, Rockville Institute, debrarog@rockvilleinstitute.org |
| Laura Kettel Khan, Centers for Disease Control and Prevention, ldk7@cdc.gov) |
| Nicola Dawkins, Macro International Inc, nicola.u.dawkins@macrointernational.com |
| Abstract: In 2007, the Robert Wood Johnson Foundation funded the Centers for Disease Control and Prevention (CDC) Foundation, which provided funds to the CDC and Macro International to conduct 25 evaluability assessments (EAs) on selected initiatives to improve the eating habits and physical activity levels of children. EAs provide a systematic way to determine if an initiative is ready for rigorous evaluation. Developed by Joseph Wholey and colleagues, EAs received significant attention 30 years ago but later fell into disuse (Rog 1985; Trevisan 2007). Today the method is seeing renewed interest, particularly for quickly and cheaply gauging whether an initiative is plausible and feasible, and whether investment of evaluation resources is appropriate. This session will share the method for selecting initiatives and implementing the EA process, process for identifying and training 33 professionals to conduct EAs, and lessons and products of the process. Discussants will reflect on the project to date. |
| Evaluability Assessments of Interventions to Prevent Childhood Obesity: What, Why, and How |
| Holly Wethington, Centers for Disease Control and Prevention, eyk5@cdc.gov |
| Nicola Dawkins, Macro International Inc, nicola.u.dawkins@macrointernational.com |
| Seraphine Pitt-Barnes, Centers for Disease Control and Prevention, spe6@cdc.gov |
| Diane Dunet, Centers for Disease Control and Prevention, dol0@cdc.gov |
| David Cotton, Macro International Inc, david.a.cotton@macrointernational.com |
| Leah Robin, Centers for Disease Control and Prevention, ler7@cdc.gov |
| Jo Anne Grunbaum, Centers for Disease Control and Prevention, jpg9@cdc.gov |
| Laura Leviton, Robert Wood Johnson Foundation, llevito@rwjf.org |
| Laura Kettel Khan, Centers for Disease Control and Prevention, ldk7@cdc.gov) |
| Evaluability assessments (EAs) provide a systematic way to determine if an intervention is ready for rigorous evaluation and identify refinements that may be needed prior to formal evaluation. This method gauges an initiative's plausibility and feasibility, stakeholder agreement on objectives, data systems and capacity to collect data, and consistency of key elements with program theory. In 2007, the Early Assessment of Programs and Policies to Prevent Childhood Obesity project conducted 25 EAs on programs and policies that address the physical, economic, or social environment to prevent childhood obesity. Three focus areas were selected: programs or policies in the after school/daycare settings; programs or policies to increase access to healthier foods in urban, low-income communities; and school district local wellness policies. This presentation will describe the EA methodology and will demonstrate the value of using a systematic process to conduct rapid assessments to identify those most suited for full-scale evaluation. |
| Teaching Evaluability Assessment Methodology |
| Thearis Osuji, Macro International Inc, thearis.a.osuji@macrointernational.com |
| Nicola Dawkins, Macro International Inc, nicola.u.dawkins@macrointernational.com |
| Holly Wethington, Centers for Disease Control and Prevention, eyk5@cdc.gov |
| Seraphine Pitt-Barnes, Centers for Disease Control and Prevention, spe6@cdc.gov |
| Diane Dunet, Centers for Disease Control and Prevention, dol0@cdc.gov |
| David Cotton, Macro International Inc, david.a.cotton@macrointernational.com |
| Leah Robin, Centers for Disease Control and Prevention, ler7@cdc.gov |
| Jo Anne Grunbaum, Centers for Disease Control and Prevention, jpg9@cdc.gov |
| Laura Leviton, Robert Wood Johnson Foundation, llevito@rwjf.org |
| Laura Kettel Khan, Centers for Disease Control and Prevention, ldk7@cdc.gov) |
| In 2007, the Early Assessment of Programs and Policies to Prevent Childhood Obesity project conducted evaluability assessments (EAs) of programs and policies with the potential to improve the eating habits and physical activity levels of children. One goal was to develop a network of evaluation professionals with the skills to conduct evaluability assessments. This presentation will focus on the process of identifying and training 33 professionals in the EA methodology, results from a post EA survey of the evaluators, and insights from the process of coordinating a large number of evaluation professionals in 3 months. The training process highlighted: the benefit of a diverse team approach; the distinction between EA and full evaluation; the critical nature of strong qualitative interview skills; the significant time commitment for evaluators; and the need for centralized oversight for methodologic integrity among evaluators. |
| Lessons Learned: Practical Tips for Conducting Evaluability Assessments |
| Seraphine Pitt-Barnes, Centers for Disease Control and Prevention, spe6@cdc.gov |
| Holly Wethington, Centers for Disease Control and Prevention, eyk5@cdc.gov |
| Karen Cheung, Macro International Inc, karen.cheung@macrointernational.com |
| Nicola Dawkins, Macro International Inc, nicola.u.dawkins@macrointernational.com |
| Diane Dunet, Centers for Disease Control and Prevention, dol0@cdc.gov |
| David Cotton, Macro International Inc, david.a.cotton@macrointernational.com |
| Leah Robin, Centers for Disease Control and Prevention, ler7@cdc.gov |
| Jo Anne Grunbaum, Centers for Disease Control and Prevention, jpg9@cdc.gov |
| Laura Leviton, Robert Wood Johnson Foundation, llevito@rwjf.org |
| Laura Kettel Khan, Centers for Disease Control and Prevention, ldk7@cdc.gov) |
| The Early Assessment of Programs and Polices to Prevent Childhood Obesity project offers key lessons learned from quickly conducting large numbers (25-30 annually) of evaluability assessments (EAs). These lessons are offered to inform others interested in or using this methodology. Among the lessons learned are: (1) clear communication is critical with evaluators and sites about roles, expectations, and the EA site visit process; (2) site documents often do not represent the program accurately; (3) involving the site in the selection of stakeholder interviewees is key to ensure comprehensive perspectives; (4) logic models are an appropriate and enlightening method for examining program goals and activities; (5) standard reporting formats provide clarity and consistency by evaluators; and (6) brief technical assistance calls and formal site reports are highly valued, low cost incentives for participation and useful tools for program/policy improvement. |
| Impact of the Evaluability Assessment Process: Early Returns |
| Laura Leviton, Robert Wood Johnson Foundation, llevito@rwjf.org |
| Holly Wethington, Centers for Disease Control and Prevention, eyk5@cdc.gov |
| Seraphine Pitt-Barnes, Centers for Disease Control and Prevention, spe6@cdc.gov |
| Diane Dunet, Centers for Disease Control and Prevention, dol0@cdc.gov |
| Leah Robin, Centers for Disease Control and Prevention, ler7@cdc.gov |
| Jo Anne Grunbaum, Centers for Disease Control and Prevention, jpg9@cdc.gov |
| David Cotton, Macro International Inc, david.a.cotton@macrointernational.com |
| Nicola Dawkins, Macro International Inc, nicola.u.dawkins@macrointernational.com |
| Laura Kettel Khan, Centers for Disease Control and Prevention, ldk7@cdc.gov) |
| The Early Assessment of Programs and Polices to Prevent Childhood Obesity project had four notable consequences in its first year of operation. (1) The process revealed more than ten policies and programs worthy of more rigorous evaluation. (2) Three such policies and programs rose to the top of the list of priorities: a day care policy for prevention of childhood obesity, a program designed to bring supermarkets back to underserved inner city environments, and an incentive program for poor families to buy fruit and vegetables at farmers' markets. (3) Most sites visited found the feedback helpful for program improvement. (4) Based on site visitor reports and expert panel discussions, themes and research issues emerged for all policy and program types that were assessed. This process might easily be emulated for other policies and programs to find out "what works." |