|
Session Title: Advances in Stakeholder Consultation for Evaluation Quality
|
|
Panel Session 864 to be held in Lone Star C on Saturday, Nov 13, 2:50 PM to 4:20 PM
|
|
Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG
|
| Chair(s): |
| Laura Leviton, Robert Wood Johnson Foundation, llevito@rwjf.org
|
| Discussant(s):
|
| Laura Leviton, Robert Wood Johnson Foundation, llevito@rwjf.org
|
| Abstract:
Evaluation quality depends in part on including consultation with the many stakeholder groups that have an interest in a program or its evaluation. Yet more concrete guidance is needed for identifying stakeholders, for engaging them in meaningful ways, for efficient ways to incorporate their suggestions into evaluation planning, and for reengaging them to make sense of findings. The three presentations represent many years of practitioner experience in doing so. Hallie Preskill will present experience to date in using a concrete, step-by-step process to engage stakeholders. Bill Trochim will reflect on years of experience in using concept mapping for this purpose. Amelie Ramirez will describe the latest in a long-established series of Delphi surveys with Latino community leaders and researchers, setting priorities for research and evaluation on childhood obesity prevention in Latino children. Discussion will focus on the cross-cutting principles and practices that affect evaluation quality.
|
|
Strategies for Involving Stakeholders in Developing Evaluation Questions
|
| Hallie Preskill, FSG Social Impact Advisors, hallie.preskill@fsg-impact.org
|
|
By soliciting the opinions, concerns and priorities of stakeholders early in the evaluation process, the results are more likely to address stakeholders’ information needs and be useful for a range of purposes, among them to improve program effectiveness, to affect policy decisions and/or to instigate behavioral change. Engaging a wide range of stakeholders in the question development process also provides opportunities to question assumptions, explore competing explanations, and develop consensus around what it is the evaluation should address. Finally, recommendations that result from an evaluation in which stakeholders have been involved are more likely to be accepted by a broader constituency and implemented more fully and with less resistance. This presentation will describe the following five-step process for involving stakeholders in developing evaluation questions: 1) preparing for stakeholder engagement, 2) identifying potential stakeholders, 3) prioritizing stakeholders, 4) considering potential stakeholders’ motivations for participating, and 5) selecting a stakeholder engagement strategy.
|
|
|
Mapping Stakeholder Views of Evaluation Questions and Plans
|
| William M Trochim, Cornell University, wmt1@cornell.edu
|
|
There is consensus in evaluation that stakeholders need to be integrally involved in the development of evaluation questions and plans but it is not clear how we might do this most effectively. One strategy is to use structured methodologies that enable stakeholders to create visual representations such as maps or models that reflect their thinking. Two such modeling approaches are considered. The first, structured concept mapping, involves stakeholders in brainstorming a set of ideas (such as the questions that might be addressed in an evaluation), individually sorting and rating those ideas and then developing maps using multivariate statistical methods (multidimensional scaling and hierarchical cluster analysis). The second uses a structured “protocol” to generate a traditional “columnar” logic model and causal pathway model of the program and its relationship to outputs and outcomes. The challenges to accomplishing these kinds of structured methods as a foundation for stakeholder-driven evaluation are discussed.
| |
|
A Nationwide Delphi Process to Set Priorities for Research and Evaluation to Prevent Obesity Among Latino Children
|
| Amelie Ramirez, University of Texas Health Science Center at San Antonio, gallion@uthscsa.edu
|
| Kipling Gallion, University of Texas Health Science Center at San Antonio, gallion@uthscsa.edu
|
|
Latino children have high obesity rates, and there is an urgent need for research and evaluation to address the epidemic. Salud America! is the Robert Wood Johnson Foundation Research Network to Prevent Obesity Among Latino Children, focusing on policy and environmental solutions to Latino childhood obesity. To identify priority research and evaluation on effective strategies, Salud America! undertook a Delphi survey with over 1,000 Latino community leaders and researchers interested in the issue. The Delphi survey, a widely used method for consensus-building, went through five main steps: identifying main research areas to be assessed; selecting participants; designing and pilot-testing the questionnaire; administering the three-round survey between May 1 and July 30, 2008 (monitoring participation, analyzing data, and providing feedback); and reporting results (http://www.salud-america.org/Files/Delphi_Executive_Summary.pdf). The Delphi survey results helped establish a research and evaluation agenda that guided a competitive grant process and funded 20 U.S. scientists to conduct pilot research projects.
| |