|
Session Title: Evaluation Capacity Building (ECB) Models, Measures, And Outcomes: Taking Stock to Forge Ahead
|
|
Panel Session 521 to be held in PRESIDIO B on Friday, Nov 12, 9:15 AM to 10:45 AM
|
|
Sponsored by the Organizational Learning and Evaluation Capacity Building TIG
|
| Chair(s): |
| Tina Taylor-Ritzler, Dominican University, tina.ritzler@gmail.com
|
| Discussant(s):
|
| Hallie Preskill, FSG Social Impact Advisors, hallie.preskill@fsg-impact.org
|
| Abstract:
Evaluation capacity building (ECB) has become an important process for organizations to respond to a myriad of accountability demands. As such, the ECB literature has grown to include models, measures and reported outcomes. This panel provides an analysis of what we know about current ECB efforts and identifies future directions for ECB research. The first presentation by Labin et al. reports the results of a research synthesis of the ECB literature. The second by Taylor-Ritzler et al. reports the results of a mixed-methods ECB model validation study and discusses a validated survey. The third presentation by Suarez-Balcazar et al. reports the results of a qualitative study that was conducted to further specify elements of the model presented in the second presentation and discusses implications of the study findings for future model validation efforts. Finally, discussant Hallie Preskill identifies implications of the presentations for current and future ECB research and practice.
|
|
Research Synthesis of ECB Literature: An Evidence-based Review
|
| Susan Labin, Independant Consultant, susan@susanlabin.com
|
| Jen Duffy, University of South Carolina, jenniferlouiseduffy@gmail.com
|
| Abraham Wandersman, University of South Carolina, wandersman@sc.edu
|
|
The pressure for evidence-based practice continues to burgeon and create expectations for organizations to engage in evaluation. This leads to a greater demand for evaluation capacity building (ECB) and for evaluating ECB. This study uses research synthesis, a core methodology used for current evidence-based reviews, to systematically code and assess the ECB literature. Existing ECB theory and frameworks provide the conceptual basis and a logic model integrates the concepts and provides the causal structure for study questions. Eighty empirical examples of ECB in the literature were reviewed and coded in an effort to systematically describe ECB strategies, how these strategies were evaluated, what outcomes were reported, and how contextual and implementation factors may have affected these strategies or their outcomes. The presentation builds on our previous research by focusing on synthesis findings related to the evaluation methods used and the variables associated with ECB outcomes.
|
|
|
Results and Implications of a Mixed-methods ECB Model Validation Study
|
| Tina Taylor-Ritzler, Dominican University, tina.ritzler@gmail.com
|
| Yolanda Suarez-Balcazar, University of Illinois at Chicago, ysuarez@uic.edu
|
| Edurne García-Iriarte, Trinity College Dublin, edurne21@yahoo.com
|
|
The purpose of this presentation is to describe the development, validation results and implications of a mixed-methods ECB model. The presenters will summarize current ECB models and measures and show how their model and measures represent a synthesis of the literature. Specifically, synthesis model and measurement components include individual factors (awareness of the benefits of evaluation, motivation, and competence), organizational factors (leadership, learning climate and resources), cultural and contextual factors, and evaluation capacity outcomes (use of evaluation processes and findings). Measures include organizational document reviews and staff interviews and surveys. Model validation results differed for quantitative and qualitative methods. Specifically, quantitative results revealed a model that included individual, organizational and outcome factors, whereas qualitative results showed cultural and contextual factors to also be important. The implications of the study findings for further developing a synthesis model and measure of ECB will be discussed.
| |
|
Using Qualitative Methods to Further Specify Contextual and Cultural Elements of ECB Processes
|
| Yolanda Suarez-Balcazar, University of Illinois at Chicago, ysuarez@uic.edu
|
| Tina Taylor-Ritzler, Dominican University, tina.ritzler@gmail.com
|
|
Several evaluation scholars (e.g., Cousins et al., 2007; Preskill & Boyle, 2009) and results of the second presentation in this panel have noted that cultural and contextual elements affect ECB processes. However, the specific elements of culture and context, and how these affect ECB processes, have not yet been adequately specified. As a result, the presenters conducted a review of the evaluation literature to assess how context and culture have been described and assessed. They will describe the literature in this area and the results of a qualitative study wherein staff from community-based organizations who were responsible for evaluation activities was asked to discuss contextual and cultural factors that impact their capacity to document their programs. The presenters will also describe how the study results will be incorporated into their ECB synthesis model and measure and the next steps for validating the revised model.
| |