Search Results for All Sessions

Session Title: Using the Program Evaluation Standards, Third Edition, to Define and Enhance Evaluation Quality

Skill-Building Workshop 362 to be held in Lone Star A on Thursday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the

Presenter(s):

Sarah Kohler Chrestman, Louisiana Public Health Institute, skohler@lphi.org

Lisanne Brown, Louisiana Public Health Institute, lbrown@lphi.org

Abstract: As the greater New Orleans (GNO) region is continuing to rebuild, the need for democratized data at the neighborhood level continues to grow. Instead of maximizing efforts, there is a great deal of duplication of data collection because agencies, organizations, and residents are unaware of what is already available. The Orleans Neighborhood Health Implementation Plan (ONHIP) is working to improve the availability of data through the development and use of a public website with neighborhood specific data and interactive mapping and query capability. This presentation will discuss which data sources are publicly available, the issues to consider when democratizing data, and its benefits and challenges. Data quality is of utmost importance as poor data can cause more harm than good. Participants will learn methods, including education, to ensure the quality of the data is maintained and to reduce any opportunities to misrepresent data.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Evaluation Goes to College: The Collaborative Evaluation of a Graduate Program

Roundtable Presentation 383 to be held in BONHAM A on Thursday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG

Presenter(s):

Seriashia Chatters, University of South Florida, schatter@mail.usf.edu

EunKyeng Baek, University of South Florida, ebaek@mail.usf.edu

Thanh Pham, University of South Florida, tvpham2@mail.usf.edu

Yvonne Hunter, University of South Florida, yohunter@mail.usf.edu

Abstract: A need was identified at a large, public, US university to determine the level of satisfaction of students and faculty in a Counselor Education graduate program. There is a wealth of research available regarding the evaluation of K-12 programs. However, there is limited information regarding the evaluation of higher education programs especially evaluations applying the collaborative approach. This evaluation was conducted to identify students and faculty satisfaction level of the graduate program and to recognize the differences between evaluating a K-12 program and the evaluation of a graduate program. In order to identify the needs and concerns of all relevant stakeholders, a Collaborative Evaluation approach was utilized. We will discuss how the process and procedures of the collaborative approach were implemented and strengths and challenges of utilizing this method in evaluating a graduate program will be addressed.

Roundtable Rotation II: Working Together to Design Effective Evaluation Tools

Roundtable Presentation 383 to be held in BONHAM A on Thursday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG

Presenter(s):

Rebeca Diaz, WestEd, rdiaz@wested.org

Abstract: This presentation will discuss a collaborative approach to developing effective evaluation instruments with key stakeholders carrying out a federal education grant. The main goals of the grant are to increase teacher content knowledge in U.S. history, enhance teacher practice, and increase student learning. The presenter has seven years of experience evaluating these federal grants designed to provide professional development for history teachers, and continues to explore new methods to effectively measure teacher outcomes. The evaluation approach, which consists of both qualitative and quantitative methods, is collaborative in nature. The evaluator employs an approach that involves not only project leaders but also the teachers involved in the program.

Session Title: San Antonio River Improvements Project: Field Trip to Ecosystem Restoration Sites

Demonstration Session 384 to be held in OFF SITE FIELDTRIP on Thursday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Environmental Program Evaluation TIG

Presenter(s):

Annelise Carleton-Hug, Trillium Associates, annelise@trilliumassociates.com

Abstract: This demonstration session offers an opportunity to learn about the massive multi-agency river restoration efforts underway on the San Antonio River just south of downtown San Antonio. The session includes a site visit and walking tour of the Eagleland and Mission Reach portions of the river that were previously channelized for flood control. The current project is restoring a more natural channel morphology and native plants. The tour will be lead by a specialist from the San Antonio River Authority, and discussions will include the pre-project assessment, project goals, evaluation and monitoring plans. Additional topics will be the challenges of ecosystem restoration and conservation in the urban interface, and addressing recreation uses. Space for this demonstration/field trip is limited.

Session Title: Education Evaluation: Connecting Professional Development to Changes in Classroom Practice

Multipaper Session 385 to be held in BONHAM C on Thursday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Chair(s):

Bianca Montrosse, Western Carolina University, bianca.montrosse@gmail.com

Discussant(s):

Susan Connors, University of Colorado, Denver, susan.connors@ucdenver.edu

Applying Guskey’s Model for Evaluating Professional Development to a Math and Science Partnership Program: Successes and Challenges in Collecting Data Across Schools and Grade Levels

Presenter(s):

Carol Haden, Magnolia Consulting LLC, carol@magnoliaconsulting.org

Jane Kirkley, Northern Arizona University, jane.kirkley@nau.edu

Abstract: This paper describes the successes and challenges of applying Thomas Guskey’s 5-level model for evaluating professional development to a program funded by Arizona’s Math and Science Partnership program. Professional development was provided to two cohorts of K-8 teachers in Northern Arizona across two years of the project. Program goals were to increase participants’ science content knowledge while also building skills in effective science pedagogy. Evaluation activities and data collection were particularly successful for understanding participant reactions (Level 1), participant learning (Level 2), and participant use of knowledge and skills (Level 4). Participants represented many different schools, and grade levels varied from kindergarten to middle school. These unique contextual factors led to challenges in evaluating organizational support and change (Level 3) and student learning outcomes (Level 5). We describe data collection activities at each level and how lessons learned in year one informed revisions to evaluation activities for year two.

Using the Transtheoretical Model of Readiness for Change to Evaluate the True Impact of Evidence-based Professional Development in the K-12 Setting

Presenter(s):

Christa Smith, Kansas State University, christas@ksu.edu

Katherine Sprott, Kansas State University, krs8888@ksu.edu

Abstract: For professional development interventions in the K-12 setting that are delivered in varying degrees, a holistic program evaluation that measures true attitudinal and behavioral change at individual and systems levels is necessary. The Transtheoretical Model (TTM) of stages of change (Prochaska and DiClemente, 1983, 1984, 1986) was applied as a framework to assess the impact of evidence-based professional development interventions for K-12 school personnel. This presentation will demonstrate the use of the TTM framework as illustrated through the program evaluation of evidence-based interventions implemented by a US Department of Education funded Equity Assistance Center. The presentation will discuss the application of the TTM framework in applying relevant metrics to measure the stages of attitude, knowledge, and behavior change and significant relations to appropriate demographic variables and the ultimate equity program outcome of establishing race, gender, and national origin equity in school environments.

Determining the Validity of tEacher Self-Reports as a Cost-Effective Strategy to Evaluate Teacher Performance

Presenter(s):

Kasey McCracken, David Heil & Associates Inc, kmccracken@davidheil.com

Gina Magharious, David Heil & Associates Inc, gmagharious@davidheil.com

Joe Sciulli, National Science Teachers Association, jsciulli@nsta.org

Abstract: This paper reports findings from a validation study of teacher-reported use of instructional strategies, based on independent classroom observations. The validation study was undertaken to support the evaluation of the Mickelson ExxonMobil Teachers Academy — a professional development experience serving approximately 500 teachers annually that is designed to improve elementary school teachers’ use of inquiry-based instructional strategies for teaching science. The evaluation of the Academy includes measuring changes in teachers’ reports of their instructional strategies from before to after their participation in the Academy and one year later. During Spring 2009 and 2010, independent classroom observations were conducted for a sample of 48 teachers. Teachers’ ratings of their instructional strategies were correlated with observers’ ratings. However, teachers tended to provide more favorable ratings of their instructional strategies than did the independent observers. Findings from the validation study inform strategies for designing quality educational evaluations that balance research validity and cost effectiveness.

Qualitative Analysis of Changes in Teachers' Knowledge, Beliefs and Classroom Practices Based on Three Years of Professional Development

Presenter(s):

Carol Baldassari, Lesley University, baldasar@lesley.edu

Sabra Lee, Lesley University, slee@lesley.edu

Rosalie Torres, Torres Consulting Group, rosalie@torresconsultinggroup.com

Abstract: This presentation details analysis and reporting methods for teacher case studies conducted as part of NSF’s Mathematics and Science Partnership Program. This program funds partnerships between universities and school districts to improve teacher quantity and quality. We conducted in-depth case studies of four mathematics teachers, focusing on their participation in an immersion program of mathematics professional development and their subsequent transfer of learning to the classroom. The data collected, over 1.5 to 2.5 years, included observations of their professional development sessions, review of papers they wrote, interviews with both the teachers and the faculty who taught them, and classroom observations. The data analysis and case study writing methods to be presented helped reveal the significant impact of contextual factors (such as school- and/or district-level circumstances, and teachers’ backgrounds and experiences) that bear substantially on teacher learning from professional development and how they are able to transfer it to the classroom.

Session Title: Addressing Schools as Organizations in Educational Evaluation

Multipaper Session 386 to be held in BONHAM D on Thursday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Chair(s):

Diane Binder, The Findings Group LLC, diane.binder@thefindingsgroup.com

Discussant(s):

Chad Green, Loudoun County Public Schools, chad.green@loudoun.k12.va.us

Documenting Patterns in Teacher’s Relationships: Their Impact on K-12 Education

Presenter(s):

Kathy Gullie, State University of New York at Albany, kp9854@albany.edu

Abstract: When looking at the impact of teachers’ professional development on student outcomes, there appears to be an evaluation gap between documenting involvement in professional development and documenting outcomes that represent the acquisition of skills and the transference of those skills to the real world. Current leadership models that stress a stakeholder based approach to the establishment of peer to peer relationships suggests possible solution to the development of evaluation criteria that will help bridge this gap. The purpose of this paper is to report on successful methods of documenting K-12 teacher interactions and their relationship to instructional practices and students’ subsequent outcomes. Findings are based on the result of two consecutive, three year Mathematics Science Partnership grants that unitized leadership theory for pattern matching when analyzing case studies, interviews, observation and focus group data

The Use of School Climate Data for School Improvement

Presenter(s):

Sarah Gareau, South Carolina Educational Policy Center, gareau@mailbox.sc.edu

Diane Monrad, University of South Carolina, dmonrad@mailbox.sc.edu

John May, University of South Carolina, mayjr@mailbox.sc.edu

Karen Price, South Carolina Educational Policy Center, pricekj@mailbox.sc.edu

Diana Mindrila, South Carolina Educational Policy Center,

Ishikawa Tomonori, University of South Carolina, ishikawa@mailbox.sc.edu

Abstract: Previous research suggests that school climate data can be very useful in understanding the complex dynamics of the relationships between organizational-level contexts and evaluation outcomes. While measures of school success are essential for schools to show progress under state and federal accountability requirements, assessing school climate as a critical element of school improvement has received only passing interest from policy makers. The purpose of the current collaborative work is the analysis of 2008 and 2009 school climate surveys and the development of 4-year school climate profiles (2006-2009) focused on low-performing schools, including a 4-year comparison of mean factor scores by organizational level, percentile ranks of survey factor scores by organizational level, and item-level percentage agreement indices. A discussion of the profiles’ development, meaning, and use for evaluation quality provides a practical application of school climate data.

From Monitoring to Evaluation and Back Again: Implications for Organizational Leadership, Budget, and Success

Presenter(s):

Lisa Schmitt, Austin Independent School District, lschmitt@austinisd.org

Karen Cornetto, Austin Independent School District, kcornett@austinisd.org

Lindsay Lamb, Austin Independent School District, lindsay.lamb@austinisd.org

Abstract: Over the past decade, the Austin ISD Board of Trustees adopted a policy governance management model, under which district administration reported monthly the district’s status on indicators viewed as evidence that policies outlining student expectations and district operations were implemented. This piecemeal monitoring approach, however, led the Board to request a new format that considered a holistic picture of performance at each level (elementary, middle, and high). The transition from performance monitoring towards a systemic academic program evaluation resulted in a unique collaboration among evaluators, district administrators, and campus administrators who provided the Board an integrated, user-friendly report describing internal research on what matters most to student achievement at each level, evidence on programs that influence what matters, and plans for addressing what matters most effectively in the future. This session outlines the challenges and rewards of transitioning from monitoring to evaluation, and discusses the necessary relationship between the two.

The Utility of Situation Models for Capturing the Present State of a School-Wide Initiatives

Presenter(s):

Chad Green, Loudoun County Public Schools, chad.green@loudoun.k12.va.us

Abstract: Honig’s (2008) model of central offices as learning organizations suggests that administrators can cultivate six different school capacities regardless of the school’s level of engagement in teaching and learning practices (from expert to novice). These six capacities were aligned with the NSDC’s (2001) context standards to develop an exploratory conceptual framework that guided the collection, analysis, and interpretation of data from two school-wide initiatives at different stages of implementation. Each initiative served as a leverage point for documenting the coherence and alignment of the school’s program practices within the framework. The resulting situation models (Kintsch, 1988; van Dijk and Kintsch, 1983) revealed different patterns of interrelationships for each school that corresponded to their stage of program implementation.

Session Title: Evaluating the Science of Discovery in Complex Health Systems: Challenges and Opportunities

Panel Session 387 to be held in BONHAM E on Thursday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Research, Technology, and Development Evaluation TIG

Chair(s):

Alison Buchan, University of British Columbia, abuchan@medd.med.ubc.ca

Discussant(s):

Alison Buchan, University of British Columbia, abuchan@medd.med.ubc.ca

Abstract: Complex health problems such as chronic disease or pandemics require knowledge that transcends disciplinary boundaries in order to generate solutions. Such transdisciplinary discovery requires researchers to work and collaborate across boundaries, combining elements of basic and applied science. At the same time, calls for more interdisciplinary health science acknowledge that there are few metrics to evaluate the products associated with these new ways of working. The Research on Academic Research (RoAR) initiative was established to evaluate the process of discovery and impact of collaboration that emerged through the Life Sciences Institute at the University of British Columbia, a state-of-the-art facility designed to support researchers - self-organized around specific health problems rather than disciplines. A logic model depicting the factors influencing such collaboration is presented along with a multi-method evaluation plan to assist understanding of the discovery process in this new environment and develop new metrics for assessing collaborative impact.

An Evaluation Framework for Advancing the Science of Evaluating Team Science: The Research on Academic Research Initiative (RoAR)

Cameron Norman, University of Toronto, cameron.norman@utoronto.ca

Timothy Huerta, Texas Tech University, tim.huerta@ttu.edu

Sharon Mortimer, Michael Smith Foundation for Health Research, smortimer@msfhr.org

Allan Best, Michael Smith Foundation for Health Research, allan.best@in-source.ca

Alison Buchan, University of British Columbia, abuchan@medd.med.ubc.ca

Background: In 2006 the University of British Columbia opened the Life Sciences Institute (LSI), the first building of its size at UBC to be developed to support cross-disciplinary bio-sciences team research. The Research on Academic Research (RoAR) initiative was initiated in 2007 to serve as a platform for conducting exploratory research and evaluation of the effect that co-location of previously disparate researchers and institutional policies have on the organization and output of scientists. Methods: A logic model was developed to guide a multi-method evaluation that aimed to assess the outcomes associated with the re-organization of the scientists from academic department buildings into integrated problem-based research groups. These methods included a survey of psychosocial issues and social networks, an examination of publication and grant application patterns, measures of physical proximity, and investigator interviews. Discussion: Evaluation of team science activities requires a strategy that can address process and outcomes from multiple perspectives.

Advancing the Science of Evaluating Team Science: Psychosocial Factors and Related Outcomes From the RoAR Initiative

Cameron Norman, University of Toronto, cameron.norman@utoronto.ca

Timothy Huerta, Texas Tech University, tim.huerta@ttu.edu

Sharon Mortimer, Michael Smith Foundation for Health Research, smortimer@msfhr.org

Alison Buchan, University of British Columbia, abuchan@medd.med.ubc.ca

Background: Much of the research in basic science is investigator-driven and favors individual scientists or small groups. Team science is a different orientation, requiring new skills and knowledge. A survey was developed to assess the level of comfort and skills of investigators working in a team science environment over four years. Method: Investigators with the Life Sciences Institute at the University of British Columbia were surveyed on their attitudes, knowledge, collaborative skills, and perceived benefits and risks associated with various models of research were taken. Conclusions: The overall attitudes, skills and knowledge about how to do team science and its perceived value to academic work changed considerably over four years. Dr. Norman will draw on his background on health behavior change, systems science and evaluation research to discuss the measurement challenges in evaluating team science.

Advancing the Science of Evaluating Team Science: Social Network Outcomes From the RoAR Initiative

Timothy Huerta, Texas Tech University, tim.huerta@ttu.edu

Cameron Norman, University of Toronto, cameron.norman@utoronto.ca

Sharon Mortimer, Michael Smith Foundation for Health Research, smortimer@msfhr.org

Alison Buchan, University of British Columbia, abuchan@medd.med.ubc.ca

Background: The shift from individual or small-group research to team science models requires shifts in social interaction patterns. Social network analysis can help evaluate these shifts. Method: Participants were asked to indentify the investigators in the Life Sciences Institute (LSI) that they interacted with, the nature of those interactions and the degree to which it influenced their work in annual surveys over four years. Results: An increase in the absolute numbers of collaborations within LSI was observed. While overall network density increased and the frequency of inter-departmental and inter-research group grew, the number of such increases suggest a developmental and slow growing process. Conclusions: After four years the researchers in the LSI show significant alterations structure and processes used to guide interactions. Dr. Huerta will draws on more than a decade of experience with social network and systems research in exploring the role of social networks in evaluating team science. Conclusions: Taking the Donabedian model of Structure-Process-Outcome indicates that the researchers in the LSI show significant alterations structure and processes used to guide interactions after four years.

Advancing the Science of Evaluating Team Science: Scientometric-related Outcomes From the RoAR Initiative

Sharon Mortimer, Michael Smith Foundation for Health Research, smortimer@msfhr.org

Timothy Huerta, Texas Tech University, tim.huerta@ttu.edu

Bianca Cervantes, University of British Columbia, bcervantes@exchange.ubc.ca

Alison Buchan, University of British Columbia, abuchan@medd.med.ubc.ca

Background: The shift from individual or small-group research to team science models can be observed in the publications and grant application patterns of investigators over time. Method: A comparison of funding by research group in the Life Sciences Institute (LSI) was completed for the major Canadian sources: Tri-Council (equivalent to NIH), peer-reviewed grants from other sources and research contracts. Results: The data demonstrated a significant increase in Tri-Council funding in all groups, while comparison of publication metrics indicated potential shifts after 3 years. Conclusions: Team science may be more promising as a discovery strategy within certain LSI areas. It is still too early to make definitive statements about the effect of team science model at the LSI with the exception of a rise in impact of papers in certain fields. Dr. Mortimer will expand on her role as a funder and researcher to discuss team science evaluation and its implications.

Session Title: Government Evaluation TIG Business Meeting and Panel: Happy Anniversary to Us! Celebrating Twenty Years of Government Evaluation

Business Meeting with Panel Session 388 to be held in Texas A on Thursday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Government Evaluation TIG

TIG Leader(s):

Stanley Capela, HeartShare Human Services of New York, stan.capela@heartshare.org

David J Bernstein, Westat, davidbernstein@westat.com

Sam Held, Oak Ridge Institute for Science and Education, sam.held@orau.org

Chair(s):

Stanley Capela, HeartShare Human Services of New York, stan.capela@heartshare.org

Abstract: In 1990, the theme of the AEA conference was Evaluation and the Formulation of Public Policy, a topic that is central to the role of evaluation in government. During the 1990 conference, a session attended by about 20 people gathered to discuss the possibility of establishing a State and Local Government Evaluation Topical Interest Group (TIG), which was approved by the AEA Board in early 1991. In 2005, the TIG’s focus was broadened, and the name was changed to the Government Evaluation TIG. This panel will celebrate the 20th anniversary of the TIG with three highly relevant presentations, including a panel discussion with the current and past chairs of the Government Evaluation TIG, a key note address by Joe Wholey, one of the leading experts on government evaluation in the United States, and the Government Evaluation TIG’s annual business meeting.

Panel Discussion: How Has Government Evaluation Changed in the Last Twenty Years?

David J Bernstein, Westat, davidbernstein@westat.com

Maria Whitsett, Moak, Casey and Associates, mwhitsett@moakcasey.com

Rakesh Mohan, Idaho State Legislature, rmohan@ope.idaho.gov

Stanley Capela, HeartShare Human Services of New York, stan.capela@heartshare.org

For the first time ever, a panel comprised of the present and several past chairs of the Government Evaluation TIG will gather to address some questions of interest to AEA and Government TIG members: Over the past 20 years how has the role of program evaluation changed within government? How do those working on government evaluation ensure quality? How have governments in different sectors (federal, state, local, tribal/indigenous, international) defined the concept of quality in evaluation processes? Do intergovernmental funding arrangements (e.g., federal, state, local grants or contracts) change the requirements for quality evaluations, and if so, how? Stan Capela, Chair of the Government Evaluation TIG and three Past Chairs, David Bernstein, Maria Whitsett, and Rakesh Mohan, will conduct a panel discussion to consider how government evaluation has changed (or not) in the last 20 years.

Keynote Address: How Has Evaluation Changed in the Last 20 to 50 Years?

Joseph Wholey, University of Southern California, joewholey@aol.com

Since the 1970s, governments and agencies at all levels have been using performance monitoring and performance management systems, in part because monitoring of progress toward program goals fits comfortably under the definition of program evaluation. By 1990, government evaluation had already been growing for thirty years. In the 2000s, high-stakes performance monitoring spread to every state in response to the No Child Left Behind Act. Evaluation is used in government to increase transparency, strengthen accountability, support decision-making, and improve the performance and value of public programs. As an evaluator, government manager, political appointee, elected official, consultant, academic, researcher, and author, Joe Wholey has been a witness and active participant to these and every other major change in the evaluation field in the last 40 plus years. Dr. Wholey will discuss how government evaluation has changed, and what changes we might expect to see in the near future.

Government Evaluation Topical Interest Group (TIG) Business Meeting

Stanley Capela, HeartShare Human Services of New York, stan.capela@heartshare.org

David J Bernstein, Westat, davidbernstein@westat.com

Sam Held, Oak Ridge Institute for Science and Education, sam.held@orau.org

Following the panel discussion and key note address, the Government Evaluation TIG will hold its annual Business Meeting during the AEA 2010 conference. Topics include succession planning for TIG leadership, ensuring the relevance of the TIG, "hot issues" affecting the conduct of government sponsored evaluation work, and soliciting ideas for TIG-sponsored programs and services.

Session Title: Using Logic Models to Facilitate Comparisons of Evaluation Theory

Multipaper Session 389 to be held in Texas B on Thursday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Theories of Evaluation TIG

Chair(s):

Marv Alkin, University of California, Los Angeles, alkin@gseis.ucla.edu

Discussant(s):

Robin Lin Miller, Michigan State University, mill1493@msu.edu

Visual Representations of Evaluation Theories

Presenter(s):

Tanner LeBaron Wallace, University of Pittsburgh, twallace@pitt.edu

Mark Hansen, University of California, Los Angeles, markhansen@ucla.edu

Abstract: Here, we describe the development of logic models depicting selected theories of evaluation practice. We begin with a discussion of the particular theories that were chosen for our analysis, then outline the steps involved in constructing the models. Mark’s (2008) framework for research on evaluation provides the organization to this session. We present that framework and describe its relevance to the examination of evaluation theories.

Session Title: Examining the Mixing in Mixed Methods Evaluation

Multipaper Session 390 to be held in Texas C on Thursday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Evaluation Use TIG

Chair(s):

Jennifer Iriti, University of Pittsburgh, iriti@pitt.edu

Process Evaluation for Program Improvement: Lessons From the Interns for Indiana Program

Presenter(s):

Omolola A Adedokun, Purdue University, oadedok@purdue.edu

Loran Carleton Parker, Purdue University, carleton@purdue.edu

Wilella Burgess, Purdue University, wburgess@purdue.edu

Abstract: Participant perceptions of the extent to which program outcomes are achieved are undoubtedly a popular focus of both formative and summative program evaluations. However, the effects of participants’ characteristics and contextual variables on program perceptions are less explored in formative evaluation studies where the resulting feedback can be used for program improvement. The purpose of this presentation is to demonstrate how the empirical feedback from such investigations can be used to improve program implementation and the general understanding of program processes. Data for the study is from the formative evaluation of the Interns for Indiana (IfI) program, a multi-site entrepreneurial internship program designed to increase interns’ desire to work in their home state after graduation by providing them with opportunities for experiential learning in small/startup companies. Multivariate linear regression analysis was employed to examine the effects of demographic and contextual variables on participants’ perceptions of program outcomes.

Ensuring Program Quality: Lessons Learned From Implementation Evaluation of the Kentucky Alternative Certification in Special Education (KACSE) Program

Presenter(s):

Imelda Castañeda-Emenaker, University of Cincinnati, castania@ucmail.uc.edu

Norma Wheat, Campbellsville University, nrwheat@campbellsville.edu

Abstract: This presentation highlights the role of implementation evaluation in assisting the program manager of the alternative certification program to make informed decisions about the program directions. The changes in the economic situations around the school districts participating in the program posed challenges in the program’s targeted goals, which threatened the program’s major funding. Implementation evaluation and formative feedback helped the program manager in improving its programming and in being proactive to avert the negative effects of delaying placements of certified teachers with the economically-challenged school districts. Implementation evaluation provided the rationale for the program changes made and helped ensure appropriate program accountability, without jeopardizing the program status with its funder.

Evaluation Synergy: A Multi-purpose Evaluation Design for Fundraising, Applied Learning, and Community Service

Presenter(s):

Nancy Rogers, University of Cincinnati, nancy.rogers@uc.edu

Jennifer Williams, University of Cincinnati, jennifer.williams@healthall.com

Brian Powell, University of Cincinnati, powellbb@mail.uc.edu

Abstract: Most non-profit programs recognize the value of evaluation, but more often they do not have the resources to finance an evaluation. Expanding the use of evaluation to more than simply providing program feedback or outcome data could prove advantageous to small programs on a limited budget. One small violence intervention program leveraged the resources of a local university to involve students in service learning, increased community awareness, and applied learning, all while evaluating fundraising materials and raising money for the program. The presenters will share how they crafted a meaningful collaboration using social psychological research to develop and evaluate several fundraising appeals while providing an applied educational experience that resulted in multiple wins.

The Use of Evaluation in Agricultural Policy Making: The Case of Mexico

Presenter(s):

Alfredo Gonzalez Cambero, Food and Agriculture Organization of the United Nations, agonzalez@fao-evaluacion.org.mx

Salomon Salcedo Baca, Food and Agriculture Organization of the United Nations, salomon.salcedo@fao.org

Abstract: Within the framework of international technical cooperation, the Food and Agricultural Organization of the United Nations (FAO) has conducted evaluations for the Mexican Government for a period of more than ten years. Although many evaluations have been carried out in the field of agricultural policy evaluation, little use has been made of the results. While both FAO and the Mexican Government have intended to use evaluation to advance their common objectives of rural development, the evaluations themselves have been used to serve business as usual, rather than to improve the programs through the implementation of findings or to better understand the conceptual underpinnings of the programs. The paper looks into building a virtuous circle of quality-use of the evaluation, identifying factors within the evaluation process itself, as well as institutional constraints, in order to make better use of the evaluation as a valuable tool for policy making.

Session Title: Research on Evaluation Standards and Methods

Multipaper Session 393 to be held in Texas F on Thursday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Research on Evaluation TIG

Chair(s):

Matthew Galen, Claremont Graduate University, matthew.galen@cgu.edu

Social Science Standards and Ethics: Development, Comparative Analysis, and Issues for Evaluation

Presenter(s):

Linda Mabry, Washington State University, Vancouver, mabryl@vancouver.wsu.edu

Abstract: This paper presents a comparative analysis of the Guiding Principles for Evaluators (AEA, 1995), The Program Evaluation Standards (Joint Committee, 1994), and the codes of ethics and standards established by the American Sociological Association (1997), the American Psychological Association (APA, 2007), the American Sociological Association (ASA, 1997) and other social science organizations. The theoretical contexts for cross-codes analysis are Kohlberg's (1984) theory of human moral development and Rawl's theory of justice (1971); the historical context is the origins and development during the past half-century of ethical codes by governments and international agencies. Three issues specific to evaluation are discussed: (1) potential conflicts between evaluation's codes of conduct and U.S. legal requirements regarding ethics in social science, (2) the cultural sensitivity and appropriateness of codes of conduct in international and transnational evaluations, and (3) the posssibility and advisability of enforcing evaluation codes of conduct.

Insight into evaluation practice: Results of a content analysis of designs and methods used in evaluation studies published in North American evaluation focused journals

Presenter(s):

Christina Christie, University of California, Los Angeles, tina.christie@ucla.edu

Dreolin Fleischer, Claremont Graduate University, dreolin@gmail.com

Abstract: To describe the recent practice of evaluation, specifically method and design choices, we performed a content analysis on 117 evaluation studies published in eight North American evaluation-focused journals for a 3-year period (2004-2006). We chose this time span because it follows the scientifically-based research (SBR) movement, which prioritizes the use of randomized controlled trials (RCTs) to study programs and policies. The purpose of this study was to determine the designs and data collection methods reportedly used in evaluation practice in light of federal guidelines enacted prior to 2004. Results show that, in contrast to the movement, non-experimental designs dominate the field, that mixed-methods approaches just barely nudged out qualitative methods for most commonly used, and that the majority of studies where statistical significance was reported indicated mixed-significance.

Standards for Evidence-based Practices and Policies: Do Campbell Collaboration, Cochrane Collaboration, and What Works Clearinghouse Research Reviews Produce the Same Conclusions?

Presenter(s):

Chris Coryn, Western Michigan University, chris.coryn@wmich.edu

Michele Tarsilla, Western Michigan University, michele.tarsilla@wmich.edu

Abstract: Interventions intended to ameliorate, eliminate, reduce, or prevent some persistent, problematic feature of the human condition have existed for millennia. In a climate of increasingly scarce resources and greater demands for accountability, now, more than ever, policy makers and practice-based disciplines and professions are consistently seeking high-quality, non-arbitrary, and defensible evidence for formulating, endorsing, and enforcing “best” policies and practices. In the last few decades, randomized experiments, randomized controlled trials, and clinical trials, universally have become the standard for supporting inferences and claims regarding the efficacy, effectiveness, and, to a lesser extent, generalizability, of such actions. In this presentation the authors will present a study of the degree to which the standards applied by major repositories for evidence-based practices and policies produce the same conclusions about the same studies.

Can Systematic Measurement of an Evaluations’ Goodness of Fit and its Influence Determine Quality?

Presenter(s):

Janet Clinton, University of Auckland, j.clinton@auckland.ac.nz

Abstract: Given the expanding and influencing role of evaluation it is critical that the quality of the evaluation process be more fully scrutinized. While we monitor the quality of evaluation processes it is rare to consider outcomes that are attributable to the evaluation. To understand quality we must take in to account the impact of evaluation on a program’s effectiveness and efficiency. This paper uses a heuristic to illustrate how program components and evaluation processes can be combined to produce an explanation of effectiveness. The goodness of fit of an evaluation process and an evaluations influence can be analysed with appropriate weightings to produce information to ensure quality judgements occur. A number of evaluation cases are used to demonstrate a method of monitoring and measuring evaluation influence. It is argued that a judgement of quality evaluation lies in systematic measurement of an evaluations’ goodness of fit and its influence.

Session Title: Evaluation Methodology

Multipaper Session 394 to be held in CROCKETT A on Thursday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Extension Education Evaluation TIG

Chair(s):

Paul Pope, Texas A&M University, ppope@aged.tamu.edu

Developing Surveys for Low-Literate Adults Receiving Extension Education Classes

Presenter(s):

Karen Franck, University of Tennessee, Knoxville, kfranck@utk.edu

Abstract: Developing effective surveys for low-literate adults is a challenge for program evaluators. This paper examines patterns of missing values in surveys completed by low-income adults. The survey included 3 question types: language only behavior questions, language only attitude questions, and combined photographic and language behavior questions. Over 300 adults completed surveys at the beginning of an Extension nutrition education intervention. This was a diverse, high risk group (30% minorities, 79% food insecure, 35% high school dropouts, 80% unemployed). Participants with lower levels of education and those who spoke English as a second language were more likely to skip questions about attitudes (“It costs more to eat healthy foods) and write-in responses about behaviors even when combined with photographs (“How many servings of fruit do you eat each day?” with a photograph of fruit.) This paper will discuss implications of these findings for developing evaluation tools for low-literate adults.

Optimizing Conditions for Success: An Extension Case Study in Cross-program Surveys

Presenter(s):

Gwen Willems, University of Minnesota, wille002@umn.edu

Abstract: Surveying is a highly popular method that has been used for decades to gather evaluative data. The advantages are numerous: it’s a relatively low-cost and straight-forward way to obtain data from many people in a short period of time, as described by Gary Henry. One of the difficulties is standardizing surveys across a variety of programs. Much of the literature and discussion of surveys gives attention to survey design, error reduction, the audience, and sampling of respondents. This presentation will step back, instead doing a meta-analysis that focuses on a process and environment that led eventually to successful design and adoption of cross-program end-of-educational-session and follow-up surveys for a section of the University of Minnesota Extension. The presenter will describe this case study, the multi-activity process she used with Extension educators, challenges in that environment, and factors that contributed to success of the cross-program survey initiative.

Creating a Cost-Benefits Analysis Calculator for Extension Nutrition Education Programs

Presenter(s):

Karen Franck, University of Tennessee, Knoxville, kfranck@utk.edu

Joseph Donaldson, University of Tennessee, jldonaldson@tennessee.edu

Abstract: The 1998 Virginia Tech cost-benefits analysis study remains the standard for measuring the economic impact of Expanded Food and Nutrition Education Program (EFNEP)—a federally funded national nutrition education program for low-income parents. Since 1998, several other states have used these methods to estimate the economic impacts of EFNEP in their communities. This paper will discuss a project that built on the Virginia Tech study in 3 important ways: first, the 1998 direct and indirect costs were updated to reflect current dollars; the target audience was expanded to include all adults who receive nutrition education programming through a university Extension program; third, an on-line calculator was created to capture the economic impacts of nutrition programs at county, regional, and state levels. This calculator provides program evaluators with an effective method to measure economic impact, to compare the impact of programs between different areas, and to identify areas for program improvement.

Utilizing the Delphi Method to Identify Competencies and Training to Help Reduce Turnover Among County Extension Faculty

Presenter(s):

Diane Craig, University of Florida, ddcraig@ufl.edu

Abstract: Turnover of county Extension faculty costs Extension millions of dollars per year (Ramlall, 2004). Employee turnover occurs for a multitude of reasons including the lack of job satisfaction, organizational commitment, and job embeddedness (Phillips & Connell, 2003). These factors can be addressed through proper training and strategic techniques that improve employee critical competencies, socialization and job satisfaction. The University of Florida conducted a Delphi study to explore the perceptions of county extension faculty regarding job satisfaction, training competencies, and social connectedness during the first three years of hire. A Delphi study is designed to gain consensus among experts on a given topic, with our experts being our county Extension faculty and their supervisors. The goal of this study was to determine the optimal training content and training schedule for newly hired Extension faculty in order to increase job satisfaction and job embeddedness and decrease turnover.

Session Title: Process Lessons for Applied Research and Evaluation from Capital Projects

Demonstration Session 395 to be held in CROCKETT B on Thursday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Business and Industry TIG

Presenter(s):

Kate Rohrbaugh, Independent Project Analysis, krohrbaugh@ipaglobal.com

Abstract: Capital projects in industry are projects that require the investment of significant capital to maintain or improve a capital asset. In this demonstration, the presenter will provide an overview on the practices that are considered adequate planning in the world of capital projects and identify the parallels (of which there are many) for applied research and evaluation. These parallels were identified during the development of a research work process intended to improve and maintain the intellectual assets of Independent Project Analysis (IPA), a management consulting firm in Virginia that offers evaluation services and conferences for companies in the process industries. During this demonstration, the audience will become familiar with the practices and the phases of capital projects and how they apply to research and evaluation. Additionally, the presenter will identify areas of divergence and share implementation challenges.

Session Title: Improving the Quality of Peacebuilding Evaluation

Demonstration Session 396 to be held in CROCKETT C on Thursday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the International and Cross-cultural Evaluation TIG

Presenter(s):

Cheyanne Scharbatke-Church, Child Development Associate, cheyanne.church@tufts.edu

Abstract: This demonstration session will outline 2 new tools specific to the evaluation of peacebuilding in the international arena. The demonstration will be targeted at evaluators working in conflict and post-conflict settings evaluating peacebuilding projects, though all international social change projects may find the demonstration has utility. The presentation is based on the results of over 7 years of collaborative learning with peacebuilding practitioners and donors, led by the Reflecting on Peace Practice project of CDA. This collaborative learning included, but was not limited to, over 15 evaluations that sought to apply these tools and lessons as well as the experience of including key findings into the OECD-DAC guidance on peacebuilding evaluation. The demonstration will focus specifically on how to assess the effect of peacebuilding programming at the societal level, oft called peace writ large. It will also demonstrate how to adopt a systemic approach to the evaluation of peacebuilding.

Session Title: Haiti: Challenges in Emergency Response and Recovery Bring Challenges (and Innovation) in Evaluation

Panel Session 397 to be held in CROCKETT D on Thursday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Disaster and Emergency Management Evaluation TIG

Chair(s):

Nan Buzard, American Red Cross, buzardn@usa.redcross.org

Abstract: On January 12, 2010, a magnitude 7 earthquake struck the Haitian coast 10 miles from the capital of Port-au-Prince, causing massive damage and significant loss of life. The American Red Cross (ARC) delegates working in Haiti were among the first to respond, and the public was generous in providing funds to ARC for emergency response and recovery programs. In coordination with the International Federation of Red Cross and Crescent Societies (IFRC), the International Committee for the Red Cross, and UN and US Government agencies, ARC continued the immediate emergency response while an IFRC needs assessment team deployed to conduct field assessments in eleven sector areas, to begin defining medium-term recovery priorities. This panel by ARC staff (including the Lead of the IFRC Recovery Assessment Team) will discuss how this unprecedented urban disaster challenged existing models of needs assessment, data collection and analysis, and M&E design, leading to useful innovation.

Six Weeks After Haiti Disaster: The Challenge of Leading a Multi-donor Emergency Recovery Needs Assessment

Michael Zeleke, American Red Cross, zelekem@usa.redcross.org

When the Haiti Recovery Assessment Team of the International Federation of Red Cross and Red Crescent Societies (IFRC) was organized, the presenter was recruited as an advisor. Several days before departure, he was asked to the lead the team. During the next week, the team size grew from an estimated 6 to over 28, as various Partner National Red Cross Societies who were donating funds and supplies asked to join the team. The challenge of managing this large team also encompassed: seeking model approaches for sector and geographic coverage; question definition and analysis; coordination challenges (with UN and other international actors); deployment of multisectoral teams to respect counterpart and survey respondent time; and working within resource and time constraints in a climate of high expectations. This presentation will give a flavor of how these challenges were met and what can be learned from them for future urban disasters.

Innovation in Collecting and Analyzing Geographic Information: Immediate Contributions to Recovery Efforts and Potential Contributions to Monitoring and Evaluation of Results in Haiti

Dale Hill, American Red Cross, hilldal@usa.redcross.org

After the Haiti earthquake, worldwide interest and compassion catalyzed an unusual set of collaborative initiatives focused on geographic information, made possible also by technological innovation. For example, 500 technical experts from over 22 countries were mobilized to interpret data to assess the earthquake impact in Haiti to feed into a comprehensive damage assessment, accomplished in weeks, not months. Also, some non-profit organizations bringing a different skill set to disaster needs used social networking to develop dynamic technical solutions implemented in days by volunteers, such as usable street maps of Port-au-Prince. These tools and the information that resulted from them, were put to immediate use on the ground in Haiti. The presenter brings perspectives from both development and relief organizations to examine whether these innovative tools devised for near-term damage and needs assessment can also be applied over the long term to monitoring and evaluation of relief and recovery projects.

Haiti: From Response to Recovery: Determining Sectoral Priorities and Beginning the Monitoring Process

Amy Gaver, American Red Cross, gavera@usa.redcross.org

The magnitude of damage from the Haiti earthquake required immediate response on a huge scale in emergency aid such as food, medical assistance, safe water supplies, sanitation, and shelter. But the disaster also destroyed assets and sources of livelihood required for longer term resumption of economic development-- medical facilities, schools (87% destroyed in Port-au-Prince) and markets. Learning from its experience with other disaster relief operations, the American Red Cross (ARC) response program emphasized the transition from emergency relief to recovery early on. But to respond to the emergency and early recovery needs, ARC needed to accelerate and innovate in several of its traditional sectors such as cash transfer programs and livelihood support for beneficiaries. The presenter, the ARC manager in charge of recovery programming will discuss how the IFRC needs assessment was used to help define sectoral priorities and ensure early monitoring systems supported learning from innovative programs.

Recovery Program Monitoring and Evaluation Design: Challenge of an Impacted Population on the Move in Haiti

Christine Connor, American Red Cross, connorch@usa.redcross.org

The Haiti earthquake was unprecedented since it occurred so close to the major urban center of a country with great poverty. Not only homes serving as shelter, but major buildings and facilities serving Government and the market economy were destroyed, affecting livelihoods and services. A large number of people remain displaced, and some have opted to stay with relatives in areas outside of Port-au-Prince. As recovery efforts continue, the “market pull” of jobs and improved services can attract residents to resettle outside their original home base. This creates special challenges for monitoring and evaluation (M&E) design. The presenter will draw on her experience with other projects involving refugees and displaced persons to discuss how M&E design for projects supporting Haiti’s shifting displaced population presents similarities, and how it presents differences which need to be taken into account in the context of the Haiti earthquake recovery program.

Session Title: Student Centered Issues in Evaluation

Multipaper Session 398 to be held in SEGUIN B on Thursday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the Graduate Student and New Evaluator TIG

Chair(s):

Thelma Woodard, University of Tennessee, Knoxville, twoodar2@utk.edu

Guidelines for Conducting a High-Quality Mixed Methods Dissertation

Presenter(s):

Rebecca Glover-Kudon, University of Georgia, gloverku@uga.edu

Abstract: Mixed-methods research has emerged as a distinct methodology. As a result, graduate students in the evaluation field are increasingly interested in learning how to combine quantitative and qualitative methods in a single study. Because faculty mentors have typically specialized in either quantitative or qualitative methods, they may lack experience in conducting mixed-methods research and feel reluctant to guide students’ mixed-methods endeavors. This paper summarizes the literature on how to conduct and produce high-quality mixed-methods research and suggests criteria for assessment. Specifically, the paper details core elements of mixed-methods research proposals including study design and procedures, explicit rationale for data mixing, and standard notation for data prioritization, sequencing, and integration. Features and challenges of various designs are also discussed. The intended audience for this paper and presentation are graduate students contemplating or conducting mixed-methods studies and faculty members who advise them.

Assessing the Needs of Students in an Evaluation: Statistics and Measurement Doctoral Program- Results and Lessons Learned

Presenter(s):

Susanne Kaesbauer, University of Tennessee, Knoxville, skaesbau@utk.edu

Thelma Woodard, University of Tennessee, Knoxville, twoodar2@utk.edu

Abstract: A needs assessment was developed to assess the perceptions and needs of current and former students in a Ph.D. program in Evaluation, Statistics and Measurement. The goal of the needs assessment is to inform faculty of current and former student’s perceptions of their experiences, as well as the strengths, weaknesses and opportunities for strengthening the program. This presentation will describe the aggregated results of the data collected using the needs assessment. In addition, lessons learned from the development of the needs assessment and the data collection process will be discussed.

Evaluation Quality: A Model for Reflecting on Evaluation for Evaluation Students

Presenter(s):

Thelma Woodard, University of Tennessee, Knoxville, twoodar2@utk.edu

Andrea Souflee, United Way of Dallas, asouflee@unitedwaydallas.org

Abstract: This reflective practice model provides a framework for evaluation students to reflect on the quality of their evaluation skills and development. Developers of professional evaluator programs express the importance of developing a reflective practice. Proponents of evaluator competencies suggest reflecting on individual strengths and weaknesses benefit evaluator development (Stevahn, King, Ghere, & Minnema, 2006). Developing a reflective practice can be a challenge because defining reflective practice is a challenge. There are many abstract and complex descriptions of reflection. Moreover, there have been many conflicting reflective practice definitions that have been offered in the literature over several decades. While reflection is acknowledged as an important part of professional development, if reflective practice instruction is lacking, students have to determine how to reflect on their own. This presentation will detail a model that evaluation students may use to develop a professional practice.

Session Title: Impact Evaluation at the Millennium Challenge Corporation (MCC): Theory, Application, and Complications

Panel Session 399 to be held in REPUBLIC A on Thursday, Nov 11, 4:30 PM to 6:00 PM

Sponsored by the International and Cross-cultural Evaluation TIG

Chair(s):

Marc Shapiro, Millennium Challenge Corporation, shapiromd@mcc.gov

Discussant(s):

Jack Molyneaux, Millennium Challenge Corporation, molyneauxjw@mcc.gov

Abstract: The Millennium Challenge Corporation (MCC) is committed to conducting rigorous independent impact evaluations of its programs as an integral part of its focus on results. MCC expects that the results of its impact evaluations will help guide future investment decisions and contribute to a broader understanding in the field of development effectiveness. MCC’s impact evaluations involve a variety of methods chosen as most appropriate to the context. This panel first provides an overview of evaluation at MCC. This includes the defining the evaluation objectives, the number and variety of approaches to evaluation supported, the criteria that underlie decisions about whether and how to evaluate, and the linkages between decisions to fund projects and eventual evaluation results. Next, the panel provides three examples of evaluations being conducted across sectors and involving different methods across countries. The presenters will discuss the challenges involved in implementing these evaluations and lessons learned.

MCC Project Evaluations: To Err Is Human

Jack Molyneaux, Millennium Challenge Corporation, molyneauxjw@mcc.gov

MCC was established in January 2004 with the objectives of promoting economic growth and reducing poverty by learning about, documenting and using approaches that work. MCC plans to complete 35 rigorous impact evaluations of international development projects over the next two to three years, and the rate of project evaluations likely will double in the following years. This growing pipeline of rigorous evaluations reflects a critical component to achieving these objectives. Another essential component, MCC’s cost benefit approach, defines the objectives of these impact evaluations. The results of these emerging evaluations will soon shape the selection and design of future projects. This presentation will briefly present the Economic Rate of Return (ERR) Analyses that forecast the expected project costs and benefits to influence project selection and define the impact evaluation objectives. Examples of these models will provide the motivation for the impact evaluations presented in the following presentations.

On the Rights Path to Evaluating a Property Rights Project

Marc Shapiro, Millennium Challenge Corporation, shapiromd@mcc.gov

In peri-urban rangelands, Mongolia’s tradition of open access pasture use, combined with the influx of migrants’ herds, has led to severe overgrazing. The Peri-urban Property Rights Project funded by MCC introduces a system of leasing peri-urban rangelands to herder groups and provides infrastructure and training to improve livestock management, productivity and, income. The project’s impact evaluation uses a randomized selection process to determine which herder groups will receive available leasing slots and attempts to measure spillovers/externalities. Data collection involves household surveys and direct measures of changes in land quality. Key problems encountered include large population shifts making irrelevant early designs, lack of ownership of evaluation design by the implementing unit, and a multiple-step design that ended up reducing power when herder productivity changing exogenously.

A Scholarly Assessment of the Impact of Scholarships

Rebecca Tunstall, Millennium Challenge Corporation, tunstallrh@mcc.gov

Average educational attainment in El Salvador’s northern zone is only 4.3 years of schooling, more than 1.5 years lower than the rest of the country. To increase school enrollment and to keep teenagers in schools, MCC is providing scholarships to poor students to cover the cost of books, uniforms, room and board, and transportation. To evaluate the impact of the scholarships, the Government of El Salvador agreed to conduct a lottery of applicants who meet the eligibility criteria. The students not selected in the lottery become the control group for the evaluation. Data collection involves administrative data from the Ministry of Education on grade completion and graduation in addition to household surveys to track employment and income. Key problems encountered during implementation of the evaluation include lower demand for scholarships than initially anticipated, poor management of implementation contractors, and political pressure to provide scholarships to the control group.

An Agricultural Evaluation Standing Out, If Not Outstanding, in the Field

Mamuka Shatirishvili, Millennium Challenge Account-Georgia, m.shatirishvili@mcg.ge

Marc Shapiro, Millennium Challenge Corporation, shapiromd@mcc.gov

The Agribusiness Development Activity in the Republic of Georgia awards grants to small farmers, value-adding enterprises, and service centers that sell to farms. The impact evaluation examines the project’s effects on income and job creation. For farmers, this was planned as an experimental design across nine rounds of grantees. Those randomly selected initially received grants immediately, while others receive grants later. Farm service centers and value-adding enterprise grantees are evaluated by pairing recipients to similar control enterprises using propensity score matching. Data collection involves augmenting the Georgian Department of Statistics’ household survey and using local contractors to collect household level information from direct and indirect beneficiaries. Problems in recruiting farmer grantees in early rounds led to a change to a pipeline design. The confounds of military conflict and the financial crisis plus project delays required adjustments to data collection schedules and extended timelines.

Session Title: Health Evaluation TIG Business Meeting and Presentation: Bridging the Evidence Gap in Obesity Prevention - A Framework to Inform Decision Making

Business Meeting Session 400 to be held in REPUBLIC B on Thursday, Nov 11, 4:30 PM to 6:00 PM