|
Session Title: Impact Evaluation at the Millennium Challenge Corporation (MCC): Theory, Application, and Complications
|
|
Panel Session 399 to be held in REPUBLIC A on Thursday, Nov 11, 4:30 PM to 6:00 PM
|
|
Sponsored by the International and Cross-cultural Evaluation TIG
|
| Chair(s): |
| Marc Shapiro, Millennium Challenge Corporation, shapiromd@mcc.gov
|
| Discussant(s):
|
| Jack Molyneaux, Millennium Challenge Corporation, molyneauxjw@mcc.gov
|
| Abstract:
The Millennium Challenge Corporation (MCC) is committed to conducting rigorous independent impact evaluations of its programs as an integral part of its focus on results. MCC expects that the results of its impact evaluations will help guide future investment decisions and contribute to a broader understanding in the field of development effectiveness. MCC’s impact evaluations involve a variety of methods chosen as most appropriate to the context.
This panel first provides an overview of evaluation at MCC. This includes the defining the evaluation objectives, the number and variety of approaches to evaluation supported, the criteria that underlie decisions about whether and how to evaluate, and the linkages between decisions to fund projects and eventual evaluation results. Next, the panel provides three examples of evaluations being conducted across sectors and involving different methods across countries. The presenters will discuss the challenges involved in implementing these evaluations and lessons learned.
|
|
MCC Project Evaluations: To Err Is Human
|
| Jack Molyneaux, Millennium Challenge Corporation, molyneauxjw@mcc.gov
|
|
MCC was established in January 2004 with the objectives of promoting economic growth and reducing poverty by learning about, documenting and using approaches that work. MCC plans to complete 35 rigorous impact evaluations of international development projects over the next two to three years, and the rate of project evaluations likely will double in the following years. This growing pipeline of rigorous evaluations reflects a critical component to achieving these objectives. Another essential component, MCC’s cost benefit approach, defines the objectives of these impact evaluations. The results of these emerging evaluations will soon shape the selection and design of future projects. This presentation will briefly present the Economic Rate of Return (ERR) Analyses that forecast the expected project costs and benefits to influence project selection and define the impact evaluation objectives. Examples of these models will provide the motivation for the impact evaluations presented in the following presentations.
|
|
|
On the Rights Path to Evaluating a Property Rights Project
|
| Marc Shapiro, Millennium Challenge Corporation, shapiromd@mcc.gov
|
|
In peri-urban rangelands, Mongolia’s tradition of open access pasture use, combined with the influx of migrants’ herds, has led to severe overgrazing. The Peri-urban Property Rights Project funded by MCC introduces a system of leasing peri-urban rangelands to herder groups and provides infrastructure and training to improve livestock management, productivity and, income. The project’s impact evaluation uses a randomized selection process to determine which herder groups will receive available leasing slots and attempts to measure spillovers/externalities. Data collection involves household surveys and direct measures of changes in land quality. Key problems encountered include large population shifts making irrelevant early designs, lack of ownership of evaluation design by the implementing unit, and a multiple-step design that ended up reducing power when herder productivity changing exogenously.
| |
|
A Scholarly Assessment of the Impact of Scholarships
|
| Rebecca Tunstall, Millennium Challenge Corporation, tunstallrh@mcc.gov
|
|
Average educational attainment in El Salvador’s northern zone is only 4.3 years of schooling, more than 1.5 years lower than the rest of the country. To increase school enrollment and to keep teenagers in schools, MCC is providing scholarships to poor students to cover the cost of books, uniforms, room and board, and transportation. To evaluate the impact of the scholarships, the Government of El Salvador agreed to conduct a lottery of applicants who meet the eligibility criteria. The students not selected in the lottery become the control group for the evaluation. Data collection involves administrative data from the Ministry of Education on grade completion and graduation in addition to household surveys to track employment and income. Key problems encountered during implementation of the evaluation include lower demand for scholarships than initially anticipated, poor management of implementation contractors, and political pressure to provide scholarships to the control group.
| |
|
An Agricultural Evaluation Standing Out, If Not Outstanding, in the Field
|
| Mamuka Shatirishvili, Millennium Challenge Account-Georgia, m.shatirishvili@mcg.ge
|
| Marc Shapiro, Millennium Challenge Corporation, shapiromd@mcc.gov
|
|
The Agribusiness Development Activity in the Republic of Georgia awards grants to small farmers, value-adding enterprises, and service centers that sell to farms. The impact evaluation examines the project’s effects on income and job creation. For farmers, this was planned as an experimental design across nine rounds of grantees. Those randomly selected initially received grants immediately, while others receive grants later. Farm service centers and value-adding enterprise grantees are evaluated by pairing recipients to similar control enterprises using propensity score matching. Data collection involves augmenting the Georgian Department of Statistics’ household survey and using local contractors to collect household level information from direct and indirect beneficiaries. Problems in recruiting farmer grantees in early rounds led to a change to a pipeline design. The confounds of military conflict and the financial crisis plus project delays required adjustments to data collection schedules and extended timelines.
| |