| Session Title: Reaping Randomized Results on the Ranch: Rigorous Impact Evaluation Designs and Preliminary Results From Agricultural Interventions in Three Millennium Challenge Corporation (MCC) Compacts Countries |
| Multipaper Session 572 to be held in El Capitan B on Friday, Nov 4, 8:00 AM to 9:30 AM |
| Sponsored by the International and Cross-cultural Evaluation TIG |
| Chair(s): |
| Marc Shapiro, Millennium Challenge Corporation, shapiromd@mcc.gov |
| Abstract: The Millennium Challenge Corporation (MCC) is committed to conducting rigorous independent impact evaluations of its programs as an integral part of its focus on results. MCC expects that the results of its impact evaluations will help guide future investment decisions and contribute to a broader understanding in the field of development effectiveness. MCC's impact evaluations involve a variety of methods chosen as most appropriate to the context. This panel provides a brief overview of evaluation at MCC generally to frame the overall context. Next, the panel provides examples of evaluations being conducted in the agricultural sector across three MCA compact countries. The presenters will discuss the context, evaluation design, challenges involved in implementing these evaluations, and preliminary results. Although projects for MCC are designed to be context specific rather than to best allow cross-project/country comparisons, the panel will discuss lessons learned across and within countries from an evaluation design perspective. |
| Impact Evaluation at MCC: An Overview |
| Marc Shapiro, Millennium Challenge Corporation, shapiromd@mcc.gov |
| Mawadda Damon, NORC, damon-mawadda@norc.org |
| MCC was established in January 2004 with the objectives of promoting economic growth and reducing poverty by learning about, documenting and using approaches that work. MCC plans to complete 35 rigorous impact evaluations of international development projects over the next two years, and the rate of project evaluations likely will double in the following years. The results of these emerging evaluations are intended to shape the selection and design of future projects. This short presentation will provide a brief overview of impact evaluation at MCC. |
| Thinking Small: Impact of Business Services on the Economic Wellbeing of Small Farmers in Nicaragua |
| Anne Pizer, Millennium Challenge Corporation, pizerar@mcc.gov |
| The Rural Business Development (RBD) project aims to increase profits and wages in farms and non-farm businesses by providing technical and financial assistance to small and medium farms and agribusinesses to help them transition to higher profit activities. The impact evaluation will estimate the change in beneficiary household income and other welfare measures attributed to the project. The evaluation relies on the randomized sequencing (pipeline design) of 80 to 100 communities, with half selected randomly to receive services early and half later. Interim impact evaluation results (baseline and mid-term) found that the average increase in RBD household incomes is small (2 percent above the change in household income for those not yet receiving treatment) and not statistically different from zero. The very small magnitude of change in incomes may reflect the limited amount of time between the provision of services and the measurement of change. |
| Sewing the Seeds for Impact Evaluation Success: Problems and Preliminary Results from a Georgian Agricultural Project |
| Marc Shapiro, Millennium Challenge Corporation, shapiromd@mcc.gov |
| The Agribusiness Development Activity in the Republic of Georgia awards grants to small farmers, farm service centers that serve communities, and value-adding enterprises. The impact evaluation examines the project's effects on income and job creation for farmers through a pipeline experimental design used across nine rounds of grantees. Those selected in the first random drawing received grants immediately, while others receive grants later. Farm service center and value-adding enterprise grantees are being evaluated by matching grant recipients to similar enterprises in the comparison group using propensity score matching. Data collection involves augmenting the Georgian Department of Statistics' household survey and using local contractors to collect household level information from direct and indirect beneficiaries. The confounds of military conflict and the financial crisis as well as project delays have required adjustments. Preliminary double difference results for the first seven rounds of grantees will be discussed. |
| Has Beans - Moving from Rice and Beans to Radishes and Bell Peppers: The Impact of High-Value Horticulture Training on Farmer Income in Honduras |
| Algerlynn Gill, Millennium Challenge Corporation, gillaa@mcc.gov |
| Varuni Dayarantna, NORC, dayaratna-varuni@norc.org |
| George Caldwell, NORC, jcaldwell9@yahoo.com |
| The MCC-funded Farmer Training and Development Activity in Honduras provided technical assistance to farmers to transition from subsistence crops to high-value horticultural crops for domestic sale and international export. The impact evaluation assesses the training's effects on household income and production levels, comparing farmers and communities who received training and those who did not. Double-difference estimates will be formulated using two approaches, one involving comparison of a randomly selected treatment and control group of communities and one using a model-based method, due to implementation challenges that necessitated adjustments to the evaluation plan. Results from the evaluation and how the methodology evolved to meet changing conditions on the ground will be discussed. Differences in "monitoring data" collected by the implementer and evaluation data collected through independent surveys also will be presented, to demonstrate how impact evaluations with counterfactuals can correct initial over-estimations of results. |