| Session Title: Evaluating the Impacts of International Democracy Projects and Programs |
| Multipaper Session 212 to be held in Conference Room 13 on Thursday, Nov 3, 8:00 AM to 9:30 AM |
| Sponsored by the International and Cross-cultural Evaluation TIG |
| Chair(s): |
| Krishna Kumar, United States Department of State, kumark@state.gov |
| Abstract: This panel will focus on the complex and controversial topic of evaluating international democracy promotion projects and programs. Such interventions are often, though not always, different from other social and economic interventions in terms of their nature, political settings and impacts. Most of them focus on institution building, and not on service delivery. They affect and can unsettle existing power relations and therefore are often viewed with suspicion, if not hostility, by many governments. Above all, their effects are not always visible at least in the short run. The Panel will present three original papers on the subject from the senior officials of the State Department, National Democratic Institute and International Republican Institute. |
| Applicability of Experimental and Quasi-Experimental Designs to Democracy Promotion Projects and Programs |
| Krishna Kumar, United States Department of State, kumark@stte.gov |
| A recent study "Improving Democracy Assistance: Building Knowledge through Evaluation and Research" undertaken by National Research Council has made a case for using experimental and quasi-experimental designs in democracy evaluations. The paper questions this recommendation and identifies a set of conceptual, methodological and practical obstacles which make it extremely difficult to use these designs for most democracy interventions. It argues that measuring of democracy assistance by using these designs can be even counterproductive to the extent many effects of interventions cannot be quantified. Moreover, such designs may result in ignoring unintended or long-term effects of democracy interventions. The paper stresses the need for developing alternative approaches to impact evaluations which generate reliable and relevant information about the outcomes while promoting indigenous ownership of democracy interventions. |
| Capturing Indigenous Democratic Values Through Participatory Evaluation |
| Linda Stern, National Democratic Institute, lstern@ndi.org |
| "Democracy" is not a static objective, but a dynamic socio-cultural and political process that reflects the values of the people practicing it. Evaluating democracy promotion requires a delicate balance between measuring the effectiveness of largely western democratic projects, and evaluating the creative agency of non-western citizens in adapting the project to their own needs, values and contexts. In striking this balance, the evaluator must understand the outcomes of a project, as well as their relationship to newly emerging political actors attempting to transform their own institutions and societies. Drawing from NDI's experience in implementing participatory baseline and midterm evaluations, this paper explores the relationship between democratic values, indigenous ownership and evaluation designs. While not eschewing experimental or quasi-experimental designs, the author advocates a rigorous collaborative approach for understanding the complexities of democratic change. |
| Assessing the Impact of Democracy and Governance Programs Using Qualitative Methods |
| Jonathan Jones, International Republican Institute, jdjones@iri.org |
| Experimental design style impact evaluations typically use quantitative methods for research. Such research designs however cannot capture the nuance and detail necessary to understand why a program did or did not achieve impact. As well, it is difficult for such methods to capture how a program caused, or responded to, any number of unanticipated factors, including interaction and spillover effects, unintended consequences (positive and negative), and paradigm shifts, such as a coup d'+¬tat, economic crisis or natural disaster. Drawing on two years of qualitative research of IRI's party development programs in eight countries, this paper illustrates that qualitative methods are well suited to capture the real story of IRI's programs on the ground, especially with respect to lessons learned from past experiences that can be applied to future programs. A quasi experimental design evaluation currently underway that is using qualitative methods, as well as an ongoing real time qualitative assessment of an IRI program in a volatile environment (using Patton's Developmental Evaluation framework), will also be discussed to further illuminate the value of qualitative research for assessing the impact of democracy and governance programs. |