| Session Title: Addressing Methodological Challenges in Evaluation of Environmental Programs and Policies |
| Multipaper Session 738 to be held in Wekiwa 9 on Saturday, Nov 14, 9:15 AM to 10:45 AM |
| Sponsored by the Environmental Program Evaluation TIG |
| Chair(s): |
| Matthew Birnbaum, National Fish and Wildlife Foundation, matthew.birnbaum@nfwf.org |
| Abstract: Evaluation has come later to the field of environmental programs and policies than to many other areas. Some of this area's important methodological challenges have had serious impacts on the concepts and practices of evaluations in this arena. This is a nascent field of evaluation, but the problems that evaluators are addressing here have significance to those outside the environmental arena. These features include: (1) conflicting and frequently long time horizons; (2) disparities in scaling; (3) problems in data credibility; and (4) the problem of a counterfactual given the inapplicability of RCT. These issues have formed the pillars of the first volume of New Directions for Evaluation (Issue #122, June 2009 release) devoted to the Environment. |
| Key Insights for the Design of Environmental Evaluations |
| Per Mickwitz, Finnish Environmental Institute, per.mickwitz@ymparisto.fi |
| Improving the future quality of program and policy evaluation in the environmental arena requires addressing four issues that emerged upon reading the earlier chapters in this special volume. Framing the evaluation requires careful consideration in choosing what the focus should be, what the context is, and what comprises potentially confounding factors. It means identifying the purpose of the evaluation and the stakeholders most important for inclusion. Addressing attribution requires recognizing the limits of establishing causality while maintaining counterfactual thinking. The frequent reality where bundles of interventions are implemented simultaneously makes precise assessment of relative impacts unrealistic and increases the importance of triangulation of qualitative and quantitative approaches. More effort is needed to increase the usefulness of evaluations. Multiple stakeholders come with their own perspectives; addressing this heterogeneity requires transparency and feedback during the design, implementation and follow-up to any evaluation study. Finally, improving the future quality of environmental evaluation will require continued deliberation and community building. Much of this is already underway, but the pluralism should still be promoted and our community enlarged. Evaluators in the environmental field further should continue to be mindful of contributions coming from other sub-fields of evaluation and applied research beyond the environmental community. |
| Design Alternatives to Evaluating the Impacts of Conservation Projects |
| Richard Margoluis, Foundations of Success, richard@fosonline.org |
| Caroline Stern, Foundations of Success, caroline@fosonline.org |
| Historically, examples of project evaluation in conservation have been rare. In recent years, however, conservation professionals have begun to recognize the importance of evaluation both for accountability and for improving project interventions. Even with this growing interest in evaluation, the conservation community has paid little attention to evaluation design. Recent literature has included some discussion of design, but it has focused primarily on experimental and quasi-experimental design and the use of counterfactuals. Real-life conservation projects, however, operate in complex and dynamic contexts and under conditions of limited resources, which limit the feasibility of counterfactual or experimental designs. There is, in fact, a wide range of design options to evaluate conservation interventions. The conservation community must educate itself about these options, and how different designs produce different results. This paper discusses evaluation design alternatives in light of the unique challenges that influence evaluation design selection in conservation. |
| Environmental Evaluation Practices and the Issue of Scale |
| Hans Bruyninckx, Catholic University, hans.bruyninckx@soc.kuleuven.be |
| Social scientists, natural scientists nor evaluators have properly defined the concept of scale for environmental problems. Environmental scale generally differs from social scale, which confounds the challenge of evaluating policies and governance arrangements in addressing environmental issues. Instead, social scales are generally based on traditional jurisdictional boundaries, which confounds effective decision making. Conversely, evaluators must be able to assess innovative governance arrangements as well as the outcomes of environmental problems since the two are interconnected. This becomes particularly true in looking at "cross-scale, social-ecological interactions." This has profound implications for policy evaluation since evaluators have to develop frameworks for connecting across various scales/levels in overcoming mismatches. Natural scientists probably need to be more humble in their ambitions, and evaluators will need to engage in interdisciplinary teams that blend the expertise of the social sciences with that from the natural sciences to assess outcomes to social and environmental scales. |
| A Perspective From Systematic Reviews in Environmental Management |
| Andrew Pullin, University of Bangor, a.s.pullin@bangor.ac.uk |
| To use environmental program evaluation to increase our effectiveness, predictive power and resource allocation efficiency, we need good data. Data require sufficient credibility in terms of fitness for purpose and quality to develop the necessary evidence base. Here we examine different elements of data credibility using experience from critical appraisal of studies on environmental interventions using systematic review methodology. We argue that critical appraisal of methodological quality is a key skill to improve both retrospective evaluation and prospective planning of monitoring of environmental programs. Greater transparency and data sharing among evaluators could facilitate rapid development in approaches to environmental evaluation that improve data credibility. |
| Issues of Scale and Monitoring Status and Trends in Biodiversity |
| Elizabeth Kennedy, Conservation International, e.kennedy@conservation.org |
| Issues of Scale and Monitoring Status and Trends in Biodiversity Designing a framework that can generate data to meet multiple audience objectives must consider interactions across levels and multiple scales of interest. One solution is to define targets using global criteria such that units are discrete. This aids identification of the types of data to collect in order to enable aggregation and reporting across relevant scales. By striving to standardize measurable targets and data requirements, we improve our ability to format monitoring information and tailor indicators for different reporting and decision making purposes. We outline the application of a nested approach presently used by Conservation International and describe some of its advantages and limitations. |