| Session Title: Moving Toward Impact: Alternative Approaches to Meeting the Premium Value of Impact Assessment in Environmental Evaluations |
| Multipaper Session 493 to be held in San Clemente on Thursday, Nov 3, 4:30 PM to 6:00 PM |
| Sponsored by the Environmental Program Evaluation TIG |
| Chair(s): |
| Angela Helman, Industrial Economics Inc, ahelman@indecon.com |
| Discussant(s): |
| Katherine Dawes, United States Environmental Protection Agency, dawes.katherine@epa.gov |
| Abstract: Experimental designs are increasingly being raised as the gold standard in environmental evaluations; however, environmental evaluators are seldom able to consistently employ methods that enable definitive causal impact claims. This pressure to reach the gold standard has necessitated evaluators' employment of diverse approaches that approximate impact estimation. Here, environmental evaluators discuss use of innovative alternative approaches that allow statistical estimation of impact. One approach employs a comparison of early joiners and late joiners to measure a "dosage effect." Another discusses a quasi-experimental approach that compares the outcomes of two similarly situated states receiving differential levels of compliance assistance. A third approach explores theoretical limitations of experimental designs in light of economic principles that render these approaches untenable. Finally, feasible and efficacious technique for identifying a counterfactual for comparison to the intervention is presented. These approaches will be discussed in context of relative value of experimental methods versus non-experimental methods. |
| Measuring Dosage Effects: Using Tenure in Program as a Variable to Assess Likely Environmental Influence on Solid Waste Management Behaviors |
| Angela Helman, Industrial Economics Inc, ahelman@indecon.com |
| In a recent evaluation conducted on a waste management program at the United States Environmental Protection Agency, a non-regulatory voluntary partnership program was able to secure a convenience sample of federal partners who had different tenure lengths in a program that assists its partners in adopting positive waste management practices. Considerable pressure had been placed this program demonstrate causal impact that could be assessed over and above other contributing factors that produce the desired outcomes, independent of program influence. By developing a precise survey instrument which measured the extent to which early joiners and late joiners differed on confounding variables, and ruling out selection threats to validity, a confident assertion could be made about the program's benefits, illustrating the utility of dosage effects as a way of evaluating complex environmental programs. |
| Non-Equivalent Control Designs and Group Comparison Approach in Assessing the Impact of Compliance Assistance Behaviors in the Autobody Sector of Two States |
| Tracy Dyke-Redmond, Industrial Economics Inc, tredmond@indecon.com |
| The evaluators of a compliance assistance program explored the effectiveness of activities to assess the long-term impact of a comprehensive compliance assistance package. A forthcoming environmental statute and pre-statute compliance assistance activities in two similarly situated states presented the opportunity for a group comparison quasi-experiment. Facilities in one group were offered a full suite of compliance assistance activities. Those in a second group were offered reduced- or no outreach. In each of the two regions, site visits by qualified personnel at independent random samples of facilities are being used to estimate performance before and after compliance assistance has been provided. The impact of compliance assistance will be assessed by comparing the change in group one performance with the change in group two performance. This is referred to in the policy evaluation literature as a "difference-in-differences" approach to assessing policy impacts. The advantages and challenges of this approach will be discussed. |
| Attributing Benefits of Non-Regulatory Programs to Environmental Change: Identifying Concerns and Alternative Solutions |
| Cynthia Manson, Industrial Economics Inc, cmanson@indecon.com |
| In this paper, an evaluator working on behalf of the United States Environmental Protection Agency discusses the economic justification for voluntary environmental programs to derive defensible measures of their positive social outcomes. The authors consider ideal experimental and statistical designs to detect and attribute benefits. We also explore a set of more practical approaches to benefit attribution that take into account the data gaps and statistical challenges that often make more rigorous approaches infeasible. These conclusions will be addressed |
| New Technique for Comparison to an Alternative: The Negotiated Alternative |
| Andy Rowe, ARCeconomics, andy.rowe@earthlink.net |
| In many settings evaluators encounter serious challenges in identifying a suitable comparison. This is particularly true in natural resource and sustainable development settings. Yet comparison to a reasonable alternative is our main approach to judging changes in outcomes of interest attributable to the intervention. This paper describes a new option, the negotiated alternative. It has been used in three different resource and environmental programs in the US: fish and freshwater, environmental enforcement and for off-road vehicle use at national seashores. It is proving a feasible technique generating valid and reliable judgments about environmental effects of decisions. It offers a new option for evaluators regardless of setting. |