 |
|||||||
|
|
Quick Links: |
EVALUATION POLICY INITIATIVE
AEA is pleased to announce a two-year initiative to assist in developing an ongoing capability to influence evaluation policy. This web page has been established to help you to learn more about our ongoing efforts.
On behalf of the Board of Directors of AEA, we are pleased to announce the formation of the AEA Evaluation Policy Task Force (EPTF). The goal of this two-year initiative is to assist AEA in developing an ongoing capability to influence evaluation policies that are critically important to the practice of evaluation. The members of the Task Force are:
In addition, AEA has contracted with George Grob, President of the Center for Public Program Evaluation, as a consultant to the Task Force. George is a longtime AEA member and senior evaluation manager with significant experience in policy development and congressional relations. He will play a key role in assisting in the planning and implementation of the Task Force’s efforts to influence evaluation policy. This group collectively brings extensive and varied experience in the profession of evaluation, the development of evaluation policy, and in AEA, and we are excited about the prospects that they will be able to make an important contribution. The term “evaluation policy” encompasses a wide range of potential topics that include (but are not limited to): when systematic evaluation gets employed, and on what programs, policies and practices; how evaluators are identified and selected; the relationship of evaluators to what is being evaluated; the timing, planning, budgeting and funding, contracting, implementation, methods and approaches, reporting, use and dissemination of evaluations; and, the relationship of evaluation policies to existing or prospective professional standards. To deal with the broad potential scope of this effort, the Task Force will, during the two year initiative, concentrate on evaluation policies in the United States Federal government, in both the legislative and executive branches. Focusing on the Federal level enables the Task Force to address evaluation policies that directly affect a broad cross-section of our membership and ultimately affects our entire field. The Task Force will make recommendations to the Board about how this scope might be extended into other sectors and areas over time. Of course, engagement of AEA members is critical to the success of such an effort. We are doing several things to connect you with this initiative. We have already established a dedicated e-mail address (evaluationpolicy@eval.org) to receive your comments and questions and we will establish a special web page on the AEA website to keep you informed on an ongoing basis. We will look for other ways to communicate with and involve you as the initiative unfolds. To help kick off this project and to engage you more personally, we would also like to invite you to a special open forum at the upcoming Evaluation 2007 annual conference in Baltimore entitled A Discussion of AEA's Evaluation Policy Initiative that will be held on Saturday, November 10, from 9:35 AM to 10:20 AM in the Versailles Room of the Radisson Plaza Lord Baltimore. At this session we will introduce this effort, and discuss the Task Force and its goals and charge in greater detail. We look forward to working with AEA members through this effort to enhance the ability of our association and profession to influence the evaluation policies that shape how we do our work. Hope to see many of you at the conference.
Hallie Preskill
William Trochim EPTF Response to OMB re: What Constitutes Strong Evidence of a Program's Effectiveness Dear AEA Colleagues, One of the most important evaluation initiatives in the United States federal government these days is the Program Assessment Rating Tool (PART) used by the Office of Management and Budget (OMB) to assess virtually every federal government program. A PART review asks approximately 25 general questions about a program's performance and management, including several questions explicitly about evaluation. The answers determine a program's overall rating which is then published on OMB's website http://www.expectmore.gov. The sometimes controversial PART system was the focus of the first AEA Public Issues Forum (see http://www.eval.org/ part.asp) and the newly-established AEA Evaluation Policy Task Force (EPTF) identified PART as a priority area. Earlier this year, the EPTF contacted Robert Shea, the Associate Director of OMB for Administration and Government Performance, and a major architect of the PART system. I went with the EPTF's Consultant George Grob to meet with Shea, with the goals of introducing the American Evaluation Association, emphasizing the important role professional evaluators can play in the systematic assessment of Federal programs, and engaging him in a discussion of the PART's evaluation approach. Shea described OMB's new initiative to review and improve the PART program and requested that we provide him with detailed comments on a key document cited in the OMB PART Guidance entitled "What Constitutes Strong Evidence of a Program's Effectiveness?" This document has been especially controversial because of the nature of the case it makes regarding the use of randomized controlled trials (RCTs) and a formal request to review it and provide a thoughtful and balanced critique of the document itself and its policy implications is exactly what the EPTF was hoping to encourage. We worked hard in less than a week to produce a balanced critique and I am delighted to share with you today our cover letter and the comments that we provided (download at http://www.eval.org/aea08.omb.guidance.responseF.pdf). We recommended that OMB develop new guidance for the evaluation components of PART that integrates evaluation more closely with information from other questions about program planning and management. This guidance should describe the variety of methods for assessing program effectiveness that are appropriate to the needs and development level of a program. We argued for a more balanced presentation of the role of RCTs, and suggested that there are important alternatives to RCTs for assessing effectiveness and that RCTs could be enhanced significantly when mixed with additional methods that enable identification of why and how observed effects occur. Finally, we called upon OMB to draw on broader expertise in the evaluation community to develop future guidance on evaluation for the PART program. We were delighted with the reception our comments received and with being invited subsequently to make a presentation to the first meeting of the newly established Evaluation Workgroup of the cross-agency Performance Improvement Council. We continue to work with OMB staff and other federal administrators on efforts to address the major evaluation concerns in PART. I particularly want to thank all the members of the EPTF-Eleanor Chelimsky, Leslie Cooksy, Katherine Dawes, Patrick Grasso, Susan Kistler, Mel Mark, and Stephanie Shipman-and our consultant George Grob, for their highly professional and energetic collaboration in preparing this document in such a short period of time. In the next newsletter we will share an interview we subsequently conducted with Robert Shea in which he describes the challenges facing the PART system, addresses the issue of the role of RCTs in program effectiveness evaluation, and describes how professional evaluators and AEA can be helpful in improving OMB PART in the future. Sincerely, Bill Trochim,2008 AEA President Evaluation 2008 conference session descriptions The following two sessions devoted to discussion of evaluation policy and the initiative are scheduled for Evaluation 2008.
Evaluation 2007 conference session: The following panel session devoted to discussion of the initiative was offered at Evaluation 2007. Click here to view the PowerPoint presentation slides.
|