Evaluation Policy Initative
In 2007, AEA began an initiative to assist in developing an ongoing capability to influence evaluation policy. This web page has been established to help you to learn more about our efforts.
- NEW! AEA responds to UN Resolution: Empowering Countries through Evaluation: Evaluation as a country-level tool for the new development agenda (October 2014)
- Updated! EPTF Evaluation Roadmap for a More Effective Government (pdf) (November 2013)
- Evaluation policy task force charge
- Letter in support of evaluation provisions in the Foreign AID Transparency and Accountability Act of 2013 (pdf) (May 2013)
- Updated! EPTF Roadmap Citations (Most recent update - December 2012)
- EPTF Call for Proposals for Consultant Services on Influencing Evaluation Policy (October 2012)
- EPTF External Evaluation Panel Review - Final Report (June 2012)
- EPTF 5-year History Evaluation Briefing Book (December 2011)
- Human Subjects Call for Comments to Dept of Health and Human Services (September 2011)
- Letter to the Editor, New York Times, from AEA President Jennifer Greene regarding evaluator independence (pdf)
- What does the 2012 Presidential Budget say about Evaluation? (pdf in eLibrary) (February 2011)
- AEA Comments on 2010 Exposure Draft of GAO's Government Auditing Standards (pdf)
- Evaluation Roadmap for a More Effective Government as approved by AEA membership (October 2010)
- AEA Comments to HRSA on Evidence of Effectiveness of Maternal, Infant, and Early Childhood Home Visiting Programs (pdf) (August 2010)
- What does the 2011 Presidential Budget say about Evaluation? (pdf) (February 2010)
- AEA Comments to OMB on the Paperwork Reduction Act (December 2009)
- AEA Presidents' Response to OMB Memorandum (see below) (October 2009 pdf)
- Link to OMB Director Peter Orszag Office of Management and Budget (OMB) Memorandum on Increased Emphasis on Program Evaluation (October 2009)
- Link to Government Accountability Office (GAO) Report on USAID Food Aid using the EPTF Evaluation Roadmap as Criteria (September 2009)
- EPTF Legislative Framework for Evaluation of Health Care Reform (pdf) (September 2009)
- EPTF Comments to House Committee (June 2009)
- EPTF Comments to Senate Finance Committee (June 2009)
- EPTF Evaluation Roadmap for a More Effective Government, Submitted to Peter Orzag, Director of the Office of Management and Budget (pdf)
- What does the 2010 Presidential Budget say about Evaluation? (February 2009) (pdf)
- EPTF handouts describing evaluation and evaluators and what they can contribute to good government (pdf)
- EPTF Response to OMB re: What Constitutes Strong Evidence of a Program's Effectiveness
- Request for Nominations of Successful, Influential, Evaluations
- Evaluation 2008 conference session descriptions
- Evaluation 2007 conference session description and slides
- Announcement letter (October 2007)
Actions you can take to get involved:
- Sign up for the EPTF Discussion List
- Contact the evaluation policy taskforce via email at firstname.lastname@example.org
- Rakesh Mohan
- Katherine Dawes
- George Grob, co-chair
- Denise Roosendaal
- Mel Mark, co-chair
- Cheryl Oros (Consultant)
- Stephanie Shipman
- Kathryn Newcomer (Board Liaison)
- Jonathan Breul
- Rakesh Mohan
- Cynthia Clapp Winceck
- Katrina Bledsoe
- Nick Hart
- Mary Hyde
Dear AEA Colleagues,
One of the most important evaluation initiatives in the United States federal government these days is the Program Assessment Rating Tool (PART) used by the Office of Management and Budget (OMB) to assess virtually every federal government program. A PART review asks approximately 25 general questions about a program's performance and management, including several questions explicitly about evaluation. The answers determine a program's overall rating which is then published on OMB's website http://www.expectmore.gov. The sometimes controversial PART system was the focus of the first AEA Public Issues Forum (see http://www.eval.org/ part.asp) and the newly-established AEA Evaluation Policy Task Force (EPTF) identified PART as a priority area.
Earlier this year, the EPTF contacted Robert Shea, the Associate Director of OMB for Administration and Government Performance, and a major architect of the PART system. I went with the EPTF's Consultant George Grob to meet with Shea, with the goals of introducing the American Evaluation Association, emphasizing the important role professional evaluators can play in the systematic assessment of Federal programs, and engaging him in a discussion of the PART's evaluation approach.
Shea described OMB's new initiative to review and improve the PART program and requested that we provide him with detailed comments on a key document cited in the OMB PART Guidance entitled "What Constitutes Strong Evidence of a Program's Effectiveness?" This document has been especially controversial because of the nature of the case it makes regarding the use of randomized controlled trials (RCTs) and a formal request to review it and provide a thoughtful and balanced critique of the document itself and its policy implications is exactly what the EPTF was hoping to encourage.
We worked hard in less than a week to produce a balanced critique and I am delighted to share with you today our cover letter and the comments that we provided (download here). We recommended that OMB develop new guidance for the evaluation components of PART that integrates evaluation more closely with information from other questions about program planning and management. This guidance should describe the variety of methods for assessing program effectiveness that are appropriate to the needs and development level of a program. We argued for a more balanced presentation of the role of RCTs, and suggested that there are important alternatives to RCTs for assessing effectiveness and that RCTs could be enhanced significantly when mixed with additional methods that enable identification of why and how observed effects occur. Finally, we called upon OMB to draw on broader expertise in the evaluation community to develop future guidance on evaluation for the PART program.
We were delighted with the reception our comments received and with being invited subsequently to make a presentation to the first meeting of the newly established Evaluation Workgroup of the cross-agency Performance Improvement Council. We continue to work with OMB staff and other federal administrators on efforts to address the major evaluation concerns in PART.
I particularly want to thank all the members of the EPTF-Eleanor Chelimsky, Leslie Cooksy, Katherine Dawes, Patrick Grasso, Susan Kistler, Mel Mark, and Stephanie Shipman-and our consultant George Grob, for their highly professional and energetic collaboration in preparing this document in such a short period of time.
In the next newsletter we will share an interview we subsequently conducted with Robert Shea in which he describes the challenges facing the PART system, addresses the issue of the role of RCTs in program effectiveness evaluation, and describes how professional evaluators and AEA can be helpful in improving OMB PART in the future.
2008 AEA President
On behalf of the Board of Directors of AEA, we are pleased to announce the formation of the AEA Evaluation Policy Task Force (EPTF). The goal of this initiative is to assist AEA in developing an ongoing capability to influence evaluation policies that are critically important to the practice of evaluation.
The members of the Task Force are:
- Eleanor Chelimsky
- Leslie Cooksy
- Katherine Dawes
- Patrick Grasso
- Susan Kistler
- Mel Mark
- Stephanie Shipman
- William Trochim, chair
In addition, AEA has contracted with George Grob, President of the Center for Public Program Evaluation, as a consultant to the Task Force. George is a longtime AEA member and senior evaluation manager with significant experience in policy development and congressional relations. He will play a key role in assisting in the planning and implementation of the Task Force’s efforts to influence evaluation policy. This group collectively brings extensive and varied experience in the profession of evaluation, the development of evaluation policy, and in AEA, and we are excited about the prospects that they will be able to make an important contribution.
The term “evaluation policy” encompasses a wide range of potential topics that include (but are not limited to): when systematic evaluation gets employed, and on what programs, policies and practices; how evaluators are identified and selected; the relationship of evaluators to what is being evaluated; the timing, planning, budgeting and funding, contracting, implementation, methods and approaches, reporting, use and dissemination of evaluations; and, the relationship of evaluation policies to existing or prospective professional standards. To deal with the broad potential scope of this effort, the Task Force will, during the two year initiative, concentrate on evaluation policies in the United States Federal government, in both the legislative and executive branches. Focusing on the Federal level enables the Task Force to address evaluation policies that directly affect a broad cross-section of our membership and ultimately affects our entire field. The Task Force will make recommendations to the Board about how this scope might be extended into other sectors and areas over time.
Of course, engagement of AEA members is critical to the success of such an effort. We are doing several things to connect you with this initiative. We have already established a dedicated e-mail address (email@example.com) to receive your comments and questions and we will establish a special web page on the AEA website to keep you informed on an ongoing basis. We will look for other ways to communicate with and involve you as the initiative unfolds.
To help kick off this project and to engage you more personally, we would also like to invite you to a special open forum at the upcoming Evaluation 2007 annual conference in Baltimore entitled A Discussion of AEA's Evaluation Policy Initiative that will be held on Saturday, November 10, from 9:35 AM to 10:20 AM in the Versailles Room of the Radisson Plaza Lord Baltimore. At this session we will introduce this effort, and discuss the Task Force and its goals and charge in greater detail.
We look forward to working with AEA members through this effort to enhance the ability of our association and profession to influence the evaluation policies that shape how we do our work. Hope to see many of you at the conference.
2007 AEA President
2008 AEA President and Evaluation Policy Task Force