EVALUATION POLICY INITIATIVE

In 2007, AEA began an initiative to assist in developing an ongoing capability to influence evaluation policy. This web page has been established to help you to learn more about our efforts.

• Composition

• Evaluation policy task force charge

• Updated! EPTF Roadmap Citations (Most recent update - December 2012)

• EPTF Call for Proposals for Consultant Services on Influencing Evaluation Policy (October 2012)

• EPTF External Evaluation Panel Review - Final Report (June 2012)

• EPTF 5-year History Evaluation Briefing Book (December 2011)

• Human Subjects Call for Comments to Dept of Health and Human Services (September 2011)

• Letter to the Editor, New York Times, from AEA President Jennifer Greene regarding evaluator independence (pdf)

• What does the 2012 Presidential Budget say about Evaluation? (pdf in eLibrary) (February 2011)

• AEA Comments on 2010 Exposure Draft of GAO's Government Auditing Standards (pdf)

• Evaluation Roadmap for a More Effective Government as approved by AEA membership (October 2010)

• AEA Comments to HRSA on Evidence of Effectiveness of Maternal, Infant, and Early Childhood Home Visiting Programs (pdf) (August 2010)

• What does the 2011 Presidential Budget say about Evaluation? (pdf) (February 2010)

• AEA Comments to OMB on the Paperwork Reduction Act (December 2009)

• AEA Presidents' Response to OMB Memorandum (see below) (October 2009 pdf)

• Link to OMB Director Peter Orszag Office of Management and Budget (OMB) Memorandum on Increased Emphasis on Program Evaluation (October 2009)

• Link to Government Accountability Office (GAO) Report on USAID Food Aid using the EPTF Evaluation Roadmap as Criteria (September 2009)

• EPTF Legislative Framework for Evaluation of Health Care Reform (pdf) (September 2009)

• EPTF Comments to House Committee (June 2009)

• EPTF Comments to Senate Finance Committee (June 2009)

• EPTF Evaluation Roadmap for a More Effective Government, Submitted to Peter Orzag, Director of the Office of Management and Budget (pdf)

• What does the 2010 Presidential Budget say about Evaluation? (February 2009) (pdf)

• EPTF handouts describing evaluation and evaluators and what they can contribute to good government (pdf)

• EPTF Response to OMB re: What Constitutes Strong Evidence of a Program's Effectiveness

• Request for Nominations of Successful, Influential, Evaluations

• Evaluation 2009 conference session descriptions

• Evaluation 2008 conference session descriptions

• Evaluation 2007 conference session description and slides

• Announcement letter (October 2007)

Actions you can take to get involved:

• Sign up for the EPTF Discussion List

• Contact the evaluation policy taskforce via email at evaluationpolicy@eval.org

Composition: The 2013 composition of the AEA Evaluation Policy Task Force (EPTF) is:

• Eleanor Chelimsky

• Katherine Dawes

• Patrick Grasso, chair

• Susan Kistler

• Mel Mark

• Cheryl Oros (Consultant)

• Beverly Parsons

• Stephanie Shipman

EPTF Response to OMB re: What Constitutes Strong Evidence of a Program's Effectiveness

Dear AEA Colleagues,

One of the most important evaluation initiatives in the United States federal government these days is the Program Assessment Rating Tool (PART) used by the Office of Management and Budget (OMB) to assess virtually every federal government program. A PART review asks approximately 25 general questions about a program's performance and management, including several questions explicitly about evaluation. The answers determine a program's overall rating which is then published on OMB's website http://www.expectmore.gov. The sometimes controversial PART system was the focus of the first AEA Public Issues Forum (see http://www.eval.org/ part.asp) and the newly-established AEA Evaluation Policy Task Force (EPTF) identified PART as a priority area.

Earlier this year, the EPTF contacted Robert Shea, the Associate Director of OMB for Administration and Government Performance, and a major architect of the PART system. I went with the EPTF's Consultant George Grob to meet with Shea, with the goals of introducing the American Evaluation Association, emphasizing the important role professional evaluators can play in the systematic assessment of Federal programs, and engaging him in a discussion of the PART's evaluation approach.

Shea described OMB's new initiative to review and improve the PART program and requested that we provide him with detailed comments on a key document cited in the OMB PART Guidance entitled "What Constitutes Strong Evidence of a Program's Effectiveness?" This document has been especially controversial because of the nature of the case it makes regarding the use of randomized controlled trials (RCTs) and a formal request to review it and provide a thoughtful and balanced critique of the document itself and its policy implications is exactly what the EPTF was hoping to encourage.

We worked hard in less than a week to produce a balanced critique and I am delighted to share with you today our cover letter and the comments that we provided (download at http://www.eval.org/aea08.omb.guidance.responseF.pdf). We recommended that OMB develop new guidance for the evaluation components of PART that integrates evaluation more closely with information from other questions about program planning and management. This guidance should describe the variety of methods for assessing program effectiveness that are appropriate to the needs and development level of a program. We argued for a more balanced presentation of the role of RCTs, and suggested that there are important alternatives to RCTs for assessing effectiveness and that RCTs could be enhanced significantly when mixed with additional methods that enable identification of why and how observed effects occur. Finally, we called upon OMB to draw on broader expertise in the evaluation community to develop future guidance on evaluation for the PART program.

We were delighted with the reception our comments received and with being invited subsequently to make a presentation to the first meeting of the newly established Evaluation Workgroup of the cross-agency Performance Improvement Council. We continue to work with OMB staff and other federal administrators on efforts to address the major evaluation concerns in PART.

I particularly want to thank all the members of the EPTF-Eleanor Chelimsky, Leslie Cooksy, Katherine Dawes, Patrick Grasso, Susan Kistler, Mel Mark, and Stephanie Shipman-and our consultant George Grob, for their highly professional and energetic collaboration in preparing this document in such a short period of time.

In the next newsletter we will share an interview we subsequently conducted with Robert Shea in which he describes the challenges facing the PART system, addresses the issue of the role of RCTs in program effectiveness evaluation, and describes how professional evaluators and AEA can be helpful in improving OMB PART in the future.

Sincerely,

Bill Trochim,
2008 AEA President

Evaluation 2009 Session Descriptions

Evaluation 2008 Session Descriptions:

The following two sessions devoted to discussion of evaluation policy and the initiative are scheduled for Evaluation 2008.

Evaluation 2007 Session Descriptions:

The following panel session devoted to discussion of the initiative was offered at Evaluation 2007. Click here to view the PowerPoint presentation slides.

Announcement Letter:

On behalf of the Board of Directors of AEA, we are pleased to announce the formation of the AEA Evaluation Policy Task Force (EPTF). The goal of this  initiative is to assist AEA in developing an ongoing capability to influence evaluation policies that are critically important to the practice of evaluation.

The members of the Task Force are:

• Eleanor Chelimsky

• Leslie Cooksy

• Katherine Dawes

• Patrick Grasso

• Susan Kistler

• Mel Mark

• Stephanie Shipman

• William Trochim, chair

In addition, AEA has contracted with George Grob, President of the Center for Public Program Evaluation, as a consultant to the Task Force. George is a longtime AEA member and senior evaluation manager with significant experience in policy development and congressional relations. He will play a key role in assisting in the planning and implementation of the Task Force’s efforts to influence evaluation policy. This group collectively brings extensive and varied experience in the profession of evaluation, the development of evaluation policy, and in AEA, and we are excited about the prospects that they will be able to make an important contribution.

The term “evaluation policy” encompasses a wide range of potential topics that include (but are not limited to): when systematic evaluation gets employed, and on what programs, policies and practices; how evaluators are identified and selected; the relationship of evaluators to what is being evaluated; the timing, planning, budgeting and funding, contracting, implementation, methods and approaches, reporting, use and dissemination of evaluations; and, the relationship of evaluation policies to existing or prospective professional standards. To deal with the broad potential scope of this effort, the Task Force will, during the two year initiative, concentrate on evaluation policies in the United States Federal government, in both the legislative and executive branches. Focusing on the Federal level enables the Task Force to address evaluation policies that directly affect a broad cross-section of our membership and ultimately affects our entire field. The Task Force will make recommendations to the Board about how this scope might be extended into other sectors and areas over time.

Of course, engagement of AEA members is critical to the success of such an effort. We are doing several things to connect you with this initiative. We have already established a dedicated e-mail address (evaluationpolicy@eval.org) to receive your comments and questions and we will establish a special web page on the AEA website to keep you informed on an ongoing basis. We will look for other ways to communicate with and involve you as the initiative unfolds.

To help kick off this project and to engage you more personally, we would also like to invite you to a special open forum at the upcoming Evaluation 2007 annual conference in Baltimore entitled A Discussion of AEA's Evaluation Policy Initiative that will be held on Saturday, November 10, from 9:35 AM to 10:20 AM in the Versailles Room of the Radisson Plaza Lord Baltimore. At this session we will introduce this effort, and discuss the Task Force and its goals and charge in greater detail.

We look forward to working with AEA members through this effort to enhance the ability of our association and profession to influence the evaluation policies that shape how we do our work. Hope to see many of you at the conference.

Hallie Preskill
2007 AEA President

William Trochim
2008 AEA President and Evaluation Policy Task Force

American Evaluation Association│16 Sconticut Neck Rd #290 │Fairhaven MA 02719
1-888-232-2275 │1-508-748-3326 │info@eval.org