READING

    Evaluation Policy Initative

    In 2007, AEA began an initiative to assist in developing an ongoing capability to influence evaluation policy. This web page has been established to help you to learn more about our efforts.

    Actions you can take to get involved:


    Composition: The 2013 composition of the AEA Evaluation Policy Task Force (EPTF) is:

    • Eleanor Chelimsky
    • Katherine Dawes
    • Patrick Grasso
    • George Grob, co-chair
    • Susan Kistler
    • Mel Mark, co-chair
    • Cheryl Oros (Consultant)
    • Beverly Parsons
    • Stephanie Shipman

    EPTF Response to OMB re: What Constitutes Strong Evidence of a Program's Effectiveness

    Dear AEA Colleagues,

    One of the most important evaluation initiatives in the United States federal government these days is the Program Assessment Rating Tool (PART) used by the Office of Management and Budget (OMB) to assess virtually every federal government program. A PART review asks approximately 25 general questions about a program's performance and management, including several questions explicitly about evaluation. The answers determine a program's overall rating which is then published on OMB's website http://www.expectmore.gov. The sometimes controversial PART system was the focus of the first AEA Public Issues Forum (see http://www.eval.org/ part.asp) and the newly-established AEA Evaluation Policy Task Force (EPTF) identified PART as a priority area.

    Earlier this year, the EPTF contacted Robert Shea, the Associate Director of OMB for Administration and Government Performance, and a major architect of the PART system. I went with the EPTF's Consultant George Grob to meet with Shea, with the goals of introducing the American Evaluation Association, emphasizing the important role professional evaluators can play in the systematic assessment of Federal programs, and engaging him in a discussion of the PART's evaluation approach.

    Shea described OMB's new initiative to review and improve the PART program and requested that we provide him with detailed comments on a key document cited in the OMB PART Guidance entitled "What Constitutes Strong Evidence of a Program's Effectiveness?" This document has been especially controversial because of the nature of the case it makes regarding the use of randomized controlled trials (RCTs) and a formal request to review it and provide a thoughtful and balanced critique of the document itself and its policy implications is exactly what the EPTF was hoping to encourage.

    We worked hard in less than a week to produce a balanced critique and I am delighted to share with you today our cover letter and the comments that we provided (download here). We recommended that OMB develop new guidance for the evaluation components of PART that integrates evaluation more closely with information from other questions about program planning and management. This guidance should describe the variety of methods for assessing program effectiveness that are appropriate to the needs and development level of a program. We argued for a more balanced presentation of the role of RCTs, and suggested that there are important alternatives to RCTs for assessing effectiveness and that RCTs could be enhanced significantly when mixed with additional methods that enable identification of why and how observed effects occur. Finally, we called upon OMB to draw on broader expertise in the evaluation community to develop future guidance on evaluation for the PART program.

    We were delighted with the reception our comments received and with being invited subsequently to make a presentation to the first meeting of the newly established Evaluation Workgroup of the cross-agency Performance Improvement Council. We continue to work with OMB staff and other federal administrators on efforts to address the major evaluation concerns in PART.

    I particularly want to thank all the members of the EPTF-Eleanor Chelimsky, Leslie Cooksy, Katherine Dawes, Patrick Grasso, Susan Kistler, Mel Mark, and Stephanie Shipman-and our consultant George Grob, for their highly professional and energetic collaboration in preparing this document in such a short period of time.

    In the next newsletter we will share an interview we subsequently conducted with Robert Shea in which he describes the challenges facing the PART system, addresses the issue of the role of RCTs in program effectiveness evaluation, and describes how professional evaluators and AEA can be helpful in improving OMB PART in the future.

    Sincerely,

    Bill Trochim,
    2008 AEA President


    Evaluation 2009 Session Descriptions

    Session Title: Influencing Evaluation Policy and Evaluation Practice: A Progress Report From The American Evaluation Association's (AEA) Evaluation Policy Task Force

    Panel Session 495 to be held in Panzacola Section F2 on Friday, Nov 13, 10:55 AM to 11:40 AM

    Sponsored by the AEA Conference Committee

    Chair(s):

    William Trochim, Cornell University, wmt1@cornell.edu

    Patrick Grasso, World Bank, pgrasso@worldbank.org

    Discussant(s):

    Eleanor Chelimsky, Independent Consultant, oandecleveland@aol.com

    Leslie J Cooksy, University of Delaware, ljcooksy@udel.edu

    Katherine Dawes, United States Environmental Protection Agency, dawes.katherine@epamail.epa.gov

    Susan Kistler, American Evaluation Association, susan@eval.org

    Melvin Mark, Pennsylvania State University, m5m@psu.edu

    Stephanie L Shipman, United States Government Accountability Office, shipmans@gao.gov

    Abstract: The Board of Directors of the American Evaluation Association (AEA) established the Evaluation Policy Task Force (EPTF) in order to enhance AEA's ability to identify and influence policies that have a broad effect on evaluation practice and to establish a framework and procedures for accomplishing this objective. Since starting operations on September 1, 2007, the EPTF has issued key documents promoting a wider role for evaluation in the Federal Government, influenced both federal legislation and executive policy, and informed AEA members and others about the value of evaluation through public presentations and newsletter articles. In July, the Board extended the charter of the EPTF for two years with an evaluation at the end of this period. It also appointed Patrick Grasso as the new Chair, replacing Bill Trochim, who chaired the EPTF since its inception. This session will provide an update on their work and invite member input on their plans and actions.

    Introduction to the Evaluation Policy Task Force

    William Trochim, Cornell University, wmt1@cornell.edu

    This will be an overview of EPTF activities over the last two years and a summary of current plans for the future.

    Activities and Plans for the EPTF

    George F Grob, Center for Public Program Evaluation, georgefgrob@cs.com; Patrick Grasso, World Bank, pgrasso@worldbank.org

    Mr. Grob, Consultant to the EPTF, will facilitate a discussion involving EPTF members and the audience about the activities and plans of the EPTF

     

    Session Title: Introduction to Evaluation and Public Policy

    Expert Lecture Session 579 to be held in Panzacola Section F2 on Friday, Nov 13, 3:35 PM to 4:20 PM

    Sponsored by the AEA Conference Committee

    Presenter(s): George F Grob, Center for Public Program Evaluation, georgefgrob@cs.com

    Abstract: Evaluation and public policy are intimately connected. Such connections occur at national, state, and local government levels, and even on the international scene. The interaction moves in two directions: sometimes evaluation affects policies for public programs, and sometimes public policies affect how evaluation is practiced. Either way, the connection is important to evaluators. This session will explain how the public policy process works. It will guide evaluators through the maze of policy processes, such as legislation, regulations, administrative procedures, budgets, re-organizations, and goal setting. It will provide practical advice on how evaluators can become a public policy players, how they can influence policies that affect their very own profession, and how to get their evaluations noticed and used in the public arena. There will opportunities for audience discussion of sensitive topics, such as how evaluators can protect their independence in a world of compromise and deal making.

     

    Session Title: The Office of Management and Budget's New Policy on Increased Emphasis on Program Evaluation: An Open Discussion

    Think Tank Session 751 to be held in Panzacola Section H1 on Saturday, Nov 14, 10:55 AM to 11:40 AM

    Sponsored by the Presidential Strand

    Chair(s): Debra Rog, Westat, debrarog@westat.com

    Presenter(s): George F Grob, Center for Public Program Evaluation, georgefgrob@cs.com

    Discussant(s): Patrick Grasso, World Bank, pgrasso@worldbank.org

    Abstract: On October 7, 2009, OMB Director Peter Orszag issued a memorandum to the Heads of Federal Departments and Agencies on Increased Emphasis on Program Evaluation. (http://www.whitehouse.gov/omb/assets/memoranda_2010/m10-01.pdf) It focuses on impact evaluation, with initial application to social, educational, economic, and similar programs whose expenditures are aimed at improving life outcomes for individuals. It promotes rigorous, independent evaluations to be used as a key resource in determining whether government programs are achieving their objectives at the lowest possible cost. Because of the importance of this new policy. This session will provide an opportunity for open discussion among all interested AEA members.


    Evaluation 2008 Session Descriptions:

    The following two sessions devoted to discussion of evaluation policy and the initiative are scheduled for Evaluation 2008.

    Session Title: Introduction to Evaluation and Public Policy
    Demonstration Session 344 to be held in Capitol Ballroom Section 5 on Thursday, Nov 6, 3:35 PM to 4:20 PM
    Sponsored by the Presidential Strand
    Chair(s):
    William Trochim,  Cornell University,  wmt1@cornell.edu
    Presenter(s):
    George Grob,  Center for Public Program Evaluation,  georgefgrob@cs.com
    Abstract: Evaluation and public policy are intimately connected. Such connections occur at national, state, and local government levels, and even on the international scene. The interaction moves in two directions: sometimes evaluation affects policies for public programs, and sometimes public policies affect how evaluation is practiced. Either way, the connection is important to evaluators. This session will explain how the public policy process works. It will guide evaluators through the maze of policy processes, such as legislation, regulations, administrative procedures, budgets, re-organizations, and goal setting. It will provide practical advice on how evaluators can become a public policy players how they can influence policies that affect their very own profession, and how to get their evaluations noticed and used in the public arena. There will opportunities for audience discussion of sensitive topics, such as how evaluators can protect their independence in a world of compromise and deal making.

     

    Session Title: Influencing Evaluation Policy and Evaluation Practice: A Progress Report From AEA's Evaluation Policy Task Force
    Panel Session 692 to be held in Centennial Section C on Friday, Nov 7, 4:30 PM to 6:00 PM
    Sponsored by the Presidential Strand
    Chair(s):
    William Trochim,  Cornell University,  wmt1@cornell.edu
    Abstract: At its Winter 2007 meeting, the Board of Directors of the American Evaluation Association (AEA) discussed its interest in the Association enhancing its ability to identify and influence evaluation policies that have a broad effect on evaluation practice. To that end the Board established an Evaluation Policy Task Force that can advise AEA on how best to proceed in this arena. Since then, the Task Force has identified opportunities for policy influence, developed materials, overseen the hiring of a consultant, and guided the initiative in carrying out its charge. This session will provide an update on their work and seek member input on their actions and outcomes.
    Eleanor Chelimsky,  Independent Consultant,  oandecleveland@aol.com
    Leslie J Cooksy,  University of Delaware,  ljcooksy@udel.edu
    Katherine Dawes,  United States Environmental Protection Agency,  dawes.katherine@epa.gov
    Patrick Grasso,  The World Bank,  pgrasso45@comcast.net
    George Grob,  Center for Public Program Evaluation,  georgefgrob@cs.com
    Susan Kistler,  American Evaluation Association,  susan@eval.org
    Melvin Mark,  Pennsylvania State University,  m5m@psu.edu

    Evaluation 2007 Session Descriptions:

    The following panel session devoted to discussion of the initiative was offered at Evaluation 2007. Click here to view the PowerPoint presentation slides.

    Session Title: A Discussion of AEA's Evaluation Policy Initiative
    Panel Session 733 to be held in Versailles Room on Saturday, November 10, 9:35 AM to 10:20 AM
    Sponsored by the AEA Conference Committee
    Abstract: The AEA has created a new Evaluation Policy Task Force (EPTF). The goal of this two-year initiative is to assist AEA in developing an ongoing capability to influence evaluation policies that are critically important to the practice of evaluation. The term “evaluation policy” encompasses a wide range of potential topics that include (but are not limited to): when systematic evaluation gets employed, and on what programs, who is involved in evaluations and how they are engaged; how evaluators are identified and selected; the relationship of evaluators to what is being evaluated; the timing, planning, budgeting and funding, contracting, implementation, methods and approaches, reporting, use and dissemination of evaluations; and, the relationship of evaluation policies to existing or prospective professional standards. This session introduces the Task Force, describes how it has organized to accomplish this work, and elicits AEA member comments and thoughts to help in this endeavor.
    William Trochim,  Cornell University,  wmt1@cornell.edu
    Hallie Preskill,  Claremont Graduate University,  hallie.preskill@cgu.edu
    George Grob,  Center for Public Program Evaluation,  goergefgrob@CS.com

    Announcement Letter:

    On behalf of the Board of Directors of AEA, we are pleased to announce the formation of the AEA Evaluation Policy Task Force (EPTF). The goal of this  initiative is to assist AEA in developing an ongoing capability to influence evaluation policies that are critically important to the practice of evaluation.

    The members of the Task Force are:

    • Eleanor Chelimsky
    • Leslie Cooksy
    • Katherine Dawes
    • Patrick Grasso
    • Susan Kistler
    • Mel Mark
    • Stephanie Shipman
    • William Trochim, chair

    In addition, AEA has contracted with George Grob, President of the Center for Public Program Evaluation, as a consultant to the Task Force. George is a longtime AEA member and senior evaluation manager with significant experience in policy development and congressional relations. He will play a key role in assisting in the planning and implementation of the Task Force’s efforts to influence evaluation policy. This group collectively brings extensive and varied experience in the profession of evaluation, the development of evaluation policy, and in AEA, and we are excited about the prospects that they will be able to make an important contribution.

    The term “evaluation policy” encompasses a wide range of potential topics that include (but are not limited to): when systematic evaluation gets employed, and on what programs, policies and practices; how evaluators are identified and selected; the relationship of evaluators to what is being evaluated; the timing, planning, budgeting and funding, contracting, implementation, methods and approaches, reporting, use and dissemination of evaluations; and, the relationship of evaluation policies to existing or prospective professional standards. To deal with the broad potential scope of this effort, the Task Force will, during the two year initiative, concentrate on evaluation policies in the United States Federal government, in both the legislative and executive branches. Focusing on the Federal level enables the Task Force to address evaluation policies that directly affect a broad cross-section of our membership and ultimately affects our entire field. The Task Force will make recommendations to the Board about how this scope might be extended into other sectors and areas over time.

    Of course, engagement of AEA members is critical to the success of such an effort. We are doing several things to connect you with this initiative. We have already established a dedicated e-mail address (evaluationpolicy@eval.org) to receive your comments and questions and we will establish a special web page on the AEA website to keep you informed on an ongoing basis. We will look for other ways to communicate with and involve you as the initiative unfolds.

    To help kick off this project and to engage you more personally, we would also like to invite you to a special open forum at the upcoming Evaluation 2007 annual conference in Baltimore entitled A Discussion of AEA's Evaluation Policy Initiative that will be held on Saturday, November 10, from 9:35 AM to 10:20 AM in the Versailles Room of the Radisson Plaza Lord Baltimore. At this session we will introduce this effort, and discuss the Task Force and its goals and charge in greater detail.

    We look forward to working with AEA members through this effort to enhance the ability of our association and profession to influence the evaluation policies that shape how we do our work. Hope to see many of you at the conference.

    Hallie Preskill
    2007 AEA President

    William Trochim
    2008 AEA President and Evaluation Policy Task Force

    AEA responds to UN Resolution: Empowering Countries through Evaluation: Evaluation as a country-level tool for the new development agenda