|
Session Title: Benchmarking the United States' Program Assessment Rating Tool: Looking Back at Federal Program's PART Experiences and Getting Glimpses of the Future of Federal Evaluation Policy
|
|
Panel Session 260 to be held in Centennial Section C on Thursday, Nov 6, 10:55 AM to 12:25 PM
|
|
Sponsored by the Presidential Strand
and the Government Evaluation TIG
|
| Chair(s): |
| Katherine Dawes,
United States Environmental Protection Agency,
dawes.katherine@epa.gov
|
| Abstract:
This session features a series of studies related to the Program Assessment Rating Tool (PART) assessments reported by the US Office of Management and Budget (OMB) as part of the federal budget process (See http://www.whitehouse.gov/omb/part/index.html and www.expectmore.gov). Being assessed under the PART system has proved challenging for many Federal programs, and the use of measures and program evaluation has frequently been a 'point of friction' between agencies and OMB. Several federal agencies have taken in-depth looks at published PART assessments to 'benchmark' their programs PART scores, learn about best practices from other agencies' PART assessments, and to better understand OMB expectations for performance measurement and program evaluation. The three presentations in this panel illustrate the challenges of PART, and consider the effect of agencies' PART experience on the future of federal evaluation policy.
|
|
Reflections on Environmental and Natural Resource Programs' Experiences with the Program Assessment Rating Tool and What It Means for Future Federal Environmental Evaluation Policy
|
| Michael Mason,
United States Environmental Protection Agency,
mason.michael@epa.gov
|
|
The presentation will provide an overview of the PART experiences for federal environmental and natural resource programs from 2002 to 2008. Agencies included in the overview will be the Environmental Protection Agency (EPA), the Department of the Interior, the Department of Agriculture, and the National Oceanic and Atmospheric Administration. The presentation will provide a statistical summary of PART ratings and scores with a breakdown of section-by-section and question-by-question results. Trends will be noted and their implications will be discussed. In-depth analysis will also be provided on EPA's experience based on interviews with program PART coordinators. Environmental and natural resource PART results will be compared with a random selection of other federal programs in an effort to detect trends and differences. The presentation will provide a set of recommendations or options for evaluating federal programs with a focus on how environmental and natural resource programs could benefit from regular evaluations.
|
|
|
Program Assessment Rating Tool Scores and Budgetary Reactions from the Office of Management and Budget (OMB) and Congress: An Analysis of the Impact on Energy Efficiency and Renewable Energy Programs
|
| David Shen,
Consultant for the United States Department of Energy,
david.shen@ee.doe.gov
|
|
The Program Assessment Rating Tool (PART) is linked to the Federal budget process and is used as a factor in determining if programs should receive greater or lesser amounts of funding. This paper reports on the findings from a statistical analysis of PART scores for 955 Federal programs and 340 Federal programs in a cohort analysis. It also examines program PART results for eleven programs in the Office of Energy Efficiency and Renewable Energy (EERE) in the Department of Energy (DOE). The analysis examines the first and second order changes in program budgets that may be associated with their PART scores. Although much statistical variance makes it difficult to claim strong significance, a pattern of budget increases with positive scores and budget cuts with lower scores was nevertheless found across all different types of Federal programs.
| |
|
Research Evaluation and Performance Measurement: New Directions from the National Academy of Sciences
|
| Phillip Juengst,
United States Environmental Protection Agency,
juengst.phillip@epa.gov
|
|
This presentation focuses on the Environmental Protection Agency's (EPA) success in initiating dialog with other agencies to improve the performance measurement and evaluation of research programs. EPA initially struggled with the Office of Management and Budget's (OMB) Program Assessment Rating Tool (PART), and subsequently examined and implemented best practices from around the government that significantly improved PART scores. EPA found wide variation in the types of measures used by Federal agencies and engaged the National Academy of Sciences (NAS) to conduct a study of the issue. NAS' report includes broad recommendations for research programs and changing OMB's PART process. EPA and OMB are leading discussions with other agencies to review these recommendations. The NAS report and cross-agency discussions are providing new directions for how and to what extent research programs should focus on intermediate outcomes, conduct evaluations that incorporate assessments of 'investment efficiency,' and develop meaningful 'process efficiency' measures.
| |