READING

An Introduction to PART: William Trochim

The topic for this inaugural forum is the Program Assessment Rating Tool (PART), a major initiative of the Office of Management and Budget (OMB) to assess and improve federal program performance, with the goal of achieving better results from such programs. I will introduce the PART process and its website dissemination mechanism, ExpectMore.gov, and describe briefly how they operate. Then each panelist will have a chance to present their experiences with PART and their sense of its strengths and weaknesses, after which we’ll take general questions from the audience. Finally, I hope to give each panelist an opportunity for any final summary remarks.

"As would be expected with any federal program of such magnitude, the PART process and the ExpectMore.gov website have not been without controversy ... Within the evaluation profession, a lively discussion has been joined about the degree to which PART reflects good evaluation practice, and that discussion is one reason we decided to focus on this as both a timely and important issue for this conference." -Bill Trochim

Click here to hear to hear Dr. Trochim discuss an overview of open issues for PART within the evaluation community - an excerpt from the full session audio.

The Program Assessment Rating Tool (PART) can be viewed as evolving out of the Government Performance Results Act (GPRA) of 1993, which mandated the development of a system for assessing performance of all government programs. PART was developed during 2002 and is now in the 5th year of a 5-year cycle to assess all Federal programs. It has assessed 793 programs to date and expects to reach approximately 1,000 in total by the end of 2006. Programs will continue to be assessed at least once every five years.

The PART questionnaire includes 25 questions that are divided into four sections:

  • The first section of questions asks whether a program's purpose is clear and whether it is well designed to achieve its objectives.
  • The second section involves strategic planning, and weighs whether the agency establishes valid annual and long-term goals for its programs.
  • The fourth section of questions focuses on results that programs can report with accuracy and consistency.

The questions in section one, which account for 20% of the final score, ask whether the program’s purpose is clear, whether the program addresses a specific and existing problem or issue, whether it duplicates other efforts, is free of major design flaws, and is designed effectively.

Section I. Program Purpose and Design

1.1: Is the program purpose clear?
1.2: Does the program address a specific and existing problem, interest, or need?
1.3: Is the program designed so that it is not redundant or duplicative of any other Federal, State, local or private effort?
1.4: Is the program design free of major flaws that would limit the program’s effectiveness or efficiency?
1.5: Is the program design effectively targeted so that resources will address the program’s purpose directly and will reach intended beneficiaries?

The questions in section two, accounting for 10% of the final score, focus on strategic planning and ask whether the program has performance measures, targets and timeframes, annual performance measures that can assess progress with baselines and targets, whether partners are committed to working toward these goals, whether budgeting is tied to accomplishments and whether the program has taken steps to correct any deficiencies. I’ve highlighted question 2.6 which is of particular relevance to evaluators and asks whether “high-quality” independent evaluations are conducted on a regular basis.

Section II. Strategic Planning

2.1: Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?
2.2: Does the program have ambitious targets and timeframes for its long-term measures?
2.3: Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program’s long-term goals?
2.4: Does the program have baselines and ambitious targets for its annual measures?
2.5: Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?
2.6: Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?
2.7: Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program’s budget?
2.8: Has the program taken meaningful steps to correct its strategic planning deficiencies?

The program management questions in section three, which accounts for 20% of the final score, ask whether the agency collects and uses performance information, holds managers and partners accountable, whether funds are handled appropriately, if procedures are in place to measure efficiencies and cost effectiveness, whether the program collaborates and cooperates with others, has strong financial practices, and addresses identified management deficiencies.

Section III. Program Management

3.1: Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?
3.2: Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?
3.3: Are funds (Federal and partners’) obligated in a timely manner, spent for the intended purpose, and accurately reported?
3.4: Does the program have procedures (e.g., competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?
3.5: Does the program collaborate and coordinate effectively with related programs?
3.6: Does the program use strong financial management practices?
3.7: Has the program taken meaningful steps to address its management deficiencies?

The results and accountability questions in section four, which accounts for 50% of the overall rating ask whether the program demonstrates progress toward goals, achieves annual performance goals, shows improved efficiencies and compares favorably to other programs. Of special note to evaluators, the final question explicitly asks whether independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results.

Section IV. Program Results/Accountability

4.1: Has the program demonstrated adequate progress in achieving its long-term performance goals?
4.2: Does the program (including program partners) achieve its annual performance goals?
4.3: Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?
4.4: Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?
4.5: Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

In the simplest case, each of the questions is answered with a Yes or No answer and questions within sections receive equal weight, with a total of 100 points for each section. The section scores are then weighted to achieve the final score, which determines which of the four overall categories the program will be assigned to: Effective (if the score is 85-100); Moderately Effective (70-84); Adequate (50-69); or Ineffective (below 50). Regardless of overall score, a fifth possible rating of Results Not Demonstrated is given when programs do not have acceptable long-term and annual performance measures. A program also gets a rating of Results Not Demonstrated when it lacks baselines and performance data to indicate how it has been performing.

The current distribution for the 793 programs that have already been through a PART review shows that approximately a quarter of them received a rating of Results Not Demonstrated. Fifteen percent were determined to be effective. Approximately 30% were found to be Moderately Effective or Adequate. Only 4% have been determined to be Ineffective. Every part review includes an improvement plan that the program can follow to address identified performance issues.

PART: Results to Date

Rating Percent (and Number) of Programs (N = 793)

Effective: 15% (124)
Moderately Effective: 29% (231)
Adequate: 28% (219)
Ineffective: 4% (28)
Results Not Demonstrated: 24% (191)

The website ExpectMore.gov, shown in Figure 1, was launched in February 2006 and is the primary public mechanism for reporting on Federal program performance and what is being done to improve results. The site includes a variety of simple mechanisms that enable the user to identify programs that are performing or not performing, and search by keyword or topic.

As one example, Figure 2 shows a section of the report displaying programs that are considered to be performing, in the section of reviewed programs from the Department of Education.

ExpectMore.gov is designed to be a user-friendly public site, which provides simple explanations for the ratings and uses different numbers of stars to indicate the final rating category for each program. The user can then drill down on that overall rating to get more detail about how that assessment was reached and what the improvement plan is for the program. For example, Figure 3 shows an example of the rating details for the Even Start program, which was judged to be ineffective.

Figure 3. Rating details for the Even Start program.

As would be expected with any federal program of such magnitude, the PART process and the ExpectMore.gov website have not been without controversy. Several reports of the Government Accountability Office have raised concerns about implementation and advocacy groups outside the federal government, such as OMB Watch, remain critical. Within the evaluation profession, a lively discussion has been joined about the degree to which PART reflects good evaluation practice, and that discussion is one reason we decided to focus on this as both a timely and important issue for this conference.

There’s an old saying that the “devil is in the details” and that is certainly likely to be the case with a system that is as ambitious and complex as PART. It is impossible in so short an introduction to do justice to the many details and controversies that swirl around each of the specific ratings, let alone to consider the broader implications of their aggregation, categorization and representation to the public. And so, we present this first AEA Public Issues Forum in the hope that it will begin to identify some of those issues and present the experiences and reflections of evaluators who have been involved in different ways with the PART system, which has such important potential implications for the theory and practice of evaluation.

Panel comments: Michael Schooley