Panel Comments: Michael Schooley

“With the PART review, a lot of people focus on the destination of getting to the score, and getting the review done, but there is equal value, I think, in the journey. We need to think about how to use the aspects of the journey in our continued work, and as we move forward.”    -Michael Schooley

Click here to hear Michael Schooley discuss a summary of future challenges with respect to the use of PART in program assessment - an excerpt from the full session audio.

Good afternoon. I want to start by recognizing and thanking Bill for his introduction and warm remarks, and also for organizing this panel. As he mentioned, there was a lot of discussion on EvalTalk earlier this year on two different occasions, so I think we all welcome the opportunity to engage in this fruitful discussion. 

I’m going to start by adding to a few of the comments Bill has made and reiterating a few of the points that he highlighted in his introductory remarks. I’m going to talk about some of the PART reviews that have been conducted at CDC, how they have been conducted and how the agency has done on these reviews, and then I’m going to share some of my own observations and considerations as an evaluator working on some of the reviews. If you’ve heard a few other federal presentations, you’ll not be surprised to hear me say that the views I present are my own and do not necessarily represent those of the agency. 

As Bill mentioned, PART was created in 2002 – fiscal year 2004 was when it was first applied, to programs at CDC and elsewhere. It was presented as a diagnostic tool to be used by OMB to assess program performance, and its intention is to improve program performance. 

The tool, as Bill introduced, is 25 to 30 questions, divided up into four broad sections or areas. One of the questions of particular importance for evaluation is, “Does the program have a limited number of specific, ambitious, long-term performance goals that focus on outcomes and meaningfully reflect the purpose of the program?”, “Does the program achieve its annual performance goals?”, and then “Do independent and quality evaluations of the program indicate that it is effective in achieving results?”

The PART tool can be slightly adapted across a range of programs. There are seven broad areas of programs it can be applied to:

  • Competitive grant programs
  • Block/formula grant programs
  • Regulatory-based programs
  • Capital assets and service acquisition programs
  • Credit Programs
  • Research and development programs
  • Direct federal programs

Most of the work that is done at CDC falls under the competitive grants program. At CDC 23 programs have been reviewed over the past five years, and we are in the final year of evaluating programs for fiscal year 2008. Fifteen out of these 23 programs are competitive grant programs, and the funding range is anywhere from $63M to $900M. I think this is important in illustrating that there is sufficient variation in the size and scope of programs that have been reviewed, even within CDC and the public health realm.

Here are examples, in terms of content and topics, of some of the programs that have been reviewed at CDC. These include things like the child immunization program, and in comparison, at the end of the list, the global immunization program, as well as an entire agency, the Agency for Toxic Substances and Disease Registry. Other areas include state and local preparedness grants, domestic HIV/AIDS program, chronic disease prevention and health promotion programs, which represents a variety of programs addressing tobacco, cancer, heart disease, nutrition, and physical activity.

Sample CDC programs that have had PART reviews

  • Childhood Immunization Program
  • Domestic HIV/AIDS Prevention
  • State and Local Preparedness
  • Chronic Disease Prevention and Health Promotion
  • Agency for Toxic Substances and Disease Registry
  • Global Immunization

In terms of the reviews that have been conducted at CDC, most of the reviews have resulted in scores that are adequate to moderately effective. So overall, as an agency, we’ve actually done pretty well if you’re just looking at the final scores. One program was designated as effective, five programs were determined “results not demonstrated”, and we haven’t had any programs that have been determined to be ineffective.

In general, if you look across those four broad sections, the lowest scores are on the results sections – that’s where I think a lot of challenges come in with programs in providing and demonstrating results and providing some of the evaluation information. Over the past five years, the scores have increased, overall by about 10 percent. This is actually interesting, and I think Bill made an allusion to this when he said, “Ted obviously knows how to do a good PART review” - in that the agency, in some respects, may have become a better test-taker, in knowing what to provide for the review and how to prepare for the review. The other issue is that over the course of the five years we have had three OMB examiners. We had one OMB examiner for the first couple of years of reviews, another one for the second couple of years, and just recently a third examiner for the most recent year. As OMB examiners change, there is some subjectivity to the review and the exam, and that may influence the review and the scores that we receive.

And then finally, part of the PART review is looking at performance measures. These measures are generally outcome measures, mostly longer-term health outcome measures – not shorter-term measures, and not really a lot of output measures. Some samples of these measures at CDC are listed here.

 Sample performance measures at CDC

  • Number of countries in the world with endemic wild polio virus
  • For states receiving CDC funding, increase the percentage of persons with diabetes who receive annual eye and foot exams
  • Reduce the prevalence of Chlamydia among high risk women under age 25 by 15%
  • Ratio of operational costs to total program costs in countries receiving $1M+ HIV/AIDS

There’s a document on the guidance for OMB website, which documents the workshop that occurred in 2003 looking at performance measurement issues in providing measures for PART. It addresses seven broad areas around performance measurement challenges. I think these are of particular interest to program evaluators, and obviously ring true to a lot of the work we do, especially public health. As programs have been preparing for these PART reviews and preparing performance measures, some of the things that come up are:

1) Program outcomes are extremely difficult to measure. Some of the programs are looking at broad and far-reaching things like policy change and issues, which are difficult things to measure from a scientific perspective and from a specific discrete measurement perspective.
2) The programs oftentimes are one contributor to the outcome, looking at the issue of contribution versus attribution.
3) The results often will not be achieved for many years. There is a long time lag between the program implementation and the actual results.
4) The programs, especially in public health, often relate to deterrence or prevention of specific behaviors and not actually seeing a change or reduction in specific behaviors, which can be another measurement challenge.
5) Oftentimes programs have multiple purposes and funding can be used for a range of activities.
6) Looking across a variety of sites, you may see a range of things going on that are not necessarily a standard program being implemented in every site.
7) Sometimes the program purpose is administrative or process-oriented – one unique thing with many of our programs, a lot of times the leadership, management, and administration costs are a separate line item within the budget and might be looked at separately – how do you look at the outcomes for those types of things outside of the program context?

 In addition to performance measures, there is one other key outcome of a PART review, and that is the program improvement plan. That is one of the things that actually comes out of doing a review, so I wanted to share, just in terms of impact on implementation, two very brief stories about program improvement plans. These highlight what we’ve seen with some CDC programs and demonstrate some good use and application of the PART reviews. 

  • In immunization, one of the things that came out of the PART review, and going through the PART process, was an improvement plan to actually integrate and automate vaccine ordering and management by centralizing the distribution of all public-purchased vaccines, to try and get at better efficiency and delivery of vaccines to the public.
  • Within the Agency for Toxic Substances and Disease Registry, one of the things that came out of doing the PART review and assessment was a renewed commitment to evaluating the interventions by documenting effectiveness at each hazardous site to determine its particular and specific impact on public health.

There has been a lot of discussion and a lot of concern about budget implications, and how PART might be used as an “axe” or a tool to put programs on the chopping block. I can only speak to what I’ve seen and experienced at CDC – I know there are a wide range of other stories out there about programs, but I’ll share with you some of my observations at CDC. 

OMB does consider PART in developing the President’s budget. So looking at the PART review and the PART scores is one of the tools that OMB uses in the preparation of the President’s budget. However, the scores are generally not considered and do not come up in Congressional review of the budget. So, things that may go into OMB’s preparation of the President’s budget could get reinstated during Congressional review without regard for what the PART score or the PART review was. That is just one example, but generally we haven’t seen a lot of use or value for the PART scores in the Congressional review process. 

In looking at just a snapshot three-year period – and I didn’t do this statistically, I just eyeballed it – but looking at a three-year period of PART reviews and funding cycles for programs, I really can see no clear association, positive or negative, between PART scores and funding changes in programs. There were a few that went up, a few that went down, none of them significantly or substantially. I did mention earlier that CDC hasn’t had any programs that have been demonstrated ineffective, so the way our agency has done and our scoring may have contributed to that, but there has been no clear association that I could find. That said, as budget constraints continue to increase, and funds become tighter and tighter, I don’t think we can say that there isn’t great potential for PART to be increasingly used in budget decision-making. 

In thinking about implications of the PART process, and the PART review, there was a little aside in an on-line forum that I found and just couldn’t help but share. It was conducted in 2003 with Clay Johnson, then the deputy director for management at OMB, and the question was, “What is the penalty for an agency if they get a low score?” And his response was “Public shame and humiliation, and the opportunity to be questioned about it by the President.” Which I think is summed up nicely in a cartoon of the poor lone government official defending his program, where he is surrounded by cameras and microphones saying “Accountability (stinks)”.

So shifting towards the close of my presentation, here are some observations and considerations:

  • One of the key aspects of any PART review is the OMB examiner. Their investment in the process and learning about the program is critical to what comes out of the review. Just as in evaluation, the investment and involvement of the program evaluator, in working with the program, is critical to what comes out of that evaluation or that assessment.
  • Consistent interpretation of the guidance is critical. PART is designed to be an objectively applied tool and process, but there is surely subjectivity – whatever the adage is, “interpretation is 9/10ths of the law”. There is some variation in interpretation, and that is one of the things that the OMB guidance is trying to address, that variation in interpretation of the rules.
  • Finally, with respect to the examiner, knowing what is acceptable is important – having clear communication with the examiner of what they really want, what they are trying to get at, and trying to understand what they need, so that they can answer the question and score the question appropriately.

On the other side, involvement and investment of program leadership in the process is critical. PART can be an extremely resource-intensive process to go through, so having leadership engaged, involved and invested in the process – to not only conduct the review, but actually do some follow-up after the review – is of critical importance. So the leadership is very important to the process and what comes out of the review. 

In general, I think that PART overall has been a continuing boon to evaluation in Federal government, in that it clearly states and demonstrates that programs should be evaluated. That’s not a bad thing from a program evaluator’s perspective. Because of the questions, and the way that program evaluation is displayed in PART, it has elevated the importance and attention to evaluation. Now sometimes, not all attention is good attention, or you could argue that any attention is good attention – but the importance, the commitment to evaluation within the agency, I think has continually increased, and PART is one thing that has contributed to that, especially in recent years. 

During a review, there also have been dedicated resources for evaluation. When people are mobilizing for these reviews, there are often people who are pulled off of their regular jobs and dedicated to nothing but the PART review process; usually for at least six months, sometimes for much longer than a six-month period. There are workgroups that are called together, and program evaluators are pulled in and made part of that process. So there are at least, during that time, some dedicated resources for evaluation.

In general, if you look through the tools and the framework, the approach, intent and questions of the PART review, I really think that they are helpful and informative. From an evaluator’s perspective, and in applying an evaluative process to programs, it has merit and some good things that can be applied. And then, in terms of implementing it within the agency, it provides the opportunity to engage multiple internal stakeholders. When doing a PART review, there is increased attention to the program, there is a substantial opportunity for dialog among program planners, decision makers, evaluators, all coming to the table talking about issues that even in their day-to-day work they don’t normally take the time to think about or talk about.

On the other side, some of the challenges with respect to the PART review process are as follows:

  • PART is really designed, in some respects, as a one size fits all approach. There are slight adaptations that can be made across programs, as I mentioned earlier, but it really is one tool to fit the breadth of federal programs, and there is great variation in program size and scope.
  • The resources invested in evaluation and reviewing programs is mobilized during the review, but afterwards, I think there are questions and concerns about the sustainability of those resources. So when the dust settles, and everyone goes back to their regular jobs, you might forget about all of those great things you learned while going through the process and reviewing the program.
  • In thinking about the website, there are issues surrounding the reporting of findings. Even though you can drill down, a lot of people probably don’t drill down, so you just get the primarily quantitative score for the summative report, which is probably one page or less – a brief summary of the program. From a decision-maker’s standpoint or a legislator’s standpoint, there is certainly concern about understanding that there is a lot more behind that evaluation and what you need to know about that program in order to make decisions about it.
  • There has been some historical debate on evaluation evidence and realistic outcomes. While I am not going to touch on this in detail, there has been a lot of work over recent years to increase the breadth of information in evaluations that will be accepted as evidence for program evaluation, not just basing things on a medical research model paradigm, and only accepting a higher gold standard of evaluation.
  • And then finally, with respect to challenges, there is often an approach, especially from program planners or decision-makers, that performance measurement = evaluation = program improvement – if you do any one of those things, or address any one of them, you are addressing all of them. And that is obviously not the case. Performance measurement is an important tool, but that does not absolve you from doing program evaluation, and either of those does not necessarily relate to anything being used and actually seeing improvement in the program.

So in closing, I think one of my critical issues is that PART has a lot of good aspects and good components. Ensuring that there is a continued organizational commitment to evaluation, and actually institutionalization of those evaluative processes that PART brings to programs in Federal agencies, is where we have the next challenge, and where we can move forward with things. With the PART review, a lot of people focus on the destination of getting to the score, and getting the review done, but there is equal value, I think, in the journey. We need to think about how to use our learning from the journey in our continued work, and as we move forward. And then there are a lot of questions with respect to the future of PART. I don’t have the answers, any more than anyone else at this table does, but that is something for consideration - is this is going to be around in the next administration and the coming years, or is it not?

I will close by acknowledging and thanking some colleagues at CDC who helped prepare for this, including Julie Zajac, Marcia Taylor, and Thomas Chapel, as well as mention that there is a website available if you want more information on PART:

Thank you.

Panel comments: Ted Kniker