PANEL COMMENTS: MICHAEL SCHOOLEY
Click here to hear Michael Schooley discuss a summary of future challenges with respect to the use of PART in program assessment - an excerpt from the full session audio. Good afternoon. I want to start by recognizing and thanking Bill for his introduction and warm remarks, and also for organizing this panel. As he mentioned, there was a lot of discussion on EvalTalk earlier this year on two different occasions, so I think we all welcome the opportunity to engage in this fruitful discussion. I’m going to start by adding to a few of the comments Bill has made and reiterating a few of the points that he highlighted in his introductory remarks. I’m going to talk about some of the PART reviews that have been conducted at CDC, how they have been conducted and how the agency has done on these reviews, and then I’m going to share some of my own observations and considerations as an evaluator working on some of the reviews. If you’ve heard a few other federal presentations, you’ll not be surprised to hear me say that the views I present are my own and do not necessarily represent those of the agency. As Bill mentioned, PART was created in 2002 – fiscal year 2004 was when it was first applied, to programs at CDC and elsewhere. It was presented as a diagnostic tool to be used by OMB to assess program performance, and its intention is to improve program performance. The tool, as Bill introduced, is 25 to 30 questions, divided up into four broad sections or areas. One of the questions of particular importance for evaluation is, “Does the program have a limited number of specific, ambitious, long-term performance goals that focus on outcomes and meaningfully reflect the purpose of the program?”, “Does the program achieve its annual performance goals?”, and then “Do independent and quality evaluations of the program indicate that it is effective in achieving results?” The PART tool can be slightly adapted across a range of programs. There are seven broad areas of programs it can be applied to: ●
Competitive grant programs Most of the work that is done at CDC falls under the competitive grants program. At CDC 23 programs have been reviewed over the past five years, and we are in the final year of evaluating programs for fiscal year 2008. Fifteen out of these 23 programs are competitive grant programs, and the funding range is anywhere from $63M to $900M. I think this is important in illustrating that there is sufficient variation in the size and scope of programs that have been reviewed, even within CDC and the public health realm. Here are examples, in terms of content and topics, of some of the programs that have been reviewed at CDC. These include things like the child immunization program, and in comparison, at the end of the list, the global immunization program, as well as an entire agency, the Agency for Toxic Substances and Disease Registry. Other areas include state and local preparedness grants, domestic HIV/AIDS program, chronic disease prevention and health promotion programs, which represents a variety of programs addressing tobacco, cancer, heart disease, nutrition, and physical activity. Sample CDC programs that have had PART reviews ●
Childhood Immunization Program In terms of the reviews that have been conducted at CDC, most of the reviews have resulted in scores that are adequate to moderately effective. So overall, as an agency, we’ve actually done pretty well if you’re just looking at the final scores. One program was designated as effective, five programs were determined “results not demonstrated”, and we haven’t had any programs that have been determined to be ineffective. In general, if you look across those four broad sections, the lowest scores are on the results sections – that’s where I think a lot of challenges come in with programs in providing and demonstrating results and providing some of the evaluation information. Over the past five years, the scores have increased, overall by about 10 percent. This is actually interesting, and I think Bill made an allusion to this when he said, “Ted obviously knows how to do a good PART review” - in that the agency, in some respects, may have become a better test-taker, in knowing what to provide for the review and how to prepare for the review. The other issue is that over the course of the five years we have had three OMB examiners. We had one OMB examiner for the first couple of years of reviews, another one for the second couple of years, and just recently a third examiner for the most recent year. As OMB examiners change, there is some subjectivity to the review and the exam, and that may influence the review and the scores that we receive. And then finally, part of the PART review is looking at performance measures. These measures are generally outcome measures, mostly longer-term health outcome measures – not shorter-term measures, and not really a lot of output measures. Some samples of these measures at CDC are listed here. Sample performance measures at CDC ●
Number of countries in the world with endemic
wild polio virus There’s a document on the guidance for OMB website, which documents the workshop that occurred in 2003 looking at performance measurement issues in providing measures for PART. It addresses seven broad areas around performance measurement challenges. I think these are of particular interest to program evaluators, and obviously ring true to a lot of the work we do, especially public health. As programs have been preparing for these PART reviews and preparing performance measures, some of the things that come up are:
1) Program
outcomes are extremely difficult to measure. Some of the programs are looking at
broad and far-reaching things like policy change and issues, which are difficult
things to measure from a scientific perspective and from a specific discrete
measurement perspective. In addition to performance measures, there is one other key outcome of a PART review, and that is the program improvement plan. That is one of the things that actually comes out of doing a review, so I wanted to share, just in terms of impact on implementation, two very brief stories about program improvement plans. These highlight what we’ve seen with some CDC programs and demonstrate some good use and application of the PART reviews. ●
In immunization, one of the things that came out of the PART review, and going
through the PART process, was an improvement plan to actually integrate and
automate vaccine ordering and management by centralizing the distribution of all
public-purchased vaccines, to try and get at better efficiency and delivery of
vaccines to the public.
There has been a lot of discussion and a lot of concern about budget implications, and how PART might be used as an “axe” or a tool to put programs on the chopping block. I can only speak to what I’ve seen and experienced at CDC – I know there are a wide range of other stories out there about programs, but I’ll share with you some of my observations at CDC. OMB does consider PART in developing the President’s budget. So looking at the PART review and the PART scores is one of the tools that OMB uses in the preparation of the President’s budget. However, the scores are generally not considered and do not come up in Congressional review of the budget. So, things that may go into OMB’s preparation of the President’s budget could get reinstated during Congressional review without regard for what the PART score or the PART review was. That is just one example, but generally we haven’t seen a lot of use or value for the PART scores in the Congressional review process. In looking at just a snapshot three-year period – and I didn’t do this statistically, I just eyeballed it – but looking at a three-year period of PART reviews and funding cycles for programs, I really can see no clear association, positive or negative, between PART scores and funding changes in programs. There were a few that went up, a few that went down, none of them significantly or substantially. I did mention earlier that CDC hasn’t had any programs that have been demonstrated ineffective, so the way our agency has done and our scoring may have contributed to that, but there has been no clear association that I could find. That said, as budget constraints continue to increase, and funds become tighter and tighter, I don’t think we can say that there isn’t great potential for PART to be increasingly used in budget decision-making. In thinking about implications of the PART process, and the PART review, there was a little aside in an on-line forum that I found and just couldn’t help but share. It was conducted in 2003 with Clay Johnson, then the deputy director for management at OMB, and the question was, “What is the penalty for an agency if they get a low score?” And his response was “Public shame and humiliation, and the opportunity to be questioned about it by the President.” Which I think is summed up nicely in a cartoon of the poor lone government official defending his program, where he is surrounded by cameras and microphones saying “Accountability (stinks)”. So shifting towards the close of my presentation, here are some observations and considerations: ●
One of the key aspects of any PART review is the OMB examiner. Their investment
in the process and learning about the program is critical to what comes out of
the review. Just as in evaluation, the investment and involvement of the program
evaluator, in working with the program, is critical to what comes out of that
evaluation or that assessment. On the other side, involvement and investment of program leadership in the process is critical. PART can be an extremely resource-intensive process to go through, so having leadership engaged, involved and invested in the process – to not only conduct the review, but actually do some follow-up after the review – is of critical importance. So the leadership is very important to the process and what comes out of the review. In general, I think that PART overall has been a continuing boon to evaluation in Federal government, in that it clearly states and demonstrates that programs should be evaluated. That’s not a bad thing from a program evaluator’s perspective. Because of the questions, and the way that program evaluation is displayed in PART, it has elevated the importance and attention to evaluation. Now sometimes, not all attention is good attention, or you could argue that any attention is good attention – but the importance, the commitment to evaluation within the agency, I think has continually increased, and PART is one thing that has contributed to that, especially in recent years. During a review, there also have been dedicated resources for evaluation. When people are mobilizing for these reviews, there are often people who are pulled off of their regular jobs and dedicated to nothing but the PART review process; usually for at least six months, sometimes for much longer than a six-month period. There are workgroups that are called together, and program evaluators are pulled in and made part of that process. So there are at least, during that time, some dedicated resources for evaluation. In general, if you look through the tools and the framework, the approach, intent and questions of the PART review, I really think that they are helpful and informative. From an evaluator’s perspective, and in applying an evaluative process to programs, it has merit and some good things that can be applied. And then, in terms of implementing it within the agency, it provides the opportunity to engage multiple internal stakeholders. When doing a PART review, there is increased attention to the program, there is a substantial opportunity for dialog among program planners, decision makers, evaluators, all coming to the table talking about issues that even in their day-to-day work they don’t normally take the time to think about or talk about. On the other side, some of the challenges with respect to the PART review process are as follows: ●
PART is really designed, in some respects, as a one size fits all approach.
There are slight adaptations that can be made across programs, as I mentioned
earlier, but it really is one tool to fit the breadth of federal programs, and
there is great variation in program size and scope.
So in closing, I think one of my critical issues is that PART has a lot of good aspects and good components. Ensuring that there is a continued organizational commitment to evaluation, and actually institutionalization of those evaluative processes that PART brings to programs in Federal agencies, is where we have the next challenge, and where we can move forward with things. With the PART review, a lot of people focus on the destination of getting to the score, and getting the review done, but there is equal value, I think, in the journey. We need to think about how to use our learning from the journey in our continued work, and as we move forward. And then there are a lot of questions with respect to the future of PART. I don’t have the answers, any more than anyone else at this table does, but that is something for consideration - is this is going to be around in the next administration and the coming years, or is it not? I will close by acknowledging and thanking some colleagues at CDC who helped prepare for this, including Julie Zajac, Marcia Taylor, and Thomas Chapel, as well as mention that there is a website available if you want more information on PART: http://www.whitehouse.gov/omb/part/index.html. Thank you.
|