READING

Panel Comments: Ted Kniker

“PART is not about compliance, it’s about alliance. It’s about figuring out how to get yourselves aligned with OMB, in order to ensure that they understand what you’re doing and can be an advocate in the budget negotiations for you.” -Ted Kniker

Click here to hear Ted Kniker discuss the results of his experiences with the PART process, as an evaluator and consultant - an excerpt from the full session audio.

Good afternoon. It is quite an honor for me to be here for the first AEA Public Issues Forum, and I am humbled to share the dais with these three distinguished people. I feel like a rookie who has been called up to the majors. But I appear before you today as a survivor of the PART, having been personally responsible for three separate assessments, advising on numerous others, and now serving as a consultant to other Federal agencies in how to improve their performance and build evaluation capacity. And, of course, also being a “Fed”, I must have those disclaimers – so the views expressed in this presentation are my own, and do not represent the official views of the Department of State, the Department of the Treasury, the American Evaluation Association, or CBS and its affiliates (laughter). 

Now, we’ve all heard the phrase that where you stand depends on where you sit, and I think that this is certainly true from my perspective. As a program manager, and the chief for evaluation in public diplomacy for the Department of State, I was first very excited about PART. But after the first year, I thought that PART had a different meaning. I thought it stood for “Pernicious Annual Required Torture”. (laughter) I think it’s important to note here that PART is a laborious process, both for the agencies and the OMB examiners. I think we tend to forget that OMB examiners who work on PART are generally working on three to six others at the same time – so, they are not always pleased with this process either.

However, as the Bureau of Educational and Cultural Affairs gained success in the PART, and now as I’ve transitioned into my role as a consultant, I think I’ve kind of changed my view on PART to some degree, and now, it has a new meaning for me, which is “Program Analyst’s Retirement Trust”. (laughter) But in all seriousness, though, I have always considered myself a supporter of the PART, because while I’ve been an advocate of the PART, I also acknowledge that it is not perfect, and it does need to be improved. But where I’ve been a supporter of it is as a management tool -- a management checklist -- not necessarily an evaluation tool. 

The first thing that anyone should know who lives outside of the Beltway is that PART has become so ingrained in the Federal culture it is now a verb. A little history that you may not get from the general readings that you do or from OMB is that in the fiscal year 2002 budget, the President’s budget, there were programs that were given ratings. The program that I was evaluating at the time was given a “moderately effective”. Now this came before anyone had ever heard of PART, and it just showed up in the budget, and so there was a lot of uproar from several agencies about “how did you make this assessment?”, because it wasn’t transparent at all to the agencies, and OMB came back and said, “Well, we kind of did this internal thing, and we are thinking about maybe doing some kind of assessment in the future, and we just piloted this, but we didn’t tell anyone about it.” So the next year, in FY 2003, there were some pilots, and I helped to work on one of those with the Department of State on the Freedom Support Act, which was technical assistance provided to the former Soviet Union for democratic and market reform. 

Now, I’m going to talk to you mostly from my experience at the Department of State, which I only left about six months ago. And for those of you who are unfamiliar with the U.S. Department of State, its mission is to create a more secure, democratic, and prosperous world for the benefit of the American people and international community. And the Bureau of Educational and Cultural Affairs’ mission is to improve and strengthen the international relations of the United States by promoting mutual understanding between the people of the United States and other countries – a very observable and measurable thing to deal with.

The Bureau, for those of you who are unfamiliar, exchanges about 35,000 people per year through 90-plus programs. Some of these programs you may be familiar with, such as the Fulbright Scholarship program, the International Visitor Leader program. The programs range from high school to senior government officials, business leaders, and administrators at universities, and they are administered by over 100 non-profit partner organizations and 200 colleges per year. And it’s done with a budget of approximately 355 million dollars, of which the amount that went to program evaluation was .43 per cent, or about 1.5 million dollars. Now, over time, that went up a little bit, but our budget also went up a lot, so the actual percentages remained constant.

Our PART story

When we started, and when we were told that PART was coming, we were really excited about it. And the reason that we were was that we were the only bureau in the entire Department of State that a staffed and resourced internal evaluation unit. So we thought we had this thing nailed. We said, “we’ve got the evidence of effectiveness, we have everything, there can be no problems.” And when we got our first report back, our first score was a 61% “results not demonstrated”, which was kind of a shock to us. But the reason that OMB gave to us for why we had “results not demonstrated” was because we had performance measures that were not linked to long-term goals, that did not have baselines or ambitious targets, and we had an unclear strategic planning process.

So in the next year, fiscal year 2005, we made some revisions. We refine our measures, we hired a performance measurement expert to come in and really work with us to re-align our measures with our strategic plan, to help us actually draft measures that read well. We linked PART to planning, and this was also something that was being done in the Department at the time. There is a report that has come out recently by John Gilmour from the College of William and Mary that talks about integrating budget and performance, and one of the examples he uses is the Department of State, and how we had a centralized resource management office that took PART very seriously and very aggressively, and worked to ensure consistency within the organization and to build PART within our strategic planning process.

In the bureau we also implemented frequent measurement. Now one of the things that is important to point out here is while it looks like there is a year between all of these, the actual time we had between assessments and re-assessments is actually about six months. The process that OMB goes through works so that you know your first inkling of your score sometime in the late summer. Generally by October you have a good idea of what it is, and then in March again, they are in there re-assessing. So you have from basically October to March before you really know what it is they found wrong with you, and it’s written down to a point that you can take action on it.

Well, then in fiscal year 2006, as we had done all these things and continued to refine what we were doing, we ended up with a 98% effective for that first PART, and then we also had a second PART that was launched, and we came out of the gates with a 97% effective rating.

Figure 1. Evolution of PART assessment ratings for the Bureau of Educational and Cultural Affairs in the Department of State from fiscal years 2004 through 2006.

The results of PART

Now, it’s important to remember, I think, that PART is as much a process and a relationship as it is a tool, and a significant element of that process and that relationship is communication. And I always used to say that PART is not about compliance, it’s about alliance. It’s about figuring out how to get yourselves aligned with OMB, in order to ensure that they understand what you’re doing and can be an advocate in the budget negotiations for you. Better communication with OMB was probably our first result. Prior to PART, the only folks in our bureau who talked to OMB were our budget people. After that, our leadership and our evaluation people had regular contact with OMB. I probably spoke with our examiner at least once every couple of weeks, and we were constantly feeding information about evaluation and what we were doing.

We also had internal cultural change, and we moved from kind of your normal government processing – “move money out the door, make sure you spend all of your money before the end of the fiscal year” – to really a focus on results, and what it is we are trying to achieve. And I think that more than GPRA or the President’s Management Agenda, PART prompted real discussions in our senior leadership meetings and among staff about what those results were – and what they should be.

We also improved planning and focus of our programs. The bureau’s primary business line is grants, and you have ever read RFPs, basically from the 1990s all the way through to a few years ago, most of those RFPs for grants would focus on the activity that they wanted the recipient to work on – “you will implement an orientation program, you will do this, you will do that”. And it wasn’t until a couple of years ago that we started talking in those RFPs about what it is that we are trying to achieve, and what it is we want our organizations to work towards with us. So we think we had better grants after we created a stronger culture of measurement.

We also re-directed resources towards performance measurement and away from evaluation, and this is something that always bothered me. But what happened was performance measurement became so important in the PART that that collection and analyzing of data, we had to put resources into it, and because resources weren’t necessarily going up in a way that we would have liked, we had to take what we had and to re-allocate it. We doubled the staff in our evaluation unit. We originally had four people working on evaluation, and it became basically nine people. We worked back to three people doing program evaluation, and the rest working on systematic program performance measurement. 

We also increased funding for programs in evaluation. Now, that may not jibe with what I just said, but eventually over time, because we had done so well on PART, it was something that we used in our budget negotiations, and it was something that was held up within the department as, “here’s the example of a bureau that is doing it right, doing it well, and they should be rewarded”. Now, I won’t say that there is a causal link there, but it was certainly correlational. There were some other things that were happening in the environment that allowed exchange programs to gain more and a bigger share of the budget, but still there was the evidence that those kinds of programs worked.

We integrated our planning and evaluation better – and again, this is what I was talking about earlier – that in our Congressional budget submissions, we had all of our PART information in there, we had all of our measures. We spent a lot of time talking about PART in Congressional budget submissions.

We also had more respect and recognition. I think that one of the important things PART has done is make government programs transparent. It has helped to really shine a light on what we’re doing with your money. We were held up as a model, and that was really great for us. We were benchmarked by a number of different agencies and firms, and we had consultants coming in and talking to us about how we did our work, and of course then what success leads to is more work. We ended up being asked to do a lot more, as our audience and our constituents became more sophisticated, because we had really outreached to them to educate them about what we were doing. They started asking more sophisticated questions, which led to, “Oh boy, now we’ve got to start measuring this even more rigorously”.

Challenges with PART

Of course, success doesn’t come without a price. If there is no pain, there is no gain. So we had some challenges. And one thing I’ll state here is that PART is not coordinated or accepted by Congress, plain and simple. We had a particular incident when I was called to the Hill to brief a staffer on our appropriations staff about why public diplomacy in general didn’t have measures, and when I said to her that, “What we can do is show you our PART measures, and we’ll report those on a quarterly basis like we do to OMB”, the response was, “The PART is OMB’s. It is not Congress’s. I am not interested in looking at what you submit to OMB.” When we pressed a little bit further about what information they wanted, that was different from what we were submitting, we didn’t really get a response – we basically got, “Well, you figure it out.” And then when we kind of thought we had figured it out, and started sending it around, what it ended up turning out to was that it had nothing to do with results. It was, “We just want to know what you spent the money on, and what you got out of it in terms of output” – it really simply boiled down to output.

The Paperwork Reduction Act requirements, versus reporting results requirements, one of the interesting things we encountered was, as the PART kept moving, all of a sudden you noticed that you were reporting more frequently. It went from annual to quarterly. And so we had all of these quarterly management reports we had to submit, and that was coming from the budget side of OMB – yet, on the regulation side of OMB, there is the Paperwork Reduction Act, which requires that a survey of ten or more individuals conducted on behalf of or sponsored by a Federal agency must have an OMB approval. The OMB approval process takes, in our estimation and experience, six to eight months. And then on top of it, they weren’t granting generic clearances. So we had no way of getting clearances for our surveys in a timely way, in order to collect the data to give to the other side of OMB.

We saw that in the improvement plans, they went from constructive to idiosyncratic. One of the benefits is that as your OMB examiner becomes more familiar with your program over time, they in my experience typically become more of a supporter of it. But we have had three OMB examiners in four years, and so that became kind of an iffy process. But we noticed that if they stayed around too long, what would start happening is that they would tell you that how to improve was some wild idea. So we already had this 97 and 98 per cent effective ratings on our two PARTs, and all of a sudden we got lumped in to a couple of other PARTs that were being done for the Department, and then all of those together were collectively told, “You won’t succeed unless you have an over-arching strategic plan” for that particular part of the Department.

As for the role of evaluation, it was very clear to us that it was more important to focus on performance indicators than on evaluation. That first year, when we had the 61 per cent, we actually had 20 separate, independent evaluations that demonstrated effectiveness across a range of our programs that we handed in as part of our evidence, and it didn’t really count. We ended up getting, I think, a small extent rating for that. So it was very clear that either there wasn’t an understanding of evaluation, or there wasn’t an appreciation for the results that were demonstrated in an evaluation. It was all about the measures, and the indicators.

And then there were some issues with the definitions of independence and quality. We’ve definitely seen this across the board – in some cases independence is only seen as possibly a GAO or an OIG audit, because those are not funded. We had problems with one of our examiners that, even though we were an internal unit that was separate from program management, and actually still funded external evaluations, that because we funded them, that was a problem.

There was also, we found, little room for qualitative data or citizen satisfaction. One of the things that I’ve noticed is that in the Clinton-Gore team, citizen satisfaction was really the main thing that the administration was promoting – that it was important that citizens be satisfied with government services, and what government was doing. And that has kind of moved to more of what I view as, if you liken it to business, a move from customer focus to shareholder dividends. The focus is improving the dividend, as opposed to really whether the beneficiaries are supported and happy with what you are doing.

Also, OMB also – and I’m glad that we don’t have a representative here, so now I can say this – is shifting boundaries and the underlying philosophy of PART. PART has a philosophy that no program is perfect, and that you have to show improvement. And we had an issue in which a lot of our measurements showed that we were about as high as we were going to get with things. One of my bosses called it “the Lance Armstrong effect” – if you are Lance Armstrong, and you’ve won the Tour de France six or seven times, what do you do next to improve? You could win it in a few more minutes, or you could maybe win it one more time, but at that point who cares? It really doesn’t matter. And so there is not a focus from OMB on maintenance of excellence. We also found that there were requirements added after the fact, and I mentioned a couple of those.

Observations

I just wanted to make some observations that I’ve seen from other agencies, to wrap up really quickly:

  • PART is seen as compliance, instead of a way to communicate. Agencies see PART as increasing their bureaucracy or taking away from their ability to program.They struggle with output versus outcome.
  • They are confused about the role, definition, and importance of evaluation.
  • Agencies don’t want to be held accountable for societal impact, because they feel that if they are held accountable for it through the PART, they will always lose.
  • The PART rating, as was mentioned earlier, generally is dependent upon an individual examiner, although there are some processes in place at OMB to change that.

My assessment of PART

Here is my take on a quote from Shakespeare: “They say he PARTed well , and paid his score;  And so, God be with him!” I used this quote with my examiner to say, “If Shakespeare said it, and we did it, we should get something good,” and then she reminded me that this quote comes from Macbeth, which is about death and destruction. (laughter)

So, does PART improve program performance? My assessment is that results are not demonstrated. It does, however, improve goal articulation, planning, and stakeholder dialog, as well as elements of program management. It is kind of mixed as to whether it actually improves performance – one of the things you’ll see is there is no consistency on what is accepted as the measure. Some agencies are given certain measures and are told, “these are great”, while another agency will use a very similar measure and are told, “no, that is not acceptable”.

There are also issues where it does generate attention to performance measurement. Evaluation certainly increased internally, and our reputation as well as respect for what we did in the Department of State certainly increased. And there certainly now seems to be evidence that evaluation is important.

And does PART need improvement to foster the improvement that it claims to achieve? I think we need to change from a “yes/no” format to a continuum – that if they allowed a continuum of “this is where an agency is”, it would be much easier to build agencies towards improvement by providing agencies with a “ladder” of success to follow. 

PART does not include questions for assessing leadership or workforce development, two things that I think are critical for an effective organization. And it doesn’t currently allow for the contribution of programs – there still is a sense that it has to be a causation.

As for the future, I think that PART, or something like it, will continue to exist. We have gotten too far into results measurement to move away from it – in fact, over the last couple of years, the Malcolm Baldridge Quality Awards have now opened up to non-profits and the government. And ExpectMore.gov is out there now – people are reading it, people see it. And so I think there will be a continued need for these kinds of things.

Thank you.

Panel comments: Nancy Kingsbury