READING

Panel Comments: Nancy Kingsbury

“I’ve been around a long time, folks – something PART-like may survive in the next administration, but it won’t be what we have today. That’s the nature of the political process, and at the heart of this is a political process.” -Nancy Kingsbury

Click here to hear Dr. Kingsbury discuss her assessment of the future of the PART process - an excerpt from the full session audio.

I had some trepidation in following Ted Kniker – he has done presentations for my classes, so I know what he is up to, and it is going to be a hard slog to be that much more entertaining. 

My role in this panel, I think, is to step back and put some context around some of these issues. PART has, in some respects, come further than I would have expected it to when it first started, and I think it has some good things to say for itself. But we start at GAO from where our current leader, the Controller General of the United States, likes to start every single talk, which is that the United States government is on a fiscal path that is fundamentally unsustainable into the future, and we need to develop tools and techniques, and most importantly the political consensus to make changes in government programs so that we can avoid leaving our children and grandchildren in dire straits. That is kind of where we start – and so philosophically, the idea of being concerned about performance, making decisions with information, doing evaluations, and all of that sort of thing, fundamentally has some real value. Whether or not this particular tool gets us there – and I personally think not – it is certainly something we can start from. 

Let me go back to something that has been referred to both in this session and the previous session, which is PART as an outgrowth of the Government Performance and Results Act. GAO was there in the development of GPRA. We were very supportive of it, and it had some real features that we thought were important. One was that it specifically called for evaluation to be done. It called for strategic planning and performance planning to be done, and in the context of strategic planning and performance planning, it explicitly called for the results of evaluations to be reported and used. So the framework has always been there. 

Now GPRA had its downsides too. Necessarily, it was agency-focused. Arguably, a lot of the decisions that have to be made cross agency lines, and PART certainly has some of that same characteristic. GPRA called for a government-wide strategic and performance plan, which has never happened. One would hope, if there were a government-wide plan, we could begin to get at some of the interaction effects among agencies, and so forth. The President, by the way, and OMB have argued through two administrations that the President’s budget is the government-wide strategic plan, but that is not very persuasive. 

I bring to your attention that GPRA was passed at a time when the administration was in Democratic hands, and at least part of the Congress that passed the law was in Republican hands. It is interesting to me that GPRA, to its credit, specifically called for the input from stakeholders, broadly defined, in the development of strategic and performance plans – stakeholders to explicitly include, among others, program participants, state and local governments who are affected by these programs, and the Congress. And we did some early reporting on how well that happened. In the programs where it happened well, I think you moved a little closer to getting a consensus about what performance measures should be used, and how to measure the results of programs. 

You will recall, however, that the Clinton administration’s principal initiative was something called “Reinventing Government”, which in many respects was not particularly based on evidence and performance. So GPRA erupted at kind of an interesting point in time. Along comes the Bush administration, who adopts concern about results and adopts a management agenda – a huge step forward, in our view – and then develops this PART. And Ted was right, PART was developed kind of in the back rooms of OMB in its first iteration, and there was an attempt on the part of the OMB management to get GAO to go on record saying that we thought this was the greatest thing since sliced bread. We refused, tactfully, to do that at the time, but we did provide some input about issues such as whether yes/no questions are really a good idea, and things of that sort. 

But PART’s unit of analysis is the program. Now one of the things I think is not as evident to the evaluation community as might be the case is that “program” is defined as the line item in the budget. Sometimes that has correlation with what you and I as program evaluators would consider to be a program, but many times it does not. One of my favorite examples in that regard is that all of the tactical aircraft programs of all four services in the military are considered a program, subject to PART. This makes no sense at all to me, and I never have figured out how they did that. 

If a program can be identified and is coherent, then you apply these questions, but even then, it is a two way dialogue. It is the agency and OMB, with OMB being the elephant in the room. So what happened in many programs’ experience is that the performance measures that eventually got settled on were not performance measures, which under GPRA might have included the views of the state and local governments and the program participants or Congress about how to measure performance. They were, rather, ones that the OMB budget examiner liked or wanted to see. OMB budget examiners are, by and large, not trained evaluators. So it was a very ad-hoc, subjective process, and I think has largely remained so. All of the things that have been said from the agency perspective about the value of some of this process – calling attention to performance measurement, calling attention to evaluation, modestly increasing the actual investment in evaluation, I think are all things to the good – but getting there, I think, has arguably been painful for a lot of people, and taken up a lot of management resources in the government. 

As the PART process got started, and as an outgrowth of some work we had done under GPRA, we began to re-recognize, because we had been through this some years before, that there was some evaluation going on in the Federal government. It was out there, but it was pretty decentralized and pretty hidden. This effort takes off from the fact that GPRA does require evaluation and so we did a series of engagement that looked at the question of what were agencies doing in the way of evaluation, and how was that helping to frame how the thought about performance measurement, and how they thought indeed, in due time, about the PART process. 

Just for our own information, we began to assemble more or less an inventory of where evaluation was being done, and from that we established something that has come to be called the Federal Evaluators Network. GAO has sort of had some nominal leadership here, but leadership has also come out of the agencies. And that network actually began meeting before the PART process started, and so by the time the PART process started and in particular when the debate erupted about methodology and independence, a group existed that could become a focus to discuss what was going on. In that discussion, both issues were equally important.

We had a forum to begin to debate these things among the evaluator community inside the Federal government, and I think that forum has actually proved to be quite useful in helping to mediate some of the discussion about those issues. For example, OMB put up on its website and started telling agencies that no evaluation was worth anything unless it was done by GAO or the IG’s. That was just nonsense. And we worked overtime, through the Federal Evaluators Network, to develop a thought process that says, “No – excuse me – the value of an evaluation is whether or not the researchable questions and the methods you are using are suitable to the answers you are looking to get, and that just because it’s funded by the agency, so, you still can independently judge the methodologies. That was a debate which we were able to begin to have.

Regarding the other question about the only acceptable methodology being the randomized controlled trial, I think at least inside the government, the Federal Evaluators Network provided a vehicle to begin to push back at OMB on the issue. Eventually, the group came together and developed a briefing – I was going to call it training, but one doesn’t want to say that word in front of OMB examiners, they take offense – so we developed some briefings, which we have now done two series of over the last couple of years, to get OMB examiners to better understand what evaluation is, what evaluation methods are appropriate under which circumstances for what programs, and in what phase of development those programs may be. Whether we have actually influenced the decisions that have been made, I’m not sure, but it has gotten sufficient acceptance that some of this material is now actually found not only on the Federal Evaluators website, which is housed by the Environmental Protection Agency, but also on OMB’s website itself. So I think we’ve made some progress there. 

But if I could return for a moment to the idea that we really need a process sort of like this, we need it to be much more broadly grounded. It needs to include the Congress. When we went up for one of our engagements to talk with people in the Congress about what they thought of PART, a substantial number had never heard of it, and the ones that had didn’t like it very much, because they were never consulted. And it is a reality in our political world that there is a fundamental constitutional issue of the need to involve the Congress in these kinds of decisions that this administration, for whatever reason, has chosen not to do – about a whole lot of things, by the way, and not just PART. Mind you, all this debate about PART and Congress not liking it took place in a situation where the administration was Republican and so was the Congress. This administration has chosen to exert its executive authority in a way that reduces Congress’ authority, or attempts to, and Congress quite frankly, Republican or Democrat, doesn’t like that very much.

So if we are going to come back to this fiscal sustainability issue, if we are going to create an environment in which the government can develop processes and thoughts to get us to the hard decisions that we are going to need to make, in a way that we develop the political consensus they need to make them – we need a new process. Evaluators, I think, are going to have a very important role in that process. And I am sort of giving some thought to how we engage together to try to figure out how best to move this country down the path of making the kinds of decisions it needs to make. 

A couple of years ago, I was fortunate enough to be invited to a conference in Spain, sponsored by the Spanish Association of Parliamentary Lawyers, the theme of which was evaluating the impact of laws. And when I was invited, I thought, “They are talking about program evaluation”. Uh-uh. They were talking about developing methodologies in such a way that when a program or law is proposed, evidence is brought to the table that it will work, before it is passed and before it is funded. Interesting thought – we do it a little bit in this country, in the regulatory arena, doing cost-benefit analyses and the like, but we never go back and look at whether or not those were correct. 

So I think we need to open up the discussion beyond PART. I’ve been around a long time, folks – something PART-like may survive in the next administration, but it won’t be what we have today. That’s the nature of the political process, and at the heart of this is a political process. Michael mentioned earlier that he had “eyeballed” the programs in CDC to see if there was any relationship between funding decisions and the results of PART, and didn’t see any. In the context of one of the engagements we did, we actually did an econometric analysis of that, and there is basically no correlation between the results of PART and funding decisions. But that’s OK, because funding decisions, at their heart, are political decisions. And the idea of hiding behind a process like PART and trying to pretend it is objective, fact-based, etcetera, is a little misleading to what really needs to be done. And on that note, I think I’ll conclude.

Questions and answers