READING

Questions And Answers

The closing section of this forum involved open questions and dialogue between members of the audience and panel participants. The following is a transcript of this Q&A session.

Q: Thank you all, this is very educational. I’m really interested in the fact that OMB examiners have so much power, and the fact that there is an opportunity to educate them through the interactions that you have as a Federal bureaucrat. But I’d also like to know if there is any recognition within OMB that it would be valuable for their examiners to have some understanding of evaluation if they are going to evaluate evaluation, as well as evaluate agencies. 

Nancy Kingsbury: I would suggest that there has been some recognition of that need, in their willingness to accept the offer that the Federal Evaluators Network made to provide some fundamental information across examiners in different areas, to try to improve the consistency with which they come to the table to talk about evaluation. Again, the OMB budget process is intended to be a reflection of the President’s priorities and the President’s process, and so there is probably a relatively small extent to which you could influence it and move it much beyond where we’ve got it.

Q: I was intrigued by Nancy’s comments towards the end, where you seemed to be saying that PART wouldn’t survive like it is today because of the politics. You also seem to call for a new process, and to suggest that evaluators could play a role in that new process. Would you care to elaborate a little bit on what that might be? Are you suggesting that evaluators at this point might be thinking about the change in politics that will inevitably be coming down the road and anticipating that, and do you have any thoughts about what directions we might want to be taking on that?

Nancy Kingsbury: When I had the idea a few hours ago to put that thought on the table, I had a feeling someone would be asking me what I meant by it, and that is a fair enough question. I keep coming back to what my boss is trying to do, and for those of you who haven’t seen it, and haven’t had an opportunity to do so, he has engaged in a partnership with a wide spectrum of outside organizations, starting with the Concord Coalition, Brookings, Heritage, across the political spectrum, in what we internally refer to as “the fiscal wake-up tour”. And what he is trying to do is convince the American public that this problem of fiscal unsustainability is important and has to be resolved, but that it will take political consensus (with a small “p”) to do that. He has got, I think, a good theme basically that until you get the public at large to understand what the problem is, the politicians themselves won’t solve it.

So I think, in terms of what role could evaluators or anybody else play in terms of trying to struggle with, suppose that we succeed and suppose there is a groundswell of the need to make some of these decisions. Some of them, like fixing the entitlement programs, are probably beyond even evaluators’ skills, at least in the aggregate. But on a program by program basis, to look at programs about whether they are still needed now, and whether you could get constituencies and other parties like, say, local governments in the model that GPRA originally proposed, to begin to grapple with the need to make some of these hard decisions, it’s just a thought that I decided to put on the table, and start thinking about myself, because I think if something is going to happen, this is the community that maybe has an edge on the kinds of thought processes and methodologies that might be useful in doing that. That’s about as far as I’ve thought about it.

Q: I’m new to the field of evaluation, one year and counting, with a background in research, and I’m also new to AEA. I had a global question first, and then a more specific question. First, where would you like this to go? I mean, if this is a public issues forum on the PART, what do you hope to accomplish by this forum? And also, as an evaluator of one of the programs that I saw listed on one of the slides, a GEARUP (Gaining Early Awareness and Readiness for Undergraduate Programs) program in Ohio, I’m wondering what the implications of this might be for an evaluator of such a program. How might I look towards the PART as a tool for examining a program that may have already been evaluated by an OMB person, or a program that is yet to be evaluated? 

Bill Trochim: Both of these are good questions. The first one, on what we hope to get from this forum, I’ll take a crack at. As far as this forum is concerned, what led to this forum was an exchange that occurred on EvalTalk, shortly after ExpectMore.gov went live. I think it was readily apparent to anyone who was involved in EvalTalk at that point that ExpectMore.gov, which we showed a few slides of, triggered a variety of responses, some of them quite negative, about whether this PART effort and this website represented evaluation well, whether as an evaluation organization, we ought to respond to it in some way, and whether we had a responsibility to take some position with respect to this kind of an endeavor. 

At that point I was chair of the Public Affairs Committee, and we had a number of conversations where we felt we did need to respond to these kinds of situations. But it wasn’t clear that we could go through the process of engaging the membership of AEA and try to formulate some policy with respect to PART, or that this was even the right time to do that. Instead, we thought that this would be a good time for us to learn about this, and to identify what seems to be going on here from an evaluator’s perspective, and we thought that this kind of a forum would be a way for us to start this process. So this really was, and is, an experiment in mechanisms in learning for us – can we respond to situations that arise during the course of a year, identify ones that are particularly important for evaluators, focus on them in this kind of a forum, and then post the results of this so that other evaluators can also benefit from that?

Also, hopefully, I fully expect that some of the people at OMB will take a look at this forum, and review what we said here. I think Nancy already described an example of an interaction between the Federal Evaluators Network and OMB that had an impact on how they do some of the details in grading these programs and in terms of evaluation. And so I’m hoping that this will have an impact for the membership in terms of our learning about these things and engaging on these issues, and quite possibly we’ll have some impact as a form of feedback to those policymakers who might read about this. So that’s the intent of this. And I guess the other question that I heard you ask as well has to do with implications for evaluators who work with programs that have been through PART reviews, and I wonder if any of you would like to comment on that?

Michael Schooley: I think there are two things in the response. One, as a local evaluator and looking up at the list, and saying, “Hey, I’m doing local work and local evaluation with a program that might be an example of one of those competitive grant programs, a block grant or other grant type program. Looking at the PART evaluation, I think, would certainly be interesting and informative to you as a local evaluator, but is probably not going to tell you a whole lot about what you do on a day-to-day basis. It came up a little bit in a previous session on Federal assessments that PART reviews are done at a program level. What is happening locally, especially for many competitive grant programs or block grant programs, are projects that are implemented locally and individually. So looking at the evaluation of programs will be of interest to you, and it will have content for what you do in terms of the broad spectrum of your role, but it’s not going to do a lot for the projects that you might be evaluating on a day-to-day basis.

One the other hand, looking at the PART tool I think is of use and value to local evaluators. I think the process and what they have pulled together is interesting and informative. I wouldn’t say that you would take it carte blanche and apply it to your local area, but there are certainly aspects of it that are of value. Some of the techniques might add to the pool of techniques you are using locally in the work that you are doing.

Nancy Kingsbury: I would agree with that. I think the fundamental question involves the kinds of questions that need to be thought about, and it might be enlightening just to look at those and say, down at this project level, do we understand what these things are about, and do we have evidence of whatever kind, from whatever source, that could go towards answering those questions rather than trying to say yes or no to the fundamental substance of the question itself?

Q: I’ve enjoyed this panel very much, and have followed a lot of the Federal initiatives or performance measures, and I think my initial question has pretty much been answered in terms of what’s the role of this Public Issues Forum to try to improve the PART process, which I happen to believe is likely to be a good thing if the right measures are chosen. But hearing what Nancy has said about how this is likely to not be continued in a new administration, and who knows what’s going to happen after next week’s election, I’m thinking what might be the role of AEA and/or evaluators individually who know about a particular content area in trying to improve this process in the future, so that good performance measures are selected and evidence is weighed appropriately. It seems to me these are two key evaluation-oriented problems. Do you have any thoughts are to how we might contribute in the future? 

Nancy Kingsbury: Let me say this. I think it will be interesting to see if anything comes out of the election that begins to mold where this is headed. A year ago at this conference, the debate around PART was almost entirely around the randomized control design that was out there, and Bill and I were talking before this session about whether that was going to be the topic of all the questions, and it hasn’t been. That may be because of the urgency with which evaluators were reacting to the same thing that the GAO reacted to, which is that this is not an appropriate way to go around designing an evaluation tool for many, many programs, and have begun to see through the process itself in some kind of helpful way.

So going forward and looking ahead, I think if we’re lucky, there will be a debate even in the campaign for President the next time around, that we may be able to feed into or help change. It’s really hard to see that yet, but as soon as the election ends next week the presidential campaign will start. So I don’t know the answer, really, but I do sense there is something of an opportunity.

Bill Trochim: I think that the little example that Nancy just gave us, of a group of federal evaluators bothering to pick up the phone and call OMB and arrange for a meeting and engage with them, actually had some pretty profound effects, given the cost of that effort. And as part of what I hope we take away from this is the idea that as members, as individuals, as evaluators, and perhaps even as an association, if we can find ways to begin to engage more directly in dialog around these things, we might be surprised to find the degree to which we might have some direct influence on the formation of legislation on the way in which these things are implemented, practiced and so on. At the very least, that ought to be part of what we are taking part in.

Q: I was really struck by a couple of comments. One was where Ted Kniker said couldn’t we look at contribution instead of the causation, and I was also struck by Nancy Kingsbury’s comments about trying to look at the interactions among Federal agencies for a wider strategic plan across Federal agencies. I think that one of the fictions that underlies the PART is sort of like the old concept of union jurisdiction – that there is one union to one industry. But in practice it’s not like that, and it isn’t even necessarily clear it’s a bad thing that it’s not like that. So I think that there needs to be a lot more attention to whether or not the interaction and interchanges between federal agencies are productive, useful, helpful, or whether it is simply redundancy. And the PART seems to basically say it’s redundancy.

Ted Kniker: One of the things about the PART as it is currently implemented, I think, still promotes stovepiping of organizations, although in certain sectors there has been guidance from OMB that we need to broaden who is in particular programs. For example, with public diplomacy, the promotion of the U.S. image abroad, led by Karen Hughes, that the bureaus who worked on that were brought together, and there is talk about doing a united PART for that. But then there was also talk about, because there are other agencies that influence the image of the U.S. abroad, that they should be brought in as well, including the Department of Homeland Security, Department of Defense, Department of Justice, Peace Corps, and a number of these others.

So there is at least the recognition that programs and impact goes beyond just a particular agency. But one of the things that we have seen, and maybe an area where AEA can assist with, is in the Department of State, for example. The evaluation across the Department of State is not coordinated at all. PART was, but if my bureau was doing something on exchange programs, and how well particular programs worked, we didn’t really involve any other programs that might contribute to that success. So I think that one of the things we need to look at is whether there are ways to begin to coordinate evaluation across agencies, if multiple agencies are involved with a particular program.

Bill Trochim: Well, I think our time is over, and with one final pun, I’d like to thank you all for taking “PART” in this first AEA Public Information Forum!

Return to PART