|
QUESTIONS AND ANSWERS
The closing
section of this forum involved open questions and dialogue between members of
the audience and panel participants. The following is a transcript of this Q&A
session.
Q: Thank you all, this is very educational. I’m really interested
in the fact that OMB examiners have so much power, and the fact that there is an
opportunity to educate them through the interactions that you have as a Federal
bureaucrat. But I’d also like to know if there is any recognition within OMB
that it would be valuable for their examiners to have some understanding of
evaluation if they are going to evaluate evaluation, as well as evaluate
agencies.
Nancy Kingsbury:
I would suggest that there has been some recognition of that
need, in their willingness to accept the offer that the Federal Evaluators
Network made to provide some fundamental information across examiners in
different areas, to try to improve the consistency with which they come to the
table to talk about evaluation. Again, the OMB budget process is intended to be
a reflection of the President’s priorities and the President’s process, and so
there is probably a relatively small extent to which you could influence it and
move it much beyond where we’ve got it.
Q: I was intrigued by Nancy’s comments towards the end, where you
seemed to be saying that PART wouldn’t survive like it is today because of the
politics. You also seem to call for a new process, and to suggest that
evaluators could play a role in that new process. Would you care to elaborate a
little bit on what that might be? Are you suggesting that evaluators at this
point might be thinking about the change in politics that will inevitably be
coming down the road and anticipating that, and do you have any thoughts about
what directions we might want to be taking on that?
Nancy Kingsbury: When I had the idea a few hours ago to put that thought on the
table, I had a feeling someone would be asking me what I meant by it, and that
is a fair enough question. I keep coming back to what my boss is trying to do,
and for those of you who haven’t seen it, and haven’t had an opportunity to do
so, he has engaged in a partnership with a wide spectrum of outside
organizations, starting with the Concord Coalition, Brookings, Heritage, across
the political spectrum, in what we internally refer to as “the fiscal wake-up
tour”. And what he is trying to do is convince the American public that this
problem of fiscal unsustainability is important and has to be resolved, but that
it will take political consensus (with a small “p”) to do that. He has got, I
think, a good theme basically that until you get the public at large to
understand what the problem is, the politicians themselves won’t solve it.
So I think, in
terms of what role could evaluators or anybody else play in terms of trying to
struggle with, suppose that we succeed and suppose there is a groundswell of the
need to make some of these decisions. Some of them, like fixing the entitlement
programs, are probably beyond even evaluators’ skills, at least in the
aggregate. But on a program by program basis, to look at programs about whether
they are still needed now, and whether you could get constituencies and other
parties like, say, local governments in the model that GPRA originally proposed,
to begin to grapple with the need to make some of these hard decisions, it’s
just a thought that I decided to put on the table, and start thinking about
myself, because I think if something is going to happen, this is the community
that maybe has an edge on the kinds of thought processes and methodologies that
might be useful in doing that. That’s about as far as I’ve thought about it.
Q: I’m new to the field of evaluation, one year and counting,
with a background in research, and I’m also new to AEA. I had a global question
first, and then a more specific question. First, where would you like this to
go? I mean, if this is a public issues forum on the PART, what do you hope to
accomplish by this forum? And also, as an evaluator of one of the programs that
I saw listed on one of the slides, a GEARUP (Gaining Early Awareness and
Readiness for Undergraduate Programs) program in Ohio, I’m wondering what the implications of this
might be for an evaluator of such a program. How might I look towards the PART
as a tool for examining a program that may have already been evaluated by an OMB
person, or a program that is yet to be evaluated?
Bill Trochim: Both of these are good questions. The first one, on what we hope to get
from this forum, I’ll take a crack at. As far as this forum is concerned, what
led to this forum was an exchange that occurred on EvalTalk, shortly after
ExpectMore.gov went live. I think it was readily apparent to anyone who was
involved in EvalTalk at that point that ExpectMore.gov, which we showed a few
slides of, triggered a variety of responses, some of them quite negative, about
whether this PART effort and this website represented evaluation well, whether
as an evaluation organization, we ought to respond to it in some way, and
whether we had a responsibility to take some position with respect to this kind
of an endeavor.
At that point
I was chair of the Public Affairs Committee, and we had a number of
conversations where we felt we did need to respond to these kinds of situations.
But it wasn’t clear that we could go through the process of engaging the
membership of AEA and try to formulate some policy with respect to PART, or that
this was even the right time to do that. Instead, we thought that this would be
a good time for us to learn about this, and to identify what seems to be going
on here from an evaluator’s perspective, and we thought that this kind of a
forum would be a way for us to start this process. So this really was, and is,
an experiment in mechanisms in learning for us – can we respond to situations
that arise during the course of a year, identify ones that are particularly
important for evaluators, focus on them in this kind of a forum, and then post
the results of this so that other evaluators can also benefit from that?
Also,
hopefully, I fully expect that some of the people at OMB will take a look at
this forum, and review what we said here. I think Nancy already described an
example of an interaction between the Federal Evaluators Network and OMB that
had an impact on how they do some of the details in grading these programs and
in terms of evaluation. And so I’m hoping that this will have an impact for the
membership in terms of our learning about these things and engaging on these
issues, and quite possibly we’ll have some impact as a form of feedback to those
policymakers who might read about this. So that’s the intent of this. And I
guess the other question that I heard you ask as well has to do with
implications for evaluators who work with programs that have been through PART
reviews, and I wonder if any of you would like to comment on that?
Michael Schooley: I think there are two things in the response. One, as a local
evaluator and looking up at the list, and saying, “Hey, I’m doing local work and
local evaluation with a program that might be an example of one of those
competitive grant programs, a block grant or other grant type program. Looking
at the PART evaluation, I think, would certainly be interesting and informative
to you as a local evaluator, but is probably not going to tell you a whole lot
about what you do on a day-to-day basis. It came up a little bit in a previous
session on Federal assessments that PART reviews are done at a
program
level. What is happening locally, especially for many competitive grant programs
or block grant programs, are
projects
that are implemented locally and individually. So looking at the evaluation of
programs
will be of interest to you, and it will have content for what you do in terms of
the broad spectrum of your role, but it’s not going to do a lot for the
projects
that you might be evaluating on a day-to-day basis.
One the other
hand, looking at the PART tool I think is of use and value to local evaluators.
I think the process and what they have pulled together is interesting and
informative. I wouldn’t say that you would take it carte blanche and apply it to
your local area, but there are certainly aspects of it that are of value. Some
of the techniques might add to the pool of techniques you are using locally in
the work that you are doing.
Nancy Kingsbury: I would agree with that. I think the fundamental question
involves the kinds of questions that need to be thought about, and it might be
enlightening just to look at those and say, down at this project level, do we
understand what these things are about, and do we have evidence of whatever
kind, from whatever source, that could go towards answering those questions
rather than trying to say yes or no to the fundamental substance of the question
itself?
Q: I’ve enjoyed this panel very much, and have followed a lot of
the Federal initiatives or performance measures, and I think my initial question
has pretty much been answered in terms of what’s the role of this Public Issues
Forum to try to improve the PART process, which I happen to believe is likely to
be a good thing if the right measures are chosen. But hearing what Nancy has
said about how this is likely to not be continued in a new administration, and
who knows what’s going to happen after next week’s election, I’m thinking what
might be the role of AEA and/or evaluators individually who know about a
particular content area in trying to improve this process in the future, so that
good performance measures are selected and evidence is weighed appropriately. It
seems to me these are two key evaluation-oriented problems. Do you have any
thoughts are to how we might contribute in the future?
Nancy Kingsbury: Let me say this. I think it will be interesting to see if
anything comes out of the election that begins to mold where this is headed. A
year ago at this conference, the debate around PART was almost entirely around
the randomized control design that was out there, and Bill and I were talking
before this session about whether that was going to be the topic of all the
questions, and it hasn’t been. That may be because of the urgency with which
evaluators were reacting to the same thing that the GAO reacted to, which is
that this is not an appropriate way to go around designing an evaluation tool
for many, many programs, and have begun to see through the process itself in
some kind of helpful way.
So going
forward and looking ahead, I think if we’re lucky, there will be a debate even
in the campaign for President the next time around, that we may be able to feed
into or help change. It’s really hard to see that yet, but as soon as the
election ends next week the presidential campaign will start. So I don’t know
the answer, really, but I do sense there is something of an opportunity.
Bill Trochim: I think that the little example that Nancy just gave us, of a group of
federal evaluators bothering to pick up the phone and call OMB and arrange for a
meeting and engage with them, actually had some pretty profound effects, given
the cost of that effort. And as part of what I hope we take away from this is
the idea that as members, as individuals, as evaluators, and perhaps even as an
association, if we can find ways to begin to engage more directly in dialog
around these things, we might be surprised to find the degree to which we might
have some direct influence on the formation of legislation on the way in which
these things are implemented, practiced and so on. At the very least, that ought
to be part of what we are taking part in.
Q: I was really struck by a couple of comments. One was where Ted
Kniker said couldn’t we look at contribution instead of the causation, and I was
also struck by Nancy Kingsbury’s comments about trying to look at the
interactions among Federal agencies for a wider strategic plan across Federal
agencies. I think that one of the fictions that underlies the PART is sort of
like the old concept of union jurisdiction – that there is one union to one
industry. But in practice it’s not like that, and it isn’t even necessarily
clear it’s a bad thing that it’s not like that. So I think that there needs to
be a lot more attention to whether or not the interaction and interchanges
between federal agencies are productive, useful, helpful, or whether it is
simply redundancy. And the PART seems to basically say it’s redundancy.
Ted Kniker:
One of the things about the PART as it is currently implemented, I think, still
promotes stovepiping of organizations, although in certain sectors there has
been guidance from OMB that we need to broaden who is in particular programs.
For example, with public diplomacy, the promotion of the U.S. image abroad, led
by Karen Hughes, that the bureaus who worked on that were brought together, and
there is talk about doing a united PART for that. But then there was also talk
about, because there are other agencies that influence the image of the U.S.
abroad, that they should be brought in as well, including the Department of
Homeland Security, Department of Defense, Department of Justice, Peace Corps,
and a number of these others.
So there is at
least the recognition that programs and impact goes beyond just a particular
agency. But one of the things that we have seen, and maybe an area where AEA can
assist with, is in the Department of State, for example. The evaluation across
the Department of State is not coordinated at all. PART was, but if my bureau
was doing something on exchange programs, and how well particular programs
worked, we didn’t really involve any other programs that might contribute to
that success. So I think that one of the things we need to look at is whether
there are ways to begin to coordinate evaluation across agencies, if multiple
agencies are involved with a particular program.
Bill Trochim: Well, I think our time is over, and with one final pun, I’d like to thank
you all for taking “PART” in this first AEA Public Information Forum!
Return to PART
|
