Panel Comments: George Grob

Wow, it’s a real pleasure to see you all in the room and I’m glad to see there’s such a great interest in our topic. I’m not going to delve into opinions and advice for you. My job here is to simply give you a factual orientation to set the stage for the other speakers. I want to let you know in as concrete way as I can what we mean when we talk about evaluation policy.

While the topic of Federal evaluation policy seems somewhat vague, it is possible to it when you see it. I used to think about that in the following way: that sometimes when I went to the office, I really would dress up like I am now – my best suit, dark, a tie like this, and my wife would say, “Oh, George, you going to make national policy today?”

The truth of the matter is that at the end of the day, I could tell if I was involved in national policy; I literally could tell. For example, I could say, “Well, today we sent a draft bill over to the Congress,” or, “We prepared the budget for such-and-such a thing.” My goal here today is to give you concrete examples of evaluation policy. That’s how I’ll try to address it for you.

Evaluation Policy: the short version is that – it’s laws and rules. But it is really much broader than that. Rules is a big word here. Now I don’t mean just formal regulations. I mean to refer to all kinds of rules, including administrative rules that affect the practice of evaluation. This would include funding, standards, evaluation organizations and requirements for specific studies.

To press the point further, evaluation policies can be quite detailed. They might say, “Here are the questions we want you to answer, here are the methods we want you to follow. This is the kind of person we want you to get to do it and this is how much money we’re going to get you to do it and we want it done by this period of time.” Each one of these questions floats down in a natural way to all these details. But those are the kind of things we mean by policy within the government. By the way, today we’re only talking about the policies of Federal Government, although the concepts are similar in state and local governments as well.

Evaluation Policy establishes the boundaries within which we practice our profession. I would really encourage you to look at that back page of our manual for the conference. You’ll see the statement that Bill Trochim has written for the coming year. He’s got great definitions and if any of you were treated with going to the Awards Banquet, Bill also laid it out pretty well there.

Federal policies take the form of laws and regulations, and you can really see if something has happened. --- a law is passed, a regulation is issued and a regulation is a real document. You can see that one day the Secretary or some other senior official signs a regulation. Then there are drafts; they’re formal, you make comments on them. These are very precise things, such as administrative procedures—e.g., what you have to do to approve a Medicare claim; organizational charts – a new office has been created,; there wasn’t an office there, now there is an office there. At one time there was no Office of Evaluation in the Inspector General’s office and then one day there was an office. It is that real. This is not nebulous stuff that I’m talking about; it’s very, very real.

Other examples include administrative procedures and requests for proposals, to name a couple. I will give you a good example of one that I saw and was interested in for a short time. It was a request for proposals to make grants for the Abstinence Only program and they had a requirement that between 10% and 30% of the funds had to be set aside for an evaluation and they wanted it to be the random controlled study. And, by the law, it had to be done by an evaluator who was associated with a major educational institution in the state in which the grant was being made. Furthermore, the program office went on to say and it needs to be a Ph.D.. When we say that there are these details that affect how we practice, that’s an example. There are many, many more but that’s a real, live example of something you might see in an administrative document.

There are still more examples– the budget’s a big one. It may prescribe how much money will be available for evaluation for an organization.

Prevailing practices are also a form of evaluation policy. Bill has referred to this. Everybody in an office does things in a certain way.. It is not written down. If you work in that office, you’re going to do it that way. They’ll tell you.

Evaluation plans – some offices actually issue evaluation plans. They’re published. In most of the offices I worked, there was a formal plan.

Policies are established by the Congress and the Executive Branch. In the Congress, much of the work is done by committees, but the first ones, the big ones, are the general government oversight committees. There are two of them - one in the Senate and one in the House of Representatives. Then there are subject matter oversight committees. They look such things as health, education, labor, environment, and the like. And then there are the appropriations committee, which recommend how much money will be allocated for the policies and programs approved by the oversight committees.

Then there is the Government Accountability Office, GAO. That’s a congressional office, it’s not an executive office. First of all, it exists! They do evaluations. They don’t do many audits anymore. It’s almost all what they would call evaluations, and which most other people would too. They use the phrase - “performance audit”. But it is truly evaluation work, or at least most of it is.

On the executive side, you have the Office of Management and Budget. This office established PART. In addition, each Federal department or agency may have offices, money, procedures, plans for evaluation., However, some agencies have no such arrangement for evaluation.

There may also be special agencies that the Congress and Executive Branch leaders would call on to conduct evaluation studies. These include the Institute of Medicine, National Science Foundation, the Inspector Generals.

Another well-known example of evaluation policy is the Government Performance and Results Act (GPRA). That law was prepared by those oversight committees for the functioning of the entire Government. It took a dozen years to get that law passed.

Another example of evaluation policy is appropriations, including set-asides. A prominent example is the setting aside of 1% of program appropriations to be used for evaluations in many agencies within the Department of Health and Human Services. Still another example one would be the so-called “high stakes” testing policy for schools. That law would have been formulated by the congressional committees that oversee education programs.

Then there evaluation standards. A prominent example is the GAO standards for performance audits. While they are used by GAO, they are also followed by many State and Federal Agencies. In a similar vein, the Inspectors General have issued quality standards for evaluations and inspections.

Examples of Executive Branch policies include the Program Assessment Appraisal Tool. It was issued by the Office of Management and Budget OMB) and applies to almost all Federal programs and agencies. Other examples are OMB standards for and clearance of surveys, standards for the protection of human Research subjects, issued by regulation. In addition, there are evaluation units within Federal agencies. I worked in one for many years—the Office of Assistant Secretary for Planning and Evaluation at the Department of health and Human Service. Furthermore, some of the programmatic agencies within large federal departments have offices dedicated to evaluation, including the Centers for Disease Prevent and Control, the National Institutes of Health, and the Health Resources and Services Agency at HHS, to name just a few.

Another example of evaluation policy is the requirement for the used of randomized control trials for the assessment of the impact of education programs. That policy was established by regulation. It was not required by law. The Education Department came up with that rule.

Policy and practices vary considerably. Some policies apply across the whole Government - GPRA, PART, the clearance of surveys. Other policies, such as those relating to how much funding is provided for evaluation, whether there is a formal evaluation office for an agency, whether these is a requirement for evaluation plans, how studies are published vary from agency to agency.

Panel comments: Tom Chapel