|
Session Title: Getting Down to Cases: Evaluation Results and Decisions in Situ
|
|
Panel Session 246 to be held in Santa Monica on Thursday, Nov 3, 8:00 AM to 9:30 AM
|
|
Sponsored by the Quantitative Methods: Theory and Design TIG
|
| Chair(s): |
| Lee Sechrest, University of Arizona, sechrest@u.arizona.edu
|
| Abstract:
Results of evaluations are almost always generic: a program "works" (or doesn't), an intervention "is effective" (or isn't). Decisions, on the other hand, are specific: they are made by this practitioner or that one, his administrator or another one; they are made with respect to this particular school or classroom, this particular client, and so on. The process of getting from generic results to particular decisions is not necessarily straightforward. This panel will consider the basis in generic "evidence" for decisions about particular cases, including the specific hypotheses said to have been tested, the specific research questions addressed, the effect sizes claimed, and the implied confidence about the evidence. Frequentist and Bayesian approaches will examined for differences they may reveal about individual decisions, and the bases for modifying decisions about individual cases, including risks, will be examined.
|
|
Are Evaluation Results to be Taken Seriously?
|
| Sally Olderbak, University of Arizona, sallyo@u.arizona.edu
|
| Lee Sechrest, University of Arizona, sechrest@u.arizona.edu
|
|
When a program is said to be "effective," that usually means that in at least a fairly large group of persons at least somewhat homogeneous in at least some ways, those persons exposed to the program were in some ways at least somewhat better off than those persons not exposed to the program.
Such assurance of effectiveness may not be sufficient to persuade local administrators to adopt the program in their particular locale, to persuade an individual practitioner to apply the intervention to his or her clients, or to persuade an individual client to submit to the intervention. Program interventions may seem alien, but they may also seem irrelevant or inappropriate, or the results may seem ephemeral. It may also be that the intervention itself seems unfeasible in the particular instance. These reasons for exceptions in the use of otherwise promising interventions are illustrated with specific examples.
|
|
|
The Shortcomings of P Values
|
| J Michael Menke, Critical Path Institute, menke@u.arizona.edu
|
|
Usual approaches to assessing effects of programs are based on "frequentist" thinking and methods and are exemplified by null hypothesis statistical testing. Even if a program is found to have significant positive effects, those effects may not apply generally in the population involved; in fact, the effects may not apply to more than a small proportion of the sample and, presumably, the population. It is often unclear-usually because not even questioned-whether results mean that a few people were helped a lot, many helped moderately, or most helped a little. Thus, there may not be in program evaluation results a basis for any clear decision about whether, let alone how, program results should be used. And, unless the evaluation provides estimates of effects for segments of the sample, no basis will exist for decisions better tuned to local circumstances. Improvements in ways of reporting results are possible and recommended
| |
|
Can we Cover the Shortcomings of P With a Bayes Leaf?
|
| Kirsten Yaffe, University of Arizona, yaffe@u.arizona.edu
|
|
Bayesian approaches to program evaluation are often recommended as a way of improving our inferences and, presumably, generalizations. Bayesian methods and analyses are aimed at capitalizing on prior knowledge (and expectations) about phenomena in order to improve estimates of the confidence that can be placed in hypotheses after the data are in. Presumably, if those estimates are better, then confidence in generalizability of findings should be improved. That is not necessarily the case, however. Bayesian analyses may provide a better estimate of confidence in a hypothesis, but that hypothesis may still seem irrelevant or inappropriate in situ. In fact, vagueness about in just what hypothesis confidence may be placed and what further conclusions that confidence may justify seems to characterize a good many Bayesian explanations of empirical findings. But as with frequentist approaches, improvements in ways of thinking about Bayesian presentations of results are possible and can be recommended.
| |
|
Decision Making in the Face of Uncertainty
|
| Lee Sechrest, University of Arizona, sechrest@u.arizona.edu
|
|
The controversy over "clinical vs. actuarial" decisions-making has a long history, and, although it has not been completely resolved, the preponderance of the evidence seems to favor the actuarial argument. Still, there must be some cases and some circumstances in which practitioners (and their clients) may reasonably "go against the evidence." In fact, discussions of evaluation and other research results very frequently include, or conclude with, assurances that practitioners and clients must consider the evidence carefully and make their own decisions. Still, unless there is uncertainty, there is no need for decision making. Careful reading of the literature, along with equally careful consideration of "the evidence," suggests that practitioners and clients may indeed make decisions that are against, or beyond, the evidence, but they need to very good reasons for doing so. Harder thinking about the choices and their bases might help.
| |