|
Session Title: Advancing External Validity in Program Evaluation
|
|
Panel Session 389 to be held in Capitol Ballroom Section 4 on Thursday, Nov 6, 4:30 PM to 6:00 PM
|
|
Sponsored by the Theories of Evaluation TIG
|
| Chair(s): |
| Huey Chen,
Centers for Disease Control and Prevention,
hbc2@cdc.gov
|
| Discussant(s):
|
| Charles Reichardt,
University of Denver,
creichr@psy.du.edu
|
| Abstract:
Internal and external validity are vital to program evaluation. However, evaluation literature has focused attention mainly on internal validity. Few evaluators have formally addressed issues related to external validity in their evaluations. Program evaluation is an applied science. The neglect of external validity may curtail the usefulness of and limit the future development of program evaluation. The purpose of this panel is to advance external validity in program evaluation. Presenters of the panel will address conceptual and methodological issues of external validity. The presenters will: 1). Examine the limitations of the existing concepts and model of validity, 2.) Propose alternative conceptualization of external validity, 3). Propose alternative methods and models for achieving external validity, 4). Discuss future directions for dealing with internal and external validity.
|
|
External Validity, Policy and Practice
|
| Leonard Bickman,
Vanderbilt University,
leonard.bickman@vanderbilt.edu
|
|
External validity lacks the more specific 'rules' found with other forms of validity. For example, internal validity has a list of threats and statistical validity has well defined power calculations. In contrast the determination of external validity is much more in the eye of the beholder. The policy and practice issues raised by this difference will be discussed by contrasting efficacy and effectiveness research.
|
|
|
What Works Where for Whom and Why?
|
| Stewart I Donaldson,
Claremont Graduate University,
stewart.donaldson@cgu.edu
|
|
Recent debates about RCTs, gold standards, and credible evidence focus largely on internal validity concepts and how best to determine the average effects of a program, policy, or 'What Works' in general terms. However, policy and other decision makers desire credible and actionable evidence that generalizes or applies to present or future problems, programs, and policies of interest (i.e., external validity concepts). This paper will describe how the principles and procedures of program theory-driven evaluation science can be used to make evaluations more useful for decision making. A framework for increasing the external validity of primary evaluations and the syntheses of multiple evaluations will be presented and discussed.
| |
|
External Validity: Alternative Models and Practice
|
| Melvin Mark,
Pennsylvania State University,
m5m@psu.edu
|
|
External validity is a central concept in the Campbellian tradition. The term has been widely used, sometimes quite differently than Campbell intended. Even when external validity refers to the generalizability of findings, different models exist regarding how external validity is conceptualized and how it is thought to be achieved. Alternative models of external validity are selectively reviewed. I also identify different design and analysis procedures that have been suggested as ways of enhancing (or at least identifying the limits of) external validity. Fruitful directions for future work, both conceptually and in terms of evaluation practice, are described.
| |
|
External Validity, Integrative Validity Model, and Evaluative Evidence
|
| Huey Chen,
Centers for Disease Control and Prevention,
hbc2@cdc.gov
|
|
Validity has been one of the crucial issues in program evaluation. Recent controversies on the best evaluation methods and what should be considered as evidence in the evidence-based movements are related to validity. The controversies may reflect a need for re-examining the existing concepts and models of validity. Based upon the contribution by Campbell and Cronbach, this paper advances the concept of external validity and provides a comprehensive model of validity which allows evaluators to respond to the challenges more coherently. The paper also discusses sequential and concurrent evaluation approaches for evaluators to effectively deal with validity and how these strategies relate to program evaluation. Based upon the discussions, the article discusses possible directions for advancing issues related to validity in the future.
| |