|
Session Title: Quantitative Methods: Theory and Design TIG Business Meeting and Panel - Revisiting Validity: Implications for Theory, Practice, and Policy
|
|
Business Meeting with Panel Session 259 to be held in Centennial Section B on Thursday, Nov 6, 10:55 AM to 12:25 PM
|
|
Sponsored by the Quantitative Methods: Theory and Design TIG
|
| TIG Leader(s):
|
|
Patrick McKnight,
George Mason University,
pmcknigh@gmu.edu
|
|
George Julnes,
Utah State University,
george.julnes@usu.edu
|
|
Frederick Newman,
Florida International University,
newmanf@fiu.edu
|
|
Karen Larwin,
University of Akron,
drklarwin@yahoo.com
|
| Chair(s): |
| Frederick Newman,
Florida International University,
newmanf@fiu.edu
|
| Abstract:
The quantitative traditions in evaluation are based on well-established views of validity. Indeed, many debates among quantitative evaluators are based on different views of validity. This panel will examine different aspects of how we view validity and suggest implications for how we think about validity, how we safeguard it in practice, and how we can use elaborated views of validity to improve evaluation policies.
|
|
Reframing Validity for Fun and Profit
|
| George Julnes,
Utah State University,
george.julnes@usu.edu
|
|
In their seminal paper on developing our understanding of use or utilization of insights due to evaluation, Mark and Henry (2003) refer to the prior accretion of theory as being both 'overgrown and impoverished.' Theirs is a useful model of how we honor best the frameworks we have been bequeathed, taking the accumulated insights and trying to clarify and elaborate the relationships involved. This presentation attempts this with our accumulated concepts of validity, identifying dimensions that differentiate our primary types of validity and suggesting implications for evaluation policy.
|
|
|
Reconsidering Validity
|
| Ernie House,
University of Colorado,
ernie.house@colorado.edu
|
|
An inspection of the drug evaluation literature reveals that many drug studies are biased to produce the findings that the sponsoring drug companies want. Yet reviewers of these studies, while noting the deliberate biases built in, often call these studies 'high quality' because the studies meet certain formal requirements like randomization and double blinding. What can one possibly mean by 'high quality' in such circumstances? Perhaps we should reconsider what the 'validity' of studies means and whether the definition should be differently delineated.
| |
|
A Critique of the Campbellian Conceptualization of Validity and an Alternative
|
| Charles Reichardt,
University of Denver,
creichar@du.edu
|
|
Validity is a central construct in the methodological literature in both the fields of measurement and experimentation. Within the field of experimental (and especially quasi-experimental) design, the most prominent conceptualization of validity is the Campbellian approach which categorizes validity into the four types of internal, external, construct, and statistical conclusion validity. The distinction between internal and external validity was first introduced by Campbell (1957) and subsequently immortalized in Campbell and Stanley's (1963, 1966) seminal treatise on quasi-experimental design. Cook and Campbell (1976, 1979) revised Campbell and Stanley, and expanded the original bipartite categorization of validity into the current four-fold version. In turn, Cook and Campbell (1979) has been revised by Shadish, Cook and Campbell (2002), who further explicated the logic behind the four-part typology. The purpose of the present paper is to detail inconsistencies and limitations in the Campbellian conceptualization. In addition, the present discussion provides an alternative conceptualization of validity which avoids the flaws in the Campbellian approach. My intent is not to replace the Campbellian conceptualization, at least not as it is used in practice. The Campbellian approach has become so firmly entrenched over the past half century that it is difficult to imagine how any critique, no matter how compelling, could dislodge it from its deeply embedded position within the consciousness of social scientists. Nonetheless, methodologists need to be cognizant of the flaws in the current zeitgeist if rigorous theories of experimentation are to be built upon a firm foundation.
| |