|
Session Title: Generalizability Theory Applications in Program and Policy Evaluation: An Introduction and Application To Quasi-Experimental and Experimental Designs
|
|
Panel Session 213 to be held in Sebastian Section I2 on Thursday, Nov 12, 9:15 AM to 10:45 AM
|
|
Sponsored by the Quantitative Methods: Theory and Design TIG
|
| Chair(s): |
| Patrick McKnight, George Mason University, pmcknigh@gmu.edu
|
| Abstract:
Generalizability theory was originally developed by Lee Cronbach as a psychometric alternative to classical test theory. Since its inception, quantitative social scientists found that its relevance branched far beyond Cronbach's original focus. Generalizability theory enables analysts to document sources of variability in both observational and experimental designs; thus the approach works well for program evaluation in all areas. The purpose of this panel is to introduce the audience to generalizability theory methods and procedures to increase awareness of data analytic alternatives to regression, ANOVA, and other standard statistical tools. Generalizability theory may enhance program development and future evaluations by assessing which factors affect outcomes the most. Applying generalizability theory to program and policy evaluation might lead to a clearer understanding of why certain programs produce changes while others fail to deliver.
|
|
A Brief Introduction to Generalizability Theory
|
| David Kidd, George Mason University, dkidd3@gmu.edu
|
|
Generalizability Theory (GT) is a powerful and flexible statistical tool that answers many evaluation questions. GT allows the evaluator to estimate the amount of variability attributable to different predictors in several research designs and identifies the factor that most influences the outcomes of interest. This presentation will introduce the basic mechanism, interpretation, and application of GT. The audience will be shown a series of simple datasets that illustrate how GT helps identify and estimate sources of variance. Next, we will show how variability estimates are related to mean differences found using common statistical analyses. This primer will help evaluators understand how GT captures different sources of variability, and, more importantly, will highlight the utility of GT for helping evaluators better understand the nature of observed effects.
|
|
|
Generaliability Theory Applied to Studies in Naturalistic Settings
|
| David Cades, George Mason University, dcades@gmu.edu
|
|
Most evaluation takes place outside of controlled laboratories in more applied naturalistic settings. While naturalistic settings have high ecological validity, they may lack strength of inference. One way to increase our ability to draw conclusions from uncontrolled naturalistic data is the application of Generalizability Theory (GT). GT adds control to applied work. The following presents an application of GT to a naturalistic data set examining the influence of interruptions on office workers. Laboratory studies have shown that interruptions can be detrimental to our ability to complete tasks as shown by both reduced accuracy and increased performance time. Applying GT to our naturalistic data, we hope to show where the sources of variance are and which variables influence performance. This knowledge will allow us to design future programs and policies that minimize the effects of interruptions on worker productivity.
| |
|
Evaluating Randomized Designs With Generalizability Theory
|
| Matthew Kendra, George Mason University, mkendra@gmu.edu
|
|
Program evaluators aim to minimize confounding variables by controlling naturalistic settings and resorting to randomized controlled field trials. The extent that those settings can be controlled remains unclear. Generalizability theory (GT) offers us the ability to assess how well controlled situations, randomization, or any other methodological controls worked by estimating variance components. I intend to demonstrate how GT may help evaluators assess the extent that stronger designs may eliminate or enhance individual differences, participant response, or program effectiveness. GT will be used to identify the exact variance attributed to several relevant variables. By using GT, we may understand whether and why a program or policy worked. Furthermore, GT will enable us to make better design decisions for future evaluations.
| |