|
Session Title: Advances in Propensity Score Research: Improving Methods to Reduce Bias in Quasi-Experiments
|
|
Panel Session 421 to be held in Room 106 in the Convention Center on Thursday, Nov 6, 4:30 PM to 6:00 PM
|
|
Sponsored by the Quantitative Methods: Theory and Design TIG
|
| Chair(s): |
| MH Clark,
Southern Illinois University at Carbondale,
mhclark@siu.edu
|
| Abstract:
Quasi-experiments are useful for studies that need to be conducted in applied settings where random assignment to treatment groups is not practical. However, a major disadvantage in using these designs is that the treatment effects may not yield unbiased estimates. Propensity scores, the predicted probability that cases will be in a particular treatment group, are often used to help model and correct for this selection bias. The studies presented as part of this panel represent the latest findings in propensity score research. This panel will present the effectiveness of reducing bias using propensity score analysis compared to (a) conventional regression adjustment for covariates, and (b) single covariate matching using a variety of matching techniques.
|
|
Can Propensity Scores Reduce Regression to the Mean?
|
| MH Clark,
Southern Illinois University at Carbondale,
mhclark@siu.edu
|
|
Computer simulations were used to examine how various matching methods reduced regression to the mean that occurs in non-randomized experiments that use matching. Simulations were created so that the true effect size was zero, but the treatment and control groups were biased so that the control group appeared to perform better than the treatment group. The regression artifact was created by adding random error to each observed variable. A three-factor design examined the effects that type of matching, proportion of available matches, and the number of variables used to compute propensity scores had on reducing the regression artifact. It was expected that (a) full matching would be more effective than paired matching; (b) having more control units available to match to treatment units would be more effective than matching equal numbers of control and treatment units; and (c) matching on propensity scores created from several covariates would be more effective than single covariate matching.
|
|
|
Propensity Score Analysis versus Traditional Regression Adjustment for Covariates: A Monte Carlo Study
|
| Jason Luellen,
Vanderbilt University,
jason.luellen@vanderbilt.edu
|
| David S Cordray,
Vanderbilt University,
david.s.cordray@vanderbilt.edu
|
|
With regard to adjusting for confounding in nonexperimental studies, recent review papers have questioned the utility of propensity score analysis relative to conventional regression adjustment for covariates, reporting that effect estimates were often comparable for studies that presented both types of adjustments on the same data. However, these findings were tempered by the fact that many published propensity score analyses were not well implemented. This paper briefly explores the rationale behind each of the adjustment methods and presents the findings from a Monte Carlo study investigating their relative performance in terms of bias reduction and precision. The authors simulated a nonequivalent control group design and performed thousands of trials adjusting the outcome estimates for propensity scores and for the same raw covariates using regression.
| |
|
Examining the Effects of Communities In Schools Inc (CIS) on School-Level Outcomes with the Use of Propensity Score Analysis Methods: A National Application
|
| Katerina Passa,
ICF International,
kpassa@icfi.com
|
| Heather Clawson,
ICF International,
hclawson@icfi.com
|
| Susan Siegel,
Communities In Schools Inc,
ssiegel@cisnet.org
|
|
As a component of the national evaluation of Communities In Schools, Inc. (CIS), our team sought to quantify the impact of the CIS network on several academic and behavioral outcomes across elementary, middle, and high schools in seven states. Using propensity score matching techniques, we matched CIS schools with non-CIS schools on several characteristics, including attendance rates, racial/ethnic composition, dropout rates, and test scores, and then compared outcomes. The study examined schools served by CIS for 3 to 5 consecutive years between the 1998-99 and 2003-04 school years, and those that did not have exposure to the CIS program during the same period. Our findings indicate that programs in schools using the CIS model demonstrate positive outcomes for students over a three-year period. Specifically, our presentation will focus on the methodology and use of propensity score analysis in a quasi-experimental evaluation of a national program model.
| |
|
Controlling for Endogeneity Bias in Evaluating Alcoholics Anonymous’ Effect on Drinking
|
| Stephen Magura,
Western Michigan University,
stephen.magura@wmich.edu
|
|
Evaluation studies consistently report correlations between Alcoholics Anonymous (AA) participation and less drinking or abstinence. Randomization of alcoholics to AA or non-AA is impractical and difficult. Unfortunately, non-randomization studies are susceptible to artifacts due to endogeneity bias, where variables assumed to be exogenous (“independent variables”) may actually be endogenous (“dependent variables”). Such common artifacts are selection bias, where different types of people choose to participate or not participate in AA, and reverse causation, where reducing or stopping drinking may lead to increased or sustained AA participation. The paper will present a plan for using three specific statistical techniques to control for possible selection and/or reverse causation biases in a national alcoholism dataset - propensity score matching, instrumental variable analysis and structural equation modeling with cross-lagged panel analysis. This presentation will be primarily conceptual, not mathematical or statistical, and thus accessible to evaluators and others without advanced statistical training.
| |