|
Session Title: Advances and Applications in Using Propensity Scores to Reduce Selection Bias in Quasi-Experiments
|
|
Panel Session 834 to be held in Baltimore Theater on Saturday, November 10, 1:50 PM to 3:20 PM
|
|
Sponsored by the Quantitative Methods: Theory and Design TIG
|
| Chair(s): |
| M H Clark,
Southern Illinois University, Carbondale,
mhclark@siu.edu
|
| Abstract:
Quasi-experiments are useful for studies that need to be conducted in applied settings where random assignment to treatment groups is not practical. However, a major disadvantage in using these designs is that the treatment effects may not yield unbiased estimates. Propensity scores, the predicted probability that cases will be in a particular treatment group, are often used to help model and correct for this selection bias. The studies included as part of this panel present recent findings in propensity score research. This panel will present (a) a comparison of various methods for computing, using, and interpreting propensity scores; and (b) how propensity scores can be applied to quasi-experiments in which selection into treatment conditions is potentially biased.
|
|
A Simulation Study Comparing Propensity Score Methods
|
| Jason Luellen,
Vanderbilt University,
jason.luellen@vanderbilt.edu
|
|
Estimates of treatment effects from quasi-experiments are likely biased to some unknown extent due to the nonrandom assignment of study conditions to units, and evaluators are interested in methods for reducing that selection bias. Propensity score methods, which utilize an aggregate of the observed pretreatment covariates to adjust for selection bias, are now a popular option employed when analyzing the data from non-equivalent control group designs. This simulation study compares several methods of estimating propensity scores (logistic regression, classification trees, bootstrap aggregation, boosted regression, and random forests) crossed with several methods of adjusting outcomes using propensity scores (matching, stratification, covariance adjustment, and weighting). The paper is a follow-up to the talk I presented at Evaluation 2006 with additional analyses useful for helping practitioners choose from among the available propensity score methods.
|
|
|
Freshmen Interest Groups: Effects of Academic Success and Retention.
|
| Joel Nadler,
Southern Illinois University, Carbondale,
jnadler@siu.edu
|
| M H Clark,
Southern Illinois University, Carbondale,
mhclark@siu.edu
|
| Heather Falat,
Southern Illinois University, Carbondale,
hfalat@siu.edu
|
| Chad Briggs,
Southern Illinois University, Carbondale,
briggs@siu.edu
|
|
A Freshmen Interest Group program was examined using first-year students sampled over a three-year period from a mid-western university. Freshmen Interest Groups are college-level interventions in which students with similar academic interests are housed together and placed in a structured set of pre-chosen classes. The goals of the program are to increase academic performance and retention rates among college freshmen. Since students self-selected into this program, it should not be assumed that posttest only treatment effects would be unbiased. Therefore, statistical adjustments were made to reduce potential selection bias by stratifying on propensity scores. Propensity scores, which are the predicted probabilities for selecting into the program, were computed from several covariates, including personality, previous academic achievement, social skills, and family history. These adjusted results should provide less biased estimates, allowing for stronger causal conclusions than normally allowed in quasi-experiments.
| |
|
Assessing the Success and Attrition of College Students: A University 101 Study
|
| Nicole Cundiff,
Southern Illinois University Carbondale,
karim@siu.edu
|
| M H Clark,
Southern Illinois University, Carbondale,
mhclark@siu.edu
|
| Heather Falat,
Southern Illinois University, Carbondale,
hfalat@siu.edu
|
| Chad Briggs,
Southern Illinois University, Carbondale,
briggs@siu.edu
|
|
A first-year experience course at a higher education institution was evaluated to understand the effects of the program. The program's goals are geared toward promoting success at the University through retention, higher grade point averages, and enhanced academic skills. This study attempts to look at these variables over a three-year span to grasp the effectiveness of the program using a post-test only non-equivalent control group design. Because students self-select into the University 101 course, it is assumed that selection bias will be a problem in evaluating the effectiveness of the program. Therefore, statistical adjustments will be made by matching on propensity scores, which are created by aggregating covariates that we expect would influence students' selection into the program. These covariates include: high school GPA, personality, and social skills. It is expected that the propensity score adjustment will provide a less biased estimate of the program effect.
| |