|
Using Propensity Scores with Small Samples
|
| Presenter(s):
|
| William Holmes, University of Massachusetts, Boston, william.holmes@umb.edu
|
| Lenore Olsen, Rhode Island College, lolsen@ric.edu
|
| Abstract:
Propensity scores are increasingly being used in large sample studies to control for pre-group differences. Because these scores are often used to match cases, they can result in sample attrition. In smaller sample studies, such attrition leaves too few cases for meaningful analysis. An alternative approach when working with small samples is to use propensity scores as covariates to control for pre-group differences.
The presenters examine the use of propensity scores with small samples and compare their use with the alternative of using baseline measures to control for pre-group differences. The paper also presents a procedure for empirically testing whether construct integrity holds. The presentation uses data from a dosage specific study of substance abusing families receiving clinical services and coordinated case management. Program outcomes are examined, comparing the use of propensity scores with the use of time one measures alone.
|
|
The Utility of Propensity Score Matching in the Context of Evaluation
|
| Presenter(s):
|
| Corina Owens, University of South Florida, cmowens@usf.edu
|
| Connie Walker-Egea, University of South Florida, cwalkerpr@yahoo.com
|
| Abstract:
Propensity score matching has gained attention as a potential method for estimating the impact of treatment or causal treatment effects in the absence of experimental evaluations. Experimental evaluations require preparation and planning, and cannot be conducted post-hoc. Propensity score matching, as an alternative, is a quasi-experimental method that attempts to reduce the bias of treatment-effect estimates from observational studies. In these types of studies, participants have not been randomly assigned to treatment or control group, which can be a common scenario in many evaluations. This presentation will focus on the utility of propensity score matching in a program evaluation context. Specifically, two methods of matching will be illustrated: one-to-one matching and propensity grouping or strata matching.
|
|
A Comparison of Genetic Matching and Propensity Score Matching Methods for Covariate Adjustment in a Reading Intervention Program Evaluation
|
| Presenter(s):
|
| Ning Rui, Research for Better Schools, rui@rbs.org
|
| Debra Coffey, Research for Better Schools, coffey@rbs.org
|
| Abstract:
Considerable attention has been given to various matching techniques to adjust for baseline covariate imbalances in order to properly estimate the impact of a program under evaluation. However, little is known on whether these techniques tend to reliably provide accurate and consistent measures of the treatment effect. Drawing upon two years of experimental data about a comprehensive reading intervention program from a large southern school district, this paper presents a case study of comparing genetic matching, a non-parametric matching technique that applies an innovative search algorithm to assign weight to each covariate (Sekhon and Grieve, 2009), to the traditional propensity score matching as well as regression-based methods. Both propensity score and genetic matching significantly improved baseline covariate balance. Neither multiple regression nor propensity score analysis rendered statistically significant impact on student achievement. Surprisingly genetic matching overturned this conclusion and detected statistically significant yet model-dependent results about the program impact.
|
| | |