|
Application of Data Integration and Matching Techniques in Evaluation
|
| Presenter(s):
|
| Mukaria Itang'ata, Western Michigan University, mukaria.itangata@wmich.edu
|
| Abstract:
Often evaluators using non-experimental or observational data sets are faced with situations where they must conduct comparative studies between two or more programs or outcomes. In many of these situations there may be one large population compared to a much smaller foci subpopulation. In such situations, a logical way to study the subpopulations is to create one or more matched comparison samples drawn from the larger population. However, a question arises: how best to match the subpopulation participants when randomization is not possible? This presentation will compare different data integration and matching techniques to determine if these matching techniques result in differential evaluation conclusions under different experimental conditions.
|
|
Issues in Selecting Covariates for Propensity Score Adjustments
|
| Presenter(s):
|
| William Shadish, University of California Merced, wshadish@ucmerced.edu
|
| Abstract:
When randomized experiments cannot be used for ethical or practical reasons, nonrandomized experiments are often used instead. However, selection bias may result in doubt about bias of the resulting effect estimates. Various methods for adjusting those estimates have been proposed, including ordinary least squares covariate adjustment and propensity score analysis. This paper explore issues in the selection of covariates used in such adjustments. The paper will introduce a study that randomly assigned participants to be in a randomized or nonrandomized experiment, and then used various adjustments to the nonrandomized experiment to reproduce the results from the randomized experiment. The paper then explores various features of the covariates to shed light on how one might choose better covariates for such analyses.
|
|
A Comparison of Instrumental Variables and Propensity Score Approaches to Adjusting Group Treatment Selection Bias
|
| Presenter(s):
|
| Ning Rui, Research for Better Schools, rui@rbs.org
|
| Abstract:
One of the key challenges in identifying the causal effect of a group treatment has been to effectively control for group selection bias in order to properly estimate the treatment effect. Scholars have gained tremendous insights into the use of propensity score matching (PS) and instrumental variables (IV) strategies to address selection bias in observational studies. However, little is known about whether PS and IV methods are likely to generate similar estimates of treatment effects. The present study aims to compare both approaches to adjusting for selection bias in estimating the effect of community colleges on overall educational attainment using the High School and Beyond (HS&B) data. Both approaches agreed on the direction but differed substantially on the size of estimates for community college diversion effect on educational attainment. Two methods' relative advantages and disadvantages are discussed and suggestions for future studies are provided.
|
|
Use Resampling to Estimate and Compare the Selection Bias of Propensity Score Matching Techniques
|
| Presenter(s):
|
| Haiyan Bai, University of Central Florida, hbai@mail.ucf.edu
|
| Wei Pan, University of Cincinnati, wei.pan@uc.edu
|
| Abstract:
Propensity score matching has become increasingly popular in program evaluation when randomization is improbable. However, selection bias could still exist in the extant propensity score matching techniques. This present study utilizes resampling to estimate and compare the bias of the four types of commonly-used propensity score matching techniques: Nearest neighborhood, caliper, stratification, and kernel. Monte Carlo simulation study is conducted with three sample sizes (small, medium, and large) across the following four conditions: (a) without resampling; and (b) with resampling implemented (b1) before propensity score estimation, (b2) between propensity score estimation and matching, and (b3) after propensity score matching. The standard errors and confidence intervals for the treatment effects are compared on the totally 48(4x3x4) different simulation conditions. This study would provide researchers scientific advice on which matching technique performs the best under what data conditions and when to use resampling to obtain less biased estimates of the treatment effects.
|
| | | |