| Session Title: A Comparison of Bias Reduction Methods |
| Multipaper Session 140 to be held in Santa Monica on Wednesday, Nov 2, 4:30 PM to 6:00 PM |
| Sponsored by the Quantitative Methods: Theory and Design TIG |
| Chair(s): |
| Laura Peck, Abt Associates Inc, Laura_Peck@abtassoc.com |
| Discussant(s): |
| Furio Camillo, University of Bologna, furio.camillo@unibo.it |
| Abstract: While researchers use a variety of methods to estimate treatment effects in quasi-experimental designs, propensity score adjustments have become increasingly popular. Although they can be effective in reducing selection bias, they are not without their limitations. To improve statistical estimates from non-randomized studies, researchers modify propensity score methods and develop new methods. Recently, Peck, Camillo, and D'Attoma (2010) proposed a new method to reduce selection bias in quasi-experiments using an innovative clustering approach to create balanced treatment-comparison groups. Bias and treatment effects are assessed within each cluster permitting comparisons of equivalent groups without compromising the factorial structure of the data. As with any new statistical method, the popularity of this method depends on its effectiveness when compared to existing methods. Therefore, this panel will compare bias reduction between their new method and propensity score matching across three applications. |
| A Comparison of Bias Reduction Methods on Educational Outcomes |
| M H Clark, University of Central Florida, mhclark@mail.ucf.edu |
| This study compares bias reduction in quasi-experiments between Peck, Camillo, and D'Attoma's (2010) method and propensity score matching (PSM) using data from a non-randomized evaluation of a first-year college orientation seminar. Clark and Cundiff (2011) conducted an evaluation of a first-year seminar in which participants self-selected into the course. Without any statistical adjustment, program participants had lower first-year GPAs than non-participants. However, of the 25 covariates measured, 10 were not balanced between the treatment and control groups, suggesting that the treatment effect was biased. After matching cases on propensity scores, the negative effect on GPA that was originally found was nullified. While treatment groups were balanced reasonably well, nearly half of the treatment cases were dropped because they did not have suitable matches. Using the same covariates, I will examine the extent to which similar or better bias reduction and case retention occurs using Peck et al.'s clustering method. |
| How Does Welfare Use Affect Charitable Activity? A Tale of Two Methods |
| Laura Peck, Abt Associates Inc, Laura_Peck@abtassoc.com |
| Motivated by Brooks (2002, 2004), prior work (Guo & Peck, 2009) analyzed the extent to which welfare recipients engage in giving money and time to charitable causes. Using the 2003 Center on Philanthropy Panel Study (COPPS) data, we used a difference-in-difference approach to overcome sticky issues of selection bias and found evidence that public assistance receipt tends to suppress monetary donations but may increase volunteer time. The proposed paper updates that work, using the 2005 COPPS data, and applying two other methods to address the same question. Specifically, we use (1) a propensity score matching approach, and (2) a multivariate cluster-based approach to generate useful comparison groups of non-welfare-recipients to estimate the effect of welfare on charity-related outcomes. We explicitly compare the effectiveness of these approaches in reducing selection bias, in an attempt to provide methodological guidance to analysts similarly vexed by selection bias. |
| Subjective Choices in Propensity Score Applications: A Comparison With the Multivariate Cluster-based (MCB) Method |
| Ida D'Attoma, University of Bologna, ida.dattoma2@unibo.it |
| Furio Camillo, University of Bologna, furio.camillo@unibo.it |
| This study evaluates the performance of the Multivariate Cluster Based (MCB) method (Peck, Camillo & D'Attoma, 2010) in terms of bias reduction, case retention, achieved balance and subjective judgment reduction compared to Propensity Score (PS) Stratification using simulated data. Because propensity score requires subjective choices about model specification, we expect that MCB will involve less subjective judgment in developing balance between treatment and comparison cases. The analysis applies a global imbalance (GI) measure introduced in prior work to simultaneously examine balance in pre-treatment categorical covariates. We assert that the GI measure is objective because of its consideration of variance observed in the data along baseline case and the treatment assignment variable. We expect to find better bias reduction and hope to find at least similar case retention using the MCB method compared to conventional propensity score matching. |