|
Session Title: The Randomized Controlled Trial in Your Evaluation Toolkit: A Candid Discussion
|
|
Panel Session 624 to be held in Room 104 in the Convention Center on Friday, Nov 7, 1:35 PM to 3:05 PM
|
|
Sponsored by the Human Services Evaluation TIG
|
| Chair(s): |
| Jennifer Hamilton,
Westat,
jenniferhamilton@westat.com
|
| Abstract:
The randomized controlled train (RCT) is widely considered the optimal study design to minimize bias and provide the most accurate estimate of the impact of a given intervention or program. However, the design and implementation of an RTC presents a unique set of challenges. In fact, without the proper attention, a researcher may unintentionally limit the study's internal validity (the extent to which the difference between the treatment and control groups are real rather than a product of bias) or its external validity (generalizability to a wider population). Therefore, this panel is intended to raise awareness of these issues and to provide a frank discussion of possible solutions.
|
|
The Pros and Cons of a Cluster Randomized Trial
|
| Jennifer Hamilton,
Westat,
jenniferhamilton@westat.com
|
|
This presentation will use an evaluation of Newark, New Jersey's Striving Readers program, which assesses the impact of Scholastic's Read 180 curriculum, to illustrate the trade offs between randomizing at the group level versus randomizing at the level of the individual. We will share our research design along with the benefits of group level randomization, such as cost, avoidance of many control group contamination issues, ease of implementation and subgroup analysis.
However, in a cluster randomized trial, the number of randomized entities is much smaller than when individuals are randomized. This results in reduced power to detect treatment effects. This presentation will therefore go on to discuss several of the options for increasing power in cluster randomized trial such as:
- Increasing number of individuals within groups and the number of groups,
- Adding covariates to the model, and
- Stratifying groups prior to randomization
|
|
|
Student Attrition Over Time in a National Evaluation of Head Start Services
|
| Janet Friedman,
Westat,
janetfriedman@westat.com
|
|
This presentation will use the first comprehensive impact study of Head Start to illustrate how to reduce another threat to the internal validity of a RCT'participant attrition over time. This National Head Start Impact Study (HSIS) quantifies the impact of Head Start on 3-and 4-year-old children across child cognitive, social-emotional, and health domains as well as on parenting outcomes. Children were randomly assigned to either a Head Start group that had access to Head Start programs or to a non-Head Start group that could enroll in available community non-Head Start programs, selected by their parents. Presently the children are being followed through third grade.
This paper presents successful approaches to implementing a nationally representative RCT and provides practical advice for tracking participants over time so as to minimize attrition as well as strategies for minimizing control group contamination.
| |
|
Assuring Fidelity of the Treatment and the Generalizability of Findings in a Study of Mental Health Treatment
|
| Susan Azrin,
Westat,
susanazrin@westat.com
|
|
The Mental Health Treatment study sponsored by the Social Security Administration, is the largest study to evaluate Federal policy around adults with psychiatric disabilities trying to return to work. This evaluation will be used to illustrate issues regarding measuring the implementation of the treatment as well as the generalizability of findings.
In this RCT intent-to-treat design, 2,000 unemployed adults receiving disability benefits for a psychiatric disability were randomized to the treatment condition; an intervention including evidence-based supported employment and mental health services, medication management, and supplemental health insurance.
The presentation will focus on how this study incorporates unique quality management features at multiple levels to achieve and maintain high implementation fidelity. Fidelity here refers to the degree to which the intervention implementation is faithful to the practice model that research has shown effective. Achieving both high fidelity and generalizablity has proven challenging
| |
|
Confronting Threats to Validity in a Study of Alcohol Risk Prevention in a National Fraternity
|
| Scott Crosse,
Westat,
scottcrosse@westat.com
|
|
This randomized trial of alcohol risk reduction interventions will be used to describe efforts to compensate for cluster-level attrition early in the study. Ninety-eight chapters of a national college fraternity were randomly assigned to receive a standard practices intervention (waiting-list control group condition), a standard intervention (3-hour alcohol risk reduction training), or an enhanced intervention (baseline training plus two 90-minute risk reduction booster training sessions).
During the several months that transpired between randomly assigning chapters to conditions and the start of baseline data collection, the study team learned that five chapters were closing or disaffiliating themselves from the national fraternity. While the attrition occurred prior to notifying chapters about the study, it still raised concerns about internal validity. The proposed paper will discuss options for dealing with this scenario, as well as the one chosen by the study team and its impact on group equivalence at baseline.
| |