|
Session Title: Evaluation Adjustments for Imperfect Fidelity in the Implementation of Social Experiments
|
|
Panel Session 312 to be held in Suwannee 15 on Thursday, Nov 12, 1:40 PM to 3:10 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Cristofer Price, Abt Associates Inc, cristofer_price@abtassoc.com
|
| Discussant(s):
|
| Barbara Goodson, Abt Associates Inc, barbara_goodson@abtassoc.com
|
| Abstract:
In 2005, the U.S. Department of Education awarded eight Striving Readers grants to projects around the country. The goal of Striving Readers is to improve literacy skills and achievement for struggling readers. This panel includes insights from several of the research teams involved in evaluating the reading programs implemented in grantee schools. Specifically, the panel focuses on the adjustments made by the evaluation teams in response to various implementation fidelity shortcomings in the field. Presenters will discuss the types of fidelity issues faced in their respective evaluation sites, efforts made to correct these issues during the course of the evaluations, and techniques utilized during data analysis to correct for potential biases introduced by the fidelity issues. By combining experimental data with quasi-experimental statistical methods, evaluators were able to leverage more information from their study designs than otherwise would have been possible.
|
|
Enhancing the Precision of Program Evaluation Results by Converting Intention-to-Treat Estimates to Impact-on-the-Treated Estimates: Striving Readers in Newark, New Jersey
|
| Jennifer Hamilton, Westat, jenniferhamilton@westat.com
|
|
The READ 180 literacy curriculum, implemented in Newark, NJ as part of the US Department of Education's Striving Readers grant program, is being evaluated using a cluster randomized trial design. In the evaluation 19 schools have been randomly assigned to the treatment (READ 180; 10 schools) or control (standard literacy curriculum; 9 schools) groups. At issue in the evaluation is the fact that over 10% of the students assigned to treatment schools never received the READ 180 curriculum. As a result, the study has focused on the intention-to-treat effect of the program. However, program officers would like information on the effect of the program for those students who actually received the treatment. To provide such an estimate, the Bloom adjustment was utilized to convert intention-to-treat estimates into impact-on-the-treated estimates. The adjustment allows the evaluation team to account for the high mobility context within which READ 180 operates.
|
|
|
What Matters Most: Relating READ 180 Inputs to Student Outcomes
|
| Deb Coffey, Research for Better Schools, coffey@rbs.org
|
|
The randomized control trial of Scholastic's READ 180 in the Memphis Striving Readers Project showed no difference in impact during the first two years of implementation. However, various local issues and challenges led to substantial differences in the fidelity of implementation across schools in the study. At what point do we decide between "the program doesn't work for this population" and "the intervention wasn't implemented as planned and fidelity of implementation should be improved before making judgments about the program"?
Evaluators of the MSRP targeted intervention collected implementation data related to a variety of inputs in 19 classrooms in eight schools. These data are used in multiple regression analyses to determine which inputs contribute most to student outcomes. Results can be used to inform additional implementation analyses and provide recommendations to the district about where to focus improvement efforts.
| |
|
Using Fidelity Data in Impact Estimates
|
| Bonnie Faddis, RMC Research Corporation, bfaddis@rmccorp.com
|
| Margaret Beam, RMC Research Corporation, mbeam@rmccorp.com
|
|
As part of its Striving Readers grant, Portland Public schools implemented the Xtreme Reading program (developed by the Center for Research on Learning at the University of Kansas) in Grades 7-10 at 10 low achieving middle and high schools. Students were randomly assigned to treatment and control conditions within schools and grade levels. To measure fidelity of implementation, the evaluation team collected data through classroom visits, teacher surveys, and teacher logs of classroom activities, and also quantified teacher characteristics, group professional development hours, school literacy coach assistance, and professional developer assistance for each teacher. Additional data on implementation issues were collected through interviews with professional developers, district staff, school literacy coaches, and school administrators. This presentation will summarize the district's implementation fidelity issues and the evaluation team's efforts to incorporate fidelity data into the impact analyses.
| |
|
Implications for Evaluation: Barriers to Implementation Fidelity in a Randomized Controlled Trail
|
| Kimberly Sprague, Brown University, kimberley_sprague@brown.edu
|
|
The Education Alliance at Brown University is conducting a randomized controlled trial (RCT) evaluating the effectiveness of two literacy interventions on the reading achievement of students identified as "struggling readers." Adjustments to the evaluation design have resulted two forms of implementation barriers. Structural barriers include "unknowns" at the time of proposed plans and relate to "on-the-ground" operational issues including district operational challenges, administrative, school, and teacher communications, general school configuration and scheduling feasible for implementation as planned, hiring and allocation of teachers as well as intervention changes/tailoring for implementation and general issues around screening and placement as well as data documentation and tracking. Classroom implementation barriers include those related to information available from developers regarding model components by which measures can be developed to indicate fidelity, ways in which to weight components to identify levels of implementation, and what how best to indicate fidelity of implementation in general.
| |