|
Lessons Learned from the Implementation of Randomized Trials in the Field of Education
|
| Presenter(s):
|
| Jessaca Spybrook, Western Michigan University, jessaca.spybrook@wmich.edu
|
| Anne Cullen, Western Michigan University, anne.cullen@wmich.edu
|
| Monica Lininger, Western Michigan University, monica.lininger@wmich.edu
|
| Abstract:
Group randomized trials as means for establishing “what works” in education have become quite common in the field. For example, between 2002 and 2006, the Institute of Education Sciences (IES), the research branch of the US Department of Education, funded 55 studies which involved a group randomized trial, or a study in which classrooms, schools, or entire districts were randomly assigned to either receive the treatment or control condition. The current study examines the challenges research teams face when implementing group randomized trials in the field. The study is based on interviews with 35 Principal Investigators of group randomized trials funded by IES. The primary challenges researchers faced included recruitment of the sample and attrition of students and teachers. This session will discuss the lessons learned from these studies and is geared toward evaluators interested in improving the planning and implementation of group randomized trials.
|
|
Effectiveness of Selected Supplemental Reading Comprehension Interventions: Impacts on Two Cohorts of Fifth-Grade Students
|
| Presenter(s):
|
| Susanne James-Burdumy, Mathematica Policy Research, sjames-burdumy@mathematica-mpr.com
|
| John Deke, Mathematica Policy Research, jdeke@mathematica-mpr.com
|
| Julieta Lugo-Gil, Mathematica Policy Research, jlugo-gil@mathematica-mpr.com
|
| Abstract:
The National Evaluation of Reading Comprehension Interventions is testing the effectiveness of 4 reading comprehension interventions in 10 school districts and 89 schools across the United States. The study is based on a rigorous experimental design that involved randomly assigning schools to one of four interventions or to a control group. We administered reading tests to fifth-grade students at baseline and follow up, collected school records data, conducted surveys of teachers, and observed classroom instruction. Over 6,000 students and 250 teachers were included in the study’s first cohort and over 4,000 students and 180 teachers were included in the study’s second cohort. The presentation will focus on the study’s second year of findings, after the reading comprehension interventions had been implemented for one school year with both cohorts of students. For the first cohort, we will also present findings one year after the end of the intervention implementation.
|
|
Rigor Is Only Half the Story: Design Decisions and Context Influences in a Cluster-Randomized Evaluation Study
|
| Presenter(s):
|
| Andrea Beesley, Mid-continent Research for Education and Learning, abeesley@mcrel.org
|
| Abstract:
This presentation will describe the design and methodology of a two-year cluster-randomized effectiveness evaluation of a teacher professional development program in classroom assessment, and then the effects (intended and otherwise) of both the design decisions and the implementation context on the study. It will include discussions of: considerations for doing experimental research in school settings; consequences of random assignment of intact groups; challenges with multiyear implementations and experimental evaluation studies of interventions; strategies for integrating student and teacher outcomes; and issues with assessing implementation fidelity in an effectiveness evaluation. The presentation will detail the rigorous design, but also highlight the elements inherent in the school environment (context) that arguably had as much effect on the study outcomes.
|
|
Comparative Advantage in Teaching: A Randomized Experiment
|
| Presenter(s):
|
| Steven Glazerman, Mathematica Policy Research, sglazerman@mathematica-mpr.com
|
| Jeffrey Max, Mathematica Policy Research, jmax@mathematica-mpr.com
|
| Ali Protik, Mathematica Policy Research, aprotik@mathematica-mpr.com
|
| Abstract:
In this paper, we examine if high performing teachers from higher achieving schools can transfer their skills to more challenging settings when they are put in low achieving schools. In ten diverse school districts throughout the country, we identified the top 20 percent of elementary, middle school math, and English teachers and offered them $20,000 to transfer to a pool of low performing schools with teaching vacancies in the targeted grades and subjects. We randomly assigned half of these schools to a treatment group eligible for hiring one of the top-tier teachers and the other half to a control group that would fill their vacancies the way they normally would. We focus on the direct impact of high performing teachers on their own students in their new settings relative to the new hires in the control group, indirect impacts on their peers to measure spillover and within-school distribution effects.
|
| | | |