|
Session Title: Contextual Issues in a Randomized Control Group Evaluation of a School-based Intervention: Fielding an Evidence-based Intervention to Reduce Youth Gun Violence in Chicago
|
|
Panel Session 285 to be held in Lone Star D on Thursday, Nov 11, 1:40 PM to 3:10 PM
|
|
Sponsored by the Non-profit and Foundations Evaluation TIG
|
| Chair(s): |
| Wendy Fine, Youth Guidance, wfine@youth-guidance.org
|
| Abstract:
This panel will explore how a promising youth development program is being evaluated, with the goal of utilizing the results to establish an evidence-based intervention that reduces youth gun violence, a major problem affecting many American cities. Panelists will discuss the different stakeholder contexts that have left their mark on the evaluation process: a) mobilizing a group of civic-minded funders for a large scale experimental evaluation; b) establishing program stakeholder buy-in for experimental evaluation design; and c) the evaluation’s impact on program model implementation by collaborating nonprofits. Panel members will highlight challenges and key decisions made along the way, such as 1) the selection process used to identify the most promising program for the evaluation; 2) how risk indices were developed to identify the target student population – a politically sensitive issue; and 3) the approach of the service provider in maintaining schools’ support for participation.
|
|
Overcoming the Odds: Carrying Out a Large Scale Randomized Control Evaluation of a Promising Youth Violence Prevention Program in Chicago
|
| Roseanna Ander, University of Chicago, rander@uchicago.edu
|
|
Randomized control evaluations in areas of social policy such as youth violence prevention remain very rare. As a consequence there is very little gold standard, "clinical trial" evidence about what works, for whom and under what circumstances despite the fact that youth violence, like medicine, is an area where lives are at stake. From a funder’s perspective Roseanna Ander will provide a broad overview of how Becoming A Man - Sports Edition, which combines life skills building, cognitive behavioral therapy (CBT), and mentoring with sports involvement, became the largest scale randomized control evaluation of Cognitive Behavioral Therapy (CBT) to be carried out among youth in an urban area. She will discuss the roles the foundation community, policy makers, academia, the media, the non-profit sector and the presiding juvenile court judge played in bringing the project to fruition.
|
|
|
The Evaluator’s Context: Challenges to the Design and Implementation of an Experimental School-based Intervention
|
| Harold Pollack, University of Chicago, haroldp@uchicago.edu
|
|
Harold Pollack will describe the construction of the sample frame used to evaluate Becoming A Man – Sports Edition. He will describe the use of risk indices in defining the group of students served by the intervention, and algorithms to randomly assign 45 treatment groups and suitably balanced control groups in 15 schools across Chicago. He will describe the value and limitations of administrative data for program assignment. He will also describe the role and limitations of power calculations in the design of a complex intervention within an intent-to-treat framework. Given this statistical design, he will characterize refusal and dropout rates in the different treatment arms based on prior student characteristics available in administrative data. Finally, he will describe from a university researcher perspective the practical obstacles and challenges of a complex collaboration involving researchers, public schools, and social service providers.
| |
|
The Organization’s Context: Evaluation Design’s Affect on Program Implementation, Ethics and Evaluation Utility
|
| Wendy Fine, Youth Guidance, wfine@youth-guidance.org
|
|
Youth Guidance is an urban school-based nonprofit, which is accustomed to conducting internal evaluations for its various programs for quality improvement and accountability to funders. One of Youth Guidance’s programs, Becoming A Man (BAM) – Sports Edition, was selected by a panel of experts as a promising practice for addressing youth gun violence. The program is now undergoing a large-scale evaluation with the goal of developing it as an evidence-based intervention that can be replicated in at-risk communities nationally. From the nonprofit’s perspective, Wendy Fine will describe how program managers, internal evaluators, practitioners, and schools have adapted to the demands of a randomized control group design. Several questions will be addressed, including 1) what are the limits of changes to a program model to accommodate evaluation design and related implications for program replication, and 2) how does an organization address the ethical dilemma posed by having a control group.
| |