|
Session Title: Taking Stock of the Quality of Evaluation Research on School-Based Prevention Programs
|
|
Panel Session 721 to be held in PRESIDIO C on Saturday, Nov 13, 8:00 AM to 9:30 AM
|
|
Sponsored by the Alcohol, Drug Abuse, and Mental Health TIG
|
| Chair(s): |
| Scott Crosse, Westat, scottcrosse@westat.com
|
| Abstract:
The proposed session will take a high level look at the quality of evaluation research on school-based prevention, examine the relationship between aspects of quality and measures of effect, and consider the overall breadth and depth of evidence on school-based prevention. This session draws on the results of an extensive systematic review of evaluation literature on school-based programs intended to prevent youth substance abuse and school crime. To identify effective or “research-based” programs, the review assessed the quality of research methods of studies, and synthesized the results on effectiveness of methodologically strong studies. The presenters consider issues bearing on: (a) criteria for and process of assessing study quality; (b) overall quality of evaluation research on school-based programs and how quality has changed over time; (c) study characteristics, including aspects of methods, that matter most for program outcomes; and (d) overall state of evaluation research on school-based prevention.
|
|
The Methodological Quality of Evaluation Research on School-based Prevention Programs: Where Are We Now?
|
| Carol Hagen, Westat, carolhagen@westat.com
|
| Scott Crosse, Westat, scottcrosse@westat.com
|
| Michele Harmon, Westat, micheleharmon@westat.com
|
| Samantha Leaf, ISA Associates, sleaf@isagroup.com
|
| Rebekah Hersch, ISA Associates, rhersch@isagroup.com
|
|
Funding organizations and other stakeholders continue to embrace the idea that school-based prevention programming should be research based. Nearly five years ago, we assessed the methodological quality of evaluation research on the effectiveness of school-based youth ATOD and school crime prevention programs. and found that the vast majority of published evaluation research examined failed to meet standard criteria for acceptable methodological quality. For the current study, we extended our systematic review of literature to incorporate more recent literature from 2004 to 2008, which included both published and unpublished evaluations research. We screened over 8000 document abstracts and assessed the methodological quality of nearly 900 documents across 91 school-based prevention programs relevant to our study. Despite the inclusion of more recent and unpublished evaluation literature and a slightly modified methodology for assessing quality, our findings changed little from our previous systematic review.
|
|
|
Main Effects and Moderating Influences Among ‘Effective’ School-based Programs
|
| Aaron Alford, Battelle Memorial Institute, alforda@battelle.org
|
| Jim Derzon, Battelle Memorial Institute, derzonj@battelle.org
|
|
Estimates of intervention effectiveness result from the true effect of the program plus intervention- and study-introduced bias. To identify these sources of bias, a systematic review of meta-analyses documenting consequences of intervention and researcher choices was conducted. Identified influences include intervention characteristics (e.g., implementation and fidelity, duration, and interactivity); population characteristics (e.g., risk status, household status, and sample age); and study-introduced influences (e.g., measurement characteristics, attrition, and compensatory services). The influence of these moderators on effectiveness estimates assessing substance use and antisocial behavior from programs of documented effectiveness was tested using meta-regression. Although programs are generally effective in improving youth outcomes, all outcomes, other than tobacco use, varied significantly with type of intervention administrator and the presence of uncontrolled variation in implementation. Intervention duration influenced, somewhat, the effectiveness of intervention evaluations targeting smoking. These results reinforce the observations that programmatic choices influence the documented effectiveness of school-based interventions.
| |
|
Breadth and Depth of Evidence: Documenting Program Effectiveness Across a Range of Behavioral Outcomes
|
| Jim Derzon, Battelle Memorial Institute, derzonj@battelle.org
|
| Aaron Alford, Battelle Memorial Institute, alforda@battelle.org
|
|
To document the impact of available manualized programs on multiple substance use and violence outcomes, we coded evidence of program effectiveness from programs supported by multiple designed studies or implementations. Studies were screened on methodological rigor and evidence for program effectiveness was coded for eight outcomes. Of the 491 programs identified by canvassing the literature, only 46 were manualized, addressed one or more of our outcomes and were supported by multiple studies or implementations. Of these, 25 had no effect, negative effects, or mixed effects while 21 were effective for at least one outcome. Of the 21 programs reporting data for multiple outcomes, eight exhibited a significant positive effect on multiple outcomes. Only three programs reported significant positive findings across all reported outcomes. Interestingly, within reported outcomes program results are often mixed. This presentation will introduce these findings and is intended to stimulate discussion of their meaning and interpretation.
| |