|
Session Title: Standard Setting in Systematic Reviews: Examples From the What Works Clearinghouse
|
|
Panel Session 645 to be held in Santa Monica on Friday, Nov 4, 10:45 AM to 11:30 AM
|
|
Sponsored by the Quantitative Methods: Theory and Design TIG
|
| Chair(s): |
| Russell Gersten, University of Oregon, rgersten@inresg.org
|
| Discussant(s):
|
| Russell Gersten, University of Oregon, rgersten@inresg.org
|
| Abstract:
Although no simple answers are available to questions about how well programs work, systematic reviews are increasingly being used in health, education, and other arenas to make informed decisions about effectiveness in the current era of accountability. The What Works Clearinghouse, developed in 2002 to provide educators, policymakers, and the public with an easily accessible repository of information on what works in educating students, is a central and trusted source of scientific evidence for what works in education. Using a set of transparent design standards developed by experts in the field, the WWC reviews and summarizes research on the effectiveness of education programs, products, and practices. The panel will review how the WWC standards were created, how they are applied consistently, what it means to meet standards, and how evaluators can navigate common challenges.
|
|
A Short History of the WWC and its Review Standards
|
| Jill Constantine, Mathematica Policy Research, jconstantine@mathematica-mpr.com
|
|
Using a set of standards based on scientifically valid criteria, the WWC evaluates education research. To be effective, WWC reviews must be based not only on rigorous standards but also on consistent procedures to determine which research to include and to rate the literature and synthesize findings. Recognizing that randomized controlled trials are not feasible in all contexts, the WWC has developed standards for reviewing RCTs as well as quasiexperimental designs. The standards adhere to three principles-they are (1) exhaustive, (2) inclusive, and (3) well documented. The presentation will provide a description of the considerations in developing standards and the standards development process, an overview of the WWC standards, what it means to meet them, and how attrition standards were developed. We will also describe how reviews of research against the standards are consistently implemented.
|
|
|
Using Standards in Systematic Reviews
|
| Neil Seftor, What Works Clearinghouse, nseftor@mathematica-mpr.com
|
|
The core of the WWC is research design standards to identify studies with the strongest causal validity. This presentation will cover the design and reporting requirements researchers must demonstrate to meet WWC standards and discuss how to use the standards to do a systematic review. It will also look at common challenges that arise in applying standards consistently and describe statistical adjustments made by the WWC to ensure findings can be compared across studies.
| |