|
FOR
IMMEDIATE RELEASE, December 3, 2003 Evaluation
Leaders Decry Department of Education’s Proposed Evaluation Methods Leaders in the American Evaluation Association (AEA), a national association representing 3700 evaluation professionals, raised concerns this week about the U.S. Department of Education’s proposal to give preference to certain methods over others for examining the efficacy of federally funded educational programs. In response to a recent Department of Education invitation to comment in the Federal Register, the AEA issued a public statement indicating concern that applying the proposed methods would result in “fruitless expenditures on some large contracts while leaving other public programs unevaluated entirely.” The AEA statement, developed by a team of evaluation professionals headed by Dr. Linda Mabry of Washington State University Vancouver, and endorsed by the 2002, 2003, 2004, and 2005 presidents of the association, called into question the Department of Education’s privileging of randomized control group trials (RCTs) over other evaluation methods. RCTs randomly assign students - or classrooms, schools, or other units of analysis - either to participate in a particular program, such as a new literacy curriculum, or to serve as a comparison by participating in a different program or by maintaining the status quo. The aim of RCTs is to limit the influence of influences, except the program, so that if changes are observed they can be attributed to the program and not to other possible causes. While
RCTs are useful in some situations, they can also be impractical and
expensive, and may return questionable results when used in schools and
classrooms. In actuality, children,
teachers, and schools are affected by many influences such as poverty,
language, ethnicity, learning outside the classroom, parent education
and involvement, and school and district budgets. In order to identify
whether, how, and why a program works, an evaluator needs draw upon a
range of social science methods to be able to examine many factors that
may contribute to a program’s success or failure. The preferred
methods advocated by the Department of Education limit the ability of
evaluators to use the best approach as dictated by the unique context of
each educational setting. Author
Linda Mabry pointed to the ethical concerns highlighted by the AEA
statement. "One of the reasons educational evaluation has moved
beyond [the use of RCTs], while not dismissing them entirely, is that
very few educators consider it equitable to refuse participation to
children who might be helped by a program," Mabry noted, "or
to require their participation in an unproven program that takes them
away from needed instruction.” Federal law requires that review boards
weight potential risks and benefits to study participants, and studies
that compromise student welfare are prohibited. Ultimately, the
Department of Education recommendations may be in conflict with the
federal laws that protect students. The
American Evaluation Association is committed to fostering quality
education. It encourages all partners in the educational experience to
use sound evaluation practices to document accountability and to
increase educational effectiveness and efficiency. In some situations,
randomized control group trials will be feasible and in those cases such
methods ought to be considered. But in other situations, the evaluation
ought to include interviews, observations, case studies, surveys and
other strategies to understand causality and to provide the information
needed to improve the educational experience.
More information about the proposed evaluation standards may be found on the Department of Education’s website at www.ed.gov. More information on the American Evaluation Association may be found at www.eval.org. Contacts for this story include: American Evaluation Association, 2003 President Richard Krueger, University of Minnesota Phone: 612-624-6754, Email: rkrueger@umn.edu American Evaluation Association, Chair drafting committee
for DOE response Linda Mabry, Washington State University Vancouver Phone: 360-546-9428, Email: mabryl@vancouver.wsu.edu
Click here to access the AEA Statement Click
here to access the Request for Comment in the Federal Register Click here to access the NEA Response as a pdf file
|