Evaluation Leaders Decry Department of Educationís Proposed Evaluation Methods


Leaders in the American Evaluation Association (AEA), a national association representing 3700 evaluation professionals, raised concerns this week about the U.S. Department of Educationís proposal to give preference to certain methods over others for examining the efficacy of federally funded educational programs. In response to a recent Department of Education invitation to comment in the Federal Register, the AEA issued a public statement indicating concern that applying the proposed methods would result in ďfruitless expenditures on some large contracts while leaving other public programs unevaluated entirely.Ē


The AEA statement, developed by a team of evaluation professionals headed by Dr. Linda Mabry of Washington State University Vancouver, and endorsed by the 2002, 2003, 2004, and 2005 presidents of the association, called into question the Department of Educationís privileging of randomized control group trials (RCTs) over other evaluation methods. RCTs randomly assign students - or classrooms, schools, or other units of analysis - either to participate in a particular program, such as a new literacy curriculum, or to serve as a comparison by participating in a different program or by maintaining the status quo. The aim of RCTs is to limit the influence of influences, except the program, so that if changes are observed they can be attributed to the program and not to other possible causes.


While RCTs are useful in some situations, they can also be impractical and expensive, and may return questionable results when used in schools and classrooms. In actuality, children, teachers, and schools are affected by many influences such as poverty, language, ethnicity, learning outside the classroom, parent education and involvement, and school and district budgets. In order to identify whether, how, and why a program works, an evaluator needs draw upon a range of social science methods to be able to examine many factors that may contribute to a programís success or failure. The preferred methods advocated by the Department of Education limit the ability of evaluators to use the best approach as dictated by the unique context of each educational setting.


Author Linda Mabry pointed to the ethical concerns highlighted by the AEA statement. "One of the reasons educational evaluation has moved beyond [the use of RCTs], while not dismissing them entirely, is that very few educators consider it equitable to refuse participation to children who might be helped by a program," Mabry noted, "or to require their participation in an unproven program that takes them away from needed instruction.Ē Federal law requires that review boards weight potential risks and benefits to study participants, and studies that compromise student welfare are prohibited. Ultimately, the Department of Education recommendations may be in conflict with the federal laws that protect students.


The American Evaluation Association is committed to fostering quality education. It encourages all partners in the educational experience to use sound evaluation practices to document accountability and to increase educational effectiveness and efficiency. In some situations, randomized control group trials will be feasible and in those cases such methods ought to be considered. But in other situations, the evaluation ought to include interviews, observations, case studies, surveys and other strategies to understand causality and to provide the information needed to improve the educational experience.  


More information about the proposed evaluation standards may be found on the Department of Educationís website at More information on the American Evaluation Association may be found at



Contacts for this story include:


American Evaluation Association, 2003 President

Richard Krueger, University of Minnesota

Phone: 612-624-6754, Email:


American Evaluation Association, Chair drafting committee for DOE response

Linda Mabry, Washington State University Vancouver

Phone: 360-546-9428, Email:




Click here to access the AEA Statement

Click here to access the Request for Comment in the Federal Register
Click here to access the AEA cover letter to the Dept of Education
Click here to access the AERA response to the Request for Comment

Click here to access the NEA Response as a pdf file