[Federal Register: November 4, 2003 (Volume 68, Number 213)]


[Page 62445-62447]

From the Federal Register Online via GPO Access [wais.access.gpo.gov]







RIN 1890-ZA00



Scientifically Based Evaluation Methods


AGENCY: Department of Education.


ACTION: Notice of proposed priority.




SUMMARY: The Secretary of Education proposes a priority that may be 

used for any appropriate programs in the Department of Education 

(Department) in FY 2004 and in later years. We take this action to 

focus Federal financial assistance on expanding the number of programs 

and projects Department wide that are evaluated under rigorous 

scientifically based research methods in accordance with the Elementary 

and Secondary Education Act (ESEA) as reauthorized by the No Child Left 

Behind Act of 2001 (NCLB). Establishing the priority on a Department-

wide basis would permit any office to use the priority for a program 

for which it is appropriate.


DATES: We must receive your comments on or before December 4, 2003.


ADDRESSES: Address all comments about this proposed priority to Margo 

K. Anderson, U.S. Department of Education, 400 Maryland Avenue, SW., 

room 4W333, Washington, DC 20202-5910. If you prefer to send your 

comments through the Internet, use the following address: comments@ed.gov.

    You must include the term ``Evaluation'' in the subject line of 

your electronic message.


FOR FURTHER INFORMATION CONTACT: Margo Anderson. Telephone: (202) 205-3010 
or via Internet at Margo.Anderson@ed.gov.

    If you use a telecommunications device for the deaf (TDD), you may 

call the Federal Information Relay Service (FIRS) at 1-800-877-8339.

    Individuals with disabilities may obtain this document in an 

alternative format (e.g., Braille, large print, audiotape, or computer 

diskette) on request to the contact person listed under FOR FURTHER 





Invitation To Comment


    We invite you to submit comments regarding this proposed priority.

    We invite you to assist us in complying with the specific 

requirements of Executive Order 12866 and its overall requirement of 

reducing regulatory burden that might result from this proposed 

priority. Please let us know of any further opportunities we should 

take to reduce potential costs or


[[Page 62446]]


increase potential benefits while preserving the effective and 

efficient administration of the Department's programs.

    During and after the comment period, you may inspect all public 

comments about this proposed priority in room 4W333, 400 Maryland 

Avenue, SW., Washington, DC between the hours of 8:30 a.m. and 4 p.m., 

Eastern time, Monday through Friday of each week except Federal 



Assistance to Individuals With Disabilities in Reviewing the Rulemaking 



    On request, we will supply an appropriate aid, such as a reader or 

print magnifier, to an individual with a disability who needs 

assistance to review the comments or other documents in the public 

rulemaking record for this proposed priority. If you want to schedule 

an appointment for this type of aid, please contact the person listed 





    The ESEA as reauthorized by the NCLB uses the term scientifically 

based research more than 100 times in the context of evaluating 

programs to determine what works in education or ensuring that Federal 

funds are used to support activities and services that work. This 

proposed priority is intended to ensure that Federal funds are used to 

support projects and activities that are consistent with a statutory 

purpose of Department programs, and evaluated using scientifically 

based research. Establishing this priority makes it possible for any 

office in the Department to encourage or to require appropriate 

projects to use scientifically based evaluation strategies to determine 

the effectiveness of a project intervention.


Discussion of Proposed Priority


    We will announce the final priority in a notice in the Federal 

Register. We will determine the final priority after considering public 

comments on this proposed priority and other information available to 

the Department. This notice does not preclude the Secretary from 

proposing or funding additional priorities, subject to meeting 

applicable rulemaking requirements.


    Note: This notice does not solicit applications. In any year in 

which we choose to use this proposed priority, we invite 

applications for new awards under the applicable program through a 

notice in the Federal Register. When inviting applications we 

designate the priority as absolute, competitive preference, or 

invitational. The effect of each type of priority follows:

    Absolute priority: Under an absolute priority we consider only 

applications that meet the priority (34 CFR 75.105(c)(3)).

    Competitive preference priority: Under a competitive preference 

priority we give competitive preference to an application by either 

(1) awarding additional points, depending on how well or the extent 

to which the application meets the competitive preference priority 

(34 CFR 75.105(c)(2)(i)); or (2) selecting an application that meets 

the competitive priority over an application of comparable merit 

that does not meet the priority (34 CFR 75.105(c)(2)(ii)).

    Invitational priority: Under an invitational priority we are 

particularly interested in applications that meet the invitational 

priority. However, we do not give an application that meets the 

invitational priority a competitive or absolute preference over 

other applications (34 CFR 75.105(c)(1)).


Proposed Priority


    The Secretary proposes a priority for program projects proposing an 

evaluation plan that is based on rigorous scientifically based research 

methods to assess the effectiveness of a particular intervention. The 

Secretary intends that this priority will allow program participants 

and the Department to determine whether the project produces meaningful 

effects on student achievement or teacher performance.

    Evaluation methods using an experimental design are best for 

determining project effectiveness. Thus, the project should use an 

experimental design under which participants--e.g., students, teachers, 

classrooms, or schools--are randomly assigned to participate in the 

project activities being evaluated or to a control group that does not 

participate in the project activities being evaluated.

    If random assignment is not feasible, the project may use a quasi-

experimental design with carefully matched comparison conditions. This 

alternative design attempts to approximate a randomly assigned control 

group by matching participants--e.g., students, teachers, classrooms, 

or schools--with non-participants having similar pre-program 


    In cases where random assignment is not possible and an extended 

series of observations of the outcome of interest precedes and follows 

the introduction of a new program or practice, regression discontinuity 

designs may be employed.

    For projects that are focused on special populations in which 

sufficient numbers of participants are not available to support random 

assignment or matched comparison group designs, single-subject designs 

such as multiple baseline or treatment-reversal or interrupted time 

series that are capable of demonstrating causal relationships can be 


    Proposed evaluation strategies that use neither experimental 

designs with random assignment nor quasi-experimental designs using a 

matched comparison group nor regression discontinuity designs will not 

be considered responsive to the priority when sufficient numbers of 

participants are available to support these designs. Evaluation 

strategies that involve too small a number of participants to support 

group designs must be capable of demonstrating the causal effects of an 

intervention or program on those participants.

    The proposed evaluation plan must describe how the project 

evaluator will collect--before the project intervention commences and 

after it ends--valid and reliable data that measure the impact of 

participation in the program or in the comparison group.

    If the priority is used as a competitive preference priority, 

points awarded under this priority will be determined by the quality of 

the proposed evaluation method. In determining the quality of the 

evaluation method, we will consider the extent to which the applicant 

presents a feasible, credible plan that includes the following:

    (1) The type of design to be used (that is, random assignment or 

matched comparison). If matched comparison, include in the plan a 

discussion of why random assignment is not feasible.

    (2) Outcomes to be measured.

    (3) A discussion of how the applicant plans to assign students, 

teachers, classrooms, or schools to the project and control group or 

match them for comparison with other students, teachers, classrooms, or 


    (4) A proposed evaluator, preferably independent, with the 

necessary background and technical expertise to carry out the proposed 

evaluation. An independent evaluator does not have any authority over 

the project and is not involved in its implementation.

    In general, depending on the implemented program or project, under 

a competitive preference priority, random assignment evaluation methods 

will receive more points than matched comparison evaluation methods.


Executive Order 12866


    This notice of proposed priority has been reviewed in accordance 

with Executive Order 12866. Under the terms of the order, we have 

assessed the potential costs and benefits of this regulatory action.

    The potential costs associated with the notice of proposed priority 

are those


[[Page 62447]]


we have determined as necessary for administering applicable programs 

effectively and efficiently.

    In assessing the potential costs and benefits--both quantitative 

and qualitative--of this notice of proposed priority, we have 

determined that the benefits of the proposed priority justify the 


    We have also determined that this regulatory action does not unduly 

interfere with State, local, and tribal governments in the exercise of 

their governmental functions.


Intergovernmental Review


    Some of the programs affected by this proposed priority are subject 

to Executive Order 12372 and the regulations in 34 CFR part 79. One of 

the objectives of the Executive order is to foster an intergovernmental 

partnership and a strengthened federalism. The Executive order relies 

on processes developed by State and local governments for coordination 

and review of proposed Federal financial assistance.

    This document provides early notification of our specific plans and 

actions for these programs.


Electronic Access to This Document


    You may view this document, as well as all other Department of 

Education documents published in the Federal Register, in text or Adobe 

Portable Document Format (PDF) on the Internet at the following site: 


    To use PDF you must have Adobe Acrobat Reader, which is available 

free at this site. If you have questions about using PDF, call the U.S. 

Government Printing Office (GPO), toll free, at 1-888-293-6498; or in 

the Washington, DC, area at (202) 512-1530.


    Note: The official version of this document is the document 

published in the Federal Register. Free Internet access to the 

official edition of the Federal Register and the Code of Federal 

Regulations is available on GPO Access at: http://www.gpoaccess.gov/nara/index.html




(Catalog of Federal Domestic Assistance Number does not apply.)



    Program Authority:  ESEA, as reauthorized by the No Child Left 

Behind Act of 2001, Pub. L. 107-110, January 8, 2002.


    Dated: October 29, 2003.

Rod Paige,

Secretary of Education.

[FR Doc. 03-27699 Filed 11-3-03; 8:45 am]


Click here to access the AEA Statement

Click here to access the Request for Comment in the Federal Register
Click here to access the AEA cover letter to the Dept of Education

Click here to access the AEA Press Release
Click here to access the AERA response to the Request for Comment

Click here to access the NEA Response as a pdf file