|
Session Title: Evaluating United States Department of Education and National Science Foundation Programs for Improving Mathematics and Science Education: Rigor, Relevance, Context, and Challenges
|
|
Panel Session 639 to be held in Suwannee 13 on Friday, Nov 13, 4:30 PM to 6:00 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Robert Yin, COSMOS Corporation, ryin@cosmoscorp.com
|
| Abstract:
The U.S. faces the challenge of increasing its competitiveness in science, technology, engineering, and mathematics (STEM) fields. Federal initiatives, including the National Science Foundation's Math and Science Partnership Program and the U.S. Department of Education Math and Science Partnerships funded under Title IIB of No Child Left Behind, focus on teacher professional development as a means to improve student outcomes in mathematics and science. The panel for this session is comprised of educational researchers who lead national and local evaluations of these programs. Panelists will present evaluation designs, methodologies, and findings from local and national evaluations of these programs. The session will provide an opportunity to discuss issues related to these complex evaluations, including: 1) ensuring rigor and relevance of project and program evaluations; 2) recognizing contextual factors that influence implementation and outcomes; 3) addressing challenges encountered in current evaluation approaches; and 4) identifying needs for future evaluation.
|
|
Comparing Academic Performance between Program Participating Schools and Non-Program Participating Schools
|
| Kenneth Wong, Brown University, kenneth_wong@brown.edu
|
| Megan Boben, Brown University, megan_boben@brown
|
|
Study leaders will present information about this project that examines academic performance between schools that participate in the National Science Foundation's Math and Science Partnership (NSF MSP) Program and their non-participating peers. Because NSF MSP activities primarily involve teacher training and professional development in multiple grade levels, we examine school-level achievement. We address the question: When schools in a state participate in the NSF MSP Program, do their students perform better than they would have if they had not participated in the NSF MSP Program? Student achievement is measured in terms of performance on state-administered assessments in mathematics and science for specific grades for five years in a sample of over 800 schools in 9 math and science partnerships across 6 states. To control for a number of demographic variables, we employ the Mahalanobis distance matching to define an appropriate comparison school group before conducting our multivariate analysis.
|
|
|
Teacher and Student Outcomes in Nebraska's Math in the Middle Institute Partnership
|
| Stephen Meyer, RMC Research Corporation, meyer@rmcdenver.com
|
| John Sutton, RMC Research Corporation, sutton@rmcdenver.com
|
|
The Math in the Middle Institute Partnership is designed to improve student mathematics achievement by creating sustainable partnerships among individuals at the University of Nebraska-Lincoln, regional Educational Service Units (ESUs), and local school districts. Partnerships are designed to educate and support teams of outstanding mathematics teachers in grades 5-8 who become intellectual leaders in their schools, districts, and ESUs. RMC Research's quasi-experimental evaluation compares outcomes for M2 participants and their students to those of teachers and students in a comparison group. Qualitative components include interviews and focus groups; observations; and analysis of documents. Results from analyses of teacher pre/post survey data and student mathematics achievement data will be presented by directors of the external evaluation. Methodological challenges and limitations will also be discussed, including attribution of program effects, conflicts between strong evaluation designs and program implementation goals, challenges identifying valid comparison groups, and data collection challenges in rural settings.
| |
|
Balancing the Call for Evidence-Based Research Designs With Formative Evaluation to Improve Implementation of Inquiry-Based Science Teaching
|
| Catherine Callow-Heusser, EndVision Research and Evaluation LLC, cheusser@endvision.net
|
| Kenneth Wareham, Lewis-Clark State College, klwareham@lcsu.edu
|
|
The Teaching for Excellence in Science and Literacy Achievement (TESLA) project (Idaho Mathematics-Science Partnership Grant) integrates science and literacy instruction, and provides ongoing support from university faculty to classroom teachers. The evaluation included both quantitative and qualitative evidence for formative program improvement and evaluation of program effectiveness. Gains in science and language scores in participating classrooms were large and statistically significant. More importantly, grade equivalent changes for science scores on the SAT-9T standardized test (a form of the Stanford Achievement Test designed to reduce student reliance on memorized factual information and emphasize unifying concepts and themes in science and literacy) were larger than 1-year expected gains. This presentation will include an overview of the TESLA program, evaluation, and findings. Project directors will discuss how the evaluation design was influenced by contextual factors including the focus on program improvement, faculty involvement in collecting evaluative data, and a small evaluation budget.
| |
|
Meta-Evaluation in Title IIB MSP Projects and Programs: What Counts as Standards of Quality?
|
| Edith Gummer, Northwest Regional Educational Laboratory, gummere@nwrel.org
|
| Judith Devine, Northwest Regional Educational Laboratory, devinej@nwrel.org
|
|
The Montana Title IIB MSP program evaluation focused on the nature of the partnerships and the quality of project evaluation for the six projects that made up the second cohort in Montana. The nature of the partnerships has been examined using a survey that was developed based on the work of Gordon Kingsley. Stakeholders were asked questions that provided a contextual profile of their partnership, including the motivation behind stakeholder involvement, the perceived need for the partnership, and the resources that helped to sustain their efforts. The survey provided evidence on the extent to which stakeholders perceived that a collaborative model of professional development has been effective in achieving program goals. The quality of the evaluations was assessed using national guidelines and standards (i.e., Guskey, 2000; Joint Committee on Standards for Educational Evaluation, 2008). Project leaders will present results and discuss their implications.
| |