|
Session Title: Evaluating Research on Science Technology Engineering and Mathematics (STEM) Learning and Discoveries: New Program Evaluations Funded by the National Science Foundation (NSF)
|
|
Panel Session 428 to be held in Huntington C on Thursday, Nov 3, 2:50 PM to 4:20 PM
|
|
Sponsored by the Government Evaluation TIG
|
| Chair(s): |
| Leslie Goodyear, National Science Foundation, lgoodyea@nsf.gov
|
| Discussant(s):
|
| Leslie Goodyear, National Science Foundation, lgoodyea@nsf.gov
|
| Abstract:
The panel session show-cases four recently-awarded evaluations and one program monitoring system funded by the NSF to examine the research on learning in formal and informal settings. The programs being evaluated include the Discovery Research K-12 (DR-K12) program, the Research and Evaluation on Education in Science and Engineering (REESE) program, the Informal Science Education (ISE) program, and the Information Technology Experiences for Students and Teachers (ITEST) program. These programs target different segments of the STEM fields, populations, and somewhat different points on the Cycle of Research and Development (i.e., hypothesize, design, implement, evaluation, synthesis). As a result, the funded evaluations, being conducted by Westat and SRI International, share many similarities in design, but also provide different strategies tailored to the specific purposes, goals and audiences served by each program. The five presentations will share insights on their approaches, and discuss methodological challenges, and explore opportunities to advance the fields.
|
|
Catalyzing the STEM Teaching and Learning: Evaluating NSF's Research and Evaluation on Education in Science and Engineering (REESE) Program
|
| Joy Frechtling, Westat Inc, joyfrechtling@westat.com
|
| John Wells, Westat Inc, johnwells@westat.com
|
| Xiaodong Zhang, Westat Inc, xiaodongzhang@westat.com
|
| Dan Heck, Horizon Research, dheck@horizon-research.com
|
| Andy Slaughter, Westat Inc, andyslaughter@westat.com
|
|
Westat and Horizon Research, Inc., are conducting a three-year evaluation of the Research and Evaluation on Education in Science and Engineering (REESE) program in the National Science Foundation's Division of Research and Learning. Using a multi-method approach the evaluation is examining the following questions:
- To what extent is the REESE program catalyzing discovery and innovation in research in STEM learning, education, and evaluation?
- How effective is the REESE program in stimulating STEM fields to produce high-quality, rigorous research studies?
- How effective is the REESE program in supporting the accumulation of knowledge in STEM education?
- How effective is the REESE program in promoting interdisciplinary research addressing questions of STEM teaching and learning?
The presentation will present Westat's approach for evaluating this program, including challenges and how they are being addressed.
|
|
|
Resource, Technology and Model for K-12 STEM Learning and Teaching: Evaluating the Discovery Research K-12 (DR-K12) Program
|
| Xiaodong Zhang, Westat Inc, xiaodongzhang@westat.com
|
| Harris Cooper, Duke University, cooperh@duke.edu
|
| Joy Frechtling, Westat Inc, joyfrechtling@westat.com
|
| Andy Slaughter, Westat Inc, andyslaughter@westat.com
|
|
Westat is contracted for a three-year evaluation. The presentation will present our designs and approaches for addressing the evaluation questions:
1. What does the portfolio of funded projects look like?
2. What percent of development-intensive projects funded in the program employ appropriate methods to evaluate the efficacy and apply them rigorously? What were the methods used to study these projects? What have been the results of these studies?
3. What percent of resources developed by the program are found to be effective and ready for adoption at scale?
4. What are the combined effects of the projects that have been evaluated with rigorous methods? What do these combined effects contribute to the knowledge base about innovative approaches to improving STEM learning and teaching?
5. Do the resources, models, tools, and technologies developed and/or studied in projects lead to significant improvement in student learning? To significant improvement in teacher STEM competency?
| |
|
The Informal Science Education (ISE) Online Project Monitoring System (OPMS)
|
| Gary Silverstein, Westat Inc, garysilverstein@westat.com
|
| John Wells, Westat Inc, johnwells@westat.com
|
| Hannah Putman, Westat Inc, hannahputman@westat.com
|
| Melissa Bryce, Westat Inc, melissabryce@westat.com
|
|
The Online Project Monitoring System (OPMS) is a collection of surveys that Informal Science Education (ISE) projects complete at different points in time during their grant award period. The online system obtains information about a project's partner organizations, activities, beneficiaries, and impacts. A key component of the OPMS is its collection of information regarding how project activities are expected to benefit public and professional audiences, as well as the study designs and data collection methods that will be used to assess whether these benefits actually occurred. Follow-up data examine whether these anticipated outcomes were actually realized by individual projects. This information is used to examine the collective impact of the ISE portfolio of funded projects, as well as to monitor participants' activities and accomplishments.
The presentation will present Westat's approach for collecting and validating the data, as well as sharing findings for projects funded between FY 2006 and FY 2010.
| |
|
Evaluation of NSF's Informal Science Education Program, 1999-2010
|
| Vera Michalchik, SRI International, vera.michalchik@sri.com
|
| Kea Anderson, SRI International, kea.anderson@sri.com
|
| Ruchi Bhanot, SRI International, ruchi.bhanot@sri.com
|
| Julie Remold, SRI International, julie.remold@sri.com
|
| Denise Sauerteig, SRI International, denise.sauerteig@sri.com
|
|
SRI's evaluation of the NSF's Informal Science Education program focuses on its contributions to social, human, technical, and institutional capital in the field and links these contributions to learning outcomes for participants, addressing the following questions:
- What is the current state of the ISE field with regard to knowledge, practice, and institutional and professional capacity, and how can these elements best be typologized?
- In what ways and to what extent has the NSF ISE program, through its funding mechanisms, contributed to the development of the ISE field by improving the infrastructure-i.e., its institutions and organizations, its knowledge base, its professional connections, and its resources, including its capacity to conduct rigorous evaluations?
- How have changes in the field, its infrastructure, and the NSF ISE program influenced learning?
The research includes multiple data sources and a solicitation/ portfolio analysis, social network analysis, and qualitative meta-analysis.
| |
|
Program Evaluation of ITEST: Innovative Technology Experiences for Students and Teachers
|
| Patrick M Shields, SRI International, patrick.shields@sri.com
|
|
NSF's Innovative Technology Experiences for Students and Teachers program aims to strengthen the students' learning experiences to cultivate their interest in and capacities for science, technology, engineering, and mathematics careers.
SRI International and Inverness Research have undertaken a 3-year evaluation to assess ITEST's impacts and to garner lessons for future investments. The evaluation has been designed to answer the following research questions:
1. What are the projects' impacts?
O What are the achieved outcomes for students and teachers?
O Do youth who participate in ITEST projects demonstrate greater interest in STEM activities and careers than nonparticipants?
O To what extent are project evaluations rigorous?
2. What project models are most effective in achieving student and teacher outcomes? What project characteristics contribute to these models' success?
3. How can we best characterize and describe ITEST projects (in terms of what they do, who the serve and where they take place)?
| |