| Session Title: The Evolution of Evaluation Within a National Science Foundation Program: Integrating Policy, Practice, and Vision |
| Multipaper Session 828 to be held in Laguna A on Saturday, Nov 5, 8:00 AM to 9:30 AM |
| Sponsored by the Research on Evaluation |
| Chair(s): |
| Arlen Gullickson, Western Michigan University, arlen.gullickson@wmich.edu |
| Discussant(s): |
| Connie Kubo Della-Piana, National Science Foundation, cdellapi@nsf.gov |
| Abstract: In 1993, Congress established the National Science Foundation's Advanced Technological Education program. The first request for proposals did not direct proposers to include an evaluation component in their proposals. In contrast, the 2010 program solicitation called for proposers to "establish claims as to the project's effectiveness," and describe how the evaluation will "provide evidence on the extent to which the claims are realized." This session includes (a) an examination of almost two decades of ATE program solicitations, revealing NSF's increasing value of evaluation and heightened expectations for grantees to have their projects evaluated in substantial and meaningful ways; (b) an analysis of current grantees' responses to the "claims and evidence" requirement; and (c) a discussion of results of a facilitated dialogue among ATE stakeholders about how grantees are currently addressing the requirement and what can be done to bridge the gap between NSF's demands and the priorities of the grantees. |
| Evaluation Expectations Expressed in National Science Foundation Advanced Technological Education Program Solicitations: An Analysis of Changes in De Facto Evaluation Policy since 1993 |
| Lori Wingate, Western Michigan University, lori.wingate@wmich.edu |
| The National Science Foundation is a major force in shaping policies and practices around program evaluation. Its overall growing interest in evaluation, as well as subtle shifts in evaluation priorities over time, has implications for how evaluators serve their NSF-funded clients. An examination of almost two decades of solicitations for NSF's Advanced Technological Education program reveals a significant increase in the attention to evaluation as well as the specificity of evaluation requirements placed on grantees. In 1993, there is only one reference to evaluation. In contrast, the 2010 solicitation conveys specific expectations for evaluation for each type of project and instructs proposers that they must allocate a portion of their budget to an external evaluator. Moreover, an overarching expectation is articulated as follows: "The PI should establish claims as to the project's effectiveness, and the evaluative activities should provide evidence on the extent to which the claims are realized." |
| Evaluative Claims and Evidence: Current Practice |
| Carl Westine, Western Michigan University, carl.d.westine@wmich.edu |
| In the grant-making world, there is a growing emphasis on improving the usefulness of evaluations in terms of providing meaningful feedback to grantees, as well as serving the primary funder's role in securing financial support from donors and government appropriations. Large funding bodies like the National Science Foundation are making increasingly specific requirements for evaluation. In a 2011 survey of all NSF Advanced Technological Education grantees, principal investigators were asked to "Provide an example of a claim or impact related to their project/center." Additionally, they were asked, "What is the evidence to support this claim or impact?" A qualitative analysis of survey responses indicates a lack of clarity and consistency among grantees' perceptions as to what constitutes evaluative claims and evidence. In an era of shallower pockets and tightening budgets, understanding how to properly make claims and provide clear evidence of effectiveness is essential for both grantees and program funders. |
| Articulating a Vision for Program-wide Evaluative Claims and Evidence: Giving Voice to Stakeholders |
| Jane Ostrander, Truckee Meadows Community College, ostranderjane@mac.com |
| Peggie Weeks, Lamoka Educational Consulting, pegweeks@gmail.com |
| The National Science Foundation's Advanced Technological Education program has been in existence for nearly two decades. By meeting in person annually and collaborating across projects, this program's grantees have developed a sense of community. They share a common vision about their roles in improving technological education and are deeply aware of the challenges associated with this work and evaluating the impact of their efforts. NSF's requirement that principal investigators establish claims about their effectiveness and provide evidence on the extent to which the claims are realized was issued with little guidance. In this presentation, we share the results of a facilitated dialogue among community members (grantees, evaluators, and program officers) about the extent to which grant-level evaluations are meeting this "claims and evidence" requirement and what needs to happen to bridge the gap between the demands of NSF and the priorities of the grantees. |