| Session Title: Working to Address the Evaluation Needs of More of the People More of the Time: Conducting Evaluation in Multi-site and Multi-context Settings |
| Multipaper Session 844 to be held in Mencken Room on Saturday, November 10, 3:30 PM to 5:00 PM |
| Sponsored by the Cluster, Multi-site and Multi-level Evaluation TIG |
| Chair(s): |
| Frances Lawrenz, University of Minnesota, lawrenz@umn.edu |
| Discussant(s): |
| Frances Lawrenz, University of Minnesota, lawrenz@umn.edu |
| Abstract: The Evaluation Center is currently conducting the eighth annual survey of the National Science Foundation's Advanced Technological Education (ATE) program. The ATE program goals are to increase the nation's capacity to provide advanced technological education and to increase the number and quality of skilled technicians in the workforce. This survey collects information about the general characteristics of the program's grantees and their work activities, accomplishments, and impacts. This multipaper session presents highlights of data collected in 2006 from more than 160 individual projects and sites throughout the nation. The first presentation will discuss information about project-level evaluation practices occurring in the ATE program. The second presentation will focus on findings related to workforce needs assessment activities. A final presentation will discuss how the Center has addressed the challenges of conducting an online survey and evaluation across multisite and multicontext settings and examine possible future directions for the survey. |
| Results From the National Science Foundation (NSF) Advanced Technological Education (ATE) Survey: Project-level Evaluation Practices |
| Arlen Gullickson, Western Michigan University, arlen.gullickson@wmich.edu |
| This paper presents highlights of project-level evaluation practices in NSF's ATE program. The first section provides an overview of ATE expectations for evaluation and how projects report having met those requirements. Section 2 describes PI perceptions of the utility of their evaluations and the extent to which these perceptions are related to the characteristics described in Section 1. Section 3 focuses on the activities of external evaluators-PI satisfaction with these evaluators, the relationship between PI ratings and standards for sound program evaluations, whether the PIs view their evaluations as meeting ATE intellectual merit requirements, and PIs' characterizations of the attributes of their external evaluators. Section 4 summarizes findings reported in previous sections to identify strengths and weaknesses of project-level evaluations and to suggest changes for improvement in current evaluation practices. |
| Evaluation of Workforce Needs Assessments Conducted by Advanced Technological Education Projects and Centers |
| Liesel Ritchie, Western Michigan University, liesel.ritchie@wmich.edu |
| This presentation reviews workforce needs assessment activities conducted by ATE grantees. The process is considered an evaluative activity with potential to concentrate the expertise of evaluators, increasing their capacity to make more meaningful summative evaluation statements about ATE program impacts. Using 2006 survey data, this paper discusses ways in which current evaluation activities might incorporate, support, and utilize workforce needs assessment to assist in determining project impact. Findings reveal that (1) a majority of grantees gather workforce needs assessment information, although there is a substantial dependency on 'weak' forms of needs assessment; (2) centers are much more likely than projects to conduct workforce needs assessment once funding is awarded; (3) a variety of factors appear to be associated with whether or not grantees conduct workforce needs assessment; and (4) the purposes for which workforce needs assessment information is considered most useful by grantees include developmental and formative evaluation activities. |
| Progression and Challenges of Conducting an Annual Evaluation in Multi-site Settings |
| Barbara Wygant, Western Michigan University, barbara.wygant@wmich.edu |
| The final paper presents an overview of the annual ATE survey process and breaks down the steps of the overall work plan and its related tasks. Major discussion will focus on the survey design and review processes to illustrate the time and effort needed in the early stages of survey implementation. The importance and significance of review panel input will be discussed, along with highlights of the most recent year's activities, which involved major revisions to the annual survey as a way to improve upon previous years' data gathering efforts. The challenges in conducting this multisite evaluation and how they were addressed will be highlighted as will future opportunities in the survey implementation and analysis efforts. The project managers will also discuss getting the sample audience on board and prepared for the survey launch, conducting the survey, and data analysis techniques. |