| Session Title: Issues in Measuring Students' Interest in Science, Technology, Engineering, Mathematics and Related Careers: Using Multiple Data Sources in Evaluations of National Science Foundation Innovative Technology Experiences for Students and Teachers (ITEST) Programs |
| Multipaper Session 641 to be held in Suwannee 15 on Friday, Nov 13, 4:30 PM to 6:00 PM |
| Sponsored by the Pre-K - 12 Educational Evaluation TIG |
| Chair(s): |
| Torie Gorges, SRI International, torie.gorges@sri.com |
| Abstract: This session offers perspectives and strategies used in evaluations of five National Science Foundation Innovative Experiences for Students and Teachers (ITEST) programs. ITEST programs seek to engage students from diverse backgrounds in science, technology, engineering, and mathematics (STEM) fields with the goal of increasing the numbers of youth who seek out careers in these areas. In each of these programs, researchers have faced the challenge of evaluating students' changing interest in STEM and STEM careers. We have found that instruments such as surveys do not always capture subtle shifts. In this session, we will discuss the methods we have used to capture small changes and growing interests, including modifications to survey items and the addition of instruments such as parent focus groups, interviews with program staff, and observations. The session offers practical advice from the field to evaluators of educational programs, particularly those focused on STEM and STEM careers. |
| Measuring Subtle Shifts: Career Interests of Middle School Girls in a Technology Fluency Informal Education Program |
| Torie Gorges, SRI International, torie.gorges@sri.com |
| Melissa Koch, SRI International, melissa.koch@sri.com |
| Reina Fujii, SRI International, reina.fujii@sri.com |
| Build IT is a technology fluency program for middle school girls developed by SRI International and Girls Inc. of Alameda County. One of the program's goals is to encourage participants to explore and pursue careers in STEM fields. To do this, girls create technologies and meet women working in STEM fields. One of the challenges for evaluators has been measuring subtle shifts in girls' interest in STEM and related careers. We found that staff who work with the girls daily saw more interest in STEM than we picked up via surveys and interviews. For example, students loved creating video games stuck to answers like "teacher" and "pediatrician" when asked by evaluators what careers interested them. In this paper, we discuss modifications to interview questions and additional data sources, such as staff feedback and observations, which enable us to better capture small shifts in career interests as well as big changes. |
| Tracking Students' Knowledge of and Interest in Science, Technology, Engineering, and Mathematics Careers |
| Ann Howe, Independent Consultant, achowe@earthlink.net |
| The SUCCEED Apprenticeship Program enables middle school students of diverse racial and ethnic backgrounds and both genders to become proficient in the use of information technologies within the context of science, technology, engineering, and mathematics (STEM). One of the stated goals of the two-year program is to build and maintain excitement for STEM with the expectation that this will eventually lead to careers in one of these fields. We have tracked students' career knowledge and interests over the two-year span through surveys and repeated interviews of students and mentors. In addition to providing information about career interests these methods have also allowed us to form tentative conclusions about the elements of the project that tend to reinforce students' interest in STEM careers. Modification of our methods as three cohorts have moved through the program will be discussed. |
| Possible Selves and Career Goals: Multiple Ways of Exploring the Question "What Do You Want To Be When You Grow Up?" |
| Karen L Yanowitz, Arkansas State University, kyanowitz@astate.edu |
| Tanja McKay, Arkansas State University, tmckay@astate.edu |
| C Ann Ross, Arkansas State University, cross@astate.edu |
| Staria S Vanderpool, Arkansas State University, svand@astate.edu |
| CSI: Creating Student Investigators uses forensic science to engage middle and high school students in a week-long intensive STEM enrichment experience. This paper presents different ways of looking at STEM interests, career aspirations, and connections between these methods. Students generated a list of careers goals, and completed general survey regarding STEM interest and activities. Self-efficacy, or the belief that one's actions will produce desired outcomes, may influence aspirations and thus, students completed a science self-efficacy scale. Finally, students completed a possible self measure by describing (open-ended) what they thought would be most likely true about themselves in the future. Possible selves, part of general identity development, may encompass a variety of life goals. Information gained from the possible self measure thus can place career and STEM goals in context with other goals. Discussion will focus upon triangulating these measures and the information that each provides when considering student career development. |
| Examining Career Interest for Youth in the Detroit Area Pre-college Engineering Program |
| Shannan McNair, Oakland University, mcnair@oakland.edu |
| Margaret Tucker, Detroit Area Precollege Engineering Program, mtucker@dapcep.org |
| Jason Lee, Detroit Area Precollege Engineering Program, jdlee@dapcep.org |
| Documenting the career interests of youth is important to precollege program goals but not as simple as it looks. Surprisingly, our challenges begin at the beginning, deciding how to categorize responses to an open ended-question about "what do you want to do as an adult?" or deciding what to include in a list of possible careers. Comprehensive lists do not always take into account all levels of STEM careers. For example, is designing computer software, the same as a job repairing personal computers? Also, newer career categories may not be included in existing lists. In addition, respondents may have the same career in mind, but call it by one name on a pre-program survey and another on a post-program survey. Challenges related to asking the question, documenting the response, categorizing the response and measuring change are discussed. Panel presenter will promote interaction by eliciting ideas for solutions to the issues. |
| Measuring Students' Science, Technology, Engineering and Math (STEM) Interest and Learning through Data Triangulation |
| Karen Manuel, Puget Sound Center for Teaching, Learning and Technology, kmanuel@psctlt.org |
| There are many challenges in evaluating the impact of an after school or summer program on students' science, technology, engineering and math (STEM) learning and interest, as well as their future course and career choices. This paper will describe how TechREACH measures middle school students' changing career interests and interest in STEM in general, and triangulates data sources as a means of assessing program impact. The TechREACH evaluation employs a mixed methods approach, including pre- and post- student surveys, focus groups, teacher surveys and interviews, parent surveys, math/science teacher surveys, observation, grade collection, and information about students' subsequent involvement in STEM activities in high school. The evaluation examines interest and achievement in STEM coursework, proficiency in computer technology, knowledge of and interest in STEM degrees and careers, and likelihood to pursue future STEM courses and careers. |