|
Session Title: Evaluation of Adult Science Media: Retrospect and Prospect
|
|
Panel Session 306 to be held in Centennial Section C on Thursday, Nov 6, 1:40 PM to 3:10 PM
|
|
Sponsored by the Distance Ed. & Other Educational Technologies TIG
|
| Chair(s): |
| Saul Rockman,
Rockman et al,
saul@rockman.com
|
| Abstract:
The National Research Council's Board on Science Education is exploring the past, present and future of informal science education (ISE). As a component of this effort, several of the presenters developed a background paper on Media-based Science Learning in Informal Environments ( http://www7.nationalacademies.org/bose/Rockman_et al_Commissioned_Paper.pdf ). The commissioned paper reviews evaluations of science television and radio programs and IMAX movies designed for the adult learner and compares the evaluation strategies to those used in studying children's media. It also recommends evaluation strategies that will capture impacts beyond the 'water cooler discussion' stage. More recent efforts by the authors and their colleagues illustrate some of the new evaluation strategies ' at least new to this area of media studies ' that provide insights into the study general audience, television-based, science learning. Strategies include longitudinal studies with repeated measurement, panel studies, and transfer tasks. The findings of these studies will also be presented.
|
|
Retrospective Review of Evaluations of Adult Science Media
|
| Saul Rockman,
Rockman et al,
saul@rockman.com
|
|
Over the past 20 years, NSF, as well as corporate and foundation funders have supported science television and radio programs. National Geographic Specials, NOVA, and programs on the Discovery Channel have informed the general public about issues of historical, current, and future interest in science topics. Many of the initiatives have had both formative and summative evaluations attached to them, with reports back to the producers and the funding agencies. While formative studies have contributed significantly to the improvement in programming, studies of program outcomes and effects have been traditionally ineffective in establishing actionable findings. As the head of an organization conducting many of these evaluations, and as the senior author of the NRC paper on media and science learning, I have been concerned that the strategies supported have been wasteful. In contrast to children's media studies, we start with a different structure and a constraint of short term outcomes.
|
|
|
Informal Science Evaluation Methodologies
|
| Jennifer Borse,
Rockman et al,
jennifer@rockman.com
|
|
The constraints on the methods and procedures of studying informal science media materials has been a function of both the funding patterns and the production cycle of the programs broadcast on radio and television. There are evaluation strategies that can provide a greater understanding of both cumulative impact and immediate outcomes that have not been part of the standard arsenal of evaluation tools for media researchers. We identify and encourage the application of several evaluation approaches that can further the goals of the funding agencies and provide a greater understanding of impact for program developers. These include both longitudinal strategies and transfer tasks that can capture audience reactions and actions in the real world. We show how these strategies are in the best interests of all interested parties. The presenter, a co-author of the NRC paper, has been an evaluator of numerous NSF-funded informal science education projects.
| |
|
Assessing Knowledge of Exploring Time
|
| Kristin Bass,
Rockman et al,
kristin@rockman.com
|
|
This presentation will describe the design and implementation of an online assessment of the Exploring Time television programs, a two-hour prime-time television special that 'reveals the unseen world of natural change and probes the deep mysteries of Time' (Exploring Time, 2008). The assessment included fixed-choice and open-ended tasks that assessed viewers' understanding of the content in the film and their ability to transfer that knowledge to new situations. We will share our process of construct identification, question creation, and instrument review or validation (Wilson, 2005). We'll talk about what worked well, what we wish we had done differently and what we learned about assessing adult learning in science media. We'll also share the results of our study and discuss what the instrument contributed to our overall evaluation. Our presentation is intended to encourage evaluators and funders to demand and develop instruments that are sensitive to the complexities of adult learning.
| |
|
Survey and Panel Studies of Quest Science Programming
|
| Monnette Fong,
Rockman et al,
monnette@rockman.com
|
|
Quest is a science and nature show based in the northern California geographic area, funded by NSF and several foundations. San Francisco's KQED disseminates the materials on three platforms: television, radio, and the web. We will describe our surveys and panels used to evaluate the influence of Quest on adult learning and behavior, and acquire data about sustained interest and activities. A combination of surveys and panels are conducted with Bay Area citizens to explore their media consumption and behavior associated with science and environmental activities. Each month for four successive months, panelists are asked to visit the website and complete a questionnaire about their science, environment and nature activities for the past two weeks, a sample participates in a telephone interview. Through the surveys and this panel, we obtain granular data about the influence of Quest, in all its dissemination strategies. The presenter is the manager of these studies.
| |