|
Session Title: Values and Valuing in After-School Program Evaluation: Perspectives From Local, District, and Statewide Evaluators
|
|
Panel Session 445 to be held in San Simeon B on Thursday, Nov 3, 2:50 PM to 4:20 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Krista Collins, Claremont Graduate University, krista.collins@cgu.edu
|
| Discussant(s):
|
| James Sass, Rio Hondo College, jimsass@earthlink.net
|
| Abstract:
After-school programs present evaluators with a maelstrom of conflicting values. In addition to the typical concerns of compliance, effectiveness, and improvement, after-school program evaluators must address the tension between child/youth development outcomes and academic achievement, respond to the conflicting needs of funders and program providers, promote evaluation use in a funding environment that prioritizes mandatory reporting, and maintain methodological rigor while being flexible in response to ever-changing demands. The presenters address these issues from different perspectives: two community-based organizations (one long-estabished and one rapidly growing) serving schools in multiple districts, an urban school district managing programs provided by 30 community-based organizations in 400 schools, and the nation's largest statewide after-school initiative. Turning principles into practice, they discuss actual decisions and techniques for providing high-quality evaluations in the context of conflicting values. Specifically, the presenters address issues of evaluation design, data collection, analysis and interpretation, utilization and consultation, and budget management.
|
|
Incorporating Client Values in After-school Program Evaluations: Results From Woodcraft Rangers' Nvision After-School Program Quality Assessment and Student Outcomes Evaluation
|
| Kristen Donovan, EVALCORP Research & Consulting, kdonovan@evalcorp.com
|
| Lisa Garbrecht, EVALCORP Research & Consulting, lgarbrecht@evalcorp.com
|
|
Our evaluation approach involves working with clients to assess and quantify what is most valuable to them, then providing consultation on where implementing such information would be of highest use and value. Over the past several years, EVALCORP has collaborated with Woodcraft Rangers' after-school program, which is provided at more than 60 schools across Los Angeles County, to measure and establish benchmarks for valuing the quality of its Nvision program model and link it to student outcomes. Findings from the assessment were used by Woodcraft Rangers to enhance its program, staffing, and other decisions. Using the Woodcraft Rangers' Assessment of Program Quality and Youth Outcomes study as an example, this presentation will discuss how valuing is a key component throughout the evaluation process - guiding which evaluation design is implemented, the types of data collection methods employed, and how the data are ultimately put to use
|
|
|
Valuing the Role of Children and Youth in Local After-School Evaluations
|
| Tiffany Berry, Claremont Graduate University, tiffany.berry@cgu.edu
|
| Ana Campos, After School All-Stars, anacampos@la-allstars.org
|
|
Evaluations of after-school programs often exclude a "whole child" perspective from the evaluation process given that many frequently (1) preference academic achievement (in the form of test scores) over the social/emotional development of children; (2) ignore the contextual variables surrounding the child (their perceptions of neighborhood safety, how children spend their time after school when not in the program, etc.); and (3) disregard the notion that children actively choose their environments (reason they join). During this presentation, the Executive Director of After School All-Stars, Los Angeles, as well as their external evaluator from Claremont Graduate University will explain the development, rationale, and theory behind our "whole child" evaluation approach, discuss measurement issues using this approach, and illustrate how our approach has facilitated program improvement throughout the organization.
| |
|
Responding to Stakeholders' Values and Priorities in After-School Program Evaluation: Tools and Strategies for Presenting Data to Managers, Policy-Makers, and Funders
|
| Harry Talbot, Los Angeles Unified School District, harry.talbot@lausd.net
|
| Steven Frankel, Research Support Services, stevefrankel@ca.rr.com
|
|
Program managers and research directors must play multiple roles if they are to be truly effective. For example, program managers must manage the activities of after-school programs serving thousands of students as well as provide data and policy recommendations supporting the multimillion dollar budgets of these programs. Program managers need unique reporting methods, rapid turnaround times, and priorities that can be shifted on a dime. Research directors must provide an ongoing stream of findings and recommendations that will improve the programs and repay funding for evaluations. They must also develop evaluation strategies that identify promising findings and trends regardless of whether these lines of investigation were included in the original research design. Using examples from Beyond the Bell within the Los Angeles Unified School District, this presentation will describe methods of meeting these multiple demands without compromising objectivity, creativity, and fixed-price budgets.
| |
|
The Role of Values and Stakeholder Perspectives in Conducting and Reporting Statewide Evaluations of After-School Programs
|
| Denise Huang, University of California, Los Angeles, dhuang@cse.ucla.edu
|
|
Since the provisions of California Proposition 49 became effective, annual funding for the After School Education and Safety Program (ASES) has increased from $120 million to $550 million. This funding requires the California Department of Education to contract for an independent statewide evaluation on the effectiveness of programs receiving funding. The National Center for Research on Evaluation, Standards, and Student Testing (CRESST) has taken on the responsibility of this task, and is conducting two statewide evaluations of after-school programs: one for programs serving elementary and middle school students; and the second for programs serving high school students. Using these studies as examples, CRESST will discuss the purpose of evaluations from the State's perspective and from the practitioners' points of views, and values and valuing in evaluation, as well as how evaluators can be forthright and transparent about the values that are being promoted during and in reporting the study.
| |