|
Session Title: Addressing Complexity: Conducting Evaluation of a Comprehensive Math-Science Partnership (MSP)
|
|
Panel Session 291 to be held in Room 111 in the Convention Center on Thursday, Nov 6, 10:55 AM to 12:25 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Cynthia Tananis,
University of Pittsburgh,
tananis@education.pitt.edu
|
| Abstract:
The Math Science Partnership of Southwest Pennsylvania (MSP) is a comprehensive partnership funded by the National Science Foundation (NSF) in 2003 to address K-12 science and mathematics reform. This project includes 53 school districts, four institutions of higher education (IHE's), and four regional educational service agencies known as Intermediate Units (IU's). The evaluation investigates the effectiveness of this partnership, its impact on institutional practices and policies at partner educational institutions, changes in math and science instruction, and changes in student course taking and achievement outcomes.
This panel offers four distinct but inter-related presentations that describe various aspects of the evaluation and the strategies used to guide the discussion of key elements related to reform sustainability. These papers include a focus on the partnership of three evaluation agencies (through higher education, a private research firm, and a K-12 public entity) to provide a diverse array of evaluation strategies, evaluation planning and responsiveness through logic modeling, the exploration of various individual educator characteristics that are descriptive of high levels of reform implementation, and the variables that appear connected to broader sustainable reform at a systemic level.
|
|
Evaluation as a Collective Effort: Diversity to Best Serve Client Needs
|
| Cynthia Tananis,
University of Pittsburgh,
tananis@education.pitt.edu
|
|
More commonly today, funding agents expect potential grantees to partner with other organizations in a common mission and agenda of activity. As the resultant partnerships and collaboratives grow in complexity of interaction and implementation, so too, does the evaluation agenda and the need to manage a vast array of data. Our approach to address the program (tm)s complexity and its need to be responsive to evaluation demands included the development of an evaluation team, drawing on expertise from three different evaluative organizations and perspectives. As a result of this approach, we are able to offer depth of experience and analytical expertise in numerous methodological approaches including case study, interviews, survey research, and extensive quantitative statistical modeling across various participation and student achievement data. While ripe with benefit to and for the client, the inclusion of a large number of evaluators and evaluation perspectives and methods also presents challenges in practice. This paper identifies and explores some of the promising approaches we have used to develop a methodological diversity and appreciation of disparate expertise as well as some of the dilemmas and challenges we have faced as we seek to provide high quality, useable evaluative knowledge for our client and the field at large.
|
|
|
The Continuing Evolution of Logic Modeling for the Math Science Partnership of Southwest Pennsylvania
|
| Valerie Williams,
RAND Corporation,
valerie@rand.org
|
| Cynthia Tananis,
University of Pittsburgh,
tananis@education.pitt.edu
|
| John Pane,
RAND Corporation,
jpane@rand.org
|
| Stuart Olmsted,
RAND Corporation,
olmsted@rand.org
|
|
A critical component of our evaluation of the MSP has been the development of a logic model to communicate the project's operations, theory of action and intended outcomes. Our efforts to develop a logic model for the MSP began in the early proposal stage and have continued to evolve over the first five years of the project. The logic model has not only been a tool for communication, helping to provide a common language and write our annual reports but more importantly, a tool to clarify our evaluation activity. It has served as a mirror to reflect our changing conceptualization of the project, and in so doing, has challenged our understanding of the MSP, requiring us to deal with the difficult "if-then" questions that are inherently embedded in any logic model.
As the project has moved into its last funded year (though it expects to operate for at least one additional year), issues related to scaling up and sustainability have moved into a more prominent focus. This focus has implications in prioritization of both program and evaluation activity, potential indicators of outcomes, and how the culminating evaluation report will be organized. This paper explores these latter developments.
| |
|
A Comparative Study of Individual-Level School Reform Implementation
|
| Keith Trahan,
University of Pittsburgh,
kwt2@pitt.edu
|
| Cara Ciminillo,
University of Pittsburgh,
ciminill@education.pitt.edu
|
| Cynthia Tananis,
University of Pittsburgh,
tananis@education.pitt.edu
|
|
The key reform strategy of the Math Science Partnership (MSP) of Southwest Pennsylvania is the professional development (content knowledge, pedagogical knowledge and practice, and leadership skills) of a cadre of teacher leaders (TL's) who are charged with further developing and supporting professional learning communities (PLC's) in their schools to implement math and science reform. These key actors' perceptions of themselves as a competent education reformers are essential to the ultimate scaling up and sustainability of the reform initiative.
This presentation reports findings from a research study of educational leaders' perception of self as an educational reformer. Individual characteristics of successful reform include: increased content and pedagogical knowledge, developed leadership skills, improved decision-making, and expanded collaboration. Data are collected through a variety of methods including open-ended interviews, a sustainability survey and the MSP Management Information System database.
The study uses a visual heuristic to analyze participants' perception of self as a program reformer. Drawing from individual data, we create a composite of a high fidelity implementer. Findings may shed light on implementation characteristics that support educational reform initiatives, helping to target activities and resources on participants that are more likely to have a greater impact on program implementation.
| |
|
Exploring Collective Variables Related to Systemic Educational Reform Sustainability
|
| Cynthia Tananis,
University of Pittsburgh,
tananis@education.pitt.edu
|
| Cara Ciminillo,
University of Pittsburgh,
ciminill@education.pitt.edu
|
| Keith Trahan,
University of Pittsburgh,
kwt2@pitt.edu
|
|
The Math Science Partnership (MSP) of Southwest Pennsylvania focuses its reform efforts on both teachers and administrators in the K-12 sector. The project assumes the need for both individual agency among educators as well as collective action to provide levers for sustainable and extended systemic change.
As the project has moved into its last funded year (though it expects to operate for additional years with extended and additional funding), issues related to scaling up and sustainability have moved into a more prominent focus. The evaluation activities associated with the project are seeking to address these issues through clustering findings from various evaluation activities and strategies to inform the development of a model of interaction. Findings may shed light on implementation characteristics that support educational reform initiatives, helping to target activities and resources on strategies and system pressure points that are more likely to have a greater impact on program implementation.
| |