|
Session Title: Intermeshing Cogs at Work: Experiences and Lessons Learned From State and Local Educational Program Evaluations
|
|
Panel Session 618 to be held in the Granite Room Section A on Friday, Nov 7, 1:35 PM to 3:05 PM
|
|
Sponsored by the Pre-K - 12 Educational Evaluation TIG
|
| Chair(s): |
| Kathleen Toms,
Research Works Inc,
katytoms@researchworks.org
|
| Abstract:
This panel consists of members of Research Works, Inc., an independent research and evaluation company which consistently evaluates programs at both the State and local levels. We, as principal investigators, research associates and research assistants contributing to multiple evaluations at both levels simultaneously have been struck by the lack of coordination between these two levels of evaluation. This panel will discuss if these levels should be collaborating with and informing each other. Is the State evaluation merely a meta-evaluation of the local studies? Should local evaluators be collecting the data that State evaluators need even if it means their implementation evaluations are not able to be completed? We propose some ways to facilitate a more coordinated approach and will ask the audience for their experiences in navigating this situation from either or both perspectives, and to answer the question: Is there an ideal interaction across system levels of evaluation?
|
|
A State Evaluator's Effect on Local Program Evaluations
|
| Elizabeth Whipple,
Research Works Inc,
ewhipple@researchworks.org
|
|
This panelist will discuss the issues facing a State evaluation as it attempts to utilize local evaluation efforts. The particular perspective presented here is from the project director of the State evaluation of the 21st Century Community Learning Center Program in New York. What is the best way to collect data relevant to the State evaluation's purpose from local evaluators that does not compromise the efforts of the local evaluator, does not cause time-consuming busywork for the local evaluator and that could also prove to be useful to the purposes of the local evaluation? We propose that it is imperative to listen to local evaluators but also to hold local evaluators to high standards thus using the role of the State evaluator to positively influence evaluation capacity building at both state and local levels.
|
|
|
The Tree, the Forest and the New Evaluator: Experiences and Lessons Learned from Working on State and Local Evaluations from the Perspective of a New Evaluator
|
| Josh De La Rosa,
Research Works Inc,
jdelarosa@researchworks.org
|
|
The role of a local evaluator differs yet should complement the duties of a state evaluator and vice versa. Each level of the program delivery system seems to have different objectives for the intervention of interest, often with multiple sub-levels. However, understanding the nuances in the different stakeholders’ objectives is a challenging task. One junction point is an understanding of the purpose of the evaluation as it relates to the purpose of the intervention as it is defined at each level of the system. This panelist will put forth the experience of a new evaluator/graduate student contributing to local evaluations of five Math Science Partnership Programs while supporting the 21st Century Community Learning Centers statewide evaluation. The presentation will focus on the presenter’s trouble switching lenses from project to project. Also, the presentation will inform of lessons learned from working on both the micro and macro levels.
| |
|
The Increased Importance of Local Evaluators on Direct Federally Funded Educational Grants
|
| Mathew Loatman,
Research Works Inc,
mloatman@researchworks.org
|
|
This panelist will focus on the relationship between the local evaluator and the federal evaluation of the Carol White Physical Education Program. The panelist will discuss how the relationship was navigated between the client and the Federal Government. The panelist has spent a considerable amount of time coordinating the data collected to satisfy GPRA reporting requirements with that collected for the local evaluation. The aim of coordinating these activities has been to make data collection easier and more meaningful for the client. This has been fairly easy since there is no state level evaluation activity that would make the overall evaluation more complicated. Also, the evaluator on this project monitored professional development that was targeted at developing the ability of all the teachers to support the achievement of the project results with all students resulting in changes in teacher practice.
| |
|
Shelf Art 101: Do Differences in Evaluation Requirements Affect the End Use of the Evaluation?
|
| Carolynn Woiler,
Research Works Inc,
cwoiler@researchworks.org
|
|
The GPRA Act of 1993, although increasing the need for evaluators, changed the dynamic and context of evaluations. As a result, this new evaluator has found that self mandated evaluations tend to be more collaborative as compared to federally mandated evaluations which, in the experience of the evaluator, tend to be less utilized by the stakeholders. This panelist will focus on the new evaluator's perspective of the relationship between a federally mandated evaluation and compare it to a local evaluation where no evaluator is required but was requested by the grantee. The perspective presented is from the local evaluator of a federally funded Teaching American History grant compared to a locally funded after school program. Data collected by the local evaluator of the federally funded program is often not used by the grantee to inform midcourse corrections of the program. On the other hand, in the case of the locally funded program which does not have an evaluation mandate, the evaluation has a much more significant effect on the program.
| |
|
Different Perspectives of the Mountain: Understanding the Relationships Between State and Local Evaluators From the View of a State and Local Evaluator
|
| Jeff Wasbes,
Research Works Inc,
jwasbes@researchworks.org
|
|
This panelist will focus on the new evaluator's perspective on the relationship between two levels of evaluative activities: state and local. The panelist's perspective stems from serving as researcher on a state level evaluation of the 21st Century Community Learning Center Program for New York, as well as researcher on a local evaluations of several Mathematics Science Partnership Grant (Title II Part B) initiatives. It seems logical to assume that data collected at the local level should help to inform information being collected at the State level evaluation, and vice versa. In the case of the MSP Projects, clear communication channels have not been established that allow for this exchange, with State evaluators presenting their role as technical assistance to the under qualified local evaluators. Further, collected data has not been systemically calibrated to purposefully inform both levels of evaluation, so we do not know if it would work. For these reasons, understanding the role in each evaluation, as well as switching between them, has proven difficult.
| |