| Session Title: Selecting Measures and Instruments for Studying Fidelity and Outcomes in Education: Lessons Learned From an Evaluation of a Teacher Professional Development Program |
| Multipaper Session 399 to be held in Ventura on Thursday, Nov 3, 1:35 PM to 2:20 PM |
| Sponsored by the Pre-K - 12 Educational Evaluation TIG |
| Chair(s): |
| Castle Sinicrope, Social Policy Research Associates, csinicrope@spra.com |
| Yasuyo Abe, Berkeley Policy Associates, yasuyo@bpacal.com |
| Abstract: With the current emphasis on rigorous research and experimental designs in education, evaluators must often negotiate different values and perspectives on measuring implementation and outcomes. While outcomes continue to be a primary focus in education evaluation, there is a growing recognition of the importance of understanding fidelity of implementation. In this session, presenters share lessons learned from measuring implementation fidelity and outcomes in a random assignment evaluation of a multi-year teacher professional development program. The presentation broadly reflects on balancing different stakeholder values in selecting measures and instruments to answer questions about fidelity and outcomes. Specific challenges to be covered in the session include identifying critical program components, selecting fidelity criteria, specifying outcome measures, and deciding between developing, adapting, or adopting existing instruments. This session will also share lessons learned from timing data collection efforts relative to program delivery and reporting implementation and outcome findings. |
| Lessons Learned From Documenting Implementation Fidelity: Developing Implementation Fidelity Measures for a Teacher Professional Development Program |
| Vanora Thomas, Berkeley Policy Associates, vanora@bpacal.com |
| Yasuyo Abe, Berkeley Policy Associates, yasuyo@bpacal.com |
| There is a growing recognition of the value of measuring fidelity of implementation in education research. An understanding of fidelity of implementation, the degree to which an intervention is delivered as intended, is essential to a comprehensive understanding of intervention outcomes and the identification of contextual factors that support or hinder implementation. However, few published studies provide details on the development and use of program specific fidelity measures for intensive school-based interventions. This presentation will outline the lessons learned from measuring implementation fidelity in a multi-year random assignment evaluation of a teacher professional program. Topics to be covered include the identification of the program's critical components and processes, selection of fidelity criteria, and the development of instruments to measure and monitor implementation. This session will also include the challenges associated with the collection of detailed program records and the potential roadblocks faced when presenting implementation findings. |
| Measuring Program Outcomes: Defining and Focusing Outcomes for a Teacher Professional Development Program |
| Castle Sinicrope, Social Policy Research Associates, csinicrope@spra.com |
| While outcomes are widely studied in education, selecting and measuring outcomes continues to pose challenges for evaluators. This presentation reflects on challenges faced in defining and measuring teacher and student outcomes in a recent multi-year teacher professional development evaluation. Key challenges included identifying outcomes that were aligned with the program theory of change and the values and priorities of the different stakeholders. Topics to be covered during the presentation include selecting short-term versus long-term outcomes, limiting and prioritizing outcomes to minimize multiple comparisons, timing data collection relative to program implementation, and reporting outcome findings. This presentation will also reflect on three additional considerations when selecting teacher and student outcomes for evaluating teacher professional development programs: 1) using established, national student assessments versus local, state-level student assessments; 2) adapting existing instruments versus developing new instruments; and 3) challenges posed by under- and over-alignment of instruments with programs. |