The Challenges and Successes of Incorporating Participatory Methods within a Summative Program Evaluation by Karen Glueckert and Julia Holton

Date: Monday, March 2, 2026

Hello, AEA365 community! Liz DiLuzio here, Lead Curator of the blog. This week is Individuals Week, which means we take a break from our themed weeks and spotlight the Hot Tips, Cool Tricks, Rad Resources and Lessons Learned from any evaluator interested in sharing. Would you like to contribute to future individuals weeks? Email me at AEA365@eval.org with an idea or a draft and we will make it happen.

We are Karen Glueckert and Julia Holton, Senior Research Associates at the University of Cincinnati Evaluation Services Center. We conduct evaluations for a variety of projects, primarily focusing on educational evaluations. We want to share some challenges, successes, and lessons learned from including participatory methods within a more traditional statewide evaluation.

Project Description

The purpose of the evaluation was to provide the state with a summary of the impact of federal funds used by 51 educational grantees to support students most negatively impacted by COVID-19. An initial landscape analysis across the state was conducted to gather information retroactively on the multitude of funded projects already implemented by grantees. The decision to include participatory methods was made collaboratively with the state leadership team using key findings from the landscape analysis. These findings included grantees expressing a desire to build capacity around evaluation and that both access to and interpretation of student level data was challenging. Additionally, many grantees were not directly responsible for implementing the funded interventions and the project outcomes for grantees were unique to district needs. Based on the findings a decision was made to use a case study multi-site method for the evaluation. A Rad Resource that informed our decision is Lawrenz & Huffman’s (2003) article.

How we Used Participatory Methods:

State Needs and Grantees Informed the Logic Model

• An advisory group of grantee leaders helped develop a logic model unique to the project that contained both student and non-student outcomes.

Grantees Selected Their Case for the Report

• Grantees were tasked with identifying one example (case) of their use of funds that best supported state priorities. While grantees could select a case that did not have student outcomes, they were given guidance to help them prioritize the selection of an example that had student outcomes.
• Grantees submitted an initial case study plan using the logic model and identified the data they intended to use.
• To support grantees in building evaluation capacity, the evaluation team met one-on-one with grantees and cooperatively revised and finalized case study plans.

Grantees Wrote Their Narrative for the Report

• Using the final case study plans, the grantees wrote their own narratives for the inputs, outputs, and outcomes of the logic model that were included in the report and submitted the relevant data.
• After analyzing the data submitted by grantees the evaluation team again met individually with grantees to support them with data interpretation and to ensure alignment between the narrative and the reported outcomes.

Successes:

• Grantees were able to tell their story of how they used the funds and how those funds impacted students, teachers, and schools.
• The logic model developed for this project can be used by grantees for future projects.
• Grantees grew in their evaluation capacity and in their ability to interpret data.
• The state received a summative report that focused on state priorities.

Challenges:

• Lack of having final control over report dissemination has limited the availability of the report for grantee use.

Lessons Learned:

• Be open to finding several avenues to share the report with participants. We were able to provide access by sharing a link to an online platform as part of a conference presentation.
• Think through multiple ethical dilemmas such as amplifying participant voices while not having control over report dissemination. The AEA Guiding Principles are a Rad Resource to guide you.
• Be clear with your participatory participants regarding what you have control over and what you do not.
• Be reflective as an organization afterwards to think through what you may have done differently.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.