Health Evaluation TIG Week: Harnessing GenAI to Bolster Evaluation Question Development by Reagan Pearce, Eliza Carroll, Juliette Berlin, and Amy Shim

Date: Wednesday, July 16, 2025

Hey there! We’re Reagan Pearce, Eliza Carroll, Juliette Berlin, and Amy Shim from Deloitte’s Evaluation and Research for Action (ERA) Center of Excellence (CoE), writing on behalf of the Health Evaluation TIG. We’re an interdisciplinary bunch with backgrounds in public health, strategy, and evaluation. You’ve heard from our colleagues this week on how to use Generative AI (GenAI) in describing your program. Today, we’re excited to share how we are using GenAI chatbots to help us craft effective evaluation questions!

Consider the following scenario: You are an evaluator conducting an outcome evaluation for a Federally Qualified Health Center (FQHC) program aimed at improving the oral health status of children in the rural Southeast. We’ll walk through the following prompts to brainstorm potential evaluation questions, identify gaps, and introduce different perspectives to our evaluation scenario.

• Brainstorming with Bots: It is common to use several evaluation questions for a specific program, especially if there are several indicators available to provide insight into program efficacy and impact. Using GenAI tools can be a great way to brainstorm and refine evaluation questions that are most relevant to your program and evaluation goals. When brainstorming questions with a GenAI tool, you’ll also want to provide context to your evaluation, such as the program’s background and focus, the evaluation’s purpose, and the intended methods.
  • HOT TIP #1: Ask GenAI to provide three alternative evaluation questions for [your program’s focus] that seek to measure [your specific outcomes of interest].
  • IN ACTION: I am interested in whether the FQHC program has improved oral health outcomes for participating children. Please develop 3-5 evaluation questions that can be answered using the following data available.
• Spotting the Gaps with A-Eye: GenAI tools can act like a keen-eyed detective, helping identify gaps in evaluation questions. By mapping your current questions to established frameworks or domains, AI can pinpoint areas that might need more attention.
  • HOT TIP #2: Ask GenAI to identify gaps in the evaluation question using a [specific framework or toolkit].
  • IN ACTION: Using the Rural Health Information Hub’s “Rural Services Integration Toolkit” as a framework, identify any gaps related to the scope of outcomes, population specificity, etc. in the evaluation question, “What are the observed changes in oral health outcomes, such as reduction in cavities and emergency dental visits among children participating in the program?”
• Playing Devil’s Algorithm: Lived experience is a crucial point of view that everyone brings to their work, but it can sometimes result in a limited understanding. GenAI can challenge assumptions and offer counterpoints, acting as our very own devil’s advocate to help you build more robust evaluation questions.
  • HOT TIP #3: Ask GenAI to suggest assumptions to mitigate or alternative viewpoints for the evaluation question.
  • IN ACTION: Using published outcome evaluations in the rural health domain, identify assumptions I should mitigate or alternative viewpoints I should consider to make the evaluation question, “What are the observed changes in oral health outcomes, such as reduction in cavities and emergency dental visits among children participating in the program?” more robust.

Moving Forward with GenAI

Like all things with GenAI, evaluators should use a human in the loop approach to analyze outputs consistently and critically for biases. The purpose of GenAI is not to replace the valuable work of evaluators, but to bolster your work and contributions.

By leveraging GenAI, evaluators can brainstorm relevant questions, identify gaps, and challenge assumptions to create more comprehensive and insightful evaluations. These approaches not only save time but also provide diverse perspectives and robust reviews, which ultimately leads to a more effective and impactful evaluation.

This posting contains general information only, does not constitute professional advice or services, and shall not be used as a basis for any decision or action that may affect your business. Deloitte shall not be responsible for any loss sustained by any person who relies on this posting. As used in this posting, “Deloitte” means Deloitte Consulting LLP, a subsidiary of Deloitte LLP. Please see www.deloitte.com/us/about for a detailed description of our legal structure. Certain services may not be available to attest clients under the rules and regulations of public accounting. Copyright © 2025 Deloitte Development LLC. All rights reserved.

The American Evaluation Association is hosting Health Evaluation TIG Week with our colleagues in the Health Evaluation Topical Interest Group. The contributions all this week to AEA365 come from our Health Evaluation TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.