Health Evaluation TIG Week: Transforming Indicator Development with GenAI by Juliette Berlin, Krysta Medearis, Amy Shim, Sarah Bergman

Date: Thursday, July 17, 2025

Hello, we are Juliette Berlin, Krysta Medearis, Amy Shim, and Sarah Bergman, from Deloitte’s Evaluation and Research for Action (ERA) Center of Excellence (CoE), writing on behalf of the Health Evaluation TIG. In this series of blog posts, we are walking through the steps of Deloitte’s evaluation framework and describing how artificial intelligence (AI) can support evaluators with each step. With the recent buzz surrounding Generative AI (GenAI), there is increasing interest in how to best leverage emerging Large Language Models (LLMs) to make evaluations more efficient. Our most recent post explored the use of GenAI in developing evaluation questions. Today we’re going to explore its use in indicator development.

Hot Tips

Imagine having a brainstorming partner who can offer you fresh perspectives and innovative ideas for developing indicators. That’s the power of LLMs and GenAI tools! Recently, we used prompt engineering with a GenAI tool to craft indicators for a client. This approach streamlined our process and provided deeper insights into potential outcomes of these roles. From this experience, we’ll share how using the four components of prompt engineering (e.g., persona, task, context, and format) can result in tailored indicators.

Prompt Component	Example Language	Hot Tip
Persona: When crafting a prompt, assign a persona (e.g., a target user) to the LLM to guide the tone and focus of the output.	“You are a program manager working to assess the outcomes of a newly funded position for a large grant.”	Aligning the output with a user improves model “reasoning” resulting in clearer, more accurate responses that are tailored to the defined role.
Task: Define the task the model is being asked to perform.	“I will enter a position title with their key responsibilities and intended job outcomes. I’d like you to generate possible indicators that could be used to evaluate the position.”	Clearly stating what you’re asking the model to do and providing an example of “good” output will generate better responses.
Context: Provide relevant context to help the model better understand the task being performed.	“The definition of an indicator is: A marker of accomplishment or progress that is specific, observable, and measurable, and show the progress made towards achieving a specific outcome in a work plan.”	Provide definitions, documentation, and examples where appropriate, as LLMs are general and not tailored to specific domains.
Format: The format of the output (e.g., paragraphs, bullets, tables) should be defined.	“Number each possible indicator and include a bullet for the definition and how it will be measured. Limit output to the four most relevant indicators to the position.”	Specifying the format and amount of the output ensures that generated content meets your specific needs, improves clarity, and reduces the need for further editing.

Lessons Learned

Iterate: Are you finding in the output that the model didn’t interpret part of your prompt as intended? That’s okay! Don’t be afraid to iterate on your prompts; changing small details can have a big impact on the output.
Use a “human in the loop” approach: Always review model outputs carefully. Ask yourself, do the generated indicators make sense for your purpose? Do you need to further define or research the resulting indicators to make sure they are measurable based on the data available? Remember, GenAI models aren’t perfect, and you should always verify their output.
Consider if indicators can be used across grantees, where appropriate: In our analysis, several positions had similar responsibilities and expected outcomes. We applied generated indicators to multiple positions where logical, both to standardize indicators where possible and reduce energy costs from GenAI use.

This posting contains general information only, does not constitute professional advice or services, and shall not be used as a basis for any decision or action that may affect your business. Deloitte shall not be responsible for any loss sustained by any person who relies on this posting. As used in this posting, “Deloitte” means Deloitte Consulting LLP, a subsidiary of Deloitte LLP. Please see www.deloitte.com/us/about for a detailed description of our legal structure. Certain services may not be available to attest clients under the rules and regulations of public accounting. Copyright © 2025 Deloitte Development LLC. All rights reserved.

The American Evaluation Association is hosting Health Evaluation TIG Week with our colleagues in the Health Evaluation Topical Interest Group. The contributions all this week to AEA365 come from our Health Evaluation TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.