Nonprofit and Foundation TIG Week: Lessons from Using AI for Learning and Evaluation by June Han and Juan Clavijo

Date: Monday, June 16, 2025

We are June Han and Juan Clavijo, evaluators at ORS Impact and members of an AI workgroup focused on AI exploration, learning, integration, and governance. Part of our role has been to test different AI tools and facilitate organizational learning. This experience has prompted some lessons we want to share with the field.

At ORS, we value transparency and relationships, and we have a policy of asking clients for permission before we use AI for analysis on a project.

Rad Resource

Principles are the ways in which we make our values actionable. We developed ORS Principles for Generative AI Use to guide us as we learn about and adopt AI tools, develop use cases and processes for verifying AI outputs, and strengthen our data governance and systems.

Use Cases and Pilots

We have an internal ORS policy of encouraging open exploration of AI tools with publicly available data, which is different than how we approach AI use with client data, internal data, and data collected from human research subjects. For data that is not publicly available, we have much stricter guidelines.

One way we’ve used AI tools with publicly available information is for literature reviews and scans. In 2023, we used ChatGPT-4 as a research assistant tool for this ORS literature on narrative change, allowing us to expand our scope and see a “roadmap” of relevant bodies of literature and key content sources much sooner. More recently, we’ve used Genei to generate summaries of articles and reports and Consensus as an AI search engine for research. While AI is a useful research assistant tool, it requires a human to verify and think critically about the AI-generated outputs.

We are piloting Microsoft 365 Copilot, which offers the same enterprise data protection as all other Microsoft 365 commercial offerings. MS 365 Copilot allows us to remain within our tenant, which means our data remains our data. This aligns with our principles and our care around data privacy, security, and sovereignty. We are piloting Copilot for tasks such as generating draft versions of executive summaries for reports and uncovering insights faster in Excel. To align with our values, we do not use Copilot on project work without first getting permission from the client to use AI for analysis.

Lessons Learned

• AI tools have been helpful when leveraged for specific tasks like transcription, translation, and summarization. Using AI to speed up routine tasks can free up time for things like creative problem solving, higher-level thinking, and relationship building.
• At the present time, AI has difficulty with applying abstract reasoning, understanding nuance and context, and making connections beyond the training data. As a result, we currently do not use AI as much for synthesis and analysis of complex systems change work. Instead, we rely on our staff to bring in concepts and frameworks from research or our prior work to enhance our understanding of the data and produce deeper insights for our clients.

Hot Tips

• A helpful framing for us has been to consider what is our mission and how AI can support it, rather than asking what is our AI strategy?
• Generative AI turns any text into data that you can then query. It can be helpful to use AI-generated output as a starting point and to view generative AI as a thought partner and assistant for writing and research.
• To summarize long excerpts, it is more effective to prompt AI to summarize the content section-by-section. Asking generative AI to summarize a long document will often result in outputs that are too high-level to be useful.

The American Evaluation Association is hosting Nonprofits and Foundations Topical Interest Group (NPFTIG) Week. The contributions all this week to AEA365 come from our NPFTIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.