Artificial intelligence is no longer a future consideration—it’s already shaping how policies are designed, programs are evaluated, and public value is delivered. AEA's Summer Workshop, happening Wednesday, June 24 in Washington, DC, will focus on practical frameworks for assessing AI tools based on performance, alignment with organizational values, risk, and real-world impact.
Official Description | Event Details | Pricing
Speaker: Kevin Hong, Core Collaborator with MERL Tech Initiative, and Associate Adjunct Professor with School of International and Public Affairs at Columbia University
The AI landscape is crowded, fast-moving, and full of bold promises. And it is increasingly overwhelming to know which tool to use. For evaluators, the stakes of choosing the wrong tool go beyond lost productivity and raise ethical, professional, and personal concerns. This hands-on workshop cuts through the hype by equipping participants with a practical framework for assessing AI tools on their own terms. We will examine what "performance" really means in the use of AI for evaluations, and how to weigh it against considerations of privacy, transparency, equity, and accountability. You'll leave with concrete tools, hands-on experience, and greater confidence to choose AI tools that reflect who you are as a practitioner. This workshop is designed for those who have a basic understanding of how large language models work, have some experience and comfort with using them, and want to use AI responsibly.
Key Takeaways:
AI Use Disclosure: Claude (Sonnet 4.6) helped with drafting this blurb and brainstorming catchy workshop titles. The content of the workshop, however, is created by humans.
With so much hype about generative AI as well as the current state of our sector, many of us are feeling quite overwhelmed about AI, whether to use it and if so, how to use it, including which tools to use. I often see people choose a tool based on “vibes”. At the beginning, most people used ChatGPT because it was better known, not because they thought it performed better than other models. Then people switched to Claude in response to #QuitGPT, although in my opinion, the corporate and political practices of Anthropic, the maker of Claude, have converged with those of OpenAI, the maker of ChatGPT, over time. I do believe in political activism around AI. I just didn’t see many evaluators, well, evaluate AI tools based on relevant criteria such as performance, hallucination, fit for purpose, etc., to decide which ones to use.
Another mistake I often observe is an assumption that AI would automate the work we need to do. It is important to recognize use cases for AI as automation vs. augmentation. AI can be quite useful in automating some tasks. But in most cases, for our work as evaluators, AI is better suited for augmenting, not automating. Therefore, it is important to examine critically for what tasks you wish to use AI, how you integrate it into your workflow and styles, and to choose an AI tool accordingly.
This workshop is intended for those who have a basic understanding of how LLMs or large language models work and have some experience and comfort with using them, including principles of prompt engineering. This foundational knowledge about AI helps us jump right into deeper conversations about what criteria and tools we should use for choosing an AI tool. More importantly, the workshop participants should bring their desire to use AI responsibly! We didn’t become evaluators to get rich. Instead, we care about our values and the impact we can create, and the AI tool we use should reflect these values. If anyone is uncertain whether they know LLMs or AI well enough to take this training, please reach out to me (kevin@merltech.org), and we can chat! There is a lot of imposter syndrome around AI, and I would like to be as inclusive as possible. You probably know more than you think.
MERL Tech and AEA have a long-standing partnership because there is a mutual recognition of how important the issues of data governance, privacy, and risk are for evaluators. Concerns for these issues are particularly acute with AI, given its hunger for training data and growing (and deserved) distrust of the companies behind these tools. AI apps are popping up everywhere with bold claims of improved productivity and better insights, but they raise many questions about how they protect your data, whether they use your data to train their products, and whether any sensitive information could get disclosed, etc. We will practice a set of questions to ask vendors to make these issues more tangible for research.
There is such a competitive narrative around “adopt AI or perish.” But many of us feel uneasy with it based on how these AI models are created with stolen intellectual properties and exploitative labor practices, how it runs on enormous environmental costs, and how they might not even work all that well with hallucinations and biases. Most of us would agree that we need to use AI responsibly, and it could feel abstract on what “responsible” means here and how one can go about doing it without getting a PhD in computer science or mathematics. This workshop is about showing concrete and approachable ways to translate our values into practices in ways that put people first, not the other way around.
This workshop is about equipping evaluators with practical knowledge and toolkits to choose AI tools more empirically, confidently, and in a value-aligned manner. Participants will learn the criteria and sources of robust data to evaluate AI foundation models. For example, we will practice how to use tools such as model system cards, benchmarks/leaderboards, and academic research to compare different models for hallucination and biases. We will also look at an AI vendor assessment toolkit so the participants know what questions to ask not only about performance but also about privacy, security, and responsible data practices. In addition, we will apply the lens of automation vs. augmentation and reflect critically on where in our workflow AI is best suited, and therefore, the participants come out of the workshop with better clarity on what they wish to use AI for and with what criteria we should evaluate AI tools.
I am very excited that we are having this workshop in person. This past year or so has been an eventful ride for the AEA community, and I am looking forward to learning together with my peers face-to-face. In addition, I strongly believe that one of the most important aspects of these AEA gatherings is an opportunity to learn from our peers and that it will be highly instructional to hear from others have approached this topic and how they plan to apply the content of the training.
AI Disclosure: No AI was used to write this Q&A.
Wednesday, June 24 9:30 a.m.-4:00 p.m.
AEA Headquarters 2001 K Street NW, 3rd Floor North Washington, D.C. 20006
Early-Bird (Ends 5/31/26)
Register Now
Cancellation Policy
AEA must receive all cancellation requests in writing no later than Friday, June 5, 2026. Please email cancellation requests with the reason for cancellation to info@eval.org . All cancellation requests will be reviewed by AEA, and if approved, will be processed within three weeks of the original request. Refunds that are approved will be subject to a $50 cancellation fee.
No refunds are accepted and all sales are final after June 5, 2026.