Integrating Technology in Evaluation TIG Week: Using ChatGPT to Develop and Iterate on Focus Group Protocols by Noah Goodman

Date: Wednesday, May 28, 2025

Hello. My name is Noah Goodman and I’m a research associate at EDC’s Center for Children & Technology and lead at my own evaluation firm Ed Investigations. In today’s post, I’m going to share how I used ChatGPT to develop and iterate on a focus group protocol for a summative evaluation.

My Process

In order to develop the initial focus group protocol, I:

Worked with the client to develop a research matrix, including research questions, subconstructs, and indicators.
Reviewed program documents and previous survey data to find examples of data relevant to the research questions—particularly quotes from teachers discussing relevant outcomes.
Created a project (you need to have a paid version of ChatGPT to do this), where I uploaded notes from early meetings with the client, program descriptions, and info from my research matrix as reference documents I could call upon.
Asked ChatGPT if it could “help me brainstorm 10-20 questions I could ask in the focus group for each of my research questions,” which I then culled through to identify those questions that felt most useful.
Worked with ChatGPT to refine specific questions or key concepts to make them more explicit. For example, the program staff wanted to know about the impact on teachers’ knowledge of current science practices and advancements in the field. I asked ChatGPT to list different types of practices or advancements that teachers might have gained awareness of so I could provide examples and start thinking about possible categories of information.
Asked ChatGPT to help me brainstorm different approaches to running the focus group, such as techniques for generating conversation amongst participants, including using vignettes or having participants write out responses or reflections before discussing.
Tested the protocol with ChatGPT to explore likely responses to each question and develop a list of probes.

Then, after each focus group, I worked with ChatGPT to revise the protocol to help me get more of what I wanted, to identify ideas or themes I might dive into in subsequent groups, and to identify holes in my data I wanted to plug. In order to do this, I:

Uploaded my researcher notes from the focus group alongside de-identified transcripts.
Asked it for suggestions on issues I was experiencing, such as the need to balance diving into one person’s experience while not losing other participants’ attention.

Lessons Learned

This iterative process of using ChatGPT as a spit-ball partner was useful for me as a researcher working by myself.
In her blog post in this series, Valerie Ehrlich discusses chatting with your data as an analytic approach. I found it useful to use manual coding of my researcher notes as an initial entry into my data to (a) familiarize myself with the data I had, (b) identify pulse quotes that seemed particularly important, and (c) to map initial emergent themes. I liked exploring how combining these two analytic tools—coding and chatting with my data—allowed me to present informed follow-up questions to ChatGPT and assess the ideas it provided, noting which of them were based on misconceptions of what teachers said.

Cool Tip

Mocking interviews with ChatGPT was useful in helping me think through the possible responses that teachers might give to any given question and likely probes I could ask. I used prompts like the following: “Could you help me simulate one of these focus groups? I’d like you to pretend you’re a high school AP Biology teacher working with a neuroscientist. Could you role play with me?” ChatGPT was generally better able to intuit what I was getting at with each question than actual participants, so I had to be aware of that bias.
I was able to work through several challenges that I had—in particular around building conversation amongst participants. ChatGPT offered a number of approaches with example probes, including thematic grouping of stories, paired or small group sharing before the whole group, using storytelling as a jumping-off point, interspersed reflection prompts, and structured story swapping. It then provided me with an example flow to balance storytelling and conversation. ChatGPT was also able to revise these suggestions based on my experience in the initial focus groups.
I was able to engage ChatGPT in revising questions I had asked that didn’t produce the information I was looking for by adding actual responses from teachers and noting some of the challenges I saw.

As a solo-researcher, who doesn’t get as many opportunities to discuss my initial impressions from data and analysis as I would like, ultimately what I got from working with ChatGPT was a back and forth that helped me keep my process and thinking lively and creative.

The American Evaluation Association is hosting Integrating Technology in Evaluation (ITE) TIG Week. The contributions all this week to AEA365 come from ITE TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.