Date: Monday, July 14, 2025
Hi, we are Eliza Carroll, Will Minor, and Lauren Toledo from Deloitte’s Evaluation and Research for Action (ERA) Center of Excellence (CoE), writing on behalf of the Health Evaluation TIG. For evaluators, literature reviews are often the first step, especially on fast-paced projects, to quickly understand a program’s context. So, we’re kicking off the week by focusing on this part of the evaluation lifecycle, and we’re eager to explore ways to summarize scientific literature more efficiently and accurately. Recently, our team explored machine learning and AI-assisted literature review approaches while conducting formative assessments of violence prevention strategies. During these reviews, we had the opportunity to compare an AI-assisted approach to screening, evaluating, and analyzing titles and abstracts and full texts with more “traditional” human-led review methods.
Depending on the volume of literature and type of review, each step of the review process can require significant time and effort when using traditional methods. For our violence prevention projects, our database search returned tens of thousands of peer-reviewed articles that required title and abstract review. To gain efficiencies and review the articles with a short turnaround time, we used machine learning and a Generative AI literature review tool to accelerate the review and analysis processes. The graphic below shows our process for one of our reviews and compares machine learning/AI review time estimates to a traditional review where applicable.
AI-assisted literature reviews use advanced technologies to streamline the process. The first major advantage is the significant reduction in time and labor. We used Python to screen thousands of abstracts for relevancy, narrowing them down to a couple hundred in just 12 hours by training the model with inclusion and exclusion criteria and conducting random manual reviews to validate the tool’s accuracy.
Following a manual full text review for accuracy, we used a separate AI tool employing language processing models to derive insights from the included articles. This tool helped us code prompts and extract relevant information from hundreds of articles in two weeks, cutting time spent on this step by over 60%! The AI’s ability to quickly process and analyze large volumes of text allowed us to focus on more critical aspects of the review, such as client support and thoughtful writing.
Finally, we utilized Sidekick, a Deloitte Generative AI tool, to synthesize the findings. Sidekick compared responses extracted by Elicit across articles and identified cross-cutting thematic findings. This process reduced time spent on synthesis by around 80%!
Traditional literature reviews provide an opportunity for evaluators to engage with each document and offer a deep, hands-on understanding of the material but can be time-consuming and labor-intensive. Machine learning and AI-assisted reviews provide significant time savings and efficiency, especially for reviews that include a large volume of articles, allowing researchers to focus on higher-level analysis and synthesis.
We found that while the AI-assisted approach saved us a lot of time, these methods are not without challenges. The initial setup and learning curve for using AI tools can be a barrier for some evaluators, especially as organizations are just beginning to explore this frontier. Training machine learning models and developing effective prompts can be an iterative process that requires careful attention to ensure accuracy. In addition, human oversight is necessary to validate results.
Both traditional and AI-assisted literature reviews have their merits. The choice depends on your project’s specific needs and constraints. Have you used AI in literature reviews? Share with the group in the comments!
This posting contains general information only, does not constitute professional advice or services, and shall not be used as a basis for any decision or action that may affect your business. Deloitte shall not be responsible for any loss sustained by any person who relies on this posting. As used in this posting, “Deloitte” means Deloitte Consulting LLP, a subsidiary of Deloitte LLP. Please see www.deloitte.com/us/about for a detailed description of our legal structure. Certain services may not be available to attest clients under the rules and regulations of public accounting. Copyright © 2025 Deloitte Development LLC. All rights reserved.
The American Evaluation Association is hosting Health Evaluation TIG Week with our colleagues in the Health Evaluation Topical Interest Group. The contributions all this week to AEA365 come from our Health Evaluation TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.