Conducting Rigorous Evaluations in Operating Justice Agencies by Chip Coldren

Date: Friday, July 25, 2025

Hello, AEA365 community! Liz DiLuzio here, Lead Curator of the blog. This week is Individuals Week, which means we take a break from our themed weeks and spotlight the Hot Tips, Cool Tricks, Rad Resources and Lessons Learned from any evaluator interested in sharing. Would you like to contribute to future individuals weeks? Email me at AEA365@eval.org with an idea or a draft and we will make it happen.

Greetings colleagues. My name is Chip Coldren. I am a Senior Fellow at the CNA Center for Justice Research and Innovation (JRI), where we engage in a range of research, training and technical assistance, and organizational innovation efforts in the justice arena.

JRI conducts rigorous evaluations of justice innovations in field settings, in operating law enforcement and corrections agencies. Notable and ongoing evaluations include: an evaluation of body worn cameras in the Las Vegas Metro Police Department, in the Loudoun County (VA) Adult Detention Center, and in the Minnesota Department of Corrections, as well an evaluation of procedural justice training in an adult jail in Massachusetts, and an evaluation of a new law enforcement field officer training curriculum. Each of these evaluations features a randomized controlled trial (RCT) conducted for a year or more in an operating justice agency.

The challenges to successfully conducting RCTs in working justice agencies are numerous, and substantial, especially compared to RCTs in lab settings. They include, principally: ethics considerations regarding randomization of treatment, risks of contamination effects (e.g., when treatment and control research subjects work together on the job), other research design and fidelity issues (e.g., subject recruitment and attrition, adhering to research timelines), history effects and agency contingencies (e.g., shooting incidents, lockdowns, other emergencies requiring deviations from normal working routines), agency staff turnover (e.g., loss of research project ‘champions’ or points of contact, change in agency leadership), and more….

Here, we intend to encourage, not discourage, RCTs in operational justice settings. We affirm the validity of other rigorous evaluation research designs and understand that not all research questions require an RCT to be answered with veracity. At the same time, we push for the most rigorous designs possible in our efforts to contribute to the ever-growing evidence base in justice research. Below, we discuss several practices and approaches to rigorous evaluation in justice agencies, based on our experiences with the research efforts noted above, and based on our Center’s involvement in several national training and technical assistance initiatives – most notably, the Smart Policing Initiative, National Public Safety Partnership, and Project Safe Neighborhoods, to mention a few.

Lessons Learned

Here are some thoughts and recommendations for conducting RCTs in operating justice agencies:

• Remember the risks of doing harm, or having no positive effects, by continuing to implement untested programs or policies. Randomization (or quasi-experimentation) with treatments and other interventions is still the best way to determine effects. Sometimes we have to discuss these matters directly, and at length, with justice agency leaders, helping them see that their current operations may be at cross-purposes with their goals and missions, or that they don’t ‘know’ if their work has the intended effects.
• There are several options for randomization designs and procedures, in addition to randomizing individual research subjects to treatment and control groups. In some instances, we have randomized by organizational units, or by work shifts, or by supervisors. There are pros and cons to every randomization approach, and each should be supported by a statistical power analysis.
• It is important to monitor agency adherence to randomization and research protocols closely. Such monitoring methods will help operationalize and measure the effects of contamination and errors encountered in random assignments, and this will allow for the estimation of how deviations from research design (e.g., contamination) effect the research initiative as well as outcome measures.
• There may be nothing more crucial than the trusting and collaborative relationship you build with the agency you study. In one sense, this goes without saying. In another sense, thinking about implementing and successfully completing RCTs in operating agencies, collaboration must be a goal and an objective of the research – something that is intentionally developed and nurtured, something that is resourced in different ways, something that is tended to throughout the project. It is also something that can be measured.

Much, much more can be said, and put into practice, regarding the successful implementation of sound, rigorous, impactful, and helpful research in operating justice agencies, and we believe the appetite for such research is growing in the field. Please consider consulting the resources provided below, and please reach out to our Center if you would like more information or would like to discuss this approach to evaluation research more directly.

Rad Resources

• The Benefits of Body Worn Cameras, RCT with the Las Vegas Metro Police Department
• The Benefits of Body Worn Cameras in a Local Jail Setting (podcast), RCT with the Loudoun County, VA Adult Detention Center
• Visit the CNA Center for Justice Research and Innovation

Please reach out to me if you would like further information or would like to discuss these matters in more detail: James R. “Chip” Coldren, Ph.D. – coldrej@cna.org.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.