Date: Tuesday, June 30, 2026
Hello! We are Selam Stephanos and Stewart I. Donaldson from Claremont Graduate University. In this post, we share one lesson from our recent review of published Research on Evaluation (RoE) studies. RoE is scholarship that studies evaluation itself, how it is practiced, what methods are used, and what makes it effective.
One key takeaway: critical appraisal tools can help RoE researchers think more clearly about methodological decisions, not just synthesize evidence after studies are completed.
As more RoE is published, questions about study quality become increasingly important. RoE shapes how evaluators think about methods, evidence, and practice. Yet readers are not always given enough information to judge how much confidence to place in a study’s findings.
Critical appraisal tools can help. These are structured checklists designed to assess the credibility and limitations of empirical studies. Examples include the Joanna Briggs Institute (JBI) critical appraisal checklists and the Mixed Methods Appraisal Tool (MMAT), which are commonly used in systematic reviews to evaluate the trustworthiness of included studies. In a field as methodologically diverse as evaluation, these tools also provide a shared way to think about study credibility across qualitative, quantitative, mixed-methods, and review designs.
In our review of RoE studies published in the American Journal of Evaluation between 2014 and 2024, we applied design-appropriate appraisal tools to assess study quality. Many studies demonstrated moderate to high quality, which is encouraging. However, we also noticed recurring issues: qualitative studies sometimes lacked clear descriptions of researcher reflexivity, quantitative studies did not always fully address potential confounders, mixed-methods studies showed uneven quality across components, and review studies frequently did not follow standard systematic review procedures.
A critical appraisal tool is not just a scorecard used after a study is finished. It can also help researchers anticipate design risks, think through methodological choices, and communicate those decisions more transparently.
When planning a RoE study, review a relevant appraisal checklist before finalizing your design. Start by asking: What kind of study am I conducting? And what do I want an appraisal tool to help me do? The best tool is not always the most familiar one, but the one that fits your study’s design and purpose. For example, a qualitative checklist might prompt you to clarify your theoretical perspective, while a mixed-methods tool might help you plan how to integrate your study’s components more intentionally.
Of course, no single tool fits every type of RoE. Empirical checklists may be less useful for purely conceptual work. But for empirical RoE, they offer a helpful structure for reflecting on design, analysis, and reporting. Used thoughtfully, critical appraisal tools can help make RoE easier to interpret, easier to synthesize, and ultimately more useful for strengthening evaluation practice.
JBI Critical Appraisal Tools – A suite of design-specific checklists (for qualitative, cross-sectional, RCT, and other study types) widely used in systematic reviews to assess methodological quality.
Mixed Methods Appraisal Tool (MMAT) – A single tool that can appraise qualitative, quantitative, and mixed-methods studies, making it especially useful when reviewing a methodologically diverse body of work.
Critical Appraisal Skills Programme (CASP) Checklists – Accessible, beginner-friendly checklists for different study designs, helpful for evaluators who are newer to critical appraisal.
The American Evaluation Association is hosting Research on Evaluation (ROE) Topical Interest Group Week. The contributions all this week to AEA365 come from our ROE TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.