Date: Wednesday, July 8, 2026
We’re Jacob Gray (Assistant Professor of quantitative methodology) and Kimberly Menendez (Managing Editor of the Journal of Behavioral Health Services & Research). Both at USF’s Florida Center for Behavioral Health Workforce, we share lessons learned on aligning evaluation questions with data and transparent reporting for publishable manuscripts.
Avoid waiting too long to talk to a methodologist. Evaluators often ask for help with their data analysis after having already finished data collection. The hardest part about these discussions is explaining they collected data that won’t answer their evaluation questions or they collected data in an unusable format. When this happens, the evaluation questions often must be revised or reduced in scope, because once data are collected, you have what you have.
Doing only a little bit of research on the methods. Good evaluators will often stay informed on the most appropriate methods for their evaluation question. However, every evaluation question and design are subtly different, and the methods should reflect this. Unfortunately, many evaluators choose a method based on similarity to other studies’ evaluation questions, but they often fail to consider whether those methods are appropriate for their unique data.
Ignoring methods problems altogether. We methodologists have a bit of a curmudgeonly reputation. When we get focused on the details of a method, it’s not because we enjoy picking apart designs, but rather the method has to be calibrated to answer the research question. Ignoring methodological issues can lead to research questions that remain unanswered, even when the researcher claims they have addressed it.
Treating assumptions as nuisances. Any intro statistics course will include discussions about assumptions related to skewness, independence, and measurement. These assumptions are built into the foundation of statistical testing and have clear implications for how to interpret results. More importantly though, violations of these assumptions can be important signals to investigate in the data. Why is variable X skewed? What are the implications for different group variances? These aren’t just nuisances to address but potential future directions for research.
Attempts to hide misalignment usually backfire. When authors try to downplay or smooth over misalignment between the original question and the data collected, reviewers often sense the tension. This creates suspicion. Once trust is weakened, even minor issues can escalate, frequently triggering harsher and more critical reviews across the board. (The dreaded “Reviewer 2” effect).
Precise language protects credibility. If an evaluation only partially draws on implementation science (IS), either fully commit to the framework and methods or avoid the terminology. Sprinkling in IS keywords or frameworks when the study didn’t genuinely apply them automatically routes the manuscript to expert IS reviewers, who will quickly flag the mismatch. This can result in rejection or major revisions, depending on Reviewer 2’s mood that day.
Leave the kitchen sink in the evaluation report. Strong evaluation manuscripts usually succeed because they make one clear contribution rather than trying to report everything that happened in the project. When authors attempt to cover too much ground, the central insight gets lost and the paper feels scattered. Identifying a specific, useful angle and developing it well makes transparent reporting easier and produces a stronger, publishable manuscript.
Honest limitations build trust. A clear, well-written limitations section is one of the best credibility tools an author has. Openly describing design constraints, timing realities, or places where the data only partially answered the original question increases trust with reviewers. Vague language has the opposite effect. Transparency ultimately serves both rigor and usability and can also help temper Reviewer 2’s more paltry objections.
The American Evaluation Association is hosting Behavioral Health TIG Week with our colleagues in Behavioral Health Topical Interest Group. The contributions all this week to AEA365 come from our Behavioral Health TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.