Date: Saturday, June 14, 2025
Hi friends, my name is John LaVelle. I am currently at the University of Minnesota, where I am conducting a research on evaluation (RoE) study about what is expected of our doctoral students. I teach advanced classes that emphasize RoE, such as advanced evaluation theory/theorycrafting and the research on evaluation doctoral seminar. Today I’ll share a few of my lessons learned over the years.
One of the most difficult things about RoE is that there are so many directions you can go, and too many options can be hard to manage. Making the topic straightforward by limiting options (for now) is key; I recommend starting with a RoE framework, such as the one Mel Mark proposed in 2008 for studying evaluation theory. For example, RoE could look at the contexts in which an evaluation is conducted, the activities and outcomes of an evaluation process, and professional issues such as accreditation, competencies, or professional development; and understanding evaluators themselves is also an important topic. There might be some overlap, but the categories are a place to start.
Next, look at the literature to see what has been done before. Don’t be afraid to critique your predecessors, and ask yourself how well the published studies hold up to methodological and conceptual scrutiny. When you find a place where the research is limited, ask yourself: do you want to try to describe a phenomenon? Perhaps you want to predict relationships or try to classify the phenomenon into groups. Maybe you want to do a deep dive into the values and perspectives of a group to better understand them.
Say, for example, we want to know more about the impostor phenomenon in evaluators. And let’s imagine this is a first study on the topic, so trying to describe it would be a good first step. To simplify things, we would go to the RoE matrix and “x” out all the things that the study is not. In this case, this study only groups in two categories, so we can leave the other eighteen alone for now.
Writing a RoE paper can be tough because there are many ways to organize your argument; I always recommend starting with a general description about what evaluation is, what type(s) of processes and models exist, and the rationale for these processes in practice. This lets you set the tone for the rest of your paper and precludes readers from inserting their assumptions about what they imagine evaluation to be.
Draw from other fields to organize your argument
When framing RoE, I strongly recommend you link your topic to another area of study, such as psychology, education, policy, or another discipline. This is because it is important to draw from their theories or models to help organize your study. Share the model(s), describe the components, and summarize the research that has been done on it. This gives your RoE study a stronger foundation because you are using an existing model and applying it to evaluation.
Once you have summarized (and critiqued) the model or theory from another field, you bring it back to evaluation to show how the model can be used to understand evaluation. Many of the models are based on available data collection tools that you can adapt with permission, which helps increase the study’s credibility.
A study on the impostor phenomenon might look something like this:
Building a better evidence base for evaluation theory: Beyond general calls to a framework of types of research on evaluation by Melvin Mark
Best wishes with your work, I look forward to learning from you!
The American Evaluation Association is hosting Research on Evaluation (ROE) Topical Interest Group Week. The contributions all this week to AEA365 come from our ROE TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.