Date: Friday, September 5, 2025
Hello, AEA365 community! Liz DiLuzio here, Lead Curator of the blog. This week is Individuals Week, which means we take a break from our themed weeks and spotlight the Hot Tips, Cool Tricks, Rad Resources and Lessons Learned from any evaluator interested in sharing. Would you like to contribute to future individuals weeks? Email me at AEA365@eval.org with an idea or a draft and we will make it happen.
Greetings from Ann Arbor Michigan. Go Blue! I’m Jonny Morell, owner of 4.669… Evaluation and Planning. We work with clients to apply the behavior of complex systems to program planning and evaluation.
What good is knowing that you are evaluating a complex system? Does that knowledge help you build a causal path through your model? Does it help you identify outcomes? Does it help you anticipate patterns of change? No. No to all of these questions. What about insight concerning pattern, predictability, and how change happens? Again no.
“I am evaluating a complex system” is akin to “I will analyze my data with statistics.” It puts you in the ballpark, but it does not tell you how to play the game. To apply complexity we need to know what complex system behaviors are operating in our setting, and if we don’t know, we need to make educated guesses. Here are some examples of complex system behaviors that matter for how we do our work.
Multiple paths through a system can lead to the same outcome. This is a statement about program theory, not to mention the methodology and analysis needed to see if the multiple paths are operating.
Emergent phenomena are qualitatively different from their precursors. So to measure them, we need metrics that are different from their precursors. Also, emergent phenomena introduce hiccups into efforts at back-tracing causal paths.
If we can identify the mechanisms of self-organization in a complex system, we can discern the dynamics that keep systems in an equilibrium state. Those equilibria explain both resistance to change and sustainability post-change. (Resistance and sustainability of an innovation are evil twins. Values determine which is which.)
State change behavior implies that an evaluation might have to attend to two different realities, one on either side of the change. This might require
different constructs that need to be measured and different theories of how those constructs support each other.
Because local change can affect the entire trajectory of how a system evolves, some humility is called for when we develop theories of program action and trust in the methodologies we wrap around testing those theories.
Understanding the implications of complex behaviors requires understanding their joint operations. As an example, many local changes may affect how a system evolves, but a whole family of those paths may lead to the same state change. To evaluate the program we need a methodology that detects discontinuous change. And, if the program was implemented more than once, we need to look for more than one causal path to the same state change.
All of these examples identify something about what complex systems do that have implications for planning and evaluation. All of them touch on the content and logic in models, the methodology we need to do the evaluation, and what meaning we extract from data. None of this understanding would come from the mere knowledge that we were evaluating a complex system.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.