Date: Friday, March 27, 2026
Hello, everyone! I’m John F. Akwetey, a doctoral candidate in Interdisciplinary Ph.D. in Evaluation at Western Michigan University, and a Health Technology Evaluation Expert. I spend a lot of my time thinking about how we evaluate digital health interventions. I wanted to share a lesson I’ve learned about a common pitfall I call “the utilization trap.”
I had a moment a while back that I think many of you will recognize. A stakeholder was thrilled, telling me, “We had 10,000 patients log into our new portal last month!” I congratulated them, but then I asked, “That’s great, but what did they do? Did it change how they manage their health?” The line went quiet. We had fallen into the utilization trap.
The utilization trap is when we mistake usage metrics for impact. It’s easy to do. Utilization data – logins, clicks, downloads, dashboard views – is readily available and looks great in a report. Stakeholders love it because it feels like tangible evidence of success. But it doesn’t tell us if the intervention actually empowered the user. It tells us they accessed the tool, not whether it built their skills, knowledge, or confidence to manage their health more effectively.
To avoid this trap, I’ve started using a simple three-level framework in my own work:
Most evaluations stop at level 2. The real impact – and the real story – is at level 3.
Before your next digital health evaluation, I encourage you to ask your stakeholders this one simple question: “Beyond using this tool, what specific capability do we want to build in our users?”
Are you trying to improve their understanding of their condition? Boost their confidence in talking to their doctor? Help them make more informed decisions? Once you define the target capability, you can measure it directly instead of relying on utilization as a proxy.
There are some great validated tools out there to measure empowerment. The Patient Activation Measure (PAM) is fantastic for assessing a person’s knowledge, skills, and confidence in managing their own health. For measuring confidence more directly, you can’t go wrong with a tailored self-efficacy scale based on Albert Bandura’s foundational work. And for shared decision-making, the CollaboRATE scale is a quick and effective tool.
By shifting our focus from utilization to empowerment, we can move beyond counting clicks and start measuring what really matters: whether our interventions are truly helping people lead healthier lives. Have you encountered the utilization trap in your own work? I’d love to hear how you’ve addressed it!
The American Evaluation Association is hosting Health Evaluation TIG Week with our colleagues in the Health Evaluation Topical Interest Group. The contributions all this week to AEA365 come from our Health Evaluation TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.