Date: Friday, June 19, 2026
I am Ian Hutchins, Assistant Professor of Data and Information Science at the University of Wisconsin-Madison Information School.
Toiling somewhere in the research ecosystem right now, there is a researcher who will save lives. Likely, this scientist is publishing in mid-tier journals and has never had a paper published in a highly prestigious venue. Yet, their work is already being cited by the scientists who will build on it and translate their work into lifesaving interventions. We know this because we can see them in the data, if we care to look. Under the evaluation systems most institutions use today, this researcher is largely invisible. Yet this outcome may occur despite our evaluation systems rather than because of them.
Scientists aspire to evaluation systems that are meritocratic and fair. This raises the question, fair to whom? Science has many stakeholders from enterprising researchers to taxpayers who would like to see science stimulate the economy through innovation, and patients waiting for new or better treatments for disease. At many institutions we have evaluation systems that primarily acknowledge prestige bestowed by the branding of journals and conferences. This system arose from the creation of Cell in the 1970s, marketed as the prestige journal where authors would have the last word on a subject, and later extended to families of prestige journals operationalized by the Journal Impact Factor.
By definition, this system works well for those who have survived the academic gauntlet and now shape self-reinforcing evaluation systems. However, we also know that this process overlooks tens of thousands of researchers who have never published in prestige journals but have authored equally influential work. It fails to acknowledge important contributions to clinical translation that save lives. Downstream economic innovation and patenting are nearly invisible in academic reward systems. This is problematic because it is ultimately unfair, not only to overlooked researchers, but to taxpayers who want innovation to drive a growing economy, and to patients waiting for a new hope for currently intractable diseases.
Ultimately, the evaluation measures we choose shape the science we get. To ensure fairness to all the stakeholders of the research enterprise, these should intentionally reward broadly valued outcomes, rather than producing these as a side-effect of prestige. Fortunately, scientific knowledge networks can identify who is publishing highly influential work outside prestige venues, which discoveries led to inventions, what basic research translated into clinical knowledge, and how novel therapeutics were developed. In addition, these impact-anchored evaluation measures can also be used in forecasting, to flag important bodies of work that look like they will stimulate the kinds of applied research outcomes that resonate with diverse stakeholders.
When discussing fairness in evaluation, the ugly side of research integrity problems cannot be overlooked. These are most visible in retracted papers, but likely stem from structural factors. Misinformation propagated from unreliable published work harms all the stakeholders in the research ecosystem. Here too, scientific knowledge networks may have a role to play. Our lab is studying whether the same methods that can forecast positive outcomes like breakthroughs or future FDA-approved therapeutics, can also help identify areas of heightened research integrity risk. Such methods might be able to identify structural components of research integrity risk that underpin misconduct.
The anticipated deluge of AI-generated articles can frame how these contributions are viewed. Are they a massive research integrity problem? Will they increment the frontier of knowledge somewhat, but ultimately remain self-referential? Or will these lead to faster innovation and patenting, and more new disease therapies? Impact-anchored evaluation measures can help us discern.
The American Evaluation Association is celebrating RTD TIG Week with our colleagues in the Research Technology and Development TIG. All of the blog contributions this week come from our RTD TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.