|
Session Title: Evaluating Complex Collaborative Science Initiatives: Utilization of Logic Models in Four Clinical Translational Science Institutes
|
|
Panel Session 871 to be held in El Capitan A on Saturday, Nov 5, 9:50 AM to 11:20 AM
|
|
Sponsored by the Health Evaluation TIG
|
| Chair(s): |
| Kevin Wooten, University of Houston, Clear Lake, wooten@uhcl.edu
|
| Discussant(s):
|
| Nick Smith, Syracuse University, lsmith@syr.edu
|
| Abstract:
This panel will address the use of logic models in environments that are highly complex, non-linear, process driven, and institutionally political. Drawing upon a representative sample of academic institutions receiving Clinical and Translational Science Awards (CTSA) from the National Institute of Health, four presenters will address the role of evaluation in CTSA settings that engender controversy and inquiry. Value conflicts such as the need for scientific rigor versus emergent design, outcome versus process evaluation, and empirical methods versus action research are all difficult to address using a logic model approach. Our presentations will involve using logic models to: 1) simultaneously evaluate outcomes and development; 2) integrate multiple constituencies for collaboration and a common vision; 3) develop and evaluate multidisciplinary teams in accordance with effective team science processes; and 4) utilize stakeholders and constituencies in the real-time design of evaluation tools to address relevance and dynamic institutional cultures.
|
|
Linear Logic Modeling in the Non-Linear World of Translational Research Infrastructure
|
| Janice Hogle, University of Wisconsin, jhogle@wisc.edu
|
| D Paul Moberg, University of Wisconsin, Madison, dpmoberg@wisc.edu
|
| Christina Hower, University of Wisconsin, Madison, cjhower@wisc.edu
|
| Bobbi Bradley, University of Wisconsin, Madison, bradley.bobbi@mcrf.mfldclin.edu
|
|
At the UW Institute for Clinical and Translational Research, embedded evaluators use nested logic models to: summarize complex layers of intent, assist with program improvement, encourage an evaluative perspective, communicate program achievements, and identify evaluation tasks and metrics. We think we have a handle on merit and worth, many processes and some outcomes, and formative and summative assessments. However, we continue to struggle with evaluating impacts within complex CTSA environments. CTSAs are interventions in research processes where fuzzy definitions and boundaries overlap with unpredictable emergent changes in program scope. We are not working in a simple environment of high certainty and high agreement about what to do, how to do it, and what impact to expect. Logic models, which work best in situations of static fixed cause and effect models, have become prevalent among CTSA evaluators. We address the utility of logic models within a developmental evaluation of CTSA implementation.
|
|
|
Implications of Logic Modeling in Large-scale Capacity-building Initiatives That Indicate the Need for Emergent Evaluation Designs
|
| Deborah Fournier, Boston University, fournier@bu.edu
|
|
Evaluators studying collaborative research networks grapple with the inherent complexity and highly adaptive nature of Clinical and Translational Science Institutes (CTSI). Many evaluators across the national network of 55 CTSIs use logic models prospectively to communicate a common vision across multiple projects and sites, draw formative and summative evaluation plans based on specified metrics, guide operational decision-making, and report collaborative scholarly products to the funding agency. The author will present how logic models are used at Boston University's CTSI and examine a critical question: "To what extent are logic models well-suited to evaluating the profusion of moving parts involved in such complex infrastructure development and capacity-building?" Using a systems perspective, the case shows the merit and constraints of models related to emergent evaluation designs. The paper has relevance to evaluation practice regarding complex, highly contextualized initiatives and, more specifically, the NIH nation-wide initiative on advancing Clinical and Translational Science.
| |
|
Utilizing Logic Models in Developing and Evaluating Multidisciplinary Translational Research Teams
|
| Kevin Wooten, University of Houston, Clear Lake, wooten@uhcl.edu
|
| Jean Freeman, University of Texas Medical Branch, jfreeman@utmb.edu
|
|
Much recent interest has been generated around team based science. The discipline surrounding the evaluation of team science is in its infancy, yet holds much promise to unlocking the keys to collaboration, innovation, and scientific breakthroughs. The CTSA award at the University of Texas Medical Branch is completely focused on team based science. The authors will present how logic models have been used to both evaluate team based science, as well as to evaluate team processes along suggested criteria (e.g., NIH Fieldbook on Team Science and Collaboration). Here, the use of logic models to both evaluate the science of multidisciplinary translational research teams as well as the development and evaluation of team processes purported to be of significance, will be depicted. The presentation will focus on the use of hybrid forms of logic models designed to evaluate the dynamic, non-linear, developmental sequence and processes required of effective scientific teams.
| |
|
Using an Interactive Logic Model to Evaluate a Clinical Translation Science Award Program
|
| Julie Rainwater, University of California, Davis, julie.rainwater@ucdmc.ucdavis.edu
|
| Stuart Henderson, University of California, Davis, stuart.henderson@ucdmc.ucdavis.edu
|
|
In program-driven theory evaluations, logic models articulate a program's direction and goals. However, traditional logic models can be limiting for complex programs with multiple objectives and stakeholders. Constructing a model that actively engages a range of stakeholders, illustrates a comprehensive program story, and serves as a fluid blueprint throughout the full evaluation process can reduce some of these limitations. This presentation will highlight our development of an interactive logic model, constructed with DoView software, for the UC Davis Clinical and Translational Science Center. We will show how an interactive model allows us to 1) capture the Center's high-level and program specific outcomes, 2) link multiple logic models from Center programs, and 3) build a model in real time with stakeholder input. We will discuss benefits and challenges of using interactive logic models, including the importance of empowerment and inclusion and the difficulty of creating a consistent narrative.
| |