| Session Title: Building Data Systems to Support Evaluation in a Biomedical Research and Development (R&D) Environment |
| Multipaper Session 736 to be held in Wekiwa 7 on Saturday, Nov 14, 9:15 AM to 10:45 AM |
| Sponsored by the Research, Technology, and Development Evaluation TIG |
| Chair(s): |
| James Corrigan, National Institutes of Health, corrigan@mail.nih.gov |
| Discussant(s): |
| Robin Wagner, National Institutes of Health, wagnerr2@mail.nih.gov |
| Abstract: Gaining access to high quality analytically-useful data constitutes a common challenge that NIH program evaluations must address. The widespread nature of this challenge presents an opportunity for useful electronic tools to have broad impact on both NIH program management and program evaluations. Tools in various stages of development are addressing the needs of both external and internal users/audiences. External resources address the need for public transparency and internal resources provide the evidence base for program management decisions. Examples of data systems relevant to evaluation activities will be presented, highlighting the evaluation and assessment needs that these tools address. External and internal systems, as well as NIH-wide and Institute-specific systems will be covered. The systems to be discussed are: 1) NIH Research Portfolio Online Reporting Tool (RePORT); 2) Electronic Scientific Portfolio Assistant (e-SPA); and 3) CareerTrac. |
| Data Infrastructure and Access: The National Institutes of Health Research Portfolio Online Reporting Tool (RePORT) |
| James Onken, National Institutes of Health, onkenj@mail.nih.gov |
| In this presentation, we provide an overview of the National Institutes of Health (NIH) Research Portfolio Online Reporting Tool (RePORT). Easy access to information and the lack of a data infrastructure to support complex analyses of federal research programs can be a major impediment to program evaluation, reporting, and decision support. Meaningful assessments require the integration of disparate types of information from a variety of both structured and unstructured data sources. RePORT and its Expenditures and Results system, RePORTER, are experimental efforts to integrate multiple electronic sources of data on NIH research programs and provide public access to these resources. Through this initiative, the NIH hopes to meet its goals of promoting transparency in government, communicating the nature of NIH research programs and the scientific advances they produce, and building an infrastructure to support the "science of science." |
| The Electronic Scientific Portfolio Assistant (e-SPA): A Tool for Analysis of Research Project Portfolios |
| Kevin Wright, National Institutes of Health, wrightk@mail.nih.gov |
| The Electronic Scientific Portfolio Assistant (e-SPA) is an innovative software tool designed to provide staff of the National Institute of Allergy and Infectious Diseases (NIAID) with the capability to analyze outputs of research portfolios and individual investigators. The development of e-SPA grew out of program needs to ensure accountability and transparency of information about program performance and results. The tool generates user-defined portfolios of NIAID-funded projects and provides links to multiple R&D performance indicators including funding, publications, citations, impact factors, inventions, and patents. In addition to being robust, e-SPA is flexible and customized to meet the needs of a particular analysis or application. Portfolios are continually refreshed based on existing search parameters, allowing a "dashboard" like view of portfolio performance. Program managers can refine indicators to show citation counts with or without self-citation. The analysis can include views of productivity for various time horizons, perhaps capturing precipitous declines or increases. |
| CareerTrac: An Aid to Effective Evaluation |
| Linda Kupfer, National Institutes of Health, kupferl@mail.nih.gov |
| Ramkripa Raghavan, National Institutes of Health, raghavar@mail.nih.gov |
| This presentation will address the role of a data tool, CareerTrac, in enhancing the ability of NIH program managers to evaluate, monitor and report on their training programs. CareerTrac is a web based application used to collect data and track trainees supported on grants from the Fogarty International Center (FIC). The primary goal of this system is to create a complete trainee roster for all FIC research training programs and to monitor and evaluate outputs, outcomes and impacts of the programs. In addition to quantitative data such as publications and grant funding, CareerTrac uniquely captures qualitative information (such as trainee's career highlights, their role in policy development, honors/awards conferred and leadership roles) that is otherwise difficult to track. Data from CareerTrac will be potentially used to develop benchmarks for FIC programs and will allow FIC to better compare programs with each other and with other global health training programs. |