|
Session Title: Moving Beyond Bibliometric Analysis: Emerging Evaluation Approaches at the National Institute of Environmental Health Sciences
|
|
Panel Session 240 to be held in Room 112 in the Convention Center on Thursday, Nov 6, 9:15 AM to 10:45 AM
|
|
Sponsored by the Research, Technology, and Development Evaluation TIG
|
| Chair(s): |
| Edward Liebow,
Battelle,
liebowe@battelle.org
|
| Discussant(s):
|
| Christie Drew,
National Institute of Environmental Health Sciences,
drewc@niehs.nih.gov
|
| Howard Fishbein,
Battelle,
fishbeinh@battelle.org
|
| Abstract:
Evaluations of research programs have traditionally assessed contributions to knowledge through citation analysis. The objective of many agencies that fund biomedical research is to improve health. Conventional bibliometric analyses are insufficient for measuring the many impacts not captured in peer reviewed publications. The National Institute of Environmental Health Sciences Division of Extramural Research and Training is interested in developing approaches to identify and quantify the broader impacts of its research portfolio, going beyond the publications its grantees produce. After presenting an overview of the Division's evaluation approach, the panel discusses the development of a conceptual model of comprehensive research metrics, applies this model to the agency's asthma research portfolio, and, after noting gaps and limitations of data from existing sources, describes a primary data collection effort (survey and key informant interviews) designed to fill significant gaps. Feedback from panelists and audience members about implications for research impact assessment will be encouraged.
|
|
Evaluation of Research Impacts: The National Institute of Environmental Health Sciences (NIEHS) Research Portfolio
|
| Jerry Phelps,
National Institute of Environmental Health Sciences,
phelps@niehs.nih.gov
|
|
The NIEHS mission is to reduce the burden of human illness and disability by understanding how the environment influences the development and progression of human disease. The Institute's extramural research portfolio supports basic research, clinical research, technology development, and training activities associated with the health effects of air and water pollution as well as occupational exposures to contaminants. Mr. Phelps, who manages the evaluation of portions of the Extramural Research Portfolio as the Project Officer on the Battelle contract, will provide an overview of the research portfolio and how plans for future research support are developed. He will then describe the Institute's need for analytical tools that will track program inputs, outputs, and outcomes, (especially outcomes that include more than publications) and support the Institute's efforts to understand how NIEHS research is making a difference in basic science, regulatory and public health arenas.
|
|
|
Conceptual Model of Comprehensive Research Metrics for Improved Human Health and Environment
|
| Jill Engel Cox,
Battelle,
engelcoxj@battelle.org
|
|
Evaluating scientific research programs mainly consists of near-term outputs measured through bibliometrics. Examining long-term outcomes for research programs has been particularly challenging, in part due to the serendipitous collation of scientific evidence, and in part because spatially and temporally distant outcomes are only indirectly linked to specific research findings. Dr. Engel-Cox, a senior environmental scientist at Battelle, summarizes recent efforts to build a logic model and associated metrics to measure the contribution of environmental health research programs to improvements in human health, the environment, and the economy. Expert input and literature research were used to define the components and linkages between extramural environmental health research grant programs and the outputs and outcomes related to health and social welfare, environmental quality and sustainability, economics, and quality of life. The model delineates pathways for contributions by five types of institutional partners in the research process: NIEHS, other government agencies, grantee institutions, business and industry, and community partners. Dr. Engel-Cox briefly discusses two examples and the strengths and limits of outcome-based evaluation of research programs.
| |
|
Scientific and Public Health Impacts of the NIEHS Extramural Asthma Research Program from Existing Data
|
| Shyanika Rose,
Battelle,
rosesw@battelle.org
|
|
Through support for asthma research, NIEHS aims to reduce morbidity, mortality, and other public health effects of asthma. Ms. Rose, a senior scientist and program evaluation specialist at Battelle, will describe a study to assess impacts of the NIEHS asthma research portfolio by characterizing publications resulting from these awards, as well as applications of research in clinical practice, interventions, education, and technology developments. Expert panel review, logic model development, and secondary data analyses shaped this assessment. Findings showed that the largest share of NIEHS-funded asthma research is basic scientific research, yet results are published in clinical investigation journals, increasing the likelihood of translation into practice. NIEHS provides support for genetic ontology research and health education curricula. NIEHS-funded investigators are regularly appointed to policy and legislative advisory posts. Recommendations included closing gaps in documentation of research outputs and outcomes, and employing primary data collection to better characterize uses of scientific knowledge.
| |
|
Scientific and Public Health Impacts of the NIEHS Extramural Asthma Research Program From New Primary Data
|
| Carlyn Orians,
Battelle,
orians@battelle.org
|
|
The purpose of this study is to amplify and extend the assessment of the NIEHS asthma research portfolio using primary data collection to fill in gaps in current documentation. Ms. Orians, a senior scientist and program evaluation specialist at Battelle, describes the study, which began with formative key-informant interviews to help develop a survey targeting the universe of NIEHS asthma researchers and asthma researchers supported by other federal agencies. The survey covered such topic areas as types of research support, dissemination mechanisms, and the development of public health interventions, patents and new drug applications, and commercial products. The survey also covered respondents' views on the impact of their research on community outreach and public awareness, new scientific discoveries, laws, regulations, standards, and clinical care guidelines. The program logic model also led us to ask respondents to characterize observed linkages between their work and changes in environmental quality and exposure to environmental triggers that reduce the prevalence of asthma. As a matter of policy and practice, NIEHS is weighing the time and resource requirements associated with conducting such a survey against the Institutes increased ability to answer questions concerning how its research is making a difference in the regulatory and public health arenas through non-survey methods.
| |