|
Session Title: A Roadmap for Developing a Public Health Research Portfolio Evaluation Program
|
|
Panel Session 762 to be held in Royale Board Room on Saturday, November 10, 10:30 AM to 12:00 PM
|
|
Sponsored by the Research, Technology, and Development Evaluation TIG
|
| Chair(s): |
| Robin Wagner,
Centers for Disease Control and Prevention,
rwagner@cdc.gov
|
| Abstract:
This panel presents an ongoing process for developing a program to evaluate the impact and performance of expenditures in public health research. The stages of development employed included (1) conducting a literature-based process and consulting evaluation practitioners to develop a framework for public health research evaluation; (2) identifying relevant knowledge; (3) summarizing the findings from the literature review; (4) evaluating models for gauging net payoffs from the research portfolio and (5) describing a framework to logically and empirically link outputs and outcomes of an agency's research portfolio. The process described could help provide an objective basis for decision makers to allocate limited public funds to support research when it is yielding net payoffs. Also, demonstrating the impact of a public health research portfolio may create additional public support for maintaining or increasing investments in public health research.
|
|
Overview of Methodology Used to Develop a Research Evaluation Program
|
| Jerald O'Hara,
Centers for Disease Control and Prevention,
johara@cdc.gov
|
| John Araujo,
Centers for Disease Control and Prevention,
jaraujo@cdc.gov
|
| Mona Choi,
Centers for Disease Control and Prevention,
mchoi@cdc.gov
|
| Catherine Pepper,
Centers for Disease Control and Prevention,
cpepper@cdc.gov
|
| Robin Wagner,
Centers for Disease Control and Prevention,
rwagner@cdc.gov
|
| Guijing Wang,
Centers for Disease Control and Prevention,
gwang@cdc.gov
|
| Trevor Woollery,
Centers for Disease Control and Prevention,
twoollery@cdc.gov
|
|
Increasing demands for accountability have heightened the need to rigorously assess the value of expenditures on public health research for improving public health. A process for developing a program to evaluate the impact and performance of public health research expenditures is presented. The development process revealed that (1) a rigorous, reliable, and replicable methodology to search and retrieve relevant research articles has great utility; (2) a standardized methodology to review, categorize, and summarize the relevant literature is essential to enhancing knowledge; (3) the extension of existing models and development of new models is essential to promoting the measurement of health impact in an evolving field; and (4) there is a need to identify critical data attributes and build datasets to facilitate rigorous evaluations over time. Future directions and next steps for further developing this evaluation program will be discussed.
|
|
|
MEDLINE Search Strategies vs. Relevant Retrieval: How Closely do They Match for a Research Evaluation Topic?
|
| Catherine Pepper,
Centers for Disease Control and Prevention,
cpepper@cdc.gov
|
| John Araujo,
Centers for Disease Control and Prevention,
jaraujo@cdc.gov
|
| Mona Choi,
Centers for Disease Control and Prevention,
mchoi@cdc.gov
|
| Jerald O'Hara,
Centers for Disease Control and Prevention,
johara@cdc.gov
|
| Robin Wagner,
Centers for Disease Control and Prevention,
rwagner@cdc.gov
|
| Guijing Wang,
Centers for Disease Control and Prevention,
gwang@cdc.gov
|
| Trevor Woollery,
Centers for Disease Control and Prevention,
twoollery@cdc.gov
|
|
Review of relevant literature is an integral component of research evaluation. Retrieval of relevant journal literature in MEDLINE depends on availability of appropriate Medical Subject Headings (MeSH) and/or textwords in citations' titles and abstracts. We identified 'relevant' articles through a team-based literature review that informed the development of a framework for evaluating a public health research portfolio. Team members all read 71 articles retrieved from MeSH/textword search strategies and determined article disposition (Keep, Future Reference, or Discard) based on relevance. MeSH headings within disposition categories were compared to determine an optimal search strategy for capturing relevant articles in future literature searches. Full confidence in literature searching for these topics in MEDLINE is limited by some conceptual gaps in MeSH as well as by the unavoidable lower precision of text word searching. An optimized workable search strategy for capturing relevant research evaluation articles will be discussed.
| |
|
Evaluating Public Research Investment: A Literature Review
|
| Guijing Wang,
Centers for Disease Control and Prevention,
gwang@cdc.gov
|
| John Araujo,
Centers for Disease Control and Prevention,
jaraujo@cdc.gov
|
| Mona Choi,
Centers for Disease Control and Prevention,
mchoi@cdc.gov
|
| Jerald O'Hara,
Centers for Disease Control and Prevention,
johara@cdc.gov
|
| Catherine Pepper,
Centers for Disease Control and Prevention,
cpepper@cdc.gov
|
| Robin Wagner,
Centers for Disease Control and Prevention,
rwagner@cdc.gov
|
| Trevor Woollery,
Centers for Disease Control and Prevention,
twoollery@cdc.gov
|
|
To identify methods for evaluating public health research investments and their impact, a six-member team reviewed 71 articles retrieved from MEDLINE and the grey literature. Through a structured process of assessment of defined attributes (discipline, method, model, relevance, and impact) 38 articles were identified as a core literature of public research evaluation. Themes that emerged from this analysis included
(1) a broad recognition among governments and non-governmental organizations that expenditures on research should impact health outcomes;
(2) evaluating public research investments is a complex process;
(3) several broadly defined evaluation methods consistently appeared, such as qualitative methods (e.g., peer-review) and quantitative methods (e.g., bibliometrics and cost-benefit analysis); and
(4) researchers may apply evaluation terminologies inconsistently, especially in defining and measuring research impact.
This review demonstrated a critical need to develop a standardized approach and terminology to support research portfolio investment and impact evaluation studies.
| |
|
Extending the Pay-Back Model to Incorporate Costs as Well as Benefits to Measure the Net Impacts of Organizational Expenditures on Public Health Research
|
| Trevor Woollery,
Centers for Disease Control and Prevention,
twoollery@cdc.gov
|
| John Araujo,
Centers for Disease Control and Prevention,
jaraujo@cdc.gov
|
| Mona Choi,
Centers for Disease Control and Prevention,
mchoi@cdc.gov
|
| Jerald O'Hara,
Centers for Disease Control and Prevention,
johara@cdc.gov
|
| Catherine Pepper,
Centers for Disease Control and Prevention,
cpepper@cdc.gov
|
| Robin Wagner,
Centers for Disease Control and Prevention,
rwagner@cdc.gov
|
| Guijing Wang,
Centers for Disease Control and Prevention,
gwang@cdc.gov
|
|
The Buxton and Hanney framework for the 'Pay-back Model' includes only benefits from expenditures on research (Buxton & Hanney, 1996).** To fully assess the impact or performance of a single project or a portfolio of research projects, consideration of expenditures is required. Without accounting for both costs and benefits of enabling research projects, erroneous conclusions may be drawn regarding the impact or performance of a single project or a portfolio of research projects. This work extends the Buxton-Haney framework (Pay-Back Model) to include expenditures on research and defines the concept of net impact. The enhanced framework explores the conditions under which expenditures on research derive a net impact in the single project and portfolio of projects cases.
**Buxton MJ, Hanney S (1996) How can payback from health services research be assessed?, Journal of Health Services Research & Policy, vol 1, no 1, pp 35-43.
| |
|
A Bibliometric Methodology to Inform a Logic Model for Evaluating a Public Health Research Portfolio
|
| John Araujo,
Centers for Disease Control and Prevention,
jaraujo@cdc.gov
|
| Mona Choi,
Centers for Disease Control and Prevention,
mchoi@cdc.gov
|
| Catherine Pepper,
Centers for Disease Control and Prevention,
cpepper@cdc.gov
|
| Jerald O'Hara,
Centers for Disease Control and Prevention,
johara@cdc.gov
|
| Robin Wagner,
Centers for Disease Control and Prevention,
rwagner@cdc.gov
|
| Guijing Wang,
Centers for Disease Control and Prevention,
gwang@cdc.gov
|
| Trevor Woollery,
Centers for Disease Control and Prevention,
twoollery@cdc.gov
|
|
The impact of public health research may be assessed by linking metrics of inputs, outputs, outcomes, and impact, which can form a logic model for evaluating a research portfolio. Data to study this logic model were obtained by applying a bibliometric methodology to the 2006 Recommendations of the Advisory Committee on Immunization Practices on Prevention and Control of Influenza. Approximately 90% of references in the Recommendations (n=375) were peer reviewed literature, appeared in 87 journals, and were published between 1958 and 2006. Through 1994, the average number of peer-reviewed publications by year cited by the Recommendations was 3.4¦2.6, compared to 20.1¦8.6 after 1994, and peaked at 40 publications for 2000. Approximately 75% of papers cited appeared after 1994, suggesting a 'knowledge cycle time' of 11 years. This approach illustrates a way of evaluating how peer-reviewed research can impact development of a public health guideline. Study limitations are also presented.
| |