|
Session Title: Peer Review: From Evaluating Science to Evaluating Science Policy
|
|
Panel Session 628 to be held in Room 112 in the Convention Center on Friday, Nov 7, 1:35 PM to 3:05 PM
|
|
Sponsored by the Research, Technology, and Development Evaluation TIG
|
| Chair(s): |
| Isabelle Collins,
Technopolis Ltd,
isabelle.collins@technopolis-group.com
|
| Abstract:
Peer review is one of the main elements of the evaluator's toolkit when looking at the evaluation of science and technology. However, as the focus of evaluation has shifted to evaluating the policies and programs behind the research, the use of peer review has evolved with it. New forms and new uses are emerging, some of which stretch the principles beyond their original intentions, and take the ideas into areas beyond the field of RTD. This panel looks at some of these developments and their implications in the field of science, science policy and the wider policy arena.
|
|
Papers, Projects, Programs and Portfolios: Peer Review as a Public Health Research Evaluation Tool
|
| Robin Wagner,
Centers for Disease Control and Prevention,
riw8@cdc.gov
|
| Trevor Woollery,
Centers for Disease Control and Prevention,
twoollery@cdc.gov
|
| Robert Spengler,
Centers for Disease Control and Prevention,
rspengler@cdc.gov
|
| Jerald O'Hara,
Centers for Disease Control and Prevention,
johara@cdc.gov
|
| Juliana Cyril,
Centers for Disease Control and Prevention,
jcyril@cdc.gov
|
| John Arujo,
Centers for Disease Control and Prevention,
jarujo@cdc.gov
|
|
The independent peer review process for scholarly publications is well known, but this method is also being used as an evaluation tool for research grants and projects at multiple levels, from individual projects to programs to large portfolios comprised of many programs. Peer review can be conducted for a variety of purposes including assessment of scientific merit, mission relevance and potential or actual health impact. The format of peer review is tied to its intended purpose. Reviews may be prospective or retrospective, and include stakeholders or members of the public. The strengths and weaknesses of each peer review approach will be compared, with an emphasis on identifying the types of reviews most suitable for evaluating research outputs, outcomes and impacts. Finally, since peer review arises from qualitative assessment, alternative approaches to research evaluation based on less subjective methods will be discussed.
|
|
|
Peer Review and the Open Method of Co-Ordination: Reviewing National Research and Development Policy Mixes
|
| Patries Boekholt,
Technopolis Ltd,
patries.boekholt@technopolis-group.com
|
|
The Open Method of Coordination (OMC) is a voluntary process of mutual learning between European Member States. It was first introduced at the Lisbon Council (2000) to offer means of spreading good practice and to help Member States to develop their own policies. The EU Member States represented by CREST (Committee for Scientific and Technical Research) decided to enhance mutual learning by using this OMC process. In 2006 CREST launched reviews of national R&D Policy Mixes: the full portfolio of policy instruments and strategies used in a particular country. The country reviews were conducted through peer reviews: the peers being fellow R&D policy makers from other EU countries. The presentation will discuss this peer review process applied in nine European countries in a period of two years. The pros and cons of this review method, the role of the peers and of those who volunteered to be reviewed, and the impacts will be addressed.
| |
|
Peer Review as a Policy Learning Tool
|
| Isabelle Collins,
Technopolis Ltd,
isabelle.collins@technopolis-group.com
|
| Rebecca Allinson,
Technopolis Ltd,
rebecca.allinson@technopolis-group.com
|
| Erik Arnold,
Technopolis Ltd,
erik.arnold@technopolis-group.com
|
| Barbara Good,
Technopolis Ltd,
barbara.good@technopolis-group.com
|
|
Increasing use is being made of peer review in policy fields related to the science system: in research policy, innovation policy, and higher education policy - not simply as a quality measure but as a mechanism for mutual learning. Examples include the reviews of national innovation systems by the OECD and the so-called Policy Mix Peer Reviews organized by the European Commission in the context of the 'Open Method of Coordination' of research and innovation policy and peer review of higher education policies and practices. This paper discusses strengths and weaknesses of the approach as a policy learning tool and examines to what extent peer review is transferable to more general policy contexts unrelated to the science system. Specific questions include the place of such expert opinion in the broader stakeholder analysis, tensions between expert opinion and policy objectives and between independence and engagement.
| |