|
Session Title: Multi-Level and Multi-Factor Approach to Research Program Evaluation: Designing and Implementing the Evaluation of the Europeans Union's Framework Programs for Research and Technology Development
|
|
Panel Session 280 to be held in Wekiwa 7 on Thursday, Nov 12, 10:55 AM to 12:25 PM
|
|
Sponsored by the Research, Technology, and Development Evaluation TIG
|
| Chair(s): |
| Neville Reeve, European Commission, neville.reeve@ec.europa.eu
|
| Abstract:
Evaluation of the European Union's multi billion Euro research Framework Programs (FPs) is a complex task involving multiple numbers of actors and multiple levels of activity across the EU. The European Commission has the role both of coordinating the overall approach to the evaluation of the FPs as well as supporting development of good practice and sharing of information on research evaluation in the EU. In recent years the Commission's system for research evaluation has been considerably revamped to take account of changing needs. Changes either made or envisaged include novel types of evaluation study, stronger coordination, better use of systematically collected data for the purposes of monitoring and more emphasis on the need for evaluation of long-term impacts including institutional and structural changes. This Panel aims to draw together several of the themes from the range of current activities with an interest for policy makers and practitioners.
|
|
Objectives and Constraints With Strategy for Research Program Evaluation: Experience of Developing the European Union's (EU) approach to Evaluation of the Framework Programs
|
| Peter Fisch, European Commission, peter.fisch@ec.europa.eu
|
|
European Commission research evaluation activities have evolved considerably since the start of the Framework Programs in the early 1980s. In general there has been a trend towards greater use of independent studies and quantitative data. The evaluation system is composed of actors (the European Commission, the research performers, evaluation consultants), links (networks to share information and best practice) and rules governing implementation (European Commission wide standards for evaluation and guides to support the implementation of evaluation techniques). This presentation will chart the evolution of the evaluation system and explain the major components which make up that system. It will describe the broader context and the policy objectives which have helped shape this evolution. The presentation will define a set of measures which can be used for measuring the performance of an evaluation system. It will reveal the links between evaluation implementation and context at the systemic level.
|
|
|
Evaluating the Rationale, Implementation and Achievements of a €20 Bn Research Program? Meta Evaluation of Evidence for the Evaluation of the Sixth Framework Programs
|
| Erik Arnold, Technopolis, erik.arnold@technopolis-group.com
|
|
The evaluation of the 6th Framework Program (2002-2006) was carried out in 2008 by a panel of 13 independent experts. It was based on a huge array of evidence including around 30 independent studies, self-assessments and impact assessments from the European Member States. The report from this evaluation was delivered in February 2009 and is now being widely disseminated. This presentation will examine in detail the work of the panel of experts in carrying out the evaluation. It will focus in particular on a meta-analysis of the evidence base which was used. This will provide both an analytical perspective on the challenges of combining disparate data sources but also practical rules of thumb and discussion on the pitfalls to be overcome.
| |
|
Evaluating Research Policy: Understanding and Assessing Policy To Develop The European Research Area
|
| Philippe Laredo, Manchester Business School, philippe.laredo@mbs.ac.uk
|
|
This evaluation of policy concerns the 'European Research Area dimension of Community activities'. It provides a conceptual framework of the ERA to consider its aims, roles and functions. It begins with positioning ERA within the Lisbon process, which shows that ERA is for the European Council, a means towards the central objective to become 'the most competitive knowledge based economy and society'. It then considers two approaches: one deals with the activities considered by the Commission as central to achieve it. The second examines objectives against which to benchmark FP6 i.e. designed to help realise the ERA'. Complementary discussion focuses on the rationales for building the ERA. Once account is taken of the fact that ERA is not a state but the repeated outcome of a long-lasting process of Europeanisation, four dimensions of a rationale, are analysed: fragmentation, knowledge production dynamics, innovation capabilities of existing industries, societal challenges.
| |
|
Monitoring And Assessing The Progress Of Research Programs-Implementation of performance monitoring and interim evaluation of the EU's 7th Framework Programs
|
| Neville Reeve, European Commission, neville.reeve@ec.europa.eu
|
| Costas Paleologos, European Commission, pjf@hq.cas.ac.cn
|
| Martin Ubelhor, European Commission,
|
|
The 7th Framework Program marked a huge step forward for EU research. Nor only was it larger (around 50 billion Euros) and covering a wider range of research areas, it also contained important novelties such as the European Research Council (ERC) for frontier research and Joint Technology Initiatives (JTIs) to achieve industry-led consortia of critical research mass in key technology areas. The presentation will include analysis of the separate exercises to monitor the performance and report on FP7 progress. This includes a new system of implementation monitoring based on core indicators, a progress report in 2009 and an interim evaluation in 2010. Issues to be examined will be the means to strengthen the connection between evaluation and policy and how to assess research in progress. The presentation will explore coordination issues in the light of the connected exercise being carried out by DG Information Society and Media on ICT research.
| |