| Session Title: Long-term Impacts: Evaluating Long-term Research Impacts and Valorising Evaluation Long-term |
| Multipaper Session 978 to be held in Malibu on Saturday, Nov 5, 2:20 PM to 3:50 PM |
| Sponsored by the Research, Technology, and Development Evaluation TIG |
| Chair(s): |
| Neville Reeve, European Commission, neville.reeve@ec.europa.eu |
| Abstract: Traditionally, evaluation and monitoring of the EU Framework Programmes has been aligned with the policy and decision-making processes, including for example the recent Interim evaluation of FP7. But one of the effects has been a tendency to treat evaluation results solely as inputs to these processes, with inadequate attention to securing the wider and even long-term impacts. At the same time evaluation has itself often been short term in nature, carried out with the sole intention of validating current activities. The effect has been to ignore long term impacts. This session will examine separate recent initiatives intended to address these deficits. |
| Long-term Impacts of the Framework Programs |
| Neville Reeve, European Commission, neville.reeve@ec.europa.eu |
| The traditional focus of EU evaluation work has been towards the next legislative requirement, in effect preparation of a future FP. While laudable from the perspective of ensuring evidence-based policy, the concomitant weakness has been less attention given to the analysis of the longer-term impacts. With a view to addressing this deficit a project was launched in 2010 to examine the impacts of the 4th, 5th, and 6th FPs, on a selection of key research areas. This used innovative quantitative techniques both to select and analyse different fields. Based on bibliometrics and co-word analysis, these have sought to identify breakthroughs in different fields and map these to the research carried out under the FPs. At the same time, case studies have supported more in-depth analyses. This paper will present the set up and findings from this work, including lessons and pitfalls on methodology. |
| Long-term Impacts of FP7 Evaluation Results |
| Peter Fisch, European Commission, peter.fisch@ec.europa.eu |
| The Interim evaluation of FP7 was carried out by independent experts based on an extensive set of studies, analyses, and statistics. The resulting evaluation report was presented to the European Commission in November 2011. Since then there have been extensive efforts to build-on the results of this evaluation through, amongst other things: a specific and formal exercise by the Commission to reply to the results of the evaluation; dissemination activities; formal political processes, including by the European Parliament and the European Council to fashion independent replies; a major conference to discuss findings and present new ideas; a range of further consultations intended to lead towards a future funding scheme. These efforts are aimed at valorising the results of the evaluation and monitoring and to secure a lasting basis for policy making and further evaluation work. This session will present these activities critically, examining weaknesses and strengths. |
| Indicators for Measuring Outcomes and Impacts |
| Carlos Oliveira, European Commission, carlos.oliveira@ec.europa.eu |
| Policy instruments providing EU's support to research and innovation in ICT often address a multitude of goals and objectives on various levels, ranging from the broad goals of growth and competitiveness or the contribution to addressing societal challenges such as ageing, climate change and social inclusion to very specific objectives in specific ICT technology fields (robotics, microelectronics, etc.). This results in complex and heterogeneous portfolios of projects which are combined with the need to answer to the expectations of various stakeholders (industry, policy-makers at national level, parliamentarians (EP), social groups, national governments). The presentation covers the process of identification of suitable measures (or proxies) for supporting the monitoring and ex-post evaluation of these programmes in terms of their effectiveness, efficiency, relevance, coherence, sustainability, utility, and added-value. The overall coherence of the selected indicators in terms of relevance, understandability, succinctness, contextualisation and data gathering requirements are objects of special attention. |