|
Session Title: Evaluation as an Agent of Program Change: An Example From Austria
|
|
Panel Session 727 to be held in Royale Board Room on Saturday, November 10, 9:35 AM to 10:20 AM
|
|
Sponsored by the Research, Technology, and Development Evaluation TIG
|
| Chair(s): |
| Klaus Zinoecker,
Vienna Science and Technology Fund,
klaus.zinoecker@wwtf.at
|
| Abstract:
This session demonstrates the use of evaluation as an agent of change in a major Austrian research program. The evaluation study from which it draws bears not only on program management--that is, how the program in question could be managed more effectively, but also it informs public policy-that is, should the program exist and would it benefit from restructuring. The program in question is the Austrian Genome Research Program, GEN-AU (GENome Research in AUstria), Austria's first top-down research grant program. The two presentations will treat the background of the evaluation study, its aims, methods, major findings, implications for program management and public policy, and observations about changes subsequently made in response to the findings.
|
|
The Evaluation of Genome Research Austria (GEN-AU): Overview of the Study's Aims, Structure, Methods, Results, Implications, and Impacts
|
| Klaus Zinoecker,
Vienna Science and Technology Fund,
klaus.zinoecker@wwtf.at
|
| Alfred Radauer,
Austrian Institute for SME Research,
a.radauer@kmuforschung.ac.at
|
| Brigitte Tempelmaier,
Austrian Economic Service,
brigitte.tempelmaier@univie.ac.at
|
| Iris Fischl,
Austrian Institute for SME Research,
i.fischl@kmuforschung.ac.at
|
| Roald Steiner,
Austrian Institute for SME Research,
r.steiner@kmuforschung.ac.at
|
| Rosalie Ruegg,
TIA Consulting Inc,
ruegg@ec.rr.com
|
|
This presentation provides background for understanding the program and its evaluation, and lists steps in the design process of Gen-AU. It discusses the study time frame and its influence on selection of methodological approach, identifies major players involved with GEN-AU and the types of projects funded, presents the logic chart developed for GEN-AU and relates the associated systematic identification of the program's outputs, outcomes, and impacts. With this background, findings and implications are discussed, as well as the subsequent reactions of policy makers and program managers. The presentation concludes with an account of changes that appear attributable to the study, and lessons learned.
|
|
|
Developing a Plan for Future Monitoring and Impact Analysis of Genome research Austria (GEN-AU)
|
| Rosalie Ruegg,
TIA Consulting Inc,
ruegg@ec.rr.com
|
|
Absent from the program was a plan for future program evaluation. A recommendation of the study was that appropriate steps be taken by GEN-AU to ensure that evaluation of outputs, outcomes, and impacts of the program be planned and implemented. To that end, the Logic Model developed with GEN-AU's program managers was used to construct an evaluation framework. The program's mission and goals were examined to derive key questions regarding program success. Three sets of potential indicators for monitoring progress of the program were constructed: Activity-based Performance Indicators, Output-based Performance Indicators, and Outcome-based Performance Indicators. General guidance was provided for evaluating longer-term program impacts. Specific approaches were suggested for addressing the key questions about impact. Parallel recommendations were made for supporting data collection.
| |