Return to search form  

Session Title: Tools for Useful Performance Assessment of Science and Technology Programs: An Example

Panel Session 411 to be held in International Ballroom E on Thursday, November 8, 3:35 PM to 5:05 PM

Sponsored by the Research, Technology, and Development Evaluation TIG

Chair(s):

Jerald payday loan  Hage payday loan ,  University of Maryland,  hage@socy.umd.edu

Discussant(s):

Alfred payday loan  Powell payday loan ,  National Oceanic and Atmospheric Administration,  al.powell@noaa.gov

Abstract: In this proposed panel, we plan to describe the ongoing relationship between the Center for Innovation and NOAA's Center for Satellite Applications and Research (STAR). In meeting the needs of STAR, the Center has been able to further the development of our suite of tools for the assessment and evaluation of research organizations. These tools are centered around the balanced scorecard approach, and we have been working to develop appropriate metrics for research organizations at four levels: organizational, scientific, innovation, and economic benefits. We will present work from our STAR research projects and discuss the ways that these projects offered the opportunity to explore further development of our assessment tools. The panel discussant will be Dr. Alfred Powell, Jr., Director of STAR.

A Strategic Balanced Scorecard for Publicly Funded Science and Technology Programs Gretchen payday loan  Jordan payday loan ,  Sandia National Laboratories,  gbjorda@sandia.gov There is increasing interest in stimulating scientific advance and innovation, and at the same time there are increasing requirements to demonstrate value of organizations and portfolios. However, both current knowledge of how to stimulate innovation and current performance measurement systems fall short. In this presentation, we discuss our approach consisting of a balanced scorecard of performance indicators based on the use of strategy maps and which are informed by theories of knowledge production and innovation. A balanced scorecard that is based on a strategy map is similar to a logic model/theory of change approach to describing strategy and choosing indicators. A balanced scorecard based on organizational strategy clarifies and communicates the strategy, technical focus and performance, and focused management initiatives. Balanced scorecards reflect multiple stakeholders' definitions of 'success', and thus provide more comprehensive information for policy makers and managers than ad hoc sets of indicators. In addition, we have been working to developing real-time measures of scientific progress that let managers know if performance is as expected, and if not, other indicators to show where the bottlenecks might exist. In this manner, the linkage of performance to management practice and theory is key. Our work with STAR has been critical in our efforts to combine generalizable, real time measures of S&T progress and organizational and mission goals in the balanced scorecard framework.

Perceptions of the Research Environment: Kinds of Networks, Research, and Projects Jerald payday loan  Hage payday loan ,  University of Maryland,  hage@socy.umd.edu Most existing employee attitude surveys do not focus on the specific attributes that scientists and engineers consider to be particularly important for research organizations. The Research Environment Survey (Jordan, 2005) was developed especially for research organizations and covers key attributes of organizational structure and management practices within the research environment which were identified and defined through an extensive literature review and input from fifteen focus groups that included bench scientists, engineers, and technologists, as well as their managers, across various R&D tasks. In this presentation we discuss an application of that survey at STAR and our efforts to tailor the survey. In particular, we will emphasize our analysis of those survey questions that allowed scientists to indicate their preferences for change in selected attributes of the research environment and those that captured internal networks. An important finding of the study was that those scientists that indicated a greater amount of applied work (applied scientists) also expressed the strongest preferences for an increasing amount of time to be allocated to various attributes, particularly those associated with innovative research. We also demonstrated that it is important to look at the structure of the intra-organizational networks, as we found a strong relationship between position in the network and perception of the research environment. For example, do those with high closeness have favorable perceptions of the research environment, particularly on those items that we think are favorable to fostering network connections and activity? All too often, this role of the organizational context in fostering or inhibiting the desired type of networks is overlooked. Other findings and a general discussion of the recommendations to management that are possible based on the research environment survey will be provided.

Measuring the Immeasurable: Innovation and the Economics Benefits of Satellite Data Jonathon payday loan  Mote payday loan ,  University of Maryland,  jmote@socy.umd.edu In this presentation, we discuss our work to measure two aspects of STAR's work: innovation and the economic benefits of satellite data. As we have argued elsewhere, standard measures of innovation-patents, papers and citations---are limited for a number of reasons, primarily because of the time lag between innovation and publication and they do not account for the application of the innovation. However, these measures are standards for a reason-they are easily understandable and readily accessible. In this project, we wanted to explore innovative ways to utilize these measures, as well as couple these measures of scientific innovation to their applications, that is, product or process innovations. Another project with STAR sought to quantify the economic benefits of STAR's primary work, processing and conducting research on satellite weather data. The challenge is akin to quantifying the economic benefits of any basic research, where there is little direct connection to the market. Both of these projects represent efforts to quantify some of the intangible contributions of research organizations into useable indicators that can be incorporated into performance measurement frameworks.