|
Session Title: Expert Evaluation of Federal Agencies' Program Portfolios in the Program Assessment Rating Tool (PART) Environment
|
|
Panel Session 779 to be held in Pratt Room, Section A on Saturday, November 10, 12:10 PM to 1:40 PM
|
|
Sponsored by the Government Evaluation TIG
|
| Chair(s): |
| Raymond Sinclair,
Centers for Disease Control and Prevention,
rsinclair@cdc.gov
|
| Abstract:
With the advent of PART, independent expert evaluation of federal programs has received more attention. OMB's guidance for PART reviews of federal programs requires regular assessment of program relevance, quality, and impact using expert evaluation. Expert review may be an appropriate and cost-effective evaluation method, especially for research programs that often have uncertain results. However to date, there has been little analysis of the use of independent evaluation for federal programs in the PART environment.
This panel session provides different perspectives on concurrent evaluations of a federal agency's research portfolio by the National Academies, a respected independent science advisory body. National Academies' evaluators were asked to assess both relevance and impact of the agency's programs. Up to 13 reviews are either underway or planned. The purpose of this panel is to better understand the strengths and weaknesses of independent evaluation methods, especially for research organizations subject to PART.
|
|
Expert Evaluation of the National Institute for Occupational Safety and Health's Portfolio of Research Programs in the Program Assessment Rating Tool (PART) Environment
|
| Raymond Sinclair,
Centers for Disease Control and Prevention,
rsinclair@cdc.gov
|
|
The National Institute for Occupational Safety and Health (NIOSH) is engaged in an evaluation of its research portfolio by the National Academies (NA). Multiple studies are completed (2), underway (7), or planned (4) for its workplace sector-focused research programs (e.g., agriculture, mining) and research programs that cross those sectors (e.g., traumatic injuries, respiratory diseases). The NA is recognized for high-quality, independent, expert studies of science questions and government programs. OMB's PART evaluation process requires agencies to use expert judgment review methods, yet there is much to be learned about the process and how to use such reviews to improve program management. This presentation reviews the NIOSH approach to securing useful expert evaluations by the NA, the overall review plan, the review process, results so far, communications with OMB and stakeholders, and the responses of the reviewed programs.
|
|
|
Assisting Federal Programs in Expert Evaluation: Tools and Processes
|
| Valerie Williams,
RAND Corporation,
valerie_williams@rand.org
|
|
The current emphasis on assessing program effectiveness has required many federal programs to re-think how to identify and measure program impact. This has often led to a shift from focusing on program outputs to program outcomes. Yet, for many federal programs there are few tools available to assist them in determining and demonstrating program outcomes. We report our experiences in developing a framework to assist NIOSH in its external review by the National Academies. The methods that we developed rely on the use of logic models to communicate how the program operates and identify its intended outcomes. We developed an outcome-based worksheet to categorize and trace the causal linkages from outcomes back to program activities. Finally, we have developed a structure for articulating the program impact through the use of narratives that build from the outcome worksheets. We discuss the development, implementation and some of the challenges associated with these three tools.
| |
|
The National Academies Review of the Programs of the National Institute for Occupational Safety and Health
|
| Evan Douple,
The National Academies,
edouple@nas.edu
|
|
The National Academies has a long tradition of providing independent expert advice to the nation on issues of science, technology, and medicine, including reviews of research programs. Credibility has been built on the process for the selection of committee members with emphasis on recruiting appropriate expertise (including experience in program evaluation) and avoidance of potential conflicts of interest, providing opportunities for input from affected parties and public participation in the evaluation process, ensuring an evidence-based nature in the deliberations, and a thorough review process prior to the release of the final assessment reports. In 2005, the Academies was asked by NIOSH to evaluate the impact and relevance of up to 15 research programs at NIOSH. To begin that process, the Academies appointed a framework committee that set parameters and offered guidance for the evaluations in order to promote efficiency and consistency of methods and standards across the 15 individual studies. The resulting framework document set the template for the research program evaluations, two of which have been completed and a number of others are underway. The experiences of the framework and evaluation committees will be shared.
| |
|
The Use of Independent Evaluation in the Program Assessment Rating Tool Process (PART)
|
| Norris Cochran,
United States Department of Health and Human Services,
norris.cochran@hhs.gov
|
|
Independent program evaluations play a fundamental role in PART review. As an assessment tool, PART asks programs to provide evidence of effectiveness. Independent evaluations are an important source of evidence. In the Strategic Planning and Program Results and Accountability sections of PART, independent evaluations are explicitly requested to demonstrate that programs are (1) routinely evaluating their effectiveness and relevance to the problem, interest or need, and (2) operating as intended and achieving results. In this presentation, we present the perspectives of a former OMB staff member on the task of evaluating evidence. Special emphasis is placed on the contribution of independent evaluations as a source of evidence.
| |