|
Session Title: The Science and Art of Creating Evaluation Standards That Stand the Test
|
|
Panel Session 509 to be held in Capitol Ballroom Section 6 on Friday, Nov 7, 9:15 AM to 10:45 AM
|
|
Sponsored by the AEA Conference Committee
|
| Chair(s): |
| Hazel Symonette,
University of Wisconsin Madison,
symonette@bascom.wisc.edu
|
| Abstract:
Intended as a means to introduce the 3rd edition of The Program Evaluation Standards, this session goes well beyond simple descriptions of the product. Participants get an inside perspective on this extensive and difficult development process as well as the Joint Committee's focus on quality assurance. This session explores four facets of evaluation standards: (a) explanation of the operating procedures for developing standards why they are important, how they have changed, and why they changed; (b) the procedural changes employed by the Joint Committee and input received from the evaluation and practitioner professionals; (c) the fruits of the process that will serve as an American National Standard for the next decade or more, and (d) the steps the Joint Committee takes through an independent validation panel to assure quality in both the development process and the resulting product. Participant questions and comments are encouraged.
|
|
Key Elements of Developing Sound Evaluation and How the Joint Committee Works to Achieve Them
|
| Arlen Gullickson,
Western Michigan University,
arlen.gullickson@wmich.edu
|
|
Since its inception in 1975, the Joint Committee has become the nation's preeminent developer of education evaluation standards. Its first and foremost product,' The Program Evaluation Standards.' is certified as an American National Standard by the American National Standards Institute (ANSI) and has guided evaluation planning and implementation both nationally and internationally since its first publication in 1981.
This presentation focuses on the third edition of these standards. It describes the methods the Joint Committee employs to ensure high quality in its standards and the ways in which development methods have changed since the standards were first developed. This paper tracks the genesis of the standards development process, describes the forces that have shaped its development processes, and addresses the many practical issues that can derail best intentions to assure that the standards developed actually are principles commonly agreed to in the evaluation profession.
|
|
|
Stakeholder Contributions to the Development of The Program Evaluation Standards: 3rd Edition
|
| Don Yarbrough,
University of Iowa,
d-yarbrough@uiowa.edu
|
| Flora Caruthers,
National Legislative Program Evaluation Society,
caruthers.flora@oppaga.state.fl.us
|
| Rodney Hopson,
Duquesne University,
hopson@duq.edu
|
| Lyn Shulha,
Queen's University at Kingston,
shulhal@educ.queensu.ca
|
|
A primary goal of the Program Evaluation Standards development task force is that all interested stakeholders across the 16 sponsoring organizations as well as in the evaluation communities at-large have adequate opportunities to be involved, contribute suggestions, and make recommendations. This paper describes input received during development of The Program Evaluation Standards, 3rd Edition, including results of national and targeted surveys for needs assessment and format development, the 3-phased national and international review processes and results, the on-going hearings at national conferences, and the methodology and yields of the multi-stage field trials, some of which have been supported by the National Science Foundation. The full paper provides examples of feedback received via reviews, hearings, and field trials, and how that feedback has been put to good use. Remaining opportunities to contribute via field trials, reviews and national Web-based asynchronous 'hearings' will be discussed.
| |
|
An Introduction to the Program Evaluation Standards, Third Edition
|
| Lyn Shulha,
Queen's University at Kingston,
shulhal@educ.queensu.ca
|
| Rodney Hopson,
Duquesne University,
hopson@duq.edu
|
| Flora Caruthers,
National Legislative Program Evaluation Society,
caruthers.flora@oppaga.state.fl.us
|
| Don Yarbrough,
University of Iowa,
d-yarbrough@uiowa.edu
|
|
The development process for the revised Program Evaluation Standards, 3rd Edition is nearing completion as the final reviews, hearings and field trials are completed following Joint Committee and ANSI guidelines. The near final draft is expected to be approved in September 2008 and is scheduled for publication in 2009. The 3rd edition is organized into individual chapters addressing the five key dimensions of quality in evaluations: Feasibility, Propriety, Accuracy, Utility, and Metaevaluation. In addition, it contains supporting chapters addressing the history of the standards development and development processes, what is 'new' to this edition and why a revision of the standards is needed now (and will be needed again within the next decade), how to use the standards in practical situations, and extended cases describing how multiple standards are applied in real life settings. This paper provides an introduction to the revised standards and to the 3rd edition book.
| |
|
The New Program Evaluation Standards: A First Look
|
| Marvin Alkin,
University of California Los Angeles,
alkin@gseis.ucla.edu
|
|
This presentation describes the validation panel's analysis of the Joint Committee's work in developing the standards and the quality of the standards that resulted. This panel, comprised of leading educators and evaluators and led by Dr. Alkin who is one of this nation's leading evaluation practitioners and theorists, is invested with determining whether the Joint Committee has met its own procedural expectations for development of standards and produced strong standards that truly are evaluation principles commonly agreed to by the education profession. The makeup of the panel, the procedures used, and the data employed are all described as factors important in panel determinations. This presentation directly addresses findings such as the congruence between procedures mandated for development of standards and actual procedures followed in the development process, as well as other criteria it applied to make in determining the Joint Committee's success in reaching the intended goals.
| |