|
Session Title: Cases for the Program Evaluation Standards, 3rd Edition: Reflective Practice and Problem-solving
|
|
Panel Session 255 to be held in Sebastian Section I1 on Thursday, Nov 12, 10:55 AM to 12:25 PM
|
|
Sponsored by the AEA Conference Committee
|
| Chair(s): |
| Hazel L Symonette, University of Wisconsin Madison, hsymonette@odos.wisc.edu
|
| Discussant(s):
|
| Hazel L Symonette, University of Wisconsin Madison, hsymonette@odos.wisc.edu
|
| Karen Kirkhart, Syracuse University, kirkhart@syr.edu
|
| Abstract:
A new edition of the Program Evaluation Standards, 3rd Edition, has been completed. This session presents the new standards and their applications in five integrating cases, one for each of the five attributes of high quality evaluations: utility, feasibility, propriety, accuracy, and evaluation accountability. Each case scenario provides background information followed by specific applications with each standard in the context of other standards. Each application has five focuses: stakeholders' roles, evaluation dilemmas experienced by stakeholders, dilemmas related to the specific standard, strategies to implement the standard, and how other standards and complications of addressing multiple standards come into play. In addition, an accompanying book of cases applying the standards in a wide variety of contexts is being developed. The session presents the development of the case book with invitations to become involved in developing or reviewing case applications and dimensions of quality in the cases.
|
|
Utility Illustrated through a Health Promotion Education Case
|
| Lyn Shulha, Queen's University, lyn.shulha@queensu.ca
|
|
The 3rd Edition Utility Standards are U1 Evaluator Credibility, U2 Attention to Stakeholders, U3 Negotiated Purposes, U4 Explicit Values, U5 Relevant Information, U6 Meaningful Processes and Products, U7 Timely and Appropriate Communication and Reporting and U8 Concern for Consequences. Each standard is illustrated through application to an evaluation of a multi-level program to "support the health and well being of community residents in ways that contribute to safe, active, responsive and ecologically responsible community living." Specific applications of the individual standards address the evaluations of environmental day camps for elementary school children and a leadership development project for high school students. Dilemmas include stakeholder identification issues, board member resignations, communication challenges, and differences in values including those related to the effectiveness of the evaluation. All the standards are brought to bear on the issue of how to align stakeholders' wishes and needs with evaluators' knowledge, skill, and expertise in evaluation.
|
|
|
Feasibility and Evaluation of School District Resource Use
|
| Flora Caruthers, Florida Legislature, caruthers.flora@oppaga.fl.gov
|
|
The four Feasibility Standards are F1 Project Management, F2 Practical Procedures, F3 Contextual Viability, and F4 Resource Use. The case, an evaluation of a school district's resource use, illustrates the application of the standards through reflective practice and problem solving. For example, the application of F1 Project Management is illustrated by a team meeting in which members express concern about their ability to complete the project without a plan that contains sufficient detail for all members of the team to understand when. It discusses the work products required given the scope of the project and professional experience of various members of the team. The team's approach to addressing F3 Contextual Viability is illustrated by their efforts to accommodate more than just the informal grapevine operating in and acceptable to in-group members of the school district. Necessary trade-offs and possible conflicts in trying to optimize on all standards simultaneously are addressed.
| |
|
Propriety in an Evaluation of Civic Engagement and Social Change
|
| Rodney Hopson, Duquesne University, hopson@duq.edu
|
|
Propriety refers to what is proper, fair, legal, right, acceptable, and just in evaluation practice. Embedded in the attribute are ethical, legal, and professional issues that relate to rights and responsibilities of evaluators and participants, systems of rules and laws, and roles and duties inherent in the practice of evaluation. The revised and updated Propriety Standards are P1 Responsive and Inclusive Orientation, P2 Formal Agreements, P3 Human Rights and Respect, P4 Clarity and Fairness, P5 Transparency and Disclosure, P6 Conflicts of Interests, and P7 Fiscal Responsibility. To illustrate the reflective practice and use of Propriety, the seven revised Propriety Standards are applied to a case regarding civic engagement and social change in a local community. Individual applications are used to discuss how each standard is implemented in the application. Propriety standards are used to better understand evaluation practice and the dilemmas that require application of multiple standards simultaneously.
| |
|
Accuracy in an Evaluation of Teacher Professional Development
|
| Don Yarbrough, University of Iowa, d-yarbrough@uiowa.edu
|
|
The eight updated Accuracy Standards are A1 Justified Conclusions and Decisions, A2 Valid Information, A3 Reliable Information, A4 Explicit Program and Context Portrayals, A5 Information Management, A6 Sound Designs and Analyses, A7 Explicit Evaluation Reasoning, and A8 Communication and Reporting. Each of these 8 standards are applied to the same case, an evaluation of a multi-level teacher professional development program with multiple components, including a focus of teachers' skills and knowledge related to English Language Learners. The individual applications illustrate how to apply each standard and typical trade-offs that must be considered as different standards (especially with regard to Accuracy but also referencing those from Utility, Feasibility, and Propriety) are applied. The role of different stakeholder characteristics and their potential influences on accuracy provides part of the background requiring reflection and problem solving. Issues related to the way accuracy is intertwined with stakeholders' expectations and backgrounds are discussed.
| |
|
Evaluation Accountability and Metaevaluation
|
| Don Yarbrough, University of Iowa, d-yarbrough@uiowa.edu
|
|
One of the major changes in the Program Evaluation Standards, 3rd edition, is the emphasis on "evaluation accountability through metaevaluation" as a fifth attribute of quality evaluation projects. In the 2nd edition, metaevaluation was addressed in an accuracy standard (A12). However, evaluation quality and accountability require attention to all Utility, Feasibility, Propriety, and Accuracy standards through guided reflection informed by the purposes and contingencies of the metaevaluation. In the 3rd edition, the three Evaluation Accountability standards inform this reflection. They focus on documenting the evaluation in sufficient detail to allow accountability judgments (E1 Evaluation Documentation); supporting internal formative metaevaluation for all evaluation projects (E2 Internal Metaevaluation); and encouraging formal summative metaevaluation when it is feasible and warranted (E3 External Metaevaluation). The standards are illustrated with applications to the four case studies presented in the Utility, Feasibility, Propriety, and Accuracy chapters.
| |