An Introduction to the Cultural Reading of The Program Evaluation Standards (2nd edition)
Welcome to the American Evaluation Association (AEA) site of the Final Report of A Cultural Reading of The Program Evaluation Standards (2nd edition). This is a report of the Diversity Committee of the American Evaluation Association that was approved by the AEA Board, November 2004. This site contains the primary source materials, intermediate working documents and final reports of the Diversity Committee’s Cultural Reading Task Force as well as an opening orientation to our work and its many participants.
Overview The Cultural Reading began as text conversations among a group of professional evaluators who share expertise in and concern for issues of cultural diversity and cultural context in evaluation. The purpose of the cultural reading was to review The Program Evaluation Standards (2nd edition) with respect to coverage of cultural diversity, treatment of cultural concerns, and attention to cultural competence. This conversation was intended to build knowledge among Diversity Committee members and other relevant stakeholders within the AEA; to identify specific passages that are in need of revision; and to gather ideas on relevant resources and suggested changes for input to The Joint Committee on Standards for Educational Evaluation. Throughout this reflection, culture is broadly defined, inclusive of race, ethnicity, gender, age, sexual orientation, social class, disability, language, and educational level or disciplinary background. It includes both individual characteristics and those of a group or collective (e.g., community or organizational culture).
Background At a meeting of the Diversity Committee convened by Chair Edith Thomas at the 2002 Annual Meeting of the AEA, discussion turned to agenda items for 2003. It was noted that The Program Evaluation Standards (2nd edition) would be up for consideration by the Joint Committee in fall 2003, and Karen Kirkhart expressed the strong opinion that these standards needed to be revised to update and improve their attention to dimensions of cultural diversity. This stimulated interest among members present. Kirkhart proposed that the Diversity Committee do a group reading of the Standards and record their critical reflections, exchange ideas, etc. so that the Committee would be in a more informed position to offer comment, reflect interpretations from different perspectives, and impact the process of Standards revision through established channels, via the AEA representative to the Joint Committee, Dianna Newman. The cultural reading became a 2003 action item for the Diversity Committee, chaired by Satish Verma. Karen Kirkhart volunteered to chair the work group. The Board approved this action item at its February 2003 as an element of the Diversity Committee’s 2003 Action Plan.
Participants Initial readers were members of the Diversity Committee in 2002 (when the idea was initiated) and 2003 (when Draft 1 was completed). Charles Thomas joined the reading group in Spring 2003 to provide continuity with the Building Diversity Initiative’s Task Force on Guiding Principles for Evaluators Working Across Cultures, which he chaired. In alphabetical order, the initial readers were:
Claude F. Bennett, U.S. Department of Agriculture
Denice A. Cassaro, Cornell University
Melvin E. Hall, Northern Arizona University
Stafford Hood, Arizona State University
Lester Horvath, Evaluation Consultant, Connecticut
Elmima C. Johnson, National Science Foundation
Karen E. Kirkhart, Syracuse University
Donna M. Mertens, Gallaudet University
Sharon Rallis, University of Connecticut
Edith P. Thomas, U.S. Department of Agriculture
Charles L. Thomas, George Mason University
Satish Verma, Louisiana State University
Elizabeth Whitmore, Carleton University
The work of the 2003 Task Force was supported by:
Crystal Collette, Syracuse University
David Schlesselman, Syracuse University
In 2004, the Diversity Committee, chaired by Melvin Hall, appointed a Task Force to synthesize the text generated by the initial readers and to prepare comments in formats suitable for dissemination to the Joint Committee and other relevant audiences. The members of the 2004 Task Force were (in alphabetical order):
Cindy A. Crusto, Yale University
Joan LaFrance, Mekinak Consulting
Craig W. Russon, Kellogg Foundation
Procedure The first round of the Cultural Reading began in January 2003 and concluded in August 2003, yielding roughly 100 single-spaced pages of text material, summarizing and critiquing the Standards. Members read the standards together, posting their comments to the entire group via email, then conversing and exchanging ideas around the postings. Readers created their own “archives” by saving the emails to a folder. Each standard was read in order of presentation in the book. Not everyone logged in comments on every standard, and some readers were more active than others in the process. Karen’s Spring 2003 Graduate Assistant, David Schlesselman pulled together the comments at the end of each of the four sections—Utility, Feasibility, Accuracy and Propriety—as well as an integrated document at the end. This became Draft 1. In Fall 2003, Crystal Collette, then a Graduate Assistant at Syracuse University, searched the literature and compiled a bibliography of culturally relevant references by standard. This bibliography was provided to the Joint Committee as a freestanding document in 2004 and was subsequently incorporated in the second draft of the Cultural Reading.
In 2004, moving the cultural reading from a private to a public document remained an action item of the Diversity Committee. The second round of the Cultural Reading began in January 2004 and concluded in August 2004, yielding a set of prioritized action items and summary matrixes organized by individual standard as well as by category (Utility, Feasibility, Propriety, Accuracy), a narrative summary, and an edited version of the full text. The content and format of these products are summarized below.
Cultural Reading Documents Click on the links below to view the documents in PDF format.
To download a free PDF reader go to: http://www.adobe.com/products/acrobat/readstep2.html
Current Status These documents were approved by the Diversity Committee on September 28, 2004 and sent to the AEA Board of Directors for review and approval. The Board made minor revisions to the Executive Summary to improve clarity and approved the revised documents as the Final Report of the Cultural Reading Task Force on November 3, 2004. These documents are available to support the work of AEA members, other AEA Committees, and the Task Force of the Joint Committee on Standards for Educational Evaluation charged with revising the second edition of The Program Evaluation Standards (Donald Yarbrough, Chair). The documents may also be useful to other persons or organizations that have an interest in educational evaluation. Elmima Johnson, a participant in both phases of the cultural reading, assumed the position of AEA representative to the Joint Committee in January 2005. The Diversity Committee will seek continued opportunity for input as the Joint Committee Task Force moves through the revision process.
We welcome dialogue and discussion. Comments may be addressed to: