A Cultural Reading of the Program Evaluation Standards, 2^nd edition
American Evaluation Association
Diversity Committee
Cultural Reading Task Force 

EXECUTIVE SUMMARY 

Our Work. Following the 2002 annual meeting, the Diversity Committee of the American Evaluation Association formed a Task Force to review the Program Evaluation Standards of the Joint Committee—inclusive of Standards, Overview, Guidelines, Common Errors, Illustrative Case Descriptions, Illustrative Case Analyses, and Supporting Documentation—to assess how cultural content was addressed in the second edition.[1] We approached culture broadly, inclusive of race, ethnicity, gender, age, sexual orientation, social class, disability, language, and educational level or disciplinary background. We considered both individual characteristics and those of a group or collective (e.g., community or organizational culture). Though we initially conceived of the cultural reading as an internal conversation to educate ourselves, we believe that our observations and concerns are responsive to the Joint Committee’s call for input, and we offer them respectfully for consideration in revising the Program Evaluation Standards.  

To make our process visible and to maximize the usefulness of our work to different audiences, we have synthesized our reading in multiple formats, at varying levels of detail. These are:

• Executive Summary: Narrative summary, including overview of recommended corrective actions.

• Priority Recommendations: Important corrective actions summarized for all standards, matrix format.
• Standards Overview by Categories: Summarizes the relevance to cultural competence, current strengths, concerns or limitations, and recommendations for standards by major category—Utility, Feasibility, Propriety and Accuracy, matrix format.
• Summary of Utility Standards: Summarizes the relevance to cultural competence, current strengths, concerns or limitations, and recommendations for each of the Utility standards, matrix format.
• Summary of Feasibility Standards: Summarizes the relevance to cultural competence, current strengths, concerns or limitations, and recommendations for each of the Feasibility standards, matrix format.
• Summary of Propriety Standards: Summarizes the relevance to cultural competence, current strengths, concerns or limitations, and recommendations for each of the Propriety standards, matrix format.
• Summary of Accuracy Standards: Summarizes the relevance to cultural competence, current strengths, concerns or limitations, and recommendations for each of the Accuracy standards, matrix format.
• A Cultural Reading of the Program Evaluation Standards (2^nd ed.): Narrative discussion among readers of each of the thirty standards, inclusive of Overview, Guidelines and Common Errors, Case Illustrations and Analyses, and suggestions for Supporting Documentation.
  
  

Overall Observations with Respect to Culture. In reviewing the second edition of the Program Evaluation Standards, we find scant attention to both cultural context overall and specific dimensions of human diversity. In addition to missed opportunities to infuse appropriate cultural considerations in the Standards, there are a number of entries that are culturally inappropriate or offensive by virtue of language or content. We appreciate that much of this language/content originated over two decades ago, and we applaud the Joint Committee’s intent to bring the Standards up to the current level of culturally competent professional practice. We encourage the Joint Committee to reflect on these concerns and take the following actions in rewriting the Program Evaluation Standards:  

Provide for increased cultural sensitivity.

• Correct dated (and by current usage, insulting) language with respect to cultural diversity, and update cultural examples with current research.
  
• Avoid taking a deficit approach in addressing culture—e.g., treating it as a barrier or a handicap. Include examples of ways in which culture strengthens and enriches evaluation.
  
• Either remove racist (e.g., U2), sexist (e.g., A3) and ageist (e.g., U4) illustrative cases or include explicit critique of racism, sexism, and ageism in the case analyses.
  
• Update the Standards for congruence with the Health Insurance Portability and Accountability Act (HIPAA), the Americans with Disabilities Act (ADA), and other legislation related to human rights and cultural diversity.

Strengthen references to cultural competence to illustrate its centrality to quality practice. 

• Discuss the standards in relation to the entire process of evaluation, including assumptions made early on. Culture infuses the entire evaluation process; however, many of the standards focus on the end stages of interpreting and communicating findings.
  
• Use the Overview as an opportunity to highlight the relevance of the standard to cultural context and to complex issues of public good. Avoid restrictive language that narrows the scope of the standard.
  
• Add more descriptive content on cultural context to the case illustrations. Lack of content sends an implicit message that it is not necessary to understand cultural context to practice evaluation well.
  
• Expand core principles of reliability and validity beyond narrow measurement concerns, incorporating current theory and research and addressing multicultural perspectives.
  
• Include explicit attention to cultural critique in the analyses of the case illustrations, raising questions, noticing omissions or concerns, and commenting on the use of the standard from a cultural perspective.
  
• Give greater attention to cultural diversity in operationalizing Guidelines and Common Errors, seeking to broaden the dimensions of cultural diversity that are illustrated or addressed, as well as incorporating cultural content more consistently.
  
• Expand the focus of the Standards beyond micro issues to consider mezzo (organizational) and macro (societal) issues as well, making them more relevant to issues of social justice, public good and other community/social concerns.
  
• Add a standard on evaluator self-reflection. Self-awareness is a central component of cultural competence and a basic element of responsible professional practice.
  
• Add a standard on time and timing—a complex issue of great relevance to cultural competence and one that extends far beyond dissemination of results.
  
• Add a separate standard on sampling so that issues of cultural diversity can be explored and discussed.

Avoid tacitly diminishing cultural competence by overemphasizing preordinate and traditional designs.

• Move away from the current assumption that evaluation designs are preordinate, and give more balanced attention to emergent designs. Because emergent designs are increasingly visible in culturally responsive models of practice, to omit attention to them creates cultural as well as epistemological bias.
  
• Avoid overly detailed prescriptions that imply a single (majority) approach to implementing a standard. Instead, raise issues important to consider in selecting contextually appropriate strategies.
  
• Insure even-handed treatment of multiple epistemological and methodological perspectives, including but not limited to those that are grounded in cultural standpoints. The Standards should strive to be relevant to the practice of evaluation under all models.
  
• Notice how the Standards position the evaluator in relation to the client and other stakeholders—especially consumers. Standards should be even-handed with respect to evaluator role and privilege so that they apply equally across evaluation models.

This document is a product of the Cultural Reading Task Force of the AEA Diversity Committee. It was approved by the Diversity Committee on September 28, 2004 and by the AEA Board of Directors, November 3, 2004. We appreciate thoughtful review of these points and we welcome dialogue and discussion. Comments may be addressed to the Chair of the Diversity Committee, Dr. Melvin E. Hall, the Chair-Elect of the Diversity Committee, Dr. Joan LaFrance, or the Chair of the Cultural Reading Task Force, Dr. Karen E. Kirkhart. 

November 3, 2004