|
ANNOTATED BIBLIOGRAPHY Multiculturalism and Cultural Competence in Evaluation Select References 1995-2007 Compiled by Mona Amer and Samantha Matlin, Yale University School of Medicine for the Cultural Competence in Evaluation Task Force - December 2007 The following annotated bibliography list includes references that address multiculturalism and cultural competence in evaluation research from 1995-2007. Some of the references are theoretical or provide general guidelines for cultural competency. Others are empirical studies that evaluate ethnic- or cultural-specific programs. For published articles with similar or overlapping content, only one or two articles were included from that topic area. This reference list does not address cultural competency in other areas of research or practice, such as psychotherapy or diversity training or school curriculum development. This reference list emphasizes the most recent publications from 1995-2007, and as thus, may not include key references in this area that were published before 1995 or since 2007. Content Areas: Part 1: General References on Cultural Competence in Evaluation Part 2: Illustrative Examples of Culturally Competent Evaluations PART 1: General Reference on Cultural Competence in Evaluation Multiculturalism and Cultural Competence in Evaluation Adefuin,
J. & Endo, T. (2003). Multicultural health evaluation: An
annotated bibliography. Woodland Hilss, CA: The California
Endowment. Available online:
http://www.calendow.org/ The Annotated Bibliography is meant to serve as a practical resource to those interested in deepening their understanding of multicultural evaluation in the health field and accessing examples of promising models of culturally competent evaluation. Boodoo, G.M. (1998). Addressing cultural context in the development of performance-based assessments and computer-adaptive testing: Preliminary validity considerations. Journal of Negro Education, 67, 211-219. Performance-based assessments and computer-adaptive testing are used in multiple contexts that have serious implications for those who take the tests (e.g., for placement, licensing, etc.). To provide fair assessments, cultural factors should be considered. The article highlights the key standards relevant to cultural competence included in the Standards for Educational and Psychological Testing, which was jointly produced by the American Education Research Association, American Psychological Association, and National Council on Measurement in Education. Guidelines from the Educational Testing Service (ETS), which offers the GRE exam, are listed. The author goes on to review the literature relating to culturally-competent assessment, listing guidelines offered by previous authors and providing recommendations. Note- all articles in this volume of the Journal of Negro Education relate to the theme of cultural competency in assessment and testing. Chavez, E.L., & Oetting, E.R. (1995). A critical incident model for considering issues in cross-cultural research. Failures in cultural sensitivity. International Journal of the Addictions, 30(7), 863-874. Failure to incorporate linguistic and cultural factors when conducting research with minority groups can lead to non-representative results. For this article, the authors present anecdotes from previous researchers regarding their work with minority groups. Recommendations for more culturally sensitive research are provided. Edno,
T., Joh, T., & Yu, H.C. (2003). Voices from the field: Health and
evaluation leaders on multicultural evaluation. Oakland, CA:
Social Policy Research Associates. Available online from:
http://www.calendow.org/ Based on 21 interviews with leaders in the fields of health and evaluation, five additional interviews with evaluators and a discussion with the project’s advisory committee, this report captures the perspectives, knowledge and expertise of health and evaluation leaders on culturally competent evaluation, as well as draws out implications for advancing culturally competent evaluation in the health field. English, B. (1997). Conducting ethical evaluations with disadvantaged and minority target groups. Evaluation Practice, 18, 49-54. This paper examines the dilemma of conducting evaluations of programs targeting minority groups when such an evaluation can threaten the stakeholders and consequently lead to distress and negative repercussions for the targeted groups. Threats include “intrusive threat” when the evaluation probes into the individuals’ personal lives causing undue distress, and “threat of sanction” when the evaluation brings to light practices that were previously tacitly accepted, leading to sanctions. The author discusses ways to safeguard the rights of the disadvantaged groups, emphasizing the benefits and limitations of informed consent and engaging the target groups at all stages of the evaluation. This article uses examples from programs for persons with intellectual disabilities.
Frierson, H.T., Hood, S., & Hughes, G.B. (2002). Strategies that
address culturally-responsive evaluation. In J.F. Westat, The
2002 user friendly handbook for project evaluation (pp. 63-73).
Arlington, VA: National Science Foundation. Can be obtained from:
http://www.nsf.gov/pubs/2002/ The National Science Foundation developed this handbook as a straightforward and practical guide to implementing evaluations. A special section addressing cultural responsiveness in evaluation is included. This special section discusses the need for cultural competence in evaluation, then describes in detail how to include culture in all phases of the evaluation, including: preparing for the evaluation, engaging stakeholders, identifying the purpose of the evaluation, developing effective research questions, designing the research protocol, identifying and modifying instruments, collecting data, analyzing data, and disseminating and utilizing the results. Guzmán, B.L. (2003). Examining the role of cultural competency in program evaluation: Visions for new millennium evaluators. In S.I. Donaldson & M. Scriven (Eds.). Evaluating social programs and problems: Visions for the New Millennium (pp. 167-181). Mahwah, NJ: Lawrence Erlbaum. This volume “Evaluating Social Problems…” addresses issues relating to effective evaluation of complex social problems in a variety of settings. The book traces the history of evaluation research and discusses anticipated future directions. The author of this chapter argues that evaluation projects should provide a social justice function. Four aspects of culture are described: 1) culture is an abstract and man-made idea, 2) culture refers to the setting in which a behavior takes place and is transformed, 3) culture contains adaptive values, attitudes, beliefs, behaviors, and languages, and 4) due to its importance, culture is passed down from one generation to the next. Evaluators should be aware of the dynamics of these aspects of culture, and should be careful in ensuring that their research protocols are culturally-competent. Hood, S., Hopson, R., & Frierson, H. (Eds.) (2005). The role of culture and cultural context in evaluation: A mandate for inclusion, the discovery of truth and understanding. Greenwich, CT: Information Age Publishing, Inc.
Johnson, E.C., Kirkhart, K.E., Madison, A.M., Noley, G., & Solano-Flores, G. (2007). The impact of narrow views of scientific rigor on evaluation practices for underrepresented groups. In N.L. Smith & P.R. Brandon (Eds.). Fundamental Issues in Evaluation. Each chapter of this book identifies a fundamental issue facing the field today; considers its implications for theory, method, practice, or the profession; and explores one or more approaches to dealing with the issue. The chapter referenced uses the No Child Left Behind Act of 2001 to examine scientific rigor and knowledge-based evidence to support policy decision making. Particular emphasis is placed on how the widespread call for more rigorous research may impact traditionally underrepresented groups in the American society. Kirkhart, K.E. (1995). Seeking multicultural validity: A postcard from the road. Evaluation Practice, 16, 1-12. This essay is the Presidential Address of the 1994 annual American Evaluation Association meeting in Boston. The speaker argues that there is a need for evaluators to have “multicultural validity” in evaluation theory, methodology, applications, and meta-evaluations, and that this multicultural validity is a prerequisite for social justice. The author defines multiculturalism and presents three characteristics of this concept: 1) the complexity of the definitions, 2) multiple cultural memberships, and 3) the transitional nature of culture. She then proposes three dimensions of validity: 1) methodological soundness, 2) validity of understandings from interpersonal interactions, and 3) consequential validity, which refers to the systems changes caused by the evaluation. Multicultural validity incorporates the components of multiculturalism and validity. The speaker then discusses threats to multicultural validity, including time limitations, culturally unresponsive evaluators and arrogant complacency. Madison, A. (2007). New directions for evaluation coverage of cultural issues and issues of significance to underrepresented groups. New Directions for Evaluation, 114, 107-114. This chapter examines coverage of cultural and underrepresented group issues in New Directions for Evaluation over the last twenty years. McGoldrick, M., Giordano, J., & Pearce, J.K. (Eds.). (1996). Ethnicity and family therapy (2nd ed.). New York: The Guilford Press. The United States is becoming increasingly diverse, and understanding a group’s ethnic culture can lead to more culturally-sensitive practice. Each chapter in this volume describes a specific ethnic group, discussing the overall cultural values, worldviews, attitudes, traditions central to that group. Psychological symptoms may be presented uniquely, and ways to integrate unique cultural understandings in therapy are discussed. Chapters describe the following cultures: American Indian, African origin (African American, Jamaican, Haitian, African American Muslim, Nigerian), Latino (Cuban, Mexican, Puerto Rican, Brazilian, Central American), Asian American (Chinese, Japanese, Korean, Vietnamese, Cambodian, Indonesian, Pilipino), Middle Eastern (Arab, Iranian, Lebanese, Armenian), Asian Indian, European origin (Amish, Anglo Americans, Dutch, French Canadian, German, Greek, Hungarian, Irish, Italian, Portuguese, Scandinavian), Jewish (Soviet, Israeli), and Slavic (Polish, Slovak, Russian, Czech). Mertens, D.M. (1999). Inclusive evaluation: Implications of transformative theory for evaluation. American Journal of Evaluation, 20(1), 1-14. This article focuses on the contribution that transformative theory can have in responding to a call for greater inclusiveness of marginalized groups. Transformative scholars assume that knowledge is not neutral, but is influenced by human interests, and that all knowledge reflects the power and social relationships within society, and that an important purpose of knowledge construction is to help people improve society. The author proposes the use of an inclusive model of evaluation that can address the tension between what is needed accurately represent the experiences of marginalized groups research and the traditional canons of research. Scheurich, J., & Young, M. (1997). Coloring epistemologies: Are our research epistemologies racially biased? Educational Researcher, 26(4), 4-16. This essay provides a conceptual framework for understanding the racial biases underlying the epistemologies in education research. The authors discuss five categories of racism: overt, covert, institutional, societal, and civilizational. The authors argue that the educational epistemologies arose within the social histories and contexts of the White dominant culture, and that all epistemologies are racially-biased in that they develop within the value structure of a particular culture. The negative repercussions of this bias, for example, the dismissal of epistemologies and research generated by the social histories of persons of color, are discussed. Recommendations for addressing these problems are provided. Shiu-Thornton, S. (2003). Addressing cultural competency in research: Integrating a community-based participatory research approach. Alcoholism: Clinical & Experimental Research, 27, 1361-1364. The author reviews the history and definitions of cultural competence, and extends it beyond the traditional focus of clinical and service provider cultural competence. To that end, Cross and colleagues’ (1989) framework of cultural competence values for various systems levels is discussed. This framework provides the groundwork for the author’s discussion of the role that cultural competence plays in community-based participatory research. He lists several questions that can be used in assessing five core cultural competencies in CBPR, which are: 1) developing a value for diversity, 2) conducting a cultural self-assessment, 3) understanding the dynamics of difference, 4) assessing cultural knowledge, and 5) adapting to diversity. Stanfield, J.H. (1999). Slipping through the front door: Relevant social scientific evaluation in the people of color century. American Journal of Evaluation, 20(3), 415-431. The author of this essay outlines the historical contexts of White supremacy in the academic traditions that have interfered with social scientists’ progress in assessing the different realities of the worlds around them. This includes prejudice in institutional structures and the conceptual gap between evaluators and the people they are evaluating. The author challenges that the concept of race is a human construct and racial categorizations are inaccurate and serve to perpetuate a racialized society. The necessity of re-thinking the epistemological foundations of the fields is discussed. The richness and complexities of the Black cultures are described, and the author argues in favor of structural changes, increasing empowerment, and shifting the balance of power. Sue, S. (2003). Science, ethnicity, and bias: Where have we gone wrong? In A.E. Kazdin (Ed.). Methodological issues & strategies in clinical research (3rd ed., pp. 173-188). Washington, DC: American Psychological Association, 2003. pp. 173-188. Despite increasing awareness of the need to enhance research relating to ethnic minority groups, such research continues to lag behind research for the dominant White culture in terms of quality, quantity, and funding. Sue argues that this phenomenon is largely the outcome of a dependence on “scientific psychology” which emphasizes internal over external validity. This emphasis on internal validity leads to multiple challenges for ethnic-specific research including the demonstration of cultural validity of concepts and measures used. Sue recommends 1) research with ethnic minorities emphasizing external validity and applicability of the results, 2) augmentation of current methodologies with ethnographic and qualitative techniques, and 3) examination of the concepts of race and ethnicity. Note- this chapter is a re-print from an article printed in the American Psychologist, Dec 1999, 54(12), pp. 1070-1077. Thompson-Robinson, M., Hopson, R.K., & SenGupta, S. (Eds.). (2004). In search of cultural competence in evaluation: Toward principles and practices. New Directions for Evaluation, 2004(102). San Francisco, CA: Jossey-Bass. This volume consists of several papers relating to the benefits, complexities, politics, and challenges of implementing culturally-competent evaluation. In the first article, the role of culture in the evaluation process is discussed in detail. Cases studies for a variety of settings and groups are provided next, with an emphasis on utilizing past experience to guide recommendations for future services. The author of the second article discusses his experiences as an African American evaluator. The third article discusses culturally competent services for the American Indian community. Fourth, the development and implementation of two culturally-competent HIV prevention programs for Latinos is described. The remaining chapters include an evaluation of a multicultural initiative in a Midwestern school district, cultural aspects of an Upward Bound program, and the personal reflections of an evaluator’s cultural journey. U.S. Department of Health and Human Services. (1996). Advanced methodological issues in culturally competent evaluation for substance abuse prevention (SuDoc HE 20.420:6). Assessment of Cultural Competency in Evaluation Research Beach, M.C., Price, E.G., Gary, T.L., Robinson, K.A., Gozu, A., Palacio, A, et al. (2005). Cultural competence: A systematic review of health care provider educational interventions. Medical Care, 43(4), 356-373. This article presents a literature review and analysis of thirty-four studies that evaluated interventions designed to improve the cultural competence of health professionals. They conclude that cultural competence training shows promise as a strategy for improving the knowledge, attitudes, and skills of health professionals. However, evidence that it improves patient adherence to therapy, health outcomes, and equity of services across racial and ethnic groups is lacking. Betancourt, J.R. (2003). Cross-cultural medical education: Conceptual approaches and frameworks for evaluation. Academic Medicine, 78(6), 560-569. Despite progress in the field of cross-cultural medical education, several challenges exist. Foremost among these is the need to develop strategies to evaluate the impact of these curricular interventions. This article provides conceptual approaches for crosscultural medical education, and describes a framework for student evaluation that focuses on strategies to assess attitudes, knowledge, and skills, and the impact of curricular interventions on health outcomes. Boyle, D.P. (2001). Toward a cultural competence measure for social work with specific populations, Journal of Ethnic & Cultural Diversity in Social Work, 9(3/4), 53-71. This paper assesses the need for further refinement of the concept and suggests a new approach for the development of instruments to measure it. The authors provide an overview of the construct of cultural competence, analyze the conceptual and empirical foundations of four well-known measures, and provide a rationale for the development of culture-specific instruments. Dana, R.H., Behn, J.D., & Gonwa, T. (1992). A checklist for the examination of cultural competence in social service agencies. Research on Social Work Practice, 2, 220-233. In this article the authors discuss the development and pilot implementation of a checklist of agency cultural competence. This checklist was designed as an instrument that can be used during evaluation research. The authors begin by defining cultural competence at the practitioner, consumer, agency administration, and policy-making levels. The authors then describe a compilation of 700 culturally-relevant references that they reviewed in an annotated bibliography. These papers were reviewed to generate items for the instrument relating to culturally competent staff selection, agency attitudes, services available, relationships with ethnic communities, training, and evaluation. The checklist was assessed for observer reliability, and three pilot studies using the checklist are described. Vinh-Thomas, P., Bunch, M.M., & Card, J.J. (2003). A research-based tool for identifying and strengthening culturally competent and evaluation-ready HIV/AIDS prevention programs. AIDS Education and Prevention, 15(6), 481-498. This article surveys the various ways that the concept of cultural competence has been studied, extends the concept to the field of HIV/AIDS prevention, and presents a simple-to-use instrument that operationalizes the concept for use with HIV/AIDS prevention programs. The article also explores the idea of evaluation readiness among HIV/AIDS prevention programs in the hope of eventually enlarging the pool of minority-focused HIV/AIDS programs demonstrated as effective. The resultant tool can serve as a research-based framework that: (a) serves as a cost-effective way to select promising programs—especially promising minority-focused programs—for rigorous outcome evaluation; (b) advances the field of HIV/AIDS prevention research by providing a conceptual framework for studying the relationship between cultural competence and program effectiveness; and (c) serves as a framework for building capacity in HIV/AIDS prevention programs, pointing to ways in which such programs can be strengthened. The Culturally Different and Culturally Responsive Evaluator Hood, S. (2001). Nobody knows my name: In praise of African American evaluators who were responsive. In J.C. Greene & T.A. Abma (Eds.), Responsive Evaluation (pp. 31-43). [New Directions for Evaluation, No. 92]. San Francisco, CA: Jossey-Bass. As an African American evaluator, Hood presents his thoughts regarding the role of African Americans in evaluation research, and the importance of incorporating racial factors in the evaluation process. He reviews the contributions of African American evaluators during the Tyler Years (1940-1960), and demonstrates how their early works paralleled the current concepts of responsive evaluation, including the importance of qualitative data, shared lived experiences, and responses to the critical concerns of the evaluation participants. He questions why the work of these researchers have not been acknowledged, and argues that increasing the number of evaluators of color would lead to more efficient and effective responsive evaluations. Hopson, R.K. (1999). Minority issues in evaluation revisited: Re-conceptualizing and creating opportunities for institutional change. American Journal of Evaluation, 20, 445-451. This essay proposes that the issues of race and minority status should be revisited in the institutional structures overseeing the evaluation fields. The author argues that a thorough re-examination of the racial paradigms evaluators use should be conducted in order for effective evaluation strategies (e.g., empowerment, collaborative, participatory) to be truly effective. As such, opportunities in the academic settings need to be created in order to produce a new generation of research leaders with a futuristic view. This includes recruitment and mentorship of evaluation researchers of color and organizational engagement with issues of race and disadvantage. Letiecq, B.L. & Bailey, S. (2004). Evaluating from the outside: Conducting cross-cultural evaluation research on an American Indian reservation. Evaluation Review, 28(4), 342-357
Preskill, H. S. & Russ-Eft, D. (2004). Multicultural and cross-cultural aspects of evaluation. In H. Preskill & D. Russ-Eft (Eds.). Building evaluation capacity: 72 Activities for Teaching and training (pp. 53-73). Thousand Oaks, CA: Sage. This chapter explores how the notions of cultural awareness, sensitivity, and competence affect various aspects of evaluation practice. The introduction to this chapter is brief, but a long list of resources to further explore this topic is given. The complete book was reviewed by Ross and Hopson in the American Journal of Evaluation (2006; 27; 124). Part 2: Examples of Evaluations that Examine Ethnic or Cultural-Specific Programs Considering Socio-economic and Regional Contexts Chase-Ziolek, M., & Striepe, J. (1999). A comparison of urban versus rural experiences of nurses volunteering to promote health in churches. Public Health Nursing, 16(4), 270-279. The researchers conducted program evaluations of two church-based nursing programs, one in a rural area in Iowa and the second in urban Chicago. The evaluations revealed similarities in program goals and the nurses’ faith-based values in helping their participants. However, due to cultural contexts there were some differences between the two groups. For example, in the rural program contacts with participants occurred more frequently outside of the church and the nurses were older, less ethnically diverse, less likely to work full-time, and had lower education degrees. Eng, E. & Parker, E. (1994). Measuring community competence in the Mississippi delta: The interface between program evaluation and empowerment. Health Education & Behavior, 21(2), 199-220. This paper reports on the evaluation methods employed for a health promotion program in a rural poor county of the Mississippi Delta. The evaluation took an action research approach so that the methods would not contradict or interfere with the program's empowerment agenda. The methods required a close and collaborative working relationship among evaluators and local service providers, community leaders, and program staff who defined and operationalized eight dimensions of community competence, determined the units of analysis, and developed the data collection protocol. Emphasis was placed on using the data to engage the program and three communities in a dialogue on how to confront a system with the difficult issues they faced. The findings revealed that after one year of implementation, community competence moved from social interactions internal to communities to those more externally focused on mediating with outside institutions and officials. At the same time, measures of self-other awareness and conflict containment showed a decrease or virtual nonexistence. Evans, M.E., Mejía-Maya, L.J., Zayas, L.H., Boothroyd, R.A., & Rodriguez, O. (2001). Conducting research in culturally diverse inner-city neighborhoods: Some lessons learned. Journal of Transcultural Nursing, 12(1), 6-14. The author provides recommendations for addressing challenges associated with collecting data in low-income urban neighborhoods. This is accomplished through describing evaluations of three in-family psychiatric programs for children and families in Bronx, New York. The challenges identified in conducting the evaluation included: language and acculturation variability; preserving the rights and autonomy of the participants in an ethical manner; ensuring that the children would not be at greater risk if treated at home; recruitment of staff with experience of the cultural context; unwillingness of potential staff to work in the inner city areas; staff training in cultural competence; lack of additional financial incentives for staff; safety issues; participants’ distrust and uncooperativeness in completing the assessments; missed appointments; and family dissatisfaction with the services. Robinson, B.E., Uhl, G., Miner, M., Bockting, W.O., Scheltema, K.E., Rosser, B.R.S., et al (2002). Evaluation of a sexual health approach to prevent HIV among low income, urban, primarily African American women: Results of a randomized controlled trial. AIDS Education & Prevention, 14(3,Suppl1), 81-96. A randomized clinical trial was conducted to evaluate a culture-specific HIV prevention program targeting low-income urban African American females. These females are at-risk for contracting HIV, and the prevention program, the Women’s Initiative for Sexual Health (WISH), was developed with an emphasis on cultural relevance. For example, the program incorporated community involvement, Afrocentric foods, Afrocentric visual aids, and African American group facilitators. Quantitative and qualitative approaches were used to obtain participant feedback across all levels of the development and implementation phases. Women involved in the program demonstrated greater learning of sexual information but other expected differences were not found. Challenges and recommendations are discussed. Thomas, V.G., & Stevens, F.J. (Eds.). (2004). Co-constructing a contextually responsive evaluation framework: The Talent Development Model of School Reform. New Directions for Evaluation, 2004(101). San Francisco, CA: Jossey-Bass. This volume of articles concentrates on the Talent Development model in conducting evaluations in urban school-based settings. This approach emphasizes a participatory and culturally competent approach with values in 1) engaging stakeholders, 2) co-construction, 3) responsiveness, 4) cultural and contextual relevance, and 5) triangulation of perspectives. The model is especially effective for low-income urban settings in enhancing the relevance of the evaluation and working towards social justice. The first chapter introduces the Model and the subsequent chapters provide illustrative case examples of applying the Model to evaluation practice. Whipple, E.E., & Nathans, L.L. (2005). Evaluation of a rural Healthy Families American (HFA) program: The importance of context. Families in Society: The Journal of Contemporary Social Services, 86(1), 71-82. Program implementation may be different in rural compared to urban areas due to; for example: significantly lower median wage; fragile economic systems; lower educational levels; larger geographical areas; low-density populations; few mental health and social services; and greater distance between the participants and the programs. Authors of this paper conducted an evaluation project of a Midwestern HFA program. The authors discuss the unique socio-cultural (i.e., rural) contexts and argue that such factors should be taken into account when designing rural programs. Considering Spiritual and Religious Contexts DeHaven, M.J., Hunter, I.B., Wilder, L., Walton, J.W., Berry, J. (2004). Health programs in faith-based organizations: Are they effective? American Journal of Public Health, 94(6), 1030-1036. This article reviews the literature regarding evaluation of health programs at faith-based organizations published between 1990 and 2000. The article discusses differences among the programs in terms of health problem addressed and the extent to which issues of faith were integrated in the program. The authors suggest that based on the research thus far, faith-based programs can offer effective health services, including increasing knowledge, screening for disease, and supporting uninsured individuals. Recommendations are provided for 1) increasing collaboration between health-care providers and the faith-based organizations, 2) emphasizing effectiveness over efficacy studies, and 3) building relationships with ethnically- and racially-diverse communities. Hughes, C.B. (2001). Primary care parish nursing: outcomes and implications. Nursing Administration Quarterly, 26(1), 45-59. The authors review previous studies on the effectiveness of parish nursing initiatives. This study evaluated the Primary Care Parish Nursing Practice (PCPNP) model that was established at St. Agnes Roman Catholic Church in New Jersey. For the program, a family nurse practitioner and nurse case manager were hired to provide spiritually-sensitive primary care at St. Joseph’s Hospital and Medical Center for a faith-based community. The purpose of the program was to promote health and prevent disease. The program was evaluated through the use of patient questionnaires and chart reviews. Common health and mental health concerns were listed and compared to data from other samples. In addition to program evaluation results, the authors also provide descriptive case studies of providing faith-based culturally-sensitive care. Kagimu, M., Marum, E., Wabwire-Mangen, F., Nakyanjo, N., Walkakira, Y., Y Hogle, J. (1998). Evaluation of the effectiveness of AIDS health education interventions in the Muslim community in Uganda. AIDS Education Prevention, 10(3), 215-228. In 1992, the Islamic Medical Association of Uganda designed an AIDS prevention project based on recognition of the role of the imam (mosque leader) as the teacher of family behavior and sexual values. The evaluation revealed significant increases in correct knowledge of transmission of HIV, risk reduction behaviors (e.g, condom use, number of sexual partners, and the incidence of extramarital sex. The success of this intervention is attributed to its use of the Islamic religious organizational structure as a vehicle for HIV/AIDS education. This project demonstrates that positive collaboration between health professionals and religious leaders can be achieved and will enhance the success of community AIDS prevention efforts. Stillman, F.A., Bone, L.R., Rand, C., Levine, D.M, & Becker, D.M. (1993). Heart, body, and soul: A church-based smoking-cessation program for Urban African Americans. Preventive Medicine, 22(3), 335-349. This article provides a descriptive overview of the implementation process of the Heart, Body, and Soul program. The program objective was to test strategies to reduce the prevalence of cigarette smoking among urban African Americans in East Baltimore. This study constitutes a prospective randomized trial among inner-city African Americans designed to improve quit rates among church attenders. Innovative culturally specific smoking cessation strategies mediated through lay volunteers from participating churches were implemented in the intensive intervention churches. The essential component of this successful implementation process were building trust and acceptance and providing the technical support to encourage smoking-cessation strategies. This description of the project is presented to assist others involved in church-based trials in urban African American communities. Wilhelm, I. (2002). Most charities and religious groups evaluate programs, report says. Chronicle of Philanthropy, 15(2), 57-59. The Independent Sector, a coalition of 700 charities, foundations, and corporate donors based in Washington, D.C., conducted a survey to determine if religious congregations evaluated their programs. The report found that secular groups were twice as likely to evaluate their programs as religious groups, and that religious groups were less likely to assess service satisfaction. Religious programs argued that evaluation is a challenge because outcomes are often intangible and not measurable, and the evaluation process itself would be costly in terms of finances and time. Equipment such as computers was not available at many programs, and many groups did not have the knowledge of how to conduct an evaluation. Culturally-Responsive Programs for Specific Racial and Ethnic Groups Aktan, G.B. (1999). A cultural consistency evaluation of a substance abuse prevention program with inner city African-American families. Journal of Primary Prevention, 19(3), 227-239. Cultural consistency refers to the congruence between a program (e.g., its philosophies, methodologies, policies, practices, etc.) and the cultural group it is trying to serve. This article describes an assessment of cultural consistency in the Safe Haven substance abuse prevention program for Detroit inner-city African Americans. In addition to the program’s ongoing process and outcome evaluations, an African-centered evaluation was conducted to assess for cultural consistency. Results from the evaluation helped informed modifications to the program such as using culture-specific pictures and visual aids and evaluating corporal punishment from a historical/ cultural perspective. Program evaluation after the modifications revealed greater cultural congruence. Anderson-Draper, M.H. (2006). Understanding cultural competence through the evaluation of "Breaking the Silence: A Project to Generate Critical Knowledge About Family Violence Within Immigrant Communities.” Canadian Journal of Program Evaluation, 21(2), 59-79. This article examines the topical concept of cultural competence for evaluators by presenting reflections on the evaluation of "Breaking the silence: A project to generate critical knowledge about family violence within immigrant communities" as a case example. Experiences of the internal evaluator in relation to cultural competency are explored and implications for practice are presented. Including sufficient time to build relationships, facilitating a learning process, and developing evaluator competencies are among the salient themes presented. Cervantes, R.C., & Peña, C. (1998). Evaluating Hispanic/Latino programs: Ensuring cultural competence. Alcoholism Treatment Quarterly, 16, 109-131. Focusing on alcohol and drug prevention and intervention programs, this article first presents the literature relating to culturally-sensitive strategies for serving Hispanic/Latino persons. The authors report results from a federally-funded 22-site alcohol and drug program that targeted Latino youth and families. The authors administered satisfaction questionnaires to program directors and evaluators at these sites, who reported that a program evaluation was beneficial.
Johnson, E.C. (Chair). (2002, April). The cultural context of
educational evaluation: A Native American perspective.
Proceedings of the National Science Foundation Directorate for
Education and Human Resources, Arlington, VA. Can be obtained from:
http://www.nsf.gov/pubs/2003/ Included in this volume are the proceedings of a two-day workshop aimed at a) determining the future directions for Native American evaluation and b) increasing capacity building. Key figures in the field describe their personal experiences, review accomplishments, and raise challenging questions for the years ahead during the opening statements. Papers from the four sessions, as well as discussion highlights, are included in the volume. In the first session, participants discussed issues pertaining to the academic achievement of Native American students, with an emphasis on how to narrow the achievement gap by devising culturally sensitive and context-inclusive evaluations. In the second session, representatives from federal and national agencies provided information on resource materials relevant to evaluation practices. These included the U.S. Department of Education’s Office of Indian Education and the Bureau of Indian Affairs’ Office of Indian Education Programs. The third session reviewed educational and training needs and opportunities for Native American evaluators, particularly teachers. During the fourth session participants discussed the development, maintenance, and expansion of Native American evaluator networks. Juntunen, A., Hwalek, M., & Neale, A.V. (1999). Tracking and interviewing clients at risk for HIV and substance abuse in a Latino community. Evaluation & Program Planning, 22(3), 305-312. In an effort to increase access and engagement with an HIV program located in a predominantly Hispanic community in Detroit, the authors use a culturally-sensitive approach. This included translating the questionnaires into Spanish, recruiting bicultural/ bilingual research interviewers, using effective incentives for participants ($10 gift certificates), and utilizing culturally-sensitive ways of tracking participants (i.e., sending hand-written rather than typed letters). Ways to ensure integrity of the evaluator are introduced, including: informing clients that the interview was not a member of the staff providers, distancing the interviewer from the program staff (i.e., refraining from gossip or participating in program tasks), and protecting participant confidentiality. Other recommendations are provided such as using a flexible interview protocol and exploring reasons (listed in the article) for low participation rate. Orlandi, M.A., Weston, R., & Epstein, L.G. (Eds.). (1992). Cultural competence for evaluators: A guide for alcohol and other drug abuse prevention practitioners working with ethnic/racial communities (DHHS Publication No. (ADM)92-1884). Rockville, MD: US Department of Health and Human Services. This publication addresses cultural competency for alcohol and drug programs serving ethnically diverse participants. Both conceptual frameworks and practical applications are provided for different ethnic groups, followed by an overview of cultural competence. The ethnic groups discussed are African-American, Hispanic, American Indian/ Alaska Native, and Asian/ Pacific Islander. Smith, N.L., & Jang, S. (2002). Increasing cultural sensitivity in evaluation practice: A South Korean illustration. Studies in Educational Evaluation, 28(1), 61-69. While there are numerous studies evaluating programs in the United States, professional evaluations in the international arena have only begun to increase pace. The authors discuss the definitions and qualities of culturally-sensitive evaluation research. The authors then present the differences between American evaluation practices and culturally-sensitive practices for South Koreans. The South Korean social culture is described, including worldviews, influences of Confucianism and Buddhism, clear authority structures in social groups, and respect for elders. Implications for the evaluator (e.g., challenges in evaluating older persons or negotiating stakeholder differences when explicit conflict is eschewed in the culture) are provided. The authors present the impact of South Korean communication patterns on the evaluation process and the utility and validity of American methodologies in the South Korean context. Tillman, L.C. (2002). Culturally-sensitive research approaches: An African American perspective. Educational Researcher, 31(9), 3-12. The author of this essay describes the need for using culturally sensitivity qualitative research with African Americans. The African American culture differs from European-American culture in multiple ways such as: different values and worldviews; different language patterns; and a shared cultural, historical, and political history. The author discusses a conceptual paradigm shift and implementation of culturally-sensitive research, providing examples from previous studies. Cultural Competence in Youth-Focused Programs Cablas, A. (1998). Culturally competent methods based on evaluation and research: How to use research results in systems of care. In M. Hernandez & I. Mario (Eds.). Promoting cultural competence in children's mental health services (pp. 331-355). Baltimore, MD: Paul H. Brookes Publishing. This is the final chapter in a book that discusses cultural competence across all levels of child mental health services, including: policy planning and infrastructure development, incorporating communities, specific staff training and service provision, and finally research projects. This chapter provides a theoretical framework for understanding, planning, implementing, and evaluating a culturally-competent evaluation program. The chapter includes specific examples, general recommendations, as well as in intervention outcome measure. The benefits of using the research results to guide improvements to youth systems of care are discussed. Hudley, C. (2001). The role of culture in prevention research. Prevention & Treatment, 4, Article 5, n.p. There is a growing awareness of disparities in mental health care for children, particularly low-income, urban, ethnic minority girls and adolescents. In an effort to address these disparities, the author argues in favor of utilizing ecological and culturally relevant prevention and intervention strategies aimed at empowering youth. This is demonstrated in a description of the BrainPower Program, an after-school program targeting aggression for low-income African American boys that was then modified to include Latino boys and both low- and middle-income participants. An ecological and comprehensive approach was used. Lubeck, S., Jessup, P., deVries, M. & Post, J. (2001). The role of culture in program improvement. Early Childhood Research Quarterly, 16, 499-523. This article describes the role of culture in improving the effectiveness of Head Start programs for children of different backgrounds. The authors first review definitions of culture and describe the vying camps of the “policy mechanics” who search for universal indicators of school effectiveness versus “classroom culturalists” who emphasize diversity. Using an ethnographical approach with discourse analysis, the authors assessed the role of contexts at four geographically and ethnically different Head Start programs. Based on interacting and participating with the programs for 10 months to four years, the authors provide implications for taking cultural contexts into account in school program development; for example, by appreciating both national mandates and local needs. Nastasi, B.K., Moore, R.B., & Varjas, K.M.. (2004). School-based mental health services: Creating comprehensive and culturally specific programs. Washington, DC: APA. This book presents the “participatory culture-specific intervention model (PCSIM)” as an innovative and culturally competent model aimed at developing effective school based programs that incorporate the ethnicity, race, and culture of the associated individuals and communities. The first chapter lays a theoretical framework and discusses a participatory process in developing socially valid and sustainable mental health programs in school settings. The second chapter describes how the fields of anthropology and school psychology were drawn upon for the basic concepts of PCSIM. The remaining chapters provide specific, practical, and illustrative information regarding the implementation of PCSIM, a model that incorporates a variety of methods including participant observation, expert consultation, and ongoing program improvement. Pumariega, A.J. (1996). Culturally competent outcome evaluation in systems of care for children's mental health. Journal of Child & Family Studies, 5(4), 389-397. In this guest editorial, the author discusses the area of culturally competent outcome evaluation and cross-cultural mental health research in general and its needs to be greatly developed given the culturally diverse nation we live in and the different needs of culturally diverse children and their families. Such evaluation is crucial in supporting the need for and effectiveness of culturally competent programs and special programs with a focus on particular cultural populations. The imperatives for cost effectiveness and clinical effectiveness which have been promoted by the transition to managed systems of care may actually promote the development of higher levels of cultural competence in community-based systems of care. Culturally competent care may well be the most cost-effective and clinically effective care. Rodney, L.W., Johnson, D.L., & Srivastava, R.P. (2005). The impact of culturally relevant violence prevention models on school-age youth. Journal of Primary Prevention, 26(5), 439-454. The Family and Community Violence Prevention (FCVP) Program was established in 1994 to address the escalation of youth violence among ethnic minorities. This federally funded program adapted the public health model and organized Family Life Centers throughout the country to serve youth who were considered to be at risk for violence and other abusive behaviors. The purpose of this three-year study, 1999-2002, was to determine the effectiveness of the FCVP Program's six-component curriculum in reducing violence among participants. Results from posttest scores of 2,315 youth showed girls 12 and over to be most at risk for deviant behaviors; the program was most effective with boys under age 12. Academic performance and bonding to school were protective factors whereas exposure to violence was a risk factor for all four ethnic groups studied--African Americans, Hispanics, Native Americans, and Native Hawaiians. Cultural anthropologists, public health specialists, and school officials should know that prevention programs can be designed to reflect the unique, culturally appropriate norms of specific ethnic minority groups, even as these programs address shared risk factors. The authors discuss the promising strategy of enhancing academic performance and school bonding to serve as protective factors against school violence, but they also describe age, gender, and cultural differences that must be addressed in future research. Ryan, K.E., Chandler, M., Samuels, M. (2007). What should school-based evaluation look like? Studies in Educational Evalution, 33(3-4),197-212. This article reports the results of an instrumental, mixed methods case study (Stake, 1995) of a culturally responsive school-based evaluation initiative involving fours schools. In studying culturally responsive school-based evaluation, they found some broader issues with school-based evaluation more generally. These challenges include what knowledge schools need to engage in evaluation, organizational constraints which impact what models are appropriate for school-based evaluation approaches, and the meanings of culture. Teachers and principals clearly understood culture in a different way after participating in school-based culturally responsive evaluation. They conclude that more theoretical work and field studies will need to be conducted to further assess being culturally responsive in evaluation. Disabilities and Program Evaluation Dennis, R.E., Williams, W., Giangreco, M.F., & Cloninger, C.J. (1993). Quality of life as context for planning and evaluation of services for people with disabilities. Exceptional Children, 59(6), 499-512. ABSTRACT: A review of definitions in the literature of the term quality of life suggests no agreement on its meaning except that its assessment is subjective. A quality of life framework (an optimal theory model) is offered, which includes individual, common human, and culturally specific values and needs. Theoretical approaches to quality-of-life research have varied, with some seeing value in objectifying and quantifying the concept while others believe the concept is individually unique and subjective. A combination of these approaches has also been used. Promising approaches using quality-of-life themes as a context for program planning for people with disabilities are outlined for the practitioner. Fewell, R.R. & Glick, M.P. (1996). Program evaluation findings of an intensive early intervention program. American Journal on Mental Retardation, 101(3), 233-243. An intensive early intervention program was evaluated by assessing gains made by 44 special needs children (age 2) in cognition, gross motor, fine motor, receptive language, and expressive language domains. Comparison of actual to predicted posttest scores did not reveal significant gains in any domains. In cognition, gross motor, and fine motor domains, children with less severe impairments made more progress. Filiz, M., Cakmak, A., & Ozcan, E. (2005). The effectiveness of exercise programmes after lumbar disc surgery: a randomized controlled study. Clinical Rehabilitation, 19(1). 4-11. A randomized control trial to evaluate exercise programs for low back pain was conducted in Istanbul, Turkey. Patients who had received lumbar disc surgery were randomly assigned to three groups: 1) intensive exercise and education, 2) home exercise program and education, and 3) no intervention. Measures used assessed functional disability (including returning to work), body endurance, experience of pain, and depression. The intensive exercise program produced the best results in reducing pain and disability, followed by the home exercise. However, issues of cost effectiveness were not addressed. Perkins, D.V., Born, D.L., Raines, J.A., & Galka, S.W. (2005). Program evaluation from an ecological perspective: supported employment services for persons with serious psychiatric disabilities. Psychiatric Rehabilitation Journal, 28(3), 217-24. This paper reviews 11 years of a multi-site evaluation of supportive employment programs for persons with severe psychiatric disabilities in Indiana. The evaluation assessed effectiveness, efficiency, and employee satisfaction with these programs. The employment specialist, employer, and employee (i.e., person with disability) completed different forms, including service utilization, client background, and employer satisfaction forms. The authors describe the results, including issues of costs, timing of services, types of jobs clients were employed in, as well as challenges such as missing data. Racino, J.A. (1999). Policy, Program Evaluation, and Research in Disability: Community Support For All. New York: Hawthorn Press. A reference on how community support systems can greatly assist people with diverse disabilities to live fuller lives outside of institutions. Recommendations are structured around the support and empowerment paradigm, with chapters in sections on community integration, universal access to support, and housing, families, and community support. Of interest to community service managers, policymakers, and individuals with disabilities and their families. Race, K.E., Hotch, D.F., & Packer, T. (1994). Rehabilitation program evaluation: Use of focus groups to empower clients. Evaluation Review, 18(6), 730-740. This article discusses how focus groups can meet many program assessment needs and this technique has been underused in rehabilitation program evaluation. To underscore its potential utility, this article highlights a series of focus groups conducted in a vocational training and rehabilitation setting. A total of 70 clients participated in eight focus groups intended to identify clients' perceptions of program strengths and weaknesses. After a brief summary of results, particular attention is given to discussing the advantages of using a social science model applied to focus groups and how empowerment of clients can be a central theme in such efforts. Limitations of this technique are also noted. Stewart, D. Law, M., & Russell, D. & Hanna, S. (2004). Evaluating children's rehabilitation services: an application of a programme logic model. Child: Care, Health & Development, 30, 453-462. This article describes a practical and simple method of evaluating child rehabilitation programs with the use of the logic model. The authors review program evaluation and logic model definitions and uses, then describe the outpatient rehabilitation center that was evaluated. This kindergarten-age program served children with disabilities such as cerebral palsy, spina bifida and developmental delays. A combination of qualitative and quantitative methods were used in the research, with quantitative measures assessing parents’ perceptions of the services, client satisfaction, and goal attainment. Interviews were conducted with service providers and parents. The program’s logic model guided this evaluation, and the authors describe findings and implications relating to program outputs, short-term objectives, and long-term objectives. Voekz, L.M., & Evans, I.M. (2004). Educational validity: Procedures to evaluate outcomes in programs for severely handicapped learners. Research & Practice for Persons with Severe Disabilities, 29(2), 64-76. The evaluation of student programs addressing severe learning disabilities has become increasingly critical at local and federal levels. This article reviews procedures for evaluating such programs and provides recommendations. The authors argue against the traditional approach of using single-subject designs and emphasizing internal validity. Instead, they offer three questions that should be answered throughout the course of an evaluation: 1) Did behavior change occur, and if so, was the change a function of the program?, 2) Was the intervention plan implemented as described?, and 3) Is the resultant behavior meaningful or beneficial to the child or other persons? Thus, these questions address integrity of educational program, empirical validity, and social validity. The authors discuss the assessment of these areas in detail, in addition to issues of cost-effectiveness. Gender Issues in Program Evaluation Beardsley, R.M. & Miller, M.H. (2002). Revisioning the process: A case study in feminist program evaluation. In D. Seigart & S. Brisolara (Eds.), Feminist Evaluation: Explorations and Experiences (pp. 57-70). [New Directions for Evaluation, No. 96]. San Francisco, CA: Jossey-Bass. This is an article from a special issue that focused on feminism and evaluation (several of the articles are included in this bibliography, but the entire issue pertains to this topic). A feminist paradigm was utilized in the evaluation of a women’s substance abuse prevention program. First the authors describe the feminist ideology and the need for female-specific programs since the greater portion of service consumers are women, who are also more likely to be poor. Key qualities of feminist evaluations are discussed, including: valuing women’s voices, celebration of feminist ideals, and shared feminist frames of reference. Subcomponents of each quality are detailed. The authors describe how they incorporated these feminist ideals in the evaluation of the substance abuse program, suggesting areas of further research regarding male-female differences. The role of the feminist program evaluator is discussed. The authors suggest that the processes of feminist evaluations may differ from traditional evaluations. Bergsgaard, M. (1997). Gender issues in the implementation and evaluation of a violence-prevention curriculum. Canadian Journal of Education, 22(1). 33-45. This article highlights a violence-prevention program implemented in two elementary schools and findings from two years of a three-year evaluation. Results suggest that the program may produce a decline in conflict events and offer students, as well as teachers, options for conflict resolution. But the prescribed resolution skills and strategies may be most effective in decreasing forms of conflict most common among male students; the more covert, less physical behavior typical of conflict involving female students may be more difficult for observers to detect and less likely to be targeted for resolution through program strategies. The article discuses how future editions of the Second Step intervention may need to be adapted in order to insure the transmission of learning life-skills appropriate to youths’ needs, regardless of their gender. Bond, L.A., Belenky, M.F., & Weinstock, J.S. (2000). The Listening Partners Program: An initiative toward feminist community psychology in action. American Journal of Community Psychology, 28(5), 697-730. This article describes the Listening Partners intervention and is analyzed as a synthesis of feminism and community psychology, within a developmental framework. Working from an empowerment perspective, this social action, peer group intervention supported a community of poor, rural, isolated, young, White mothers to gain a greater voice, claim the powers of their minds, and collaborate in developmental leadership—creating settings that promote their own development and that of their families, peers, and communities. High quality dialogue, individual and group narrative, and collaborative problem-solving were emphasized, in a feminist context affirming diversity, inclusiveness, strengths, social-contextual analyses, and social constructivist perspectives. Hood, D.W. & Cassaro, D.A. (2002). In D. Seigart & S. Brisolara (Eds.), Feminist Evaluation: Explorations and Experiences (pp. 27-40). [New Directions for Evaluation, No. 96]. San Francisco, CA: Jossey-Bass. Using the feminist paradigm as a mode of social inquiry, the authors explore the intersections of race/ ethnicity, socio-economic class, gender, and sexual orientation/ preference in the evaluation process. Understanding and confronting systems of social oppression are discussed. The authors argue that evaluators who can identify with the contexts of the evaluated groups may be more effective in the evaluation, and the examination of personal identity interacts with the surroundings. The authors also discuss women of color and black feminist thought. The importance of using the evaluation results to positively impact the participants served is discussed. Pillow, W.S. (2002). Gender matters: Feminist research in educational evaluation. New Directions for Evaluation, 96, The key insights and shifts in first-, second-, and thirdwave feminism are examined, along with the impact on methodological thinking of what may be broadly termed “race-based” feminisms, lesbian theory, postcolonial feminist theory, and post-structuralism. This overview provides a present-day portrait of where feminist theory is now and the continued importance of paying attention to gender, particularly in education. Salazar, L.F. & Cook, S.L. (2006). Preliminary findings from an outcome evaluation of an intimate partner violence prevention program for adjudicated, African American, adolescent males. Youth Violence and Juvenile Justice, 4(4), 368-385. This study evaluated the efficacy of a five-session intimate partner violence (IPV) prevention program for adjudicated African American male adolescents. The program was guided by feminist theory and was culturally focused and gender specific. An experimental design was used to assess changes in knowledge of IPV and patriarchal attitudes at posttest and to test for the moderating effects of committing IPV and witnessing parental IPV. Ward, K.J. (2002). Reflections on a job done: Well? New Directions for Evaluation, 96, 41-56. Seven guidelines for conducting feminist program evaluation are presented in the context of an evaluation of an adolescent gender violence prevention program. The methods and challenges of incorporating these guidelines are discussed.
|