Search Results for All Sessions

Session Title: Studying Systems Through Social Network Analysis: Empowering Institutional Change

Multipaper Session 917 to be held in Centennial Section A on Saturday, Nov 8, 4:00 PM to 5:30 PM

Sponsored by the Systems in Evaluation TIG and the Organizational Learning and Evaluation Capacity Building TIG

Chair(s):

Steve Fifield, University of Delaware, fifield@udel.edu

Abstract: This session examines the relationship of social network analysis (SNA) and systems concepts in the evaluation of initiatives to improve deeply rooted institutional practices. Key systems concepts, including perspectives, boundaries, and entangled systems, shape our use and reflections on SNA as a tool to understand and inform organizational change. We ground reflections on SNA and systems concepts in our experiences studying change initiatives in school leadership, secondary mathematics teaching, and multidisciplinary, multi-institutional scientific research. In these cases we use systems concepts to explore variations on shared themes: institutional practices in tension, evaluation design issues that led us to SNA, and shifts in our roles as evaluators as agents in communities undergoing change. By reflecting on our experiences and the results of these projects, we hope to contribute to critical conversations about and useful integration of SNA and systems concepts.

Social Network Analysis and Systems Concepts in Studies of Interdisciplinary S&T Research

Steve Fifield, University of Delaware, fifield@udel.edu

Social network analysis (SNA) is a recent addition to my studies of science and technology (S&T) research initiatives. This leads me to consider how systems concepts can inform SNA, and how SNA can put systems concepts into practice. Here I draw on ongoing evaluations of statewide interdisciplinary S&T initiatives and research on interdisciplinary S&T research centers. These are examples of changes in academic S&T research toward managed, collaborative, and interdisciplinary projects and organizations. In initiatives to catalyze the growth of research networks, SNA can multiply perspectives on processes and outcomes by representing different kinds of networks in formation. SNA can contribute to dynamic, relational understandings of changes in S&T research in combination with ethnographic methods that examine the meanings and performances of social ties in research networks. I describe this approach in a study of group interaction customs across entangled disciplinary and institutional boundaries in two S&T research centers.

Using Social Network Analysis to study Environments That Support Teacher Leadership Development

Ximena Uribe Zarain, University of Delaware, ximena@udel.edu

This study describes the use of social network analysis to evaluate the leadership configuration of mathematics teachers. The goal of this method is to use teacher collaboration data to map and explain the course of leadership and the adequate environment for a teacher to succeed as a leader. This is a shift in instructional leadership evaluation from a focus on the qualities of isolated individuals to a more contextual and systemic understanding of interrelationships and how people understand their environments. Network maps are tools to describe and reflect on organizational structure and dynamics, including the overlaps and entanglements of personal networks. To better understand the nature of relationships related to leadership among mathematics teachers, this study combined in depth interviews with network analysis. Network maps were taken back to the key players in the organization to see how they interpreted the network patterns.

Session Title: Practical Applications for Using Propensity Scores

Demonstration Session 918 to be held in Centennial Section B on Saturday, Nov 8, 4:00 PM to 5:30 PM

Sponsored by the Quantitative Methods: Theory and Design TIG

Presenter(s):

MH Clark, Southern Illinois University at Carbondale, mhclark@siu.edu

Abstract: Quasi-experiments are excellent alternatives to true experiments when random assignment is not feasible. Unfortunately, causal conclusions cannot easily be made from results that are potentially biased. Some advances in statistics that attempt to reduce selection bias in quasi-experiments use propensity scores, the predicted probability that units will be in a particular treatment group. Because propensity score research is still relatively new, many applied social researchers are not familiar with the methods, applications and conditions under which propensity scores should be used. Therefore, the proposed demonstration will present an introduction to computing and applying propensity scores using SPSS. The demonstration will include: 1. a basic method for computing propensity scores; 2. how propensity scores can be used to make statistical adjustments using matching, stratifying, weighting and covariate adjustment; 3. a discussion of known limitations and problems when using propensity score adjustments; and 4. how to improve propensity score computations.

Session Title: Successfully Managing Evaluation Management: Approaches to Common Challenges From Three Different Evaluation Perspectives

Panel Session 919 to be held in Centennial Section C on Saturday, Nov 8, 4:00 PM to 5:30 PM

Sponsored by the Presidential Strand and the Evaluation Managers and Supervisors TIG

Chair(s):

Laura Feldman, University of Wyoming, lfeldman@uwyo.edu

Abstract: This three-person panel focuses on four common challenges to successful evaluation management and supervision: 1) how to determine which evaluation projects to pursue and balancing the factors (e.g., staff, interest, organizational development) that influence these decisions; 2) how to maintain internal and external evaluation quality; 3) how to manage one's time; and 4) when and where to allocate money. The panelists offer insights based on their personal management styles and type of organization (i.e., a state agency, a University-based evaluation group, and an urban school district). Collectively, the panelists have more than 30 years of evaluation experience. Regardless of their roles or organization, they are constrained by the same resources: money, staff, and time. Presenters discuss how these constraints guide their decision making and activities. Their responses emphasize their desire to make decisions that will maintain the highest standards of evaluation practice and management.

Looking at Evaluation Management and Supervision from the Perspective of a Large Urban School District

Bret Feranchak, Chicago Public Schools, bferanchak@cps.k12.il.us

Bret Feranchak, is the Director of Program Evaluation for the Chicago Public Schools (CPS). He directs the district's internal evaluation efforts and coordinates the work of external evaluators and researchers. This work includes all aspects of program evaluation including fiscal oversight of a multimillion dollar program evaluation portfolio. His evaluation work spans numerous program areas, including the evaluation of district math, science, literacy, bilingual, early childhood, high school, and after school programs. His research has focused on quantitatively measuring the effects of educational reform initiatives, particularly those that involve professional development initiatives for teachers. He also works on issues of educational program improvement through evaluation utilization. Previously, as senior research analyst for the CPS Office of Mathematics and Science, he was a member of the senior leadership team responsible for planning and leading the Chicago Mathematics and Science Initiative and evaluating the NSF-supported Chicago Urban Systemic Program.

Looking at Evaluation Management and Supervision from the Perspective of a University-based Evaluation Group

Cindy Shuman, Kansas State University, cshuman@ksu.edu

Cindy Shuman, Ph. D., is the Lead Evaluator for the Office of Educational Innovation and Evaluation (OEIE) at Kansas State University. She is responsible for developing and implementing program evaluations for both university and statewide programs. These activities include evaluation design, development of data collection instruments and protocols, survey development and administration, field data collection, analysis of quantitative and qualitative data, and the preparation of written reports and summaries of evaluation findings. She works collaboratively with the OEIE administrative team to establish office guidelines; contributes to proposal development and client outreach, and she serves as a resource in evaluation design and methodology for other staff members. She supervises graduate assistants and assists in the training and support of project staff. Her primary interests are in program evaluation and faculty development in higher education.

Looking at Evaluation Management and Supervision from the Perspective of a Large State Agency

Carsten Baumann, Colorado Department of Public Health and Environment, cbaumann@smtpgate.dphe.state.co.us

Carsten Baumann, M. A., is the Evaluation Director for the State Tobacco Education and Prevention Partnership at the Colorado Department of Public Health and Environment. He is primarily responsible for the design and implementation of an overall plan for measuring the effectiveness of the statewide tobacco program. Baumann also consults on evaluation with other programs, overseeing many of the Department's external evaluation efforts. He has served as the Acting Director of the Prevention Services Division's PEERS (Planning, Epidemiology, Evaluation, Research, and Surveillance) Unit. Before joining the State, Baumann spent five years as a senior researcher and partner at an applied social science research company in Denver, CO. There he was responsible for managing evaluation teams and projects from youth development to substance abuse prevention. Clients included community-based organizations, foundations and Federal and State agencies.

Session Title: Communication and Cognition in Evaluation Utilization

Multipaper Session 920 to be held in Centennial Section D on Saturday, Nov 8, 4:00 PM to 5:30 PM

Sponsored by the Evaluation Use TIG

Chair(s):

William Bickel, University of Pittsburgh, bickel@pitt.edu

How do People Process Data Reports? Implications for Reporting Evaluation Findings to Stakeholders

Presenter(s):

Jill Lohmeier, University of Massachusetts Lowell, jill_lohmeier@uml.edu

Steven Lee, University of Kansas, swlee@ku.edu

Abstract: One important issue for promoting the utilization of evaluation findings is ensuring that stakeholders actually process and understand the findings. Program evaluation textbooks generally include one chapter about how to present evaluation findings. These chapters generally suggest outlines for evaluation reports and discuss the importance of personal presentations to stakeholders. However, in addition to presentations and final reports, findings are often presented in brief reports prior to the summative reports. The evaluation literature and textbooks spend little time discussing the best methods for attracting the initial attention and cognitive processing of stakeholders in these brief reports. In the current study, undergraduates were presented with a research findings briefing. Think aloud and retrospective probing procedures were utilized to determine which parts of the briefing were examined first, were studied most, and were remembered following the presentation. The findings will be discussed in terms of the implications for creating evaluation briefings.

Making Evaluation Meaningful: Creating Recommendations Clients Can and Will Use

Presenter(s):

Caitlin Scott, Northwest Regional Educational Laboratory, scottc@nwrel.org

Theresa Deussen, Northwest Regional Educational Laboratory, deussent@nwrel.org

Kari Nelsestuen, Northwest Regional Educational Laboratory, nelsestuenk@nwrel.org

Abstract: Well-crafted recommendations can increase clients' perceptions of the usefulness of evaluations and increase clients' appreciation of evaluations. Yet, many evaluation reports fail to deliver recommendations that clients can understand and use. This paper describes three aspects of crafting recommendations: 1) guidelines for determining appropriate recommendations; 2) guidelines for language used in recommendations; and 3) techniques for developing recommendations collaboratively with clients. Audience members can take the ideas from the paper and immediately apply them to their work to ensure their own reports contain meaningful, actionable recommendations. Examples in this paper come from the field of education but the guidelines and practices are applicable in other fields as well.

Exploring the Role of Communication in Evaluation Utilization

Presenter(s):

Janet Lee, University of California Los Angeles, janet.lee@ucla.edu

Abstract: Much of the research literature on how communication impacts evaluation utilization has focused on the nature and format of the report and various evaluator characteristics and actions. While communication does include all written and verbal, formal and informal sharing of information, communication is also a dynamic, interactive process. In this presentation the author explores how communication and the ways in which the presence or the lack of a strong communication/dissemination structure within an organization facilitates or hinders evaluation use among teachers in three urban area high schools. Conclusions are based on survey and interview data that has been analyzed using social network analysis along with more traditional analytic methods in order to gain a deeper understanding of how communication plays a role in evaluation utilization.

Session Title: Ethics and Evaluation: Respectful Evaluation With underserved communities

Panel Session 921 to be held in Centennial Section E on Saturday, Nov 8, 4:00 PM to 5:30 PM

Sponsored by the Multiethnic Issues in Evaluation TIG

Chair(s):

Helen Simons, University of Southampton, h.simons@soton.ac.uk

Discussant(s):

Ricardo Millett, Millett & Associates, ricardo@ricardomillett.com

Abstract: A commitment to respecting and honoring underserved communities within evaluations has drawn this panel together to present what we have learned about conducting ethical evaluations with peoples who are: indigenous, minority, disadvantaged and/or otherwise marginalized within our societies. Within these lessons the evaluators' responsibilities are made explicit about the engagement that needs to happen with these communities and the critical lens the evaluators need to maintain about the causes of their marginalization. We begin with Critical Race Theory (CRT) and how a transformative agenda can guide ethical evaluation practice. Moves towards the control of evaluation work done in Indian Country then highlights to need for respect for cultural mores and aspirations. Within MSori communities in New Zealand this respect underpins a relationship ethic that also resonates with the fourth presenters who speak to the need for underserved communities to be collaborative partners in evaluation and research.

Ethical Responsibilities in Evaluations with Diverse Populations: A Critical Race Theory (CRT) Perspective

Veronica Thomas, Howard University, vthomas@howard.edu

Recently, attention has been given to methodological considerations in evaluations of projects serving minority populations. Less explicit attention, however, has been paid to the ethical responsibilities of planning, implementing, and disseminating of such evaluations. This presentation will put forth Critical Race Theory (CRT) as a useful lens to guide evaluators in conducting ethical work in minority communities. CRT is part of a larger transformative paradigm that has implications for a research approach that expands evaluators' explicit responsibilities for dealing with a myriad of ethical issues throughout the evaluation process including (a) addressing racism and other forms of oppression, (b) considering differential power imbalances and privileges between researchers and the researched, (c) broadening conceptualizations of what counts as knowledge, (d) expanding considerations of harm to include communities (not just individual harm), and (e) working to promote social justice, equity, and democracy.

Researching Ourselves Back to Life

Joan LaFrance, Mekinak Consulting, joanlafrance1@msn.com

The myriad of research done by social, heath, and medical researchers is problematic to many American Indian people, whose tribes and families have suffered from a long history of intrusive studies that have built the reputations of anthropologists and other researchers, but brought little more than the loss of cultural ownership and exploitation to Indian people. This presentation describes the changing landscape and ethical considerations that need to be taken into account given the shift towards tribal control over what research is done and how it is done. It outlines the movement in Indian Country to take control of research agendas and describes the various processes being used in tribal communities to review and regulate research Indian Country. The presentation shares the experience and reflections of both Indigenous and non-Indigenous scholars regarding lessons learned in conducting research that is respectful of native cultural mores and responsive to community priorities.

Maintaining Indigenous Voices: Maori in Aotearoa

Fiona Cram, Katoa Ltd, fionac@katoa.net.nz

Marginalization occurs when a group of people are pushed to the periphery of a society. For Maori this was an outcome of colonization. Just as research was implicated in this push, so research plays a role in the re-centering of Maori in our own lands. The paper examines a 'community up' approach to defining evaluator conduct that forms the basis of ethical practice. The cultural values within this approach are: 1. Aroha ki te tangata (a respect for people). 2. Kanohi kitea (the face that is known). 3. Titiro, whakarongo korero (look, listen and only then speak). 4. Manaaki ki te tangata (share and host people, be generous). 5. Kia tupato (be cautious). 6. Kaua e takahia te mana o te tangata (do not trample on the status and dignity of the people). 7. Kia mahaki (be humble). These seven practices underpin a relationship ethic and facilitate community engagement.

Re-Conceptualizing Ethical Concerns in Underserved Communities

Katrina Bledsoe, Walter R McDonald and Associates Inc, kbledsoe@wrma.com

Rodney Hopson, Duquesne University, hopson@duq.edu

Underserved communities, such as those that have been considered historically disadvantaged due to social class, ethnic background, gender etc. have received secondary consideration in their involvement in the process of research and evaluation. By process, we mean the manner in which the research is conducted from research question development, to utilization of results. We maintain that the exclusion of such communities violates ethics at the most fundamental point of research: the ability to self-determine one's level of participation in every aspect of the process. In this presentation, using examples from the authors' experiences, we conceptualize the optimal research process as one that is relational and collaborative beginning at the study/project conception and continuing past its conclusion. The acknowledgement of communities as collaborative partners in research expands our understanding of what it means to be ethical in working with underserved diverse populations and communities.

Session Title: Managing the Tension between Performance Measurement and Evaluation in the Emerging Political Environment

Panel Session 922 to be held in Centennial Section F on Saturday, Nov 8, 4:00 PM to 5:30 PM

Sponsored by the Theories of Evaluation TIG

Chair(s):

George Julnes, Utah State University, george.julnes@usu.edu

Discussant(s):

Eleanor Chelimsky, Independent Consultant, oandecleveland@aol.com

Abstract: Often viewed with some suspicion by opposing proponents, performance measurement and evaluation are being brought together more frequently by government initiatives. This panel will examine how the tension between these two approaches can be managed best in the current changing political environment. The presentations and discussion will examine the challenges to effective management of performance management and evaluation and will suggest solutions for moving forward.

Tensions in Integrating Program Evaluation and Performance Measurement

Stephanie Shipman, United States Government Accountability Office, shipmans@gao.gov

To become more accountable for results, federal agencies increased both performance measurement and program evaluation activities over the past decade. Urged by GPRA, managers raised their sights from outputs to outcomes and set goals to show how programs contributed to agency missions. OMB's PART presses agencies further to obtain independent, comprehensive evaluations of program effectiveness and relevance. Clearly, program evaluations are particularly well-suited to supplement performance monitoring by exploring reasons for observed outcomes and assessing program net impact. But, the fact that evaluations are constructed to provide answers to specific questions about program performance creates additional tensions beyond simple supplementing routine monitoring. While performance monitoring usually provides feedback to program managers, evaluations may aim to answer the questions of other stakeholders - who may define program success differently, or fundamentally question the need for the program. Integrating the results of these approaches creates tensions around the roles of audience and context.

Dimensions of Use and Their Relationships with Evaluation and Performance Measurement

Patria de Lancer Julnes, Utah State University, patria.julnes@usu.edu

This presentation will discuss a theory of use of performance measurement information that builds from the experiences of evaluative efforts. The discussion centers around a model of use that focuses on the purposes, audiences, and types of knowledge use applicable to the practice of performance measurement. A classic argument made against performance measurement is the apparent lack of use of performance measurement information by decision makers. The standard response is that this argument is based on an insistence on finding evidence of instrumental use, whereas substantial evidence suggests that most use is conceptual or symbolic. This presentation discusses a broader model that includes these different forms of use and indicators that relate to them.

Tensions Between Program Evaluation and Performance Measurement: The United States Federal Experience

Kathryn Newcomer, George Washington University, newcomer@gwu.edu

Executive leaders and program managers have been asking increasingly more questions about the effectiveness of federal programs during the last decade, in part due to requirements of the Government Performance and Results Act, passed by Congress, and in part due to the assessment tool introduced by the Bush Administration�s President�s Management Agenda. Measuring programmatic performance and making judgments about results has been more transparent than ever before. The level and number of foci of dialogue about appropriate ways to measure and judge have also increased. The recent federal experience presents some good lessons about how program measurement and evaluation play out in a highly political arena that may provide useful guidance as we move to the next chapter of the federal experience with evaluation.

Session Title: Science of Science Management: Development of Assessment Methodologies to determine Research and Development Progress and Impact

Think Tank Session 924 to be held in Centennial Section H on Saturday, Nov 8, 4:00 PM to 5:30 PM

Sponsored by the Research, Technology, and Development Evaluation TIG and the Government Evaluation TIG

Presenter(s):

Deborah Duran, National Institutes of Health, durand@od.nih.gov

Abstract: Research and development funders seek to fund effective, efficient, and impactful programs. However, current methodologies are insufficient to assess the practical application of many innovative R&D programs. Under the current approach of setting planned annual milestones, assessments may indicate met or unmet; but fail to address the adaptive learning involved with the scientific discovery process. For example, projects may struggle to initially meet the planned goals; but, later adapt using sound scientific principles and discovery processes. This example highlights an important problem in science management: What patterns, pathways, or profiles can be developed to assess performance and identify intervention points? The emerging field of Science of Science Management strives to develop systematic studies to explore the complexities of science administration, to provide evidence and analytic tools for decision making, and to inform science policy.

Session Title: Foundations of Evaluation: Theory, Method, and Practice

Skill-Building Workshop 925 to be held in Mineral Hall Section A on Saturday, Nov 8, 4:00 PM to 5:30 PM

Sponsored by the Graduate Student and New Evaluator TIG

Presenter(s):

Chris LS Coryn, Western Michigan University, chris.coryn@wmich.edu

Daniela Schroeter, Western Michigan University, daniela.schroeter@wmich.edu

Abstract: This skill-building workshop is designed to provide an overview of the field and discipline of evaluation, including its theory, research, and practice perspectives. It is designed not only to provide an overview of the field and discipline, but also to generate critical thinking about evaluation theory and practice to assist attendees in formulating their own ideas about it. The presenters will provide an introduction to the foundations of evaluation, including basic concepts and definitions, evaluation's rationale and uses, the evaluation field's history and standards, alternative evaluation models and approaches, general evaluation processes and procedures of collecting, analyzing, and synthesizing information, and metaevaluation, the process of evaluating evaluations. The session is divided into mini-lectures, group discussions, and work with case examples. Each mini-lecture will be followed by exercises and discussions. Case studies will be used throughout the workshop to demonstrate core concepts, methods, and approaches.

Session Title: Improving and Applying Measurement Techniques to Identify and Account for Differences Across Social Groups

Multipaper Session 926 to be held in Mineral Hall Section B on Saturday, Nov 8, 4:00 PM to 5:30 PM

Sponsored by the Quantitative Methods: Theory and Design TIG

Chair(s):

Mary Kay Falconer, Ounce of Prevention Fund of Florida, mfalconer@ounce.org

Abstract: Recognizing the social diversity of populations served by many programs, evaluators should address how measurement can be improved and used to identify and account for differences across social groups. Adopting measurement techniques that account for differences across ethnicities or other social characteristics of program participants provides greater assurances that the measures are valid for the entire participant group. This session will cover at least two relevant approaches for improving and applying measurement techniques. In addition, information on recent efforts by product developers (testing resources) to improve several widely used measurement tools across different social groups will be shared. As a third component, presenters will be asked to list lessons learned and recommendations related to measurement when evaluating programs serving diverse populations.

Meeting the Challenge of Social Diversity in Measurement Tools

Mary Kay Falconer, Ounce of Prevention Fund of Florida, mfalconer@ounce.org

With ethnic diversity being important in target and participant populations for many programs that are being evaluated, the validity of widely used measurement tools must be addressed. Based on a survey of product and testing services, recent efforts to improve the validity of selected tools across multiple ethnic groups will be identified. The coverage of ethnic and age groups with these tools by product developers will also be indicated. The list of tools included in this survey or compilation of information will be reviewed and approved by the other presenters in the session. The objective in this presentation will be to provide one account of the 'status or progress" of measurement in meeting the challenges of evaluations of programs serving diverse ethnic populations.

Testing a Model of the "Mistreatment and Barriers to Help-Seeking by Elder Women Abused by an Intimate Other"

Frederick Newman, Florida International University, newmanf@fiu.edu

Richard Beaulauria, Florida International University,

Laura R Seff, Florida International University, emis2go@cs.com

An instrument was developed based on a qualitative analysis of 21 focus groups of women (N=134, ages 50 to 85) that represented Hispanics, White Non-Hispanics, and Black (Caribbean Haitian or Jamaicans). Using criteria of two or more persons in two or more groups having similar coded concepts to identify a "qualitatively supported factor," using Atlas.ti, we were able to identify three clusters of factors: Internal Barriers (e.g., self blame, helplessness/powerlessness, secrecy, protecting family), External Barriers (Family response, Clergy Response, Justice System Response, Community Response), and Abuser Behaviors (Isolation, Intimidation, Jealousy). We are also seeking a form of convergent validation with a standardized measure, the Conflict Tactics Scale (CTS), and comparing these three race-ethnicities on the CTS for different age groups. In addition to testing the model on 450 women (150 in each race-ethnicity) we intend to identify similarities and differences among the major ethnic and age groupings.

Using the Rasch Item Partition Model to Improve Theory-Based Evaluation

John Gargani, Gargani and Company Inc, john@gcoinc.com

I describe the Rasch item partition (RIP) model, a hierarchical generalized linear model that can be used to simultaneously estimate program impacts and gather evidence about the underlying mechanisms presumed to cause impacts. I explain how evaluators can use the RIP model to integrate theory-based evaluations with randomized trials, providing examples from randomized trials in which RIP models shed light on presumed mechanisms of change.

Session Title: New and Interesting Evaluations of Early Childhood Services and Interventions

Multipaper Session 927 to be held in Mineral Hall Section C on Saturday, Nov 8, 4:00 PM to 5:30 PM

Sponsored by the Human Services Evaluation TIG

Chair(s):

Tracy Greever-Rice, University of Missouri, greeverricet@missouri.edu

Discussant(s):

Marty Tombari, Colorado Foundation for Families and Children, mtombari@coloradofoundation.org

Assessing the Quality of Early Care for Preschool-Aged Children in An Urban Setting

Presenter(s):

Robert Fischer, Case Western Reserve University, fischer@case.edu

Donna Bryant, University of North Carolina Chapel Hill, bryant@mail.fpg.unc.edu

Ellen Peisner Feinberg, University of North Carolina Chapel Hill, peisnerf@mail.fpg.unc.edu

Liane Grayson, University of South Dakota, liane.grayson@gmail.com

Abstract: This paper reports on a study of the quality of care in early care and education programs in the Cleveland, OH area, including center-based and family child care settings. Observational and interview data were collected from a sample of 177 classrooms for 3-to-5 year olds chosen from a stratified random sample of child care centers. Data were collected by trained observers using two standardized instruments � Early Childhood Environment Rating Scale � Revised, and Caregiver Interaction Scale. The quality of care in family child care homes was examined through a review of extant data from a previous study, and administrative data on quality used to guide technical assistance and assess provider performance. Findings speak to the overall quality of care, as well as by setting type, and the factors most related to the level of quality. The paper also addresses how to use quality data to inform practice and policy.

Achieving the Dual Purpose of Accountability and Improvement: A Case of Evaluating Palm Beach County's Quality Improvement System

Presenter(s):

Xuejin Lu, Children's Services Council of Palm Beach County, kim.lu@cscpbc.org

Lance Till, Children's Services Council of Palm Beach County, lance.till@cscpbc.org

Karen Brandi, Children's Services Council of Palm Beach County, karen.brandi@cscpbc.org

Jeff Goodman, Children's Services Council of Palm Beach County, jeff.goodman@cscpbc.org

Abstract: Evaluation is to assess the value or merit of a program or other entities (Stufflebeam, 2007). Evaluation is not only for accountability but also for improvement. Our proposal uses one stream of work on the evaluation of Palm Beach County�s Quality Improvement System (QIS) to demonstrate that the dual purpose of evaluation can be achieved through asking a series of interrelated evaluation questions. Methodologies used for data collection and data analysis for specific evaluation questions will be illustrated. Evaluation findings will be presented and discussed. This proposal has both methodological and substantive implications.

Lessons Learned From an Evaluation of the Kansas Early Childhood Comprehension Systems Plan

Presenter(s):

Greg Welch, University of Kansas, gww33@ku.edu

Jackie Counts, University of Kansas, jcounts@ku.edu

Jessica Oeth, University of Kansas, jessoeth@ku.edu

Chris Niileksela, University of Kansas, chrisn@ku.edu

Rebecca Gillam, University of Kansas, rgillam@ku.edu

Karin Chang-Rios, University of Kansas, kcr@ku.edu

Abstract: This presentation will focus on lessons learned from an ongoing evaluation of the Kansas Early Childhood Comprehensive Systems (KECCS) plan. Specific attention will be placed on the Structural Equation Modeling (SEM) approach used to model school readiness using a variety of data sources. This approach is considered innovative in this realm as many states have �models� of school readiness that have not been validated via the SEM approach being utilized.

Session Title: Building Evaluation Capacity Building for Public Health: Community, CBO, and State Examples

Panel Session 928 to be held in Mineral Hall Section D on Saturday, Nov 8, 4:00 PM to 5:30 PM

Sponsored by the Organizational Learning and Evaluation Capacity Building TIG

Chair(s):

Antonia Spadaro, Centers for Disease Control and Prevention, aqs5@cdc.gov

Abstract: In community-based participatory research, partnerships form between academics and external partners, such as community groups or health departments. For the Prevention Research Centers (PRCs), funded by the Centers for Disease Control and Prevention (CDC), these partnerships often steer university researchers towards helping local organizations or state partners with evaluation projects through grants, contracts, or technical assistance, resulting in evaluation capacity-building and organizational learning, Knowledge gained from this process provides stakeholders with tools for decision-making, program improvement, and demonstrating outcomes. This session will highlight three PRCs� roles in partnering with stakeholders - the community, community-based organizations (CBOs), and state entities - for building evaluation capacity and promoting public health. The panel will describe concepts and experiences in evaluation capacity building such as the increase of stakeholders� evaluation knowledge, skills, and abilities; the growth of new projects because of these evaluation endeavors; and the development of a new AEA local affiliate.

A Multi-pronged, Strategic and Collaborative Approach to Enhancing the Capacity to Design, Conduct, Manage and Utilize Evaluation in Rural Southwest Georgia

Iris Smith, Emory University, ismith@sph.emory.edu

Sally Honeycutt, Emory University, shoneyc@sph.emory.edu

Denise Ballard, Southwest Georgia Cancer Coalition, denise.ballard@swgacancer.org

J K Barnette, Southwest Georgia Cancer Coalition, jk.barnette@swgacancer.org

Evaluation capacity building strategies are most often implemented within or between organizations or organizational units, but rarely are such strategies applied in the broader context of a geographically defined community of practitioners. This presentation will describe strategies and outcomes of an evaluation capacity building initiative undertaken by the Emory Prevention Research Center in collaboration with the SW Georgia Cancer Coalition and other community partners in a 33 county �community� in southwestern Georgia. Strategies included formative evaluation, training, technical assistance, identification and development of local evaluation resources, and �modeling� evaluation practice. Outcomes include increased knowledge and practice of evaluation and the formation of the S.W. Georgia Evaluation Association, an affiliate of AEA to support and sustain ongoing capacity building efforts in the region.

Processes and Outcomes of an Academic/Private/Community-Based Partnership to Improve Evaluation Capacities of the Human Immunodeficiency Virus (HIV) Prevention Programs in the Southeastern United States

Tabia Akintobi, Morehouse School of Medicine, takintobi@msm.edu

Elleen M Yancey, Morehouse School of Medicine, eyancey@msm.edu

The sustainability of the HIV prevention public health workforce is challenged by limited evaluation capacities and increasing funder demands for evidence-based programs. This issue is attenuated in the Southeastern United States, which experienced a 20% increase in the estimated number of new AIDS cases compared to a 7.1% increase in the United States between 2000 and 2004. The Pfizer Foundation Southern HIV/AIDS Prevention Initiative facilitated a tri-directional partnership with The Morehouse School of Medicine Prevention Research Center and 24 community-based organizations in response to these challenges between 2004 and 2006. This presentation will describe the processes implemented and reciprocal learning experienced through the strengths contributed by each member of this collaborative. Qualitative and quantitative results of capacity building activities, on-going challenges, and recommendations for evaluators supporting capacity building efforts for HIV prevention programs will also be discussed.

It�s Getting Better All the Time: Evaluating the Current, While Planning for the Future

Robert Anderson, West Virginia University, randerson@hsc.wvu.edu

Valerie Frey-McClung, West Virginia University, vfreymcclung@hsc.wvu.edu

Kimberly Horn, West Virginia University, khorn@hsc.wvu.edu

Geri Dino, West Virginia University, gdino@hsc.wvu.edu

When West Virginia began receiving its share of tobacco funds from the multi-state Master Settlement Agreement, it elected to use an external evaluator. This ultimately led to the WV Prevention Research Center initiating the Evaluation Oversight and Coordinating Unit (EOCU) with a grant from the West Virginia Bureau for Public Health (WVBPH). The EOCU is an independent evaluator for the WVBPH's Division of Tobacco Prevention (DTP). It ensures that every WV tobacco prevention activity funded by the DTP is evaluated to: determine the extent to which programs and interventions meet their goals, establish what approaches work best, and understand how to make program improvements. The EOCU ensures the state's efforts are grounded in science, responsive to communities, and accountable to policymakers. This is done by such means as developing RFPs, recommending funding criteria, proposal reviews, program monitoring, and reporting. It also provides consistent standards for assessment and criteria for success.

Session Title: Evaluating Structural Changes in Residential Children's Homes: Challenges, Strategies and Lessons Learned

Panel Session 929 to be held in Mineral Hall Section E on Saturday, Nov 8, 4:00 PM to 5:30 PM

Sponsored by the Non-profit and Foundations Evaluation TIG

Chair(s):

Toni Freeman, The Duke Endowment, tfreeman@tde.org

Abstract: The ENRICH (Environmental Interventions in Children's Homes) evaluation plan included a group randomized design, along with comprehensive process evaluation and contextual assessments, to evaluate a structural intervention designed to promote and support physical activity and healthful nutrition (eating fruits and vegetables) among children and adolescents residing in approximately 30 residential children's homes (RCHs) in North and South Carolina. ENRICH was designed to be specific to the RCH setting; however, we believe that this evaluation approach and framework are applicable to interventions in other organizational settings, including schools, worksites, churches, and other community organizations. The three sessions included in this panel will provide detailed description of the organizational settings for the intervention, including strategies used to identify key organizational characteristics important to the design of the intervention and evaluation; an overview of the intervention, evaluation framework and methodology, including evaluation challenges; and, finally, outcome evaluation results and lessons learned.

Residential Children's Homes: The Context for the ENRICH Project

Toni Freeman, The Duke Endowment, tfreeman@tde.org

Children are most commonly placed in residential care due to abuse, neglect, behavioral acting out, status offenses, pregnancy, family crisis, and substance abuse Thus, Residential Children's Homes (RCHs) play the role of 'family/home' for these underserved children. The RCHs involved in the ENRICH project vary widely on numerous characteristics including size, setting, populations served, location (two-state area) and staffing models. The Duke Endowment (funding agency) and ENRICH staff collaborated to gather information and enhance buy-in by holding a one day meeting for the CEOs of all potential participating organizations. During the meeting, representatives from both organizations provided an overview of the project, but more importantly explained the project design in detail and held small discussion groups with participants to gather information needed. This enhanced stakeholder buy-in to the design. In addition, each organization completed the ENRICH Organizational Assessment annually, and ENRICH staff closely monitored and documented organizational transitions experiences by RCHs during the project. Included in the presentation will be an overview of the organizational data used for selection, matching, and randomization to reduce heterogeneity among participating RCHs and reduce variability that would threaten statistical validity.

ENRICH Outcome Evaluation: Design and Methodology to Address Common Evaluation Challenges

Ruth Saunders, University of South Carolina, rsaunder@gwm.sc.edu

The ENRICH intervention was designed to create environmental change by partnering with the RCHs and providing training, technical assistance, and support to wellness teams (adult staff working in the RCHs) that served as organizational change agents over 2 years. The researchers used a well-defined conceptual framework to define the 'healthy home' environment. This framework guided the intervention (the change target for the wellness teams) as well as development of outcome measures to assess the environment. We will share the instrument development process designed to establish construct validity and the instruments. We used a group randomized design with two groups and pre- and post-tests to assess program effectiveness. Prior to random assignment, RCHs were matched based on RCH characteristics. This design reduces threats to internal validity. Through process evaluation we also monitored project-related activities in all RCHs as well as contextual influences that could impact project outcomes.

Outcome Evaluation Results and Lessons Learned from ENRICH Outcome Evaluation

Kelli Kenison, University of South Carolina, kenison@gwm.sc.edu

Mixed methods were used to assess the effectiveness of the ENRICH intervention with Residential Children's Homes (RCHs). The intervention had a positive impact on the physical activity environment including characteristics of physical activity provided, the social environment related to physical activity, staff wellness support, and limited changes in policies related to physical activity. Nutrition environment changes included organizational support for improved nutrition and collaboration with nutrition-related community organizations outside the RCHs. In addition to sharing results, this session will focus on the numerous lessons learned related to various design challenges inherent in structural interventions. Included will be discussion of strategies to address the need for enhancing stakeholder support for evaluation design, quickly and creatively acquiring organizational information required for matching and random assignment to condition, monitoring setting and contextual issues that may affect project implementation, outcome or design integrity, and data collection methodology for structural interventions.

Session Title: Policy, Practice, and Standards: Educational Evaluation in an Age of Accountability

Multipaper Session 930 to be held in Mineral Hall Section F on Saturday, Nov 8, 4:00 PM to 5:30 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Chair(s):

Tiffany Berry, Claremont Graduate University, tiffany.berry@cgu.edu

Examining Educational Evaluation Policy Alternatives

Presenter(s):

Katherine Ryan, University of Illinois Urbana-Champaign, k-ryan6@uiuc.edu

Abstract: Today, educational evaluation is playing a key role in the shift to a knowledge-based society (Ryan, 2005). As nations now vie for highly competitive positions within the global market place (Anderson, 2005), concerns about quality and the resources directed to education are increasing demands for information about school performance (Lundgren, 2003). This global preoccupation has lead to the heightened international and national emphasis on educational accountability. At the same time, educational accountability notions and the �machinery� of implementing accountability requirements varies across nations. In this paper, I critically examine two distinct educational evaluation policies involving accountability: a) NCLB high stakes testing (performance measurement) and b) school self-evaluation and inspection (e.g, England, Netherlands, and Pacific Rim) (Nevo, 2002; Ryan, 2005). My examination includes a brief discussion of history of their respective roles in evaluation, foundations, and theories of action in improving student learning. I also analyze the evidence concerning the effects (e.g. are students learning more?) and potential consequences (e.g., fairness, narrowing of curriculum). I close by considering whether "importing" an international accountability policy--school-self evaluation and inspection might help with both school improvement and improved student learning in the United States.

Implementation Quality as an Essential Component in Efficacy Evaluations

Presenter(s):

Tiffany Berry, Claremont Graduate University, tiffany.berry@cgu.edu

Rebecca Eddy, Claremont Graduate University, rebecca.eddy@cgu.edu

Abstract: In light of No Child Left Behind, the relative emphasis on outcomes rather than implementation data has been articulated as one of the potential consequences of the legislation on evaluation practice (Berry & Eddy, 2008). The implications this has for the future of educational research and evaluation are profound, especially if tracking and assessing implementation continues to be considered value-added for efficacy studies, particularly in the realm of educational evaluation. In fact, collecting implementation data in evaluations is as essential as a manipulation check is in experiments. The purpose of this paper is to discuss these implications, examine the benefits of collecting implementation data, and describe methods that increase the sophistication of measuring implementation data through quality ratings rather than frequency counts.

Fidelity of Implementation

Presenter(s):

Evie Chenhall, Colorado State University, evie@cahs.colostate.edu

Can Xing, Colorado State University, can.xing@colostate.edu

Abstract: The new foreign language national standards have been introduced to the K-12 teachers in a school district in the southwestern United States. Instructional practices based on the standards are designed to help students reach proficiency in foreign language. As part of a three-year U.S. Department of Education grant, which began in 2006, the purpose of the study was to measure the depth of implementation of the new national standards in K-12 instruction. The evaluation component of the study was based on the Concerns-Based Adoption Model (CBAM) process referred to as the Levels of Use (LoU) analysis of fidelity of implementation. Interviews of foreign language teachers were conducted to evaluate teachers� proficiency in using the new national standards. This presentation will include the implications of the study and examine the fidelity of implementation of the standards. Quantitative and qualitative data findings will be presented.

Effect of Professional Standards on Evaluation Practice

Presenter(s):

Teresa Brumfield, University of North Carolina at Greensboro, tebrumfi@uncg.edu

Deborah Bartz, University of North Carolina at Greensboro, djbartz@uncg.edu

Abstract: To examine how evaluation policy influences evaluation practice, this presentation proposes to address how evaluations of educational projects/programs may be affected by existing professional standards that are separate and distinct from the Guiding Principles for Evaluators. The educational project of interest�Preparing Outstanding Science Teachers (POST)�is a collaboration of a high-need comprehensive LEA, a public southeastern university�s College of Arts and Sciences, and its School of Education. The purpose of this collaboration is to develop and provide middle school science teachers with standards-based professional development opportunities in both content and pedagogy, with a focus on increasing student achievement. By examining how this project evaluation has been affected by professional development standards, national and state science standards, and test development standards, along with project evaluation guidelines, this presentation will emphasize why evaluators need to have more than basic familiarity with those professional standards that may affect their evaluations.

Session Title: Empowerment Evaluations: Insights, Reflections, and Implications

Multipaper Session 931 to be held in Mineral Hall Section G on Saturday, Nov 8, 4:00 PM to 5:30 PM

Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG

Chair(s):

Aarti Bellara, University of South Florida, bellara@coedu.usf.edu

Equalizing the Power between Participants in an Empowerment Evaluation of a Program Serving Individuals with Severe Mental Illness

Presenter(s):

Gary Walby, Ounce of Prevention Fund of Florida, gwalby@ounce.org

Abstract: An empowerment evaluation was conducted to improve services to individuals with severe mental illness at a large southeastern U.S. community mental health center. As empowerment evaluation is a highly participatory approach, additional steps were taken to fully include individuals with mental illness. Since chronic mental illness leads to disempowerment and marginalization, participants that were service recipients (SR�s) were uneasy at being peers in the evaluation process. The empowerment principle of democratic participation was emphasized to ensure that the perspective of SR�s was not overshadowed by provider, manager, or external stakeholder. Several unique strategies were developed to further enhance the participation by SR�s and to increase the acceptance of SR�s as peers by other participants that were accustomed to viewing SR�s as �clients� only. These inclusion or equalization strategies can be adapted for use in other empowerment based evaluations and could benefit other evaluators seeking to balance power across participants.

Self-determination and Improvement for Teachers Engaged in Japanese Lesson Study

Presenter(s):

Robin Smith, Florida State University, smith@bio.fsu.edu

Abstract: Japanese lesson study is a teacher-led, learning community form of professional development that is embedded within the regular role of teachers. This study investigated how lesson study enabled teachers to evaluate and direct their own professional growth in areas that they identified for improvement. The teacher benefits uncovered by this study were related to improving practice and gaining a sense of professionalism about their growth as educators. Themes that emerged from the study included the positive perceptions of lesson study as a collaborative, teacher-led process for improvement and the understanding that lesson study can instill a sense of empowerment and professionalism to those who engage in the endeavor. There also is evidence that each of the principles of empowerment evaluation was important to some degree in lesson study and that the process provided a tool for teachers to direct their own improvement, which also is the goal of empowerment evaluation.

Power in Empowerment Evaluation: A Social Conflict Perspective

Presenter(s):

Scott Rosas, Concept Systems Inc, srosas@conceptsystems.com

Abstract: The debate about the legitimacy of empowerment evaluation has occurred despite a limited theoretical examination of the broader social context in which evaluation occurs. Empowerment evaluation is rooted in the fundamental proposition that key aspects of evaluation can facilitate a shift in control and power among individuals and institutions. Using social conflict theory as a backdrop, this paper examines several empowerment evaluation components and practices in an effort to further understand the mechanisms of change through which empowering outcomes are realized. From this perspective, this paper will explore the different features empowerment evaluation, emphasize the role of the evaluator relative to the ways social institutions are organized, and highlight how social conflict supports or undermines core functions. This discussion will emphasize the importance of interests over norms and values, and how the pursuit of interests generates various types of conflict as normal aspects of social life, rather than dysfunctional occurrences.

An Action Cycle of Empowerment Evaluation: Feasibilities and Challenges

Presenter(s):

Hui-Ling Pan, National Taiwan Normal University, joypanling@yahoo.com.tw

Abstract: Empowerment evaluation has been applied to school sites with wider use. The emphasis on putting the participants back to the �driver�s seat� creates more opportunities for capacity building of schools. In order to investigate how the approach of empowerment evaluation might be employed in an eastern cultural context, in which the power structure is more hierarchical, a case study was conducted. A Voluntary Service Program of a junior high school in Taiwan was evaluated using the three steps, defining missions, taking stocks and planning for the future, proposed by Fetterman. After observations for three consecutive semesters, it was found that the evaluation implemented just like an action cycle. In the cycle, participants keep identifying their tasks and tracking what they have done. From the actions, participants re-clarified missions, having a sense of self control and molding the practical wisdom of the community. However, in a power concentrated society, some voices need to be provoked with greater efforts. And the study indicated that a program with specific goals may only have empowerment evaluation last for limited cycles when participants feel all their missions completed. It may stimulate further thinking with regard to the advantages and limitations of the empowerment evaluation.

Session Title: Overviews of Research on Observation Instruments Used in Arts Education Evaluation Studies

Panel Session 932 to be held in the Agate Room Section B on Saturday, Nov 8, 4:00 PM to 5:30 PM

Sponsored by the Evaluating the Arts and Culture TIG

Chair(s):

Suzanne Callahan, Callahan Consulting, callacon@aol.com

Abstract: In this session, we will address the research on three observation arts-education evaluation instruments: two for determining teachers' quality and one for examining the quality of arts-focused professional development. The first instrument is used to observe artist-teacher pairs who integrate the arts into elementary-school core literacy teaching. The psychometric properties of the instrument and how the findings were used in an evaluation will be addressed. The second instrument is used as an in-class observation tool for both formative and summative evaluation purposes. The instrument development and results of the findings will be presented. The final observation instrument is used to assess the quality of professional development workshops delivered to individuals who work with pre-school children. The alignment of the tool with best practices in adult education and how the tool was used to both guide observations and present the results of the observational data will be presented.

Using the Youth Program Quality Assessment Tool to Observe the Infusion of Art into Core Elementary Literacy Curriculum

Melanie Hwalek, SPEC Associates, mhwalek@specassociates.org

Mary Lou Greene, Marygrove College Institute for Art Infused Education, mgreene@marygrove.edu

Victoria Straub, SPEC Associates, vstraub@specassociates.org

Marygrove College's Institute for Art Infused Education (IAIE) supports the provision of artist-teacher teams in using art to teach core literacy curriculum in elementary schools. The Youth Program Quality Assessment (YPQA) is a validated tool developed by High/Scope Educational Research Foundation that measures the quality of programs for youth. In this evaluation, the subscales of the YPQA were first examined in relation to their use in assessing quality and outcomes of the art-infused classroom teaching. Nine subscales of the YPQA were identified as relevant to the IAIE's program. Marygrove College graduate students in education were trained to criterion on the use of these subscales and, then, observed the teacher-artist teams in 12 classrooms within seven elementary schools. This paper will present the results from the observations in terms of both psychometric properties of the nine selected subscales and the use of the findings in the evaluation of IAIE.

The Development and Findings of an In-Class Observation Instrument to Determine Teachers' Quality of Implementing Arts-Integration Techniques to Teach Elementary Age Children

Brian Lawton, University of Hawaii, blawton@hawaii.edu

Paul R Brandon, University of Hawaii Manoa, brandon@hawaii.edu

Most evaluations of arts-integration projects have focused primarily on project effects on student outcomes. Little has been done by way of examining the quality of implementation of the arts integration strategies used. The purpose of this paper is to present the development and findings of an observation instrument used to assess the quality of elementary-level teachers' implementation of arts-integration strategies to teach core academic subjects. The instrument was developed for in-class observations and serves both formative and summative evaluation purposes. The formative aspects of the instrument provide teachers with on-site feedback about the quality of their implementation and provide the program developers (the in-class observers) feedback about the quality of the professional development activities. The summative aspect of the instrument is to provide a teacher quality score, which is compared to outcome data, to determine the overall merit of the project.

Principles of Adult Learning and the Start with the Arts Professional Development Assessment Tool

Shannan McNair, Oakland University, mcnairshannan@yahoo.com

Linda Kinspel, SPEC Associates, lkinspel@specassociates.org

Melanie Hwalek, SPEC Associates, mhwalek@specassociates.org

Don Glass, VSA arts, dlglass@vsarts.edu

Start with the Arts (SWTA) is a pre-school curriculum that simultaneously uses the arts to teach pre-school literacy skills and focuses on ways to include children with disabilities in arts integrated classrooms. VSA arts, the curriculum developer, wanted to evaluate the professional development workshops that were being provided around the U.S. in how to use SWTA in pre-school child development settings. While the evaluation used multiple methods, one critical source of data was observations of SWTA workshops in action. The protocol created for these observations looked to the adult learning literature to identify observational indicators for: (1) workshop preparation, (2) workshop delivery, and (3) the workshop instructor. This paper will present the resulting observational protocol showing how it aligns with well-established principles of a effective adult learning principles.

Session Title: Nonprofit Evaluation Practice: Special Applications

Multipaper Session 933 to be held in the Agate Room Section C on Saturday, Nov 8, 4:00 PM to 5:30 PM

Sponsored by the Non-profit and Foundations Evaluation TIG

Chair(s):

Carl Hanssen, Hanssen Consulting LLC, carlh@hanssenconsulting.com

Integrating Project Design and Project Logic Into Democracy Assistance Grantmaking

Presenter(s):

Rebekah Usatin, National Endowment for Democracy, rebekahu@ned.org

Abstract: Democracy assistance presents a particular set of difficulties to the field of evaluation at both the macro and micro levels. By their very nature, these types of projects and programs are extremely difficult to evaluate and attributing causality is virtually impossible. Nonetheless, both donors and implementers of democracy assistance make considerable attempts to utilize qualitative and quantitative data to determine what difference their projects and programs are making. Without clear project design and project logic, the task of measurement is made even more difficult. There is a noticeable lack of literature pertaining to the evaluation practices of democracy assistance. This paper examines the case of one donor�s experience of integrating the concepts of project logic and project design into its grantmaking.

Evaluating the Sussex Child Health Promotion Coalition: A Case Study

Presenter(s):

Gregory Benjamin, Nemours Health and Prevention Services, gbenjami@nemours.org

Michele Lempa, OMG Consulting, michele@omgcenter.org

Marina Kaplan, Nemours Health and Prevention Services, makaplan@nemours.org

Abstract: In September, 2006 Nemours Health & Prevention Services (NHPS) partnered with several key stakeholders and local organizations to create a community-based coalition, whose aim was to address the health and well-being of children living in Sussex County, Delaware. This coalition, the Sussex Child Health Promotion Coalition has grown to include over 140 community partners. In order to evaluate the development and functioning of the coalition, a series of key informant interviews (n=14) were conducted with members of the Coalition. In addition, a web-based survey was sent to all (n=138), which yielded a response rate of almost 40%. Baseline results will be presented. From these results, not only is the Coalition able to learn from the rich data provided, but also, researchers can gain insight into implementing and evaluating this type of Coalition in a rural, low-income location, such as Sussex County, DE.

Creating a Results Framework for Child-Centered Community Development

Presenter(s):

Marjorie Willeke, Educational Service Unit 18, marj.willeke@gmail.com

Barbara Bartle, Foundation for Lincoln Public Schools, bbartle@lps.org

Jeff Cole, Nebraska Children and Families Foundation, jcole@nebraskachildren.org

Abstract: The Foundation for Lincoln Public Schools, supported by visionary leadership, places the child in the center of community development efforts. A Community Interest Assessment affirmed the need to pilot Community Learning Centers (CLCs). The Lincoln CLC Initiative is an innovative approach that links the whole community with neighborhoods, schools, and people of all ages. The core value is that lifelong learning is a civic responsibility. Unique partnerships address a myriad of issues such as safe, affordable housing, accessible health care, quality child care, and youth development. The evaluator has been at the table throughout the process, working in collaboration with the foundation and many community partners. Challenges in evaluating CLCs (including multiple funders with multiple expectations, growth in the CLC Initiative, and a small evaluation staff) are discussed. The paper concludes with the role of program evaluation in community development work.

Session Title: Walking the Tightrope: Developing Valid Tools to Measure Fidelity of Implementation That Meet Stakeholder Needs

Multipaper Session 934 to be held in the Granite Room Section A on Saturday, Nov 8, 4:00 PM to 5:30 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Chair(s):

Tania Jarosewich, Censeo Group LLC, tania@censeogroup.com

Abstract: Examination of program implementation of grant expectations is less common than is examination of student outcomes. However, without measuring fidelity of implementation, the connection between grant activities and outcomes is weak. Recognizing that a multiple site implementation provides unique opportunities for measuring fidelity of implementation, this panel will focus on methods and systems of collecting systematic data regarding implementation in statewide grant reading and math programs. The authors will describe the tensions inherent in developing valid instruments that meet client expectations and local grantees' needs, can be used by a variety of stakeholders, and may be applied to high-stakes decision-making. Panelists will focus on the methods they used to develop and validate the instruments, and to train users in collecting valid data. The papers will also reflect on the convergences and divergences of the processes used in each evaluation.

Developing Classroom-Level Measures and School-Level Measures of Implementation Fidelity

Catherine Callow-Heusser, EndVision Research and Evaluation, cheusser@endvision.net

The Bureau of Indian Education's Reading First program uses school-reported self-assessment data based on the Planning and Evaluation Tool for Effective Schoolwide Reading Programs - Revised (Kame'enui and Simmons, 2003) as a measure of implementation fidelity to contribute to decisions about continued funding. However, as the external evaluators, we felt an independent measure would likely be more aligned with student outcomes. We developed a school-based measure of implementation fidelity that included research-based indicators that aligned with the four pillars of Reading First: instructional programs and strategies, valid and reliable assessments, professional development, and instructional leadership. Additionally, a classroom-level measure of implementation fidelity is aligned with reading programs and research-based reading teaching strategies. Both classroom-level measures and school-level measures of implementation fidelity explain substantial portions of the variability in student outcomes. In this presentation, we will discuss development of the instrumentation and statistical outcomes.

Measuring Program Implementation Using a Document-Based Program Monitoring System

James Salzman, Cleveland State University, j.salzman@csuohio.edu

The Reading First - Ohio (RFO) grant indicated that implementation would be measured through a 'rubric reflective of the state's accountability system' (ODE, 2003, p. 157). The RFO Center designed a rubric to use in a document review process to hold districts accountable for attaining and sustaining fidelity. School personnel gathered artifacts and wrote summary statements for each of the grant's indicators to provide evidence of implementation for the document review. Regional consultants, supervised by the Center, reviewed the documentation multiple times each year, as both a formative and summative process. Schools that did not show progress toward full implementation after their second year in the program could lose funding. This presentation will discuss the tightrope that the Center walked in developing a tool that met the requirements of staff members of the Ohio State Department of Education and also showed strong reliability and validity.

Measures of Implementation Fidelity for Use by External Evaluators and Program Staff

Tania Jarosewich, Censeo Group LLC, tania@censeogroup.com

The Oklahoma Reading First grant did not identify how the Oklahoma State Department of Education (SDE) would monitor program implementation or evaluate district and school adherence to grant requirements. Evaluation staff and the State Department of Education team developed a statewide monitoring system that would provide a clear understanding of the strengths and needs of implementation across the state and allow state staff to make the high-stakes decision of which districts would be continued in the grant. The state evaluation team and SDE staff used a school self-assessment, a reading observation form, and site visit protocols to collect information about fidelity of implementation through a site visit at each participating Reading First school. This presentation will describe development, training, and use of the tools, and discuss the challenges inherent in developing a system in which an external evaluation team and internal project staff collect data about fidelity of implementation.

Syntheses of Local Data for Global Evaluation of Program Fidelity and Effectiveness

Elizabeth Oyer, Evaluation Solutions, eoyer@evalsolutions.com

Tania Jarosewich, Censeo Group LLC, tania@censeogroup.com

The Illinois Math and Science Partnership state evaluation framework includes five dimensions of program outcomes, including quality of professional development activities and partnerships as well as changes in teachers' content knowledge, teachers' instructional practices, and students' achievement. The cornerstone of the state-level evaluation design is the cross-site meta-analyses of local evaluation results to assess program effectiveness. The meta-analytic approach is combined with hierarchical linear modeling to analyze local and global outcomes. Measures of program implementation are a key moderating variable in analyses and are measured at the local level using a combination of logs, journals, classroom observations, and extant data. The quality of the partnerships are evaluated by triangulating a comprehensive interview protocol with artifact analyses and surveys of all key partners including teachers, local education administrators, higher education faculty, and industry partners. The presentation will discuss the issues related to balancing the needs of local evaluations with the need to provide global analyses of the statewide initiative.

Session Title: Using Indicators to Unite Partners and Players in a Common Evaluation Enterprise: Examples From the Centers for Disease Control and Prevention (CDC)

Panel Session 935 to be held in the Granite Room Section B on Saturday, Nov 8, 4:00 PM to 5:30 PM