Search Results for All Sessions

Session Title: Improving the Quality of Evaluation Practice by Attending to Context

Panel Session 102 to be held in Lone Star A on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Presidential Strand

Chair(s):

George Julnes, University of Baltimore, gjulnes@ubalt.edu

Discussant(s):

Eleanor Chelimsky, Independent Consultant, eleanor.chelimsky@gmail.com

Abstract: This panel, comprised of the three individuals involved in developing the 2009 AEA Conference Presidential Strand on context, will draw on publications developed from the strand to explore the ways in which attending to context can improve the quality of evaluation practice. Context has an influence on how we as evaluators approach and design our studies, how we carry them out, and how we report our findings. Using the five aspects of context covered by Rog in her Presidential address -- the nature of the problem being addressed, the context of the intervention being examined, the setting and broader environment in which the intervention is being studied, the parameters of the evaluation itself and the broader decision-making context—the panel will explore the ways in which attending to these areas and the dimensions within them (physical, organizational, social, cultural, tradition, and historical) may heighten the quality of evaluation practice.

Striving for Truth, Beauty, and Justice in Evaluation Practice: A Methodological Analysis of Contextually Sensitive Practice

Debra Rog, Westat, debrarog@westat.com

This first presentation will reintroduce the contextual framework offered in the 2009 Presidential address. Using the framework, it will illustrate how attending to the five areas of the framework and the dimensions within each area can inform the development and implementation of study designs and methods that embrace the standards of quality identified by House (1980). In particular, drawing on the contributions to the 2009 Presidential strand as well as other examples, the presentation will highlight context-sensitive strategies that involve stakeholder participation and involvement, provide for rigor in outcome assessments, and improve the explanatory power of studies. The presentation will analyze the extent to which these strategies appeared to enhance study credibility, were perceived to be fair by stakeholders, and resulted in findings that were considered accurate and valid. Challenges in conducting contextually-sensitive evaluation practice while balancing quality standards will be discussed and recommendations offered.

Recognizing and Reconciling Differences in Stakeholders’ Contexts as a Prelude to High Quality Evaluation: The Cases of Cultural and Community Contexts

Ross Conner, University of California, Irvine, rfconner@uci.edu

This presentation will focus on the importance of identifying differences in the contexts in which stakeholders are anchored, then reconciling these differences between and among the stakeholders in order to plan and implement high quality evaluation. Using examples from cultural and community contexts, the presenter will illustrate the types of differences among three important stakeholder subgroups: those participating in a project, those evaluating it, and those funding the project and evaluation. Drawing on their own context, people in each subgroup have viewpoints and assumptions about what constitutes a ‘successful’ project intervention and about what evidence can and should be produced to prove ‘success,’ or (to label this from the perspective of one of these subgroups, the evaluators) which designs, methods and measures are best. The lack of understanding and reconciliation of these different viewpoints and assumptions at the outset of an evaluation is one factor that hinders high quality evaluation.

Comparative Evaluation Practice and Politics: The Role of Political Context in Influencing Quality

Jody Fitzpatrick, University of Colorado, Denver, jody.fitzpatrick@ucdenver.edu

In political science and public administration, there is a strong tradition of comparative study or research. Specifically, researchers in these fields study how countries make different political or administrative choices in pursuing remedies to problems that government may address. In the United States, where evaluation has been grounded primarily in psychology and education, evaluators have not drawn from these political science traditions to study, in a systematic way, differences across countries that lead to differences in evaluation policies, practice, and, ultimately, evaluation quality. At AEA we do have a strong international presence and, in our publications, we occasionally learn of practices in other countries. This presentation, however, will draw from the comparative tradition of public administration and political science research to propose an agenda for evaluation research, one to identify political characteristics and contextual elements that may lead to different types, and qualities, of evaluation.

Session Title: Waawiyeyaa (Circular) Evaluation Tool

Skill-Building Workshop 103 to be held in Lone Star B on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Indigenous Peoples in Evaluation TIG

Presenter(s):

Andrea LK Johnston, Johnston Research Inc, andrea@johnstonresearch.ca

Abstract: Developed by Johnston Research Inc., this holistic evaluation tool, grounded in Anishnawbe traditional knowledge was created for program providers. It's a self-evaluation tool allowing programs to document both meaningful process and outcomes over time. It's also a learning tool that promotes growth and self-development among the program participants. By applying the tool at various program-milestones a full picture can be documented of the personal journeys of the participants in a systematic manner. The traditional knowledge tool provides a framework from which program participants can easily relate. Participants like the tool because the storytelling is driven by them through their own eyes and at their own pace. We will review the manual, see the 20-minute video, complete the paper and crayon exercise and incorporate our stories into an evaluation report. You will take home your story, as well as a copy of the DVD and manual. We offer additional training.

Session Title: Building a Learning Culture Within Community Initiatives and Organizations

Panel Session 104 to be held in Lone Star C on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG

Chair(s):

David Scheie, Touchstone Center for Collaborative Inquiry, dscheie@touchstone-center.com

Discussant(s):

Prudence Brown, Independent Consultant, pruebrown@aol.com

Abstract: This session explores principles and strategies for establishing a learning culture within initiatives and organizations, particularly in an urban community development context. Three case examples will be presented, each with aspirations of creating a lively, participatory learning and evaluation culture within a community organization or initiative. Ways to engage staff and project participants in planning, data collection, analysis and reflection will be examined. Tensions and pitfalls encountered in the three cases will also be considered. Challenges in navigating race and class issues – e.g. multi-cultural and lower income contexts, professional-citizen differentials in authority and credibility, and divergent interests among staff, participants, and sponsoring funders, will be given special attention. Timelines required and stages encountered in the effort to develop an effective participatory learning culture will be explored. A discussant with experience in many other learning and community change initiatives will put lessons from these three cases into broader national perspective.

Cultivating a Learning Culture in a Community Development Corporation

Kate Tilney, Hope Communit Inc, kbport@comcast.net

This presentation explores the attempts since 2008 to develop sustainable evaluation and reflection systems in an organization dedicated to rebuilding the physical and social infrastructure of a large neighborhood in Minneapolis. Over the last 15 years, Hope Community Inc. has worked to rebuild a square block just south of downtown, while also investing in building the leadership, relationships and social networks of residents. Hope’s “community engagement” work offers a variety of programming for children and adults, as well as community organizing training and systems-change efforts. Since hiring a part-time evaluation consultant who works as an internal ‘participant-observer,’ efforts to cultivate a learning culture and sustainable model have both succeeded in positively impacting the direction of Hope’s engagement work, and been challenged by issues of accountability and capacity. These will be explored, as will the potential for growth and lessons going into the future.

Building a Learning Partnership in a Neighborhood Transformation Initiative

Sue Tripathi, Making Connections, Denver, sue.tripathi@unitedwaydenver.org

This presentation explores the concept and implementation of a local learning partnership in the Annie E. Casey Foundation’s Making Connections Initiative, with ongoing efforts to build evaluation capacity and a learning community. The learning partnership sought to engage grassroots stakeholders in using data to document, assess and propel change, building capacity for learning and evaluation through the process. Three phases are explored: (1) beginning to engage stakeholders to use data to advance goals of strengthening families through neighborhood transformation; (2) creating a formal local learning partnership, a consortium of residents and others with data expertise to develop and track outcome measures and strategies to achieve results; and (3) integrating the initiative and its learning partnership into the local United Way, with potential to influence larger systems and broaden the effort to build and sustain evaluation capacity. Strengths and challenges in each phase will be examined.

Developing a Learning and Evaluation Partnership in a Regional Capacity Building Initiative

David Scheie, Touchstone Center for Collaborative Inquiry, dscheie@touchstone-center.com

This presentation will explore efforts to develop a “learning and evaluation partnership” in the Capacity Building Initiative of the Raymond John Wean Foundation. The Initiative was launched in 2007 to help revitalize the two-county Youngstown-Warren region in Ohio. In early 2009 the Foundation hired a team (led by this presenter) to guide learning and evaluation in the Initiative with a charge to evaluate progress, sharpen Initiative design and implementation, and deepen the knowledge-building capacity of the Foundation, its partners and grantees. The presentation will report on early efforts to develop learning and evaluation systems in the Initiative’s three distinct subcontexts – a neighborhood grants program, a community organizing collaborative, and a nonprofit capacity program – and to integrate these into a coherent whole for the Initiative such that the Initiative can help strengthen a culture of collaborative learning and innovation in the region overall.

Session Title: Successful Outcome Measurement in Nonprofits: Overcoming Challenges

Think Tank Session 105 to be held in Lone Star D on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Non-profit and Foundations Evaluation TIG

Presenter(s):

Lora Warner, University of Wisconsin, Green Bay, warnerl@uwgb.edu

Abstract: Successful Outcome Measurement in Nonprofits: Overcoming Challenges: How can a nonprofit organization overcome common challenges and successfully implement an effective outcome measurement system? We will present an actual case study of a local nonprofit organization attempting to implement outcome measurement The case study will illustrate common challenges faced by nonprofit organizations as they develop an outcome measurement system. Small group breakouts will each discuss a typical challenge, including: 1. Gaining the commitment of the board and top leaders 2. Increasing the organization’s evaluation capacity (staff time and expertise) 3. Identifying which outcomes to measure and developing simple, useful tools 4. Collecting and managing data efficiently 5. Using data to learn and improve the program 6. Integrating outcome data with existing management systems Through this think tank, participants will explore strategies to overcome each challenge and will take away new ideas on how to implement successful outcome measurement in nonprofits.

Session Title: Place Randomized Trials and Alternatives in a Field Setting: Examples From Psychotherapy Research

Panel Session 106 to be held in Lone Star E on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Quantitative Methods: Theory and Design TIG

Chair(s):

Lee Sechrest, University of Arizona, sechrest@u.arizona.edu

Discussant(s):

Frederick Newman, Florida International University, newmanf@fiu.edu

Abstract: Two large scale program evaluation studies researching the effects of feedback systems about patients progress in ambulatory psychotherapy have been conducted in Germany. One study used a longitudinal place randomized design where the randomization took place at the therapist level. The intervention group had access to a new feedback system and the control group did treatment as usual. The other study used a simple follow-up design without a control group. All therapists had access to the same feedback system but got feedback about their patients where they stood compared to other patients.

Evaluating a Feedback System in Ambulatory Psychotherapy Using a Longitudinal Place Randomized Trial

Manuel Voelkle, Max-Planck-Institute for Human Development Berlin, voelkle@mpib-berlin.mpg.de

In Spring 2005, the TK (a German health insurance company) started the project quality monitoring in ambulatory psychotherapy. Core of the project is a new monitoring system, which provides continuous feedback to therapists and patients on the progress of the therapy. In addition, the traditional procedure of calling in expert opinion is complemented/replaced by psychometric information. The goal of the evaluation was to determine whether the new system improves the therapy. Causal evidence was important to the stakeholders but randomization of individual patients was not possible, so we used a place randomized trial where the therapist was the unit of randomization. Large improvements in mental health (client and therapist ratings) between beginning and end of psychotherapy were found in Brief Symptom Inventory (BSI), Inventory of Interpersonal Problems (IIP), SF-12 Health Survey (SF-12), Beck Depression Inventory (BDI), impairment severity score(BSS) and Global Assessment of Functioning Scale (GAF).

Evaluating a Feedback System in Ambulatory Psychotheray Using a Classical Pre-Post Test Design With a One Year Follow-up

Andrés Steffanowski, University of Mannheim, andres@steffanowski.de

Aim of the project was evaluating a new assessment for quality assurance in ambulatory psychotherapy Bavaria, providing individual feedback about the therapeutic process in comparison to reference groups – a feature that usually in this setting is not available. Client and therapist are documenting the therapy progress using hand-held computers by answering questions about symptom severity (e.g. depression, anxiety and stress), life satisfaction, therapeutic relationship and problem domains. The results demonstrate impressive and stable effects with the treated patients. These effects lie in the upper third of internationally comparative meta-analytical studies.

Comparing Two Different Research Designs in Program Evaluation. Problems, Advantages, Pitfalls and Implications for Causal Evidence

Werner Wittmann, University of Mannheim, wittmann@tnt.psychologie.uni-mannheim.de

It is demonstrated and discussed what different kinds of unexpected problems and pitfalls showed up during the studies. The two research designs are compared with respect to threats in terms of internal, statistical conclusion, construct and external validity they have to face. The results on similar outcome instruments are compared and the implications for causal evidence and generalization of results are discussed.

Session Title: Balancing Autonomy and Uniformity in a Multi-site Evaluation: Evaluation of Program Integration Efforts at the Centers for Disease Control and Prevention (CDC)

Multipaper Session 107 to be held in Lone Star F on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Cluster, Multi-site and Multi-level Evaluation TIG

Chair(s):

Thomas Chapel, Centers for Disease Control and Prevention, tchapel@cdc.gov

On Rowing in the Right Direction: Creating an Evaluation Design for a Program Integration Effort

Presenter(s):

Thomas Chapel, Centers for Disease Control and Prevention, tchapel@cdc.gov

Abstract: In 2009 CDC’s National Center for Chronic Disease Prevention and Health Promotion (NCCDPHP) negotiated special agreements with four pilot states which gave them more flexibility in use of their chronic disease funds and allowed them to think about optimal collaborative and integrative structures that would increase the efficiency and effectiveness of their efforts. The four states offered diverse approaches to integration and collaboration but were to be held accountable to common outcomes. This session will discuss the origins, challenges, trade-offs, and final components of the evaluation approach. Common indicators were developed using the ten Essential Public Health Services as a framework. These indicators were measured in triangulated ways—objective measurement, staff surveys, key informant interviews, and social network analyses. The session will discuss the development of the framework, challenges in developing the framework, development of the indicators, and the synthesis of information from the multiple data collection sources.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Using Technological Pedagogical Content Knowledge and an Evaluation Framework to Evaluate Online Courses and Tools

Roundtable Presentation 108 to be held in MISSION A on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Distance Ed. & Other Educational Technologies TIG

Presenter(s):

Andrea Velasquez, Brigham Young University, andrea_velasquez@byu.net

David D Williams, Brigham Young University, david_williams@byu.edu

Abstract: Evaluating quality online courses has become a complex challenge as designers and evaluators are faced with multiple dimensions of the online context. This presentation will examine the use of three frameworks- an evaluation framework (Williams & Graham, 2010), Technological Pedagogical Content Knowledge (Mishra & Koehler, 2006), and ADDIE- that can be used together to help evaluators of online courses identify specific criteria and consider important questions during formative and summative evaluation of online courses. These three frameworks are presented in the context of a design case to help evaluators identify and consider relevant criteria in online course evaluations. Mishra, P., & Koehler, M. J. (2006). Technological Pedagogical Content Knowledge: A new framework for teacher knowledge. Teachers College Record, 108 (6), 1017-1054. Williams, D. D., & Graham, C. R. (2010). Evaluating e-learning. In B. McGaw, E. Baker,, & P. P. Peterson (Eds.), International Encyclopedia of Education (3rd ed.). Oxford, UK: Elsevier.

Roundtable Rotation II: Regional Education Master’s Online Training in Evaluation (REMOTE) Messages: The Job Value of a Distance Learning Graduate Program for the Pacific Region

Roundtable Presentation 108 to be held in MISSION A on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Distance Ed. & Other Educational Technologies TIG

Presenter(s):

Charles Giuli, Pacific Resources for Education and Learning (PREL), giulic@prel.org

Abstract: During 2007–2009, the University of Hawaii and the Pacific Resources for Education and Learning (PREL) offered an NSF sponsored, online, master’s degree in evaluation practice called the Regional Education Master’s Online Training in Evaluation (REMOTE)program. Previous presentations at AEA have described the challenges of implementation and the lessons learned. The purpose of this session is to present and discuss results from a follow-up study of the relevance of the program to on-the-job improvement for REMOTE participants. The study was conducted about a year after the program ended so that graduates would have time to apply the skills and knowledge they acquired in the course to their work. Graduates were asked how the program helped them understand their work and improve their decision-making. They were interviewed and completed a survey.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Increasing Nonprofit Sustainability Activities With Effective Request for Proposals (RFP's): A Mixed Methods Evaluation of RFPs as an Instrument for Success

Roundtable Presentation 109 to be held in MISSION B on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the and the Pre-K - 12 Educational Evaluation TIG

Presenter(s):

Nakia James, Western Michigan University, nakia.s.james@wmich.edu

Abstract: Nonprofit organization’s primary purpose is to provide programs and/or services. For this, they tend to rely heavily upon grant funding to sustain and deliver their programs/services. Accordingly, NPOs often generate Request for Proposals (RFPs) to procure needed services. Since most grants require an organizational assessment or program evaluation as part of their annual report, an RFP is often developed to retain the services of an external evaluator. However, though the grant may include this as a requirement, no additional information may be offered that may assist the NPO in formulating an appropriate RFP. Subsequently, they often fail to include pertinent and appropriate information for the evaluation services requested. Potential external consultants are often ill-equipped to develop an appropriate proposal due to the limitations with deficient RFPs. Clarity and inclusion of key elements are necessary and the lack of, can lead to sub-standard proposals and even unfulfilled services.

Roundtable Rotation II: Improving the Process of Reviewing Research Proposals: Reflections of a Research Review Committee

Roundtable Presentation 109 to be held in MISSION B on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the and the Pre-K - 12 Educational Evaluation TIG

Presenter(s):

River Dunavin, Albuquerque Pubic Schools, dunavin_r@aps.edu

Nancy Carrillo, Albuquerque Public Schools, carrillo_n@aps.edu

Ranjana Damle, Albuquerque Public Schools, damle@aps.edu

Abstract: Annually, dozens of evaluation and educational research proposals are submitted to Albuquerque Public Schools (APS) by universities, agencies, and individuals aspiring to conduct a remarkable range of projects. A Research Review Committee (RRC) convenes to examine submitted research and evaluation proposals for approval to conduct research at APS. Beyond ensuring projects are ethically sound, members of RRC must balance the interests of the District with research needs of applicants, grant requirements, and burden to the District and schools. Some of the questions we have considered include: What does ‘of interest and benefit to the District’ mean? Should we ensure results are made available to the District? Should research quality be a key consideration? Few guidelines are available. The purpose of this roundtable is to consider these and other questions in order to make the research review processes more systematic and efficient while providing a collaborative forum for colleagues in a research review role.

Session Title: A Tool for Designing Evaluations of Paradigm Shifts in Complex System Interventions

Skill-Building Workshop 110 to be held in BOWIE A on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Systems in Evaluation TIG

Presenter(s):

Beverly A Parsons, InSites, bparsons@insites.org

Pat Jessup, InSites, pjessup@insites.org

Marah Moore, i2i Institute Inc, marah@i2i-institute.com

Abstract: The session begins with an overview of a one-page framework for displaying a theory of change in complex systems. It focuses on patterns of change over time across multiple levels of a social system. It incorporates attention to system boundaries, relationships, and perspectives as well as system dynamics. Working in pairs or triads, the participants will practice constructing a similar visual framework based on a situation that one of them is evaluating that involves a fundamental paradigm shift (e.g., a shift from a focus on deficits to a focus on assets; from delivery of information to engaged learning; from waste to recycling). Then they will practice using their framework as the basis for designing an evaluation that helps evaluation users leverage complex dynamics in the social systems involved to achieve the desired shift with attention to sustainability and scalability.

Roundtable: Maximizing Our Collective Talent: Conversations With Senior Evaluators

Roundtable Presentation 111 to be held in BOWIE B on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Multiethnic Issues in Evaluation TIG

Presenter(s):

Tamara Bertrand Jones, Florida State University, tbertrand@fsu.edu

Pamela Frazier-Anderson, Lincoln University, pfanderson@lincoln.edu

Abstract: In many cultures community elders are respected and revered for their wisdom and years of personal and/or professional experiences, as well as the guidance provided to younger generations. As the field of evaluation continues to grow and evaluation training programs become available to increasingly diverse populations, the relationships with and experiences of senior evaluators become a valuable resource. It is rare that senior evaluators, considered experts in their respective fields, are readily accessible to those who would like to learn from them. Participants in Bertrand (2006) identified participation in professional association meetings/conferences as a major influence on the development of professional relationships. This roundtable session brings national and/or international senior evaluation leaders, novice, and mid-career evaluators to engage in evaluation discourse. The expectation is that these conversations will assist participants with improving the quality of their own evaluation activities, as well as lay the foundation for lasting professional relationships.

Session Title: Understanding, Building, and Evaluating Advocacy Capacity

Multipaper Session 112 to be held in BOWIE C on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Advocacy and Policy Change TIG

Chair(s):

Brian Quinn, Robert Wood Johnson Foundation, bquinn@rwjf.org

What is Advocacy Capacity?

Presenter(s):

Susan Sherry, Community Catalyst, sherry@communitycatalyst.org

Jacquie Anderson, Community Catalyst, anderson@communitycatalyst.org

Abstract: Community Catalyst is a national advocacy organization that works with foundations, policymakers, and state and local consumer groups on strategies to improve access to high-quality, affordable health care in the U.S. It has worked with organizations in over 40 states to help them achieve wide-reaching health policy reforms. In a W.K. Kellogg Foundation-funded study, Community Catalyst noted advocacy organizations in some states had more success advancing a consumer health agenda than those in other states. One key difference across states was different capacities of consumer organizations. In this session’s first presentation, Community Catalyst will describe the six core capacities identified as common to successful advocacy organizations and efforts. The presentation will also examine the relationship between the core capacities, reasons for building capacity across networks or systems of consumer health advocacy, and the types of technical assistance and support that can help build capacity within advocacy organizations and networks.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: A Conceptual Framework to Assess the Sustainability of Community Coalitions Post-Federal Funding

Roundtable Presentation 113 to be held in GOLIAD on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Government Evaluation TIG and the Health Evaluation TIG

Presenter(s):

Alycia Infante, University of Chicago, infante-alycia@norc.org

Jennifer Benz, University of Chicago, benz-jennifer@norc.org

Hilary Scherer, University of Chicago, scherer-hilary@norc.org

Caitlin Oppenheimer, University of Chicago, oppenheimer-caitlin@norc.org

Wilma Tilson, United States Department of Health and Human Services, wilma.tilson@hhs.gov

Abstract: We present a framework to assist evaluators with defining and measuring the sustainability of community coalitions after initial funding has ended. The federal government increasingly uses community coalitions as a programmatic approach to address emerging community health issues. The presumption is that successful community coalitions will be able to identify new resources to continue their activities and to sustain their impact in the community beyond the initial grant period. In defining sustainability, the framework considers coalition structure, goals, and activities. The framework includes a number of enabling characteristics affecting sustainability, intermediate outcomes (e.g., the expansion or retraction of the coalition, its goals or activities), and the coalition’s long-term impact on the community. Finally, the framework includes the influence of contextual factors (e.g., economy) on the sustainability of the coalition and its ability to generate outcomes. This framework is applied to a sustainability assessment of the Healthy Communities Access Program grantees.

Roundtable Rotation II: Using Community Partnerships to Reach Hard-to-Reach Populations in Health-Related Evaluations

Roundtable Presentation 113 to be held in GOLIAD on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Government Evaluation TIG and the Health Evaluation TIG

Presenter(s):

Julia Alvarez, JVA Consulting LLC, julia@jvaconsulting.com

Nancy Zuercher, JVA Consulting LLC, nancy@jvaconsulting.com

Brian O'Connell, JVA Consulting LLC, brian@jvaconsulting.com

Abstract: What is the best way to reach out to hard-to-reach populations? How can you conduct a culturally responsive evaluation and get the response rates you need? Looking for tips and advice on how to conduct a health or healthcare program evaluation that enables you to collaborate more with the communities you serve? Come share your knowledge and ask questions in this roundtable discussion where participants will have opportunities to talk through the sensitive underpinnings of health-related evaluations and brainstorm with colleagues on effective ways of reaching out to hard-to-reach populations. Session facilitators will share three unique experiences of using community partnerships as a way of reaching hard-to-reach people in health-related evaluations. Facilitators will guide conversations about the advantages and disadvantages of partnering, strategies for gathering data from these sensitive groups and the potential impacts partnerships can have on an evaluation.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Non-response Bias a Limitation: Practical Perspectives of Evaluation Quality Using Survey and Questionnaire Data

Roundtable Presentation 114 to be held in SAN JACINTO on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Quantitative Methods: Theory and Design TIG

Presenter(s):

Michelle Bakerson, Indiana University South Bend, mmbakerson@yahoo.com

Abstract: Evaluation quality from a practical stand point depends on the quality of the data gathered. Surveys and questionnaires are commonly used tools to gather data, however this type of data gathering comes with certain limitations and biases. One major limitation to this type of data is non-response bias, which exists when there is a difference in the interpretation of results that would be made regarding those who respond and those who do not respond. The bias created by non-response is a function of both the level of non-response and the extent to which non-respondents are different from respondents (Kano, Franke, Afifi & Bourque, 2008). An explanation of what occurs within survey and questionnaire data is examined using detailed alternatives of interpretation taking non-response into account by making sure the data is valid and does not contain non-response bias. Taking this extra step when examining data will help ensure quality in evaluation findings.

Roundtable Rotation II: Coding Open-Ended Survey Items: A Discussion of Codebook Development and Coding Procedures

Roundtable Presentation 114 to be held in SAN JACINTO on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Quantitative Methods: Theory and Design TIG

Presenter(s):

Heather Bennett, University of South Carolina, bennethl@mailbox.sc.edu

Joanna Gilmore, University of South Carolina, jagilmor@mailbox.sc.edu

Grant Morgan, University of South Carolina, morgang@mailbox.sc.edu

Abstract: Responses to open-ended items are generally analyzed inductively through the examination of themes. Unfortunately, key decisions in this process (such as how to segment open-ended responses and the number of codes to include in a codebook) are often smoothed over in published research articles (Draugalis, Coons, & Plaza, 1998; Lupia, 2008). To address this call for greater transparency, this round-table presentation will provide information about the decision-making process researchers from the Office of Program Evaluation (OPE) used to code open-ended items. OPE researchers will also share lessons learned in how to facilitate the coding of open-ended items among a team of researchers and ways to present findings to clients. This round-table will be useful for introducing coding procedures to novice qualitative researchers. Additionally, researchers will encourage a discussion among advanced researchers concerning key decisions in analyzing and reporting data from open-ended survey items.

Session Title: Assessing Student Learning Outcomes I: Incorporating Feeback

Multipaper Session 115 to be held in TRAVIS A on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Assessment in Higher Education TIG

Chair(s):

Jean-Marc Wise, Florida State University, jwise@fsu.edu

Determining the Quality of Student Feedback in Course Evaluations

Presenter(s):

Jean-Marc Wise, Florida State University, jwise@fsu.edu

Abstract: A common criticism of using course evaluations to evaluate teaching effectiveness is a perceived lack of feedback quality, especially when provided by undergraduate students. Instructors are wary of having these comments reviewed by administrators, claiming that students are not qualified to evaluate their teaching performance. What are the criteria they use to determine the quality of student feedback? This question was addressed by means of a survey sent to all fulltime faculty at a southeastern public university. Results showed that formative feedback is considered more useful than summative statements. Rationales, examples from class, and specific suggestions for improvement were also rated as very important criteria for assessing the quality of student feedback. Analysis of comments yielded three more factors: consistency, thoughtfulness, and consideration. Based on the findings, the paper discusses the development of a rubric that produces numeric scores for the statistical analysis of the quality of student feedback.

Student Perceptions of Providing Quality Feedback in Course Evaluations

Presenter(s):

Jean-Marc Wise, Florida State University, jwise@fsu.edu

Abstract: A central question in the debate over the low quality of student feedback in course evaluations is whether students lack the necessary skills or are simply not motivated to provide high quality feedback. In this study, the researcher surveyed a representative cross-section of undergraduate and graduate students at a southeaster public university. Quantitative items were aimed at assessing the respondents’ awareness of factors that impact the quality of their feedback, identifying what motivates them to complete course evaluations, as well as their attitudes toward course evaluations. Open-ended questions provided an opportunity to assess the consistency of numeric ratings and expand the range of reasons why some students choose to participate and others do not. The paper further describes how the results of the study can be used to create strategic in-class presentations aimed at increasing students’ motivation to participate in course evaluations and their ability to provide high quality feedback.

Maximizing Evaluation Impact in Higher Education: Using Program Evaluation Results to Measure Student Learning Outcomes

Presenter(s):

Joel Heikes, University of Texas, Austin, joel.heikes@austin.utexas.edu

Jeanette Herman, University of Texas, Austin, hermanjm@mail.utexas.edu

Abstract: Quality program evaluation is designed to provide useful results for program improvement, and it is often possible to use results for other institutional purposes including accreditation requirements. This paper provides an overview of how a well-designed, three-year, participatory evaluation of an interdisciplinary program for undergraduates was used to change processes, improve curriculum, and document success, as well as by the school/college to demonstrate student learning outcomes for accreditation compliance. Specifically, we present the evaluation design, the various data collection methods, and how results are used. We also discuss how evaluation was embedded into program processes and how stakeholders participated.

Adding an Evaluative Focus to Assessment of Higher Education Student Learning Outcomes

Presenter(s):

Jeanne Hubelbank, Independent Consultant, jhubel@evalconsult.com

Abstract: Accreditation and other organizations show increasing interest in an institute of higher education’s assessment of student learning outcomes. When asked to assess student learning outcomes, faculty often plunge directly into choosing a specific measure or tool. Sometimes there is confusion about the distinction between goals, objectives, and outcomes. Resistance to the task and unused data are common occurrences. Introducing an evaluation focus to the assessment process can result in greater participation and higher quality assessments. Rather than espouse a particular evaluation model, the presenter suggests introducing an “evaluative” way of thinking that helps clarify outcomes and addresses the needs of particular departments and individual faculty.

Session Title: Viabilities of Technologies in Evaluation Research

Multipaper Session 116 to be held in TRAVIS B on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Integrating Technology Into Evaluation

Chair(s):

Margaret Lubke, Utah State University, mlubke@gmail.com

Improving Evaluation Quality Through Use of an Interactive Database

Presenter(s):

Jan Middendorf, Kansas State University, jmiddend@ksu.edu

Aaron Schroeder, Kansas State University, aaron@ksu.edu

Sarah Bradford, Kansas State University, sbradford@ksu.edu

Valerie York, Kansas State University, vyork@ksu.edu

Abstract: As evaluator for Kan-ed, a statewide network initiative funded by the Kansas Legislature, the Office of Educational Innovation and Evaluation created and maintains the Kan-ed Membership Database, a relational database that houses all pertinent information related to the evaluation. Due to the project’s large size and the need for OEIE be able to respond quickly to client data requests, it is vital that all evaluation data is stored in one centralized location for ease of use. This high-quality, interactive tool serves to quickly generate data requests and reports for the client and other key stakeholders, while maintaining a large bank of information collected during the past seven years of the evaluation project. This presentation will provide a background on the information maintained within the database and how that information is used on a daily basis. Presenters will also discuss ways information is kept accurate and up-to-date.

Lessons Learned From Using Technology to Increase Study Participation Among Child Welfare Service Recipients

Presenter(s):

Lara Kaye, Center for Human Services Research, lkaye@uamail.albany.edu

Lynn Warner, State University of New York at Albany, lwarner@uamail.albany.edu

Rose Greene, Center for Human Services Research, rgreene@uamail.albany.edu

Corinne Noble, Center for Human Services Research, cnoble@uamail.albany.edu

Abstract: The use of personal communication technologies (e.g., email, text messaging and cell phones) may facilitate recruitment and retention of persons who are often under-represented in evaluation studies, including recipients of services that are potentially stigmatizing, such as those delivered in child welfare settings. This presentation emphasizes lessons learned when alternatives to traditional telephone and mail data collection methods were offered in a multi-county study whose main purpose was to evaluate satisfaction with strength-based services received by families at risk of child abuse or neglect. In light of results that few people provided email or text contact information, and that the great majority of satisfaction surveys were completed through traditional methods, discussion will focus on multiple barriers to using technology (e.g., access, comfort, confidentiality concerns) and implications for improving the viability of technologies in evaluation research involving under-represented groups.

Database Use in Evaluation Research: Opportunities and Challenges for Supporting Continuous Improvement of Partnerships, Programs, and Projects

Presenter(s):

William R Penuel, SRI International, william.penuel@sri.com

Barbara Means, SRI International, william.penuel@sri.com

Abstract: This paper presents examples of how evaluators can support partnerships, programs, and projects in using large-scale, longitudinal databases for continuous improvement. We argue that such databases can be valuable in supporting continuous improvement when evaluators and practitioners create partnerships in which (1) data inform program designers about potentially effective models they can adapt, (2) data are complemented by locally developed implementation and outcome measures; and (3) data from different institutional sectors enable analysis of cross-contextual change in outcomes for individuals and settings. We develop examples of each of these potential uses and detail the roles evaluators can play in supporting continuous improvement.

Evaluating Technology in Health Care: Testing the Usability of a Clinical Trial Query Tool Using Think Aloud Methods

Presenter(s):

Stuart Henderson, University of California, Davis, stuart.henderson@ucdmc.ucdavis.edu

Estella Geraghty, University of California, Davis, estella.geraghty@ucdmc.ucdavis.edu

Julie Rainwater, University of California, Davis, julie.rainwater@ucdmc.ucdavis.edu

Abstract: In the health care field, new technology is being constantly introduced to improve patient care, streamline medical record keeping, and increase clinicians’ access to information. Evaluators can play an important role in the development and improvement of new health technology by developing effective methods to evaluate its implementation and usability from the users’ point of view. In this paper, we discuss the evaluation of a clinical trial query tool (based on Harvard University’s Informatics for Integrating Biology and the Bedside-i2b2 tool) adapted by the University of California Davis Health System. To understand clinicians’ experiences using the tool, we developed an evaluation toolkit that included surveys, think alouds, and cursor movement analysis (through Camtasia Studio software). Focusing on the think aloud and cursor movement analysis, we discuss the opportunities and challenges these methods present. This project provides an example of both evaluating the use of technology and using technology in evaluation.

Session Title: Planning Programs: Allocating Scarce Resources Based on Needs Assessment

Multipaper Session 117 to be held in TRAVIS C on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Health Evaluation TIG

Chair(s):

Nicola Dawkins, ICF Macro, nicola.u.dawkins@macrointernational.com

Discussant(s):

Laura Leviton, Robert Wood Johnson Foundation, llevito@rwjf.org

Shifting Landscapes: Overview of Environmental and Policy Change Approaches and Measurement Tools for Chronic Disease Prevention

Presenter(s):

Dana Keener, ICF Macro, dana.c.keener@macrointernational.com

Amber Robinson, ICF Macro, amber.b.robinson@macrointernational.com

Abstract: As more public health initiatives are shifting attention towards systems, policies, and environments that encourage healthy behaviors for chronic disease prevention, evaluators are being asked to adapt tools and measures to fit the changing landscape of public health interventions. Accordingly, this presentation will: (1) describe a range of environmental and policy change efforts currently underway at local and national levels; (2) highlight the use of this approach in major federal initiatives such as Communities Putting Prevention to Work; (3) identify challenges and solutions associated with evaluating environmental and policy initiatives; and (4) describe existing tools that can be used to measure outcomes related to environmental and policy level change.

Session Title: Are Universal School-based Prevention Programs Effective? It Depends on the Students and Outcomes Targeted

Multipaper Session 119 to be held in INDEPENDENCE on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Human Services Evaluation TIG

Chair(s):

Wendy Garrard, University of Michigan, wgarrard@umich.edu

Discussant(s):

Ann Doucette, George Washington University, doucette@gwu.edu

Second Step Violence Prevention: A Meta-analysis of Thirty Treatment\Control Studies

Presenter(s):

Stephanie Reich, University of California, Irvine, smreich@uci.edu

Wendy Garrard, University of Michigan, wgarrard@umich.edu

Abstract: This paper presents the results of a meta-analysis of "Second Step: A Violence Prevention Curriculum" (SSVP) research spanning 17 years (1993-2010). Second Step is a universal prevention program aimed at reducing school violence and promoting children’s social competence. The curriculum focuses on three core abilities (a) empathy; (b) impulse control and problem solving; and (c) anger management and is designed for three age levels: Preschool-Kindergarten, elementary grades, and middle school . The present work is the first quantitative synthesis of the body of SSVP evaluative research. This meta-analysis of treatment-control evaluations of SSVP 1) explores main effects of the SSVP program on aggressive, antisocial, and prosocial behaviors, 2) assesses the impact of each of the three “skill-building units” (i.e., empathy, impulse control and problem solving, and anger management) on behavior in those domains, and 3) determines whether the program has stronger impacts for specific age ranges.

Session Title: Innovative Applications of Propensity Scores and Propensity Score Methodology Adjustments to Address Data Constraints

Panel Session 120 to be held in PRESIDIO A on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Quantitative Methods: Theory and Design TIG

Chair(s):

Vajeera Dorabawila, New York State Office of Children and Family Services, vajeera.dorabawila@ocfs.state.ny.us

Discussant(s):

MH Clark, Southern Illinois University, mhclark@siu.edu

Abstract: The objective of this presentation is to illustrate innovative applications of propensity scores and methodology adjustments that can be made to address data constraints. This session is of particular interest to the Quantitative Methods Topical Interest Group as it covers propensity scores in a way that will appeal to both novices and experts. It will be of interest and accesible to novices as the first presentation will discuss various propensity score matching techniques and share computer programs. Both experts and novices will find it of interest as the applications outline how data constraints and evaluation needs can be addressed through the use of propensity scores. In doing so, the presentors will describe novel applications and methods of addressing data issues.

A Step-by-Step Application Of Propensity Score Matching With Act Data

Oksana Wasilik, University of Wyoming, oksana@uwyo.edu

Anita Drever, University of Wyoming, adrever@uwyo.edu

The researchers will present a case study that uses propensity scores to control for background characteristics so that ACT scores before and the after the implementation of a state scholarship program can be compared. The focus of the paper will be not on the results, however, but rather on the process of applying propensity score matching. The presentation will appeal to novices learning how to use propensity scores as well as to experts who will have an opportunity to share their insights and recommendations concerning the presented case study. We will discuss the advantages and disadvantages associated with different matching methods (nearest-neighbor, kernel and stratification), as well as how to handle datasets that do not include optimal covariates. The presentation will involve sharing copies of the annotated STATA programs detailing the process the authors used to generate the results.

Multiple Imputation and Propensity Scores Matching to Address Data Constraints in Constructing a Control Group to Evaluate an Intervention to Reduce Recidivism of Juvenile Delinquents Released From Residential Facilities

Vajeera Dorabawila, New York State Office of Children and Family Services, vajeera.dorabawila@ocfs.state.ny.us

Leigh Bates, New York State Office of Children and Family Services,

Susan Mitchell-Herzfeld, New York State Office of Children and Family Services,

Therese Shady, New York State Office of Children and Family Services,

Do Han Kim, New York State Office of Children and Family Services,

The objective of this presentation is to demonstrate how propensity scores were utilized to address administrative data constraints when constructing a control group in a quasi-experimental design. In this study, the treatment group consisted of juvenile delinquents that received a short-term community based aftercare program upon release from New York State residential facilities. The purpose of the intervention program was to reduce rates of recidivism. In constructing the control group, the limited number of variables available and sometimes missing values in administrative databases was a data constraint. This constraint was addressed through the multiple imputation technique which maximized the utility of available variables. The next and related constraint was in identifying a match for each treatment individual. This constraint was addressed through a phased approach to nearest neighbor matching within calipers defined by the propensity score. This approach, an adjustment of that recommended by Rosenbaum and Rubin (1985), was able to balance the treatment and control groups.

An Innovative Use of Propensity Score Matching to Evaluate Alcoholics Anonymous’ Effect on Drinking

Stephen Magura, Western Michigan University, stephen.magura@wmich.edu

Evaluation studies consistently report correlations between Alcoholics Anonymous (AA) participation and less drinking or abstinence. Randomization of alcoholics to AA or non-AA is impractical and difficult. Unfortunately, non-randomization studies are susceptible to artifacts due to endogeneity bias, where variables assumed to be exogenous (“independent variables”) may actually be endogenous (“dependent variables”). A common such artifact is selection bias, where different types of people choose to participate or not participate in AA and also have different drinking outcomes. The paper will present a secondary analysis of a national alcoholism treatment data set, Project MATCH, that controls for selection into AA participation using propensity score matching (PSM). The PSM analysis is unique in statistically exploiting the random treatment assignment feature of Project MATCH, since that resulted in a correlation between treatment assignment and AA participation. The presentation will be accessible to evaluators without advanced statistical training. Supported by R21 AA017906.

Session Title: Organizational Learning and Evaluation Capacity Building TIG Business Meeting and Presentations: Advancing Quality in Evaluation Capacity Building

Business Meeting and Multipaper Session 121 to be held in PRESIDIO B on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Organizational Learning and Evaluation Capacity Building TIG

TIG Leader(s):

Michelle Baron, The Evaluation Baron LLC, michelle@evaluationbaron.com

Gary Skolits, University of Tennessee, Knoxville, gskolits@utk.edu

Stephen J Ruffini, Wested, sruffin@wested.org

Megan Bennett, Training Evaluation and Metrics, megan_bennett@cable.comcast.com

Chair(s):

Michelle Baron, The Evaluation Baron LLC, michelle@evaluationbaron.com

Gary Skolits, University of Tennessee, Knoxville, gskolits@utk.edu

Evaluation Capacity and Quality in Evaluation: Exploring Interrelations

Presenter(s):

Sebastian Lemire, Rambøll Management Consulting, setl@r-m.com

Steffen Bohni Nielsen, Rambøll Management Consulting, sni@r-m.com

Abstract: This paper focuses on the interrelations between evaluation capacity (building) (ECB) and quality in evaluation. Despite some recent contributions on ECB, differing conceptions still exist concerning what constitutes evaluation capacity building, let alone the nature of the capacity being built. This paper discusses different approaches to the definition, purpose, and methods of ECB as well as their implications on evaluation quality.

What, Me Evaluate? Building the Evaluation Capacity of School District Personnel

Presenter(s):

Patricia Lauer, Rocky Mountain Center for Health Promotion and Education, patl@rmc.org

Abstract: Evaluation is usually a requirement of grant-funded programs. Obtaining a high quality evaluation is a challenge when a grant-funded program is implemented across multiple sites. This is particularly true when data collection depends on site coordinators who have limited evaluation capacity. This paper describes evaluation technical assistance strategies and tools that were used to build the capacities of personnel from 15 school districts to evaluate their grant-funded tobacco prevention programs.

Session Title: Integrating Realist Evaluation Strategies in a Substance Abuse and Mental Health Services Administration (SAMHSA) System of Care Local Evaluation

Skill-Building Workshop 122 to be held in PRESIDIO C on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Alcohol, Drug Abuse, and Mental Health TIG

Presenter(s):

Mansoor Kazi, State University of New York at Buffalo, mkazi@buffalo.edu

Connie Maples, ICF Macro, connie.j.maples@macrointernational.com

Rachel Ludwig, Chautauqua Tapestry, mesmerr@co.chautauqua.ny.us

Abstract: This demonstration will illustrate how realist evaluation strategies can be applied in the evaluation of 100% natural samples in agencies that are providing mental health and other services to youth and families. Mental health agencies routinely collect data that is typically not used for evaluation purposes. This demonstration will include new data analysis tools drawn from both the efficacy and epidemiology traditions to investigate patterns in this data in relation to outcomes, interventions and the contexts of practice. For example, binary logistic regression can be used repeatedly with whole school databases at every marking period to investigate the effectiveness of school-based interventions and their impact on school outcomes. The demonstration will include practice examples drawn from the SAMHSA funded System of Care community mental health services for children with serious emotional disturbance and their families in Chautauqua County, New York State.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Evaluation Quality in Measuring Teacher Quality: The Impact of the Targeted Assistance Coaching Model on Local Education Agency (LEA) Improvement

Roundtable Presentation 123 to be held in BONHAM A on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Presenter(s):

Bruce Yelton, Praxis Research Inc, praxisresearchinc@gmail.com

MaryBeth Gilbert, Praxis Research Inc, marybethgilbert@bellsouth.net

Paula Plonski, Praxis Research Inc, pplonski@windstream.net

Abstract: This evaluation focused on the impact of the Targeted Assistance (TA) coaching model implemented by a large southeastern school system on instructional quality and student learning as required by No Child Left Behind legislation. The TA coaching model provides intensive coaching assistance to teachers within schools that are selected based on a greatest need data-driven rubric. Data was collected concerning teacher performance, classroom instructional behaviors, the intensity of coaching services provided, and student achievement. Evaluation quality was enhanced by a theory-driven program model constructed with stakeholder input, school administrative involvement, utilization of a growth model that included sub-group differences to measure student academic progress, and training for use of a classroom observation instrument that was analyzed for inter-rater reliability. Results to-date have shown that there have been significant positive changes in pre- to post-coaching classroom observations regarding teacher instructional behavior.

Roundtable Rotation II: Evaluating the Effects of Year-Long Professional Development on Teachers: Final Refinement of the University of California, Los Angeles (UCLA) Center X and Social Research Methodology (SRM) Evaluation Group Teacher Pre/Post Survey

Roundtable Presentation 123 to be held in BONHAM A on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Presenter(s):

Nicole Gerardi, University of California, Los Angeles, gerardi_nicole@yahoo.com

Janet Lee, University of California, Los Angeles, janet.lee@ucla.edu

Abstract: The U.S. Department of Education, state governments, school districts, universities, and private educational programs are forming partnerships to deliver large scale and individualized Professional Development (PD) to lower performing school districts and schools to raise the quality of teachers. In this Round Table we share an instrument, under development for 5 years with UCLA Center X and the SRM Evaluation Group, geared at evaluating year-long PD efforts. The survey is informed by multiple survey sources, evaluators, educational researchers, PD program directors, expert subject matter professionals, professional learning partners, and teachers. The survey has been piloted in 3 different PD programs, at 4 separate administrations, in a combined total of 33 different schools. We now believe it is time to receive feedback from the larger evaluation community. We hope to further refine the instrument in this Round Table and share the resource with other evaluators, thereby improving the practice of evaluation.

Session Title: The Intersection of Strategy and Evaluation: What Are We Learning?

Panel Session 124 to be held in BONHAM B on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Non-profit and Foundations Evaluation TIG

Chair(s):

Sarah Stachowiak, Organizational Research Services, sarahs@organizationalresearch.com

Discussant(s):

Julia Coffman, Center for Evaluation Innovation, jcoffman@evaluationexchange.org

Abstract: Increasingly, philanthropic organizations want to look beyond individual grant-level evaluations to see what they can learn across their portfolios to make strategic decisions. For evaluators, this raises important questions: What is the intersection between strategy development and evaluation? What levels of evidence are necessary? What approaches to data collection are a good fit and match strategy and budget cycles? What kinds of products are created? How do the needs of philanthropic organizations differ from public sector or non-profit organizations? What are unique dynamics or contexts in collecting information from grantees (e.g., sun-setting funding, balancing expectations among grantees)? This session will explore these questions from the perspectives of evaluation consultants with experience working on strategy-level evaluation and foundation evaluation staff. Using a “fishbowl” approach, panelists will engage in a dialogue with each other, providing real-world examples of how they wrestle with the unique opportunities and challenges associated with this work.

Perspectives From Hallie Preskill, Foundation Strategy Group (FSG) Social Impact Advisors

Hallie Preskill, Strategic Learning & Evaluation Center, hallie.preskill@fsg-impact.org

Increasingly, strategy and evaluation are being seen as two sides of a coin. For example, questions about the progress and impact of a particular strategy can help frame an evaluation, while an evaluation’s findings can inform the refinement of a strategy. This session will explore the following questions: What is the evaluator’s responsibility when at the start of an evaluation he/she realizes that the initiative is based on a faulty strategy, or worse, no strategy? What competencies are needed by evaluators who bridge the strategy-evaluation continuum? How does the developmental evaluation approach reflect the intersection of strategy and evaluation, and how does it challenge conventional evaluation wisdom, roles and approaches? While some in the evaluation profession may bristle at the notion of evaluators engaging in strategy work, it may be time for evaluators to acknowledge that we leave a lot on the table when our work stops where strategy begins.

Perspectives From Lance Potter, Bill & Melinda Gates Foundation

Lance Potter, Bill & Melinda Gates Foundation, lance.potter@gatesfoundation.org

The Bill & Melinda Gates Foundation aspires to make measurement, learning, and evaluation central to strategy development and grant making. We seek to ensure that our MLE practices are of consistently high quality and utility, while allowing programs to design measurement that is responsive to a range of needs. As a relatively young organization, we are actively evolving our measurement culture and practices. To that end, we are currently piloting foundation-wide measurement guidance. The Actionable Measurement Guidelines offer a perspective on why we measure, and guidance on designing strategically aligned measurement plans that will provide results with programmatic and strategic utility. The Guidelines help program teams select the appropriate type of measurement (e.g., monitoring versus evaluation) for a given situation, and suggest the nature of evaluation questions and methods for various categories of measurement. Through this effort, we hope to achieve rigorous, actionable, and parsimonious measurement throughout the organization.

Strategy, Evaluation and Strategic Learning at the Packard Foundation, Perspectives From Gale Berkowitz

Gale Berkowitz, David and Lucile Packard Foundation, gberkowitz@packard.org

The Packard Foundation’s commitment to effectiveness directly plays out in our grantmaking culture and practices. No strategic effort can be successful without feedback on a continuous basis and without good data. We have tried to create a culture of improvement and collaboration, with staff and grantees being allowed to make mistakes, to go in a different direction, and take corrective action. As a Foundation, we also need to be aware that the burden of evaluation can be large and so we consciously work to minimize the burden and maximize the value of evaluation for our grantees. We have been steadily shifting from evaluation for accountability to that for program improvement, or “real-time” evaluation. For us, real-time means an appropriate mix of monitoring, evaluation, and learning. Balance and feedback are essential as we try to get the highest quality and timely information, to make the best program strategy decisions possible.

Perspectives From Mayur Patel, Knight Foundation

Mayur Patel, John S and James L Knight Foundation, patel@knightfoundation.org

Are a program’s goals realistic? Are the critical assumptions on which a strategy was built valid? Are the methods used to implement a set of objectives the most effective means of achieving the desired impact? These questions are at the heart of strategy evaluation. Whereas project evaluations regularly use case study methodologies, strategy evaluations often resemble exercises in market research and comparative analysis. At the Knight Foundation this kind of assessment has helped us address: how the design and format of a grant making contest could be improved; what operating assumptions about innovation were embedded in a seed funding initiative and whether these were aligned with how certain projects were successfully developed and adopted? What the range of financing sources (private and public) were available to grantees in a particular sector and what this revealed about the role for philanthropic funding?

Perspectives From Organizational Research Services

Sarah Stachowiak, Organizational Research Services, sarahs@organizationalresearch.com

In all our evaluation consulting, we strive to work as a thought partner with our clients—outsiders with expertise in evaluation and measurement who can integrate varied points of views, offer a fresh perspective and provide useful and actionable observations and recommendations. This orientation has become increasingly important when we work with philanthropic clients to look across initiatives to inform future strategy development. At the same time, unique issues have arisen: How much and what kind of evidence is necessary for decisionmaking? What methods provide meaningful data on a useful timeline without overly burdening grantees? What products or processes are most useful to support internal planning? Based on a variety of recent experiences with strategy evaluation, such as looking across learning questions for a grant program finalizing their strategy refresh, we will share our approaches and lessons learned.

Perspectives from Tom Kelly, Annie E Casey Foundation

Tom Kelly, Annie E Casey Foundation, tkelly@aecf.or

The Annie E. Casey Foundation focuses its grantmaking on initiatives in the United States that have significant potential to demonstrate innovative policy, service delivery, and community supports for disadvantaged children and families. As Associate Director of Evaluation, Tom brings to this discussion a vantage point of thinking about strategy evaluation across a range of programs, including communications, advocacy and policy evaluation for the KIDS COUNT Network to the ten-year place-based comprehensive community change initiative, Making Connections, and more. He will also speak to lessons learned and implications from recent foundation-wide reorganization, the role of past evaluations and theory of change development, and the foundation’s results framework orientation.

Session Title: Managing Quality Through the Stages of Science, Technology, Engineering and Mathematics (STEM) Educational Evaluation

Panel Session 125 to be held in BONHAM C on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Chair(s):

Leslie Goodyear, National Science Foundation, lgoodyea@nsf.gov

Abstract: The Program Evaluation Standards and the AEA Guiding Principles represent overarching goals to foster quality in evaluation studies. Nonetheless, different contexts shape the concept of quality during the stages of a program evaluation. Consistently during evaluation studies, evaluation managers make methodological choices to resolve challenges and maintain quality. This panel will discuss a collection of challenges faced by education program evaluators at different evaluation stages: planning, data collection, data analysis, and reporting. Real-world challenges, such as sensitivity to educational settings (formal, informal, and afterschool), balancing partner needs, multisite logistics, data burden on participants, and use in reporting are situated within The Program Evaluation Standards and illustrated using STEM education evaluation case examples. Challenges presented will be balanced with successful strategies and lessons learned from practice.

Planning for Quality: Balancing Partner Needs in a Multi-site Evaluation of Science Education

Ardice Hartry, University of California, Berkeley, hartry@berkeley.edu

During the planning stages, evaluators are often faced with the challenge of incorporating the requirements and expectations of multiple stakeholders. This presentation considers issues such as balancing the public-policy related needs of an advocacy organization with the goals of rigorous and objective research; ensuring that the research and evaluation questions can be addressed within the scope of work; and planning for data collection across a wide range of sites. Through the description of an initiative to collect data on the state of science education in California, which has just completed the planning stages, the presentation will discuss the specific challenges that were faced during the planning stages, and how solutions, such educating stakeholders in evaluation through the use of logic models, were sought that balanced the need of the many partners. The presentation concludes with a discussion of planning the data collection timeline, which provides context for the next presentation.

Relation of Data Quality and Participant Burden in a Science-centered Leadership Evaluation Design

Juna Snow, University of California, Berkeley, jsnow@berkeley.edu

An illustrative case will be shared in which an evaluator began to lead and manage an in-progress evaluation design during the data collection stage. Specific challenges the evaluator will discuss relate issues of data quality and participant burden. A refined design emerged after examining the issues and returning to the question: How will this evaluation be used?

Not My Plan: Executing Someone Else’s Analysis Design While Balancing Partner Needs and Communicating Useable Findings

Ellen Middaugh, University of California, Berkeley, ellenm@berkeley.edu

Nationally funded research and development studies often require researchers and external evaluators to collaborate on data collection to serve multiple goals. Often, these partners will share the burden of data collection and may make use of the same data. However, their priorities for data collection and analysis may not coincide. Involving an internal evaluator can help provide continuity of data quality for both partners and promote utilization of evaluation findings by program or curriculum developers. This presentation will discuss two examples where an internal evaluator joined a project during the analysis and reporting phase. The following challenges, and accompanying strategies, will be discussed: (1) establishing and renegotiating spheres of responsibility, (2) communicating what the design can and cannot deliver to the client, (3) verifying and trouble-shooting data quality, and (4) translating findings.

Attention to Quality in Reporting Evaluation Findings in STEM Education Programs

Bernadette Chi, University of California, Berkeley, bchi@berkeley.edu

The Program Evaluation Standards (PES) and AEA Guiding Principles describe several aspects that infer quality in evaluation that relate to report clarity (PES U5), timeliness and dissemination (PES U6) and full disclosure of findings (PES P6). However, while these elements are important for evaluators to address, there are additional criteria that may reflect the needs and interests of clients and stakeholders that define quality evaluation products for them, including a particular focus on various stakeholders; opportunities to adapt data analysis and evaluation questions; and the opportunity to review and edit report drafts for accuracy. This presentation will discuss examples of reporting from three STEM evaluation studies across different education settings that utilized different reporting formats but a similar process to provide reports to internal clients; formative evaluation reports to external clients; and summative evaluation reports that were deemed useful and meaningful.

Session Title: Evaluating Supplementary Programs in Educational Settings

Multipaper Session 126 to be held in BONHAM D on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Chair(s):

Sheila A Arens, Mid-Continent Research for Education and Learning, sarens@mcrel.org

Discussant(s):

Javan Ridge, Colorado Springs School District 11, ridgejb@d11.org

Aligning Evaluation Tools in Curriculum and Non-curriculum-based After School Programs

Presenter(s):

Kelly Murphy, Claremont Graduate University, kelly.murphy@cgu.edu

Tiffany Berry, Claremont Graduate University, tiffany.berry@cgu.edu

Krista Collins, Claremont Graduate University, krista.collins@cgu.edu

Abstract: Durlak and Weissberg (2007) suggest that afterschool programs positively promote youth development if activities align with SAFE program features (i.e., Sequenced, Active, Focused, and Explicit). However, SAFE features were established within curriculum-based afterschool programs that taught a particular skill (e.g., self-confidence, etc.). It is unclear whether SAFE features predict positive outcomes in non-curriculum based afterschool enrichment programs. Thus, in this session, we discuss (1) the definition and how to measure each of the SAFE features, (2) if SAFE features are valid in non-curriculum based, enrichment type afterschool programs, (3) the consistency in reporting methods of SAFE features, and (4) if SAFE features should be incorporated into evaluation frameworks of afterschool programs. Implications of this paper apply to educational evaluators who grapple with measuring setting-level variables within both curriculum-based and non-curriculum school-linked programs. Our discussion will include data from an on-going external evaluation of After-School All-Stars, Los Angeles.

Implementing Bullying Prevention

Presenter(s):

Benjamin Cohen, Center for Schools and Communities, bcohen@csc.csiu.org

Heather Cecil, Center for Schools and Communities, hcecil@csc.csiu.org

Abstract: We evaluate a large scale implementation of the Olweus Bullying Prevention Program [OBPP] (Olweus, 1994) in over 40 schools in Pennsylvania to assess how change in bullying climate is related to program implementation quality. To evaluate the program, we linked over 1,100 teacher records to student (N=2,018) bullying victimization data. Teachers’ reports of classroom level program activities, and their perceptions and attitudes about the program serve as independent variables. We use ordinal regression analysis to relate different levels of teacher attitudes and activities to the probability of different bullying outcomes. Analyses revealed that certain program activities are associated with higher levels of teacher interventions in bullying incidents. We also describe a method for evaluating the quality of a school violence prevention program using a large scale evaluation system.

The Playworks Evaluation: A Randomized Study of a School-based Intervention to Promote Physical Activity and Play

Presenter(s):

Daniel Finkelstein, Mathematica Policy Research, dfinkelstein@mathematica-mpr.com

Susanne James-Burdumy, Mathematica Policy Research, sjames-burdumy@mathematica-mpr.com

Martha Bleeker, Mathematica Policy Research, mbleeker@mathematica-mpr.com

Abstract: Playworks is a physical activity program for elementary schools that places full-time coaches in low-income schools to organize activities during recess and other periods throughout the school day. These activities are designed to engage students in physical activity, foster social skills related to cooperation and conflict resolution, and reduce behavioral problems during recess that spillover into the classroom and interrupt instruction. Mathematica Policy Research is conducting a randomized study of the Playworks program at 20 schools in five cities to assess the program’s impact on student outcomes over the course of the 2010-2011 academic year. The evaluation will assess outcomes through teacher and student surveys, administrative records, and direct observation of students during recess periods. This presentation will provide an overview of the evaluation design and discuss the methodological challenges associated with measuring the impact of this school-based intervention.

Quality Assessment of Woodcraft Rangers After-School Program: Challenges and Solutions

Presenter(s):

Lisa Garbrecht, EVALCORP, lgarbrecht@evalcorp.com

Cathie Mostovoy, Woodcraft Rangers, cmostovoy@woodcraftrangers.org

Julie Slay, EVALCORP, jslay@evalcorp.com

Kristen Donovan, EVALCORP, kdonovan@evalcorp.com

Mona Desai, EVALCORP, mdesai@evalcorp.com

Abstract: Research shows an association between participation in high quality after-school programs and positive developmental outcomes for youth. However, few research studies or evaluations examine which components of program quality specifically contribute to positive outcomes. As part of a comprehensive quality assessment, evaluators measured a range of variables within and across 62 after-school program sites to determine the extent to which program quality was associated with student outcomes. A “quality model” was designed in collaboration with program staff, and factor analysis was utilized to identify categories of quality. Due to limitations related to the use of factor analysis with small samples, several other methodological approaches were used to create a unique approach to assessing quality. Setting benchmarks and testing “model” sites provided insight into how to evaluate quality and its relationship to youth outcomes. Recommendations for measurement designs and linking program implementation quality to student outcomes will be discussed.

Session Title: Environmental and Energy Evaluations: Strategies for Pursuing Innovative Approaches That Demonstrate Impact, Promote Continuous Improvement and Foster Organizational Learning

Multipaper Session 127 to be held in BONHAM E on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Environmental Program Evaluation TIG

Chair(s):

Mary McEathron, University of Minnesota, mceat001@umn.edu

Adapting Interdisciplinary Methods to Evaluate the Social Outcomes of Environmental Programs: Five Lessons from Minnesota

Presenter(s):

Karlyn Eckman, University of Minnesota, eckma001@umn.edu

Erika Rivers, Minnesota Department of Natural Resources, erika.rivers@dnr.state.mn.us

Kimberly Nuckles, Minnesota Pollution Control Agency, kimberly.nuckles@state.mn.us

Valerie Were, University of Minnesota, were@umn.edu

Abstract: In the current economic climate, public agencies have fewer resources to plan, implement and evaluate environmental programs. In Minnesota agencies are under pressure from both state legislators and new federal mandates to demonstrate results from the investment of public resources. Evaluation practice varies widely across agencies, from costly and complex indicator matrices to absence of program evaluation. Researchers successfully adapted the Knowledge, Attitudes and Practices (KAP) study method (commonly used in international community health programs) to large and small environmental projects in Minnesota. The Minnesota KAP method has proved to be a low-cost, flexible and practical tool for both planning and evaluation. It has been tested with invasive species, urban stormwater, shoreland habitat, water quality, septic and other programs. In all cases, it met or exceeded the evaluation requirements of the users. Five lessons are presented from applied evaluation research on this interdisciplinary program evaluation methodology.

Methodological Reflections and Organizational Learning Generated by a Qualitative Evaluation of a Leadership Development Program of the U.S. National Park Service

Presenter(s):

Jennifer Jewiss, University of Vermont, jennifer.jewiss@uvm.edu

Daniel Laven, National Park Service, daniel_laven@nps.gov

Nora Mitchell, National Park Service, nora_mitchell@nps.gov

Abstract: “Many leadership programs … by definition … are ventures into the unknown, aspiring beyond the tried and true. From that point of view, evaluation is about discovery, understanding what is possible to achieve, and even helping to characterize new kinds of leadership.” (Leviton, p. xii, in The Handbook of Leadership Development Evaluation, 2007). This paper explores the notion of “evaluation as discovery” as it relates to a qualitative evaluation of the Superintendents Leadership Roundtable (SLR), an innovative leadership development program of the National Park Service Conservation Study Institute. The value of the study process and findings extended beyond an assessment of the program – surfacing important observations about the complex and challenging leadership role that superintendents play and the broader contexts in which these leaders and the SLR program operate. This presentation will reflect on the organizational learning generated as a result of the chosen research methods and stakeholder engagement process.

A Task for Superwoman: Pursuit of Truth and Justice Within Energy Impact Evaluations

Presenter(s):

Mary Sutter, Opinion Dynamics Corporation, msutter@opiniondynamics.com

Abstract: In the world of energy efficiency programs, the fate of millions of dollars in penalties or rewards often lies with the results of an impact evaluation. Yet, the methods to determine net impacts are often questioned by stakeholders, entities determining the penalties and rewards have difficulty understanding the ramifications of one method versus another, and the available methods are becoming less able to discern impacts well. This paper will discuss the tightrope between pushing the envelope to use newer methods and continually educating relevant stakeholders. We will highlight the struggle to assess programs with truth and justice while working within a regulatory framework where policy choices and evaluation methods may be at odds.

Continuous Improvement in Energy Efficiency: Helping Clients Shift From One-Shot Evaluations Towards Assessing the Effectiveness Within and Across Programs

Presenter(s):

Marjorie McRae, Research Into Action, marjorie@researchintoaction.com

Jane Peters, Research Into Action, janep@researchintoaction.com

Abstract: In the field of energy conservation, continuous improvement is considered a best practice when managing energy efficiency programs. Process evaluations emphasizing continual feedback and nearly real-time data collection and analysis are recommended to utilities and energy commissions. This presentation will discuss examples of how evaluations provided fodder for the adoption of continuous improvement practices by organizations such as the Northwest Energy Alliance (NEEA), Energy Trust of Oregon, and New York State Energy Research & Development Agency (NYSERDA). It will also discuss how efforts to promote continuous improvement have been difficult to achieve within regulatory proceedings, as in California. This topic will be particularly relevant for evaluators working with resource conservation programs that must manage relationships with regulatory agencies or political organizations.

Session Title: Standards of Evidence for Evaluating Extension Programs: A Changing Picture?

Panel Session 128 to be held in CROCKETT A on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Extension Education Evaluation TIG

Chair(s):

Nicki King, University of California, Davis, njking@ucdavis.edu

Abstract: This panel session will examine a variety of perspectives regarding what types of evidence may be best suited to demonstrate program effectiveness within the Extension system. The question is particularly challenging for Extension because of its numerous partners at the federal and state levels, as well as its varied funding streams for program development and evaluation. The three presentations will each cover a distinct viewpoint, including the use of randomized trials, the challenges presented by Extension’s broad diversity of programs, and the contributions of program logic models to decisions about acceptable evidence. The evolution of Extension at the federal and state levels, in terms of organizational structure and funding patterns, creates the need to examine how determinations are made regarding program outcomes and impacts. Implications for organizational learning and evaluation capacity building will be discussed.

Randomized Trials in Extension Evaluation: How Big a Tool in the Tool Chest?

Marc Braverman, Oregon State University, marc.braverman@oregonstate.edu

Randomized trials (RTs), or experimental designs, are often promoted as the strongest type of evaluation design for establishing causal evidence for a program’s impact, and in recent years several federal agencies (e.g., NIH, SAMHSA, Department of Education) have prioritized RTs in making funding decisions on proposals. Yet most evaluators recognize there are many situations for which RTs might not be the best choice for assessing impact, due to expense, implementation challenges, ethical considerations, or other reasons. At present, the use of RTs is fairly rare in Extension evaluations, but with the recent reorganization of Extension at the federal level and the increased emphasis on competitive funding and multi-agency collaborations, the reliance on experimental designs might grow. This presentation will examine the current use of RTs in Extension, the suitability and characteristic challenges of RTs for a variety of Extension settings, and the implications for organizational capacity building.

Cooperative Extension Programs’ Capacity to Produce Evidence of Effectiveness

Suzanne Le Menestrel, United States Department of Agriculture, slemenestrel@nifa.usda.gov

Funders of programs are increasingly looking for evidence that programs are effective in producing their intended outcomes. But not all programs are capable of producing the type of evidence needed to document their effectiveness to potential funders. This is particularly true for many widely replicated programs that have popular support but only modest evidence of their efficacy. Similarly, many small scale interventions that are shown to work in highly-controlled settings have limited potential to be replicated because of the significant resources needed to replicate the intervention with high levels of fidelity. In her presentation, Dr. Le Menestrel will discuss this conundrum in detail and offer her perspectives on achieving balance between these seemingly polar extremes.

The Role of Evidence in Building Mature Program Theory

Roger Rennekamp, Oregon State University, roger.rennekamp@oregonstate.edu

Program logic models are a graphic representation of a program’s underlying theory. The linkages between inputs, outputs, and various levels of outcomes are affirmed by empirical studies, evaluation projects, experience, and intuition. The confidence one has in these linkages depends largely on the type of evidence that individual needs to establish “truth” in their own minds. Some individuals are comfortable with the truths established from their own personal experience. Others need the results of empirical studies to be convinced of truth. To what degree is epistemological pluralism valued in building assuredness that program models are sound and replicable. Dr. Rennekamp will offer intriguing insights into how “truths” might be considered as a more relative phenomenon.

Session Title: Using an Interest-Driven Project to Teach Program Planning and Evaluation

Demonstration Session 129 to be held in CROCKETT B on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Teaching of Evaluation TIG

Presenter(s):

David Diehl, University of Florida, dcdiehl@ufl.edu

Abstract: This workshop provides an overview of an interest-driven approach to teaching undergraduate and graduate program planning and evaluation. The approach optimizes student learning by connecting the course project to a social policy interest of the student’s choosing. Using each student’s proposed social program as a foundation, the project focuses on the following components: 1) Situation Statement, 2) Key Informant Interview, 3) “What Works” (evidence-based programs), 4) Program Logic Model, and 5) Evaluation Plan. In a course where students sometimes struggle with the core content, the interest-driven project approach engages the students in the key issues related to program planning and evaluation. An overview of the approach, student samples, key challenges, and lessons learned will be presented. Discussion will focus on the ways in which the approach can be adapted for different audiences. Attendees will learn practical strategies for teaching program planning and evaluation that engage students and optimize learning.

Session Title: Networking and Getting Involved With the American Evaluation Association

Skill-Building Workshop 130 to be held in CROCKETT C on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Graduate Student and New Evaluator TIG

Presenter(s):

Nicole Cundiff, University of Alaska, Fairbanks, karim@siu.edu

Cady Berkel, Arizona State University, cady.berkel@asu.edu

Abstract: This session will help new evaluators make the best use of their time at the conference, by emphasizing the development of professional connections that are essential for successful evaluation careers. First, a theoretical model linking networking to career success will be presented. Then, we will provide information about networking at the AEA conference (including concurrent sessions, TIG meetings, social events, volunteering, and hospitality suites) and specific strategies for connecting with colleagues. For example, we will overview AEA and TIG structures, addressing how to get involved in leadership positions and what to expect. Further, we will explain the conference program and use examples from the audience to demonstrate how to search for relevant sessions. Participants will leave this skill-building workshop with a strategy for making connections with colleagues, supporting their career success.

Session Title: Informing Government Policy Through Evaluation: A Cross-site Evaluation of the Self-Employment Initiative (Start-Up USA)

Multipaper Session 131 to be held in CROCKETT D on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the Government Evaluation TIG

Chair(s):

Teserach Ketema, United States Department of Labor, teserachk@yahoo.com

Discussant(s):

Richard Horne, United States Department of Labor, horne.richard@dol.gov

Evaluation of Research Demonstration Programs: Self Employment

Presenter(s):

Richard Horne, United States Department of Labor, horne.richard@dol.gov

Abstract: The Office of Disability Employment Policy was created by the Federal Government to address the low employment rate of people with disabilities. The Office provides national leadership on employment policy to the Department of Labor and other Federal agencies and policy direction for serving those with disabilities. As part of its mission, ODEP awards grants and cooperative agreements to implement research demonstration programs. ODEP funded grants/programs consist of a variety of initiatives targeted at both adults and youth with disabilities of which Self Employment is one. Richard Horne, the senior Federal official has oversight responsibility for the evaluation of ODEP programs and initiatives. His presentation will provide background information on the mission of the Office of Disability Employment Policy (ODEP) and research programs, with special emphasis on the self Employment grants and the purposes of the evaluation. He will also be the discussant providing insight about what ODEP has learned from the evaluation thus far, identifying the key findings and their implication for policy.

Session Title: Evaluation and the Complexities of International Financial Assistance Programs

Multipaper Session 132 to be held in REPUBLIC A on Wednesday, Nov 10, 4:30 PM to 6:00 PM

Sponsored by the International and Cross-cultural Evaluation TIG

A Look Inside Development: What Do the Monitoring and Evaluation Framework Designs of Foreign-Funded Urban Development Programs in Metro Manilla, Philippines Reveal?

Presenter(s):

Romeo Santos, University of the Philippines, rbsantos1@up.edu.ph

Abstract: The study analyzed a number of urban development programs in Metro Manila, Philippines that are funded by overseas development assistance (ODA) from multilateral, donor or lending agencies, such as the World Bank (WB) and the Asian Development Bank (ADB). It looked at the monitoring and evaluation (M&E) framework components; the targeted outputs, outcomes and impacts, physical development objectives, and sustainability requirements to identify patterns that describe the nature of M&E designs of each program. Review and analysis of the selected foreign-funded programs based on official documents, evaluation reports, online publications, and many other public information materials were done. Resource persons’ interviews were used to validate information. The findings show patterns and features that reveal much about the evaluation designs, as well as on the evaluators or proponents involved in the M&E formulation of the programs. Keywords: donor or loan-led urban development programs, multilateral agencies, monitoring and evaluation (M&E) structure, results-framework

Does Aid Evaluation Work? Meta-evaluation of Aid Evaluation Quality

Presenter(s):

Ryoh Sasaki, International Development Center of Japan, sasaki.ryo@idcj.or.jp

Paul Clements, Western Michigan University, paul.clements@wmich.edu

Michael Scriven, Claremont Graduate University, mjscriv1@gmail.com

Liliana Rodriguez-Campos, University of South Florida, liliana@usf.edu

Abstract: This study describes the metaevaluation of aid evaluation reports generated in a single fiscal year (Fiscal Year 2004). Sample reports (n=102) were chosen by stratified random sampling from the pool of evaluation reports (N=1034) made by 51 donor agencies. The Key Aid Evaluation Checklist (KAEC) was developed based on the well-known Key Evaluation Checklist (KEC) (Scriven, 2006). A key conclusion of this metaevaluation is this: Quality of the current aid evaluation reports in general is regarded as “satisfactory” but far from “excellent,” although some are marked as high quality (i.e., they successfully examine a set of subevaluations and offer overall evaluative conclusions, such as a single grade), while others are of very poor quality (i.e., they do not determine overall or at least some dimensional evaluative conclusions but just present analytic conclusions or fact-findings). Based on these results, recommendations for improvement of the current quality are offered.

Evaluation of the Performance-based Concessionary Resources Allocation Systems of the Multilateral Development Banks and Funds

Presenter(s):

Kenneth Watson, International Monetary Fund, ken.watson@sympatico.ca

Abstract: Most of the multilateral development banks and Funds instituted “performance based” allocation systems between 2000 and 2010 to stop the horse trading that had previously characterized the allocation of cheap loans and grants to developing countries. The new systems were rules-based, with a formula that incorporated measures of country need and country performance. The tradeoffs between need and performance were expressed by the weights of the variables in the allocation formula. It was expected that concessionary funds allocated in this new way would be more effective in promoting economic growth and poverty reduction. However the distinction between “needs” and “performance” was not easy to make in practice, and the great diversity in size and character among developing countries has raised interesting issues in regard to resource allocation by formula. The author reports three evaluations of performance-based allocation systems that he led, their results and the methodological and practical issues they raised.

Improving Development Evaluation Product Quality in Africa: Experience From the African Development Bank

Presenter(s):

Foday Turay, African Development Bank, f.turay@afdb.org

James Edwin, African Development Bank, j.edwin@afdb.org

Mohamed Manai, African Development Bank, m.manai@afdb.org

Colin Kirk, African Development Bank, c.kirk@afdb.org

Abstract: The African development Bank has, over time, been effectively instituting changes in structure, rules and procedures in support of quality improvement of its evaluation products. These measures include a well-defined self- and independent evaluation function and structure and quality assurance mechanism. Notwithstanding the improvements in evaluation product quality as revealed by the primary evaluation customers, this paper argues for an enhanced customer-focused evaluation and assesses the key challenges in pursuing such an approach. These challenges, among others, comprises meeting customers’ expectations, which are most times implicit and inconsistent, and adapting and adopting approaches and methods for gathering and analyzing credible evidence, given the weak monitoring and evaluation culture and systems in the continent. Towards addressing some of these challenges, the paper also explores how a mixed methods’ approach using theory-based evaluation can be useful.

Doing Better at Doing Well by Doing Good: Evaluating Corporate Social Responsibility Projects in Developing Countries

Presenter(s):

Catherine Elkins, RTI International, celkins@gmail.com

Abstract: Standard assessments of Corporate Social Responsibility (CSR) focus on return on investment to the firm, its stakeholders, and/or the so-called triple bottom line (people, planet, and profit). In developing countries, CSR initiatives may stretch far beyond worker satisfaction and recycling, or philanthropy, to activities intended to build social and economic capacity, promote engagement in community concerns, or explicitly strive toward sustainable local development. We explore recent evolutions in the CSR arena toward interventions that resemble international development projects and examine specific attempts to use or adapt results monitoring and evaluation tools or methods to assess these initiatives. Notwithstanding CSR’s theoretical limits, the flexible hybrid approaches possible in the nontraditional CSR project arena offer rich potential to innovate and learn at accelerated rates, and thus to enhance the impact of this new type of intervention on targeted systems, institutions, policies, or beneficiaries.

Session Title: Evaluation of the Clinical and Translational Science Awards (CTSA) Programs: A Focus on Quality

Panel Session 133 to be held in REPUBLIC B on Wednesday, Nov 10, 4:30 PM to 6:00 PM