Search Results for All Sessions

Session Title: From a Distance: Conducting Successful Long-distance Evaluations

Panel Session 401 to be held in Avalon A on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Independent Consulting TIG and the Cluster, Multi-site and Multi-level Evaluation TIG

Chair(s):

Randi Nelson, Nelson Evaluation Services, rnelson.nert@gmail.com

Abstract: The panel members will discuss lessons learned and useful resources for planning and implementing long-distance evaluations, defined as evaluations in which the client and/or programs being evaluated are geographically distant from the evaluator's primary place of business. Although each panelist will base their remarks on experiences conducting specific evaluations, the discussion focuses on strategies and resources for overcoming global challenges of conducting long-distance evaluations. The discussion highlights issues related to evaluation planning, data collection, and communication with clients and stakeholders and applies to a wide variety of program contexts and evaluation approaches. The discussion will provide useful insights to evaluators, clients, and program staff who may be involved in long-distance evaluations. Panelists invite audience members to contribute to the discussion by sharing their own experiences, perspectives, and resources to advance the practice of conducting long-distance evaluations.

Planning Long-distance Evaluations

Randi Nelson, Nelson Evaluation Services, rnelson.nert@gmail.com

Independent evaluators typically initiate planning by reading requests for proposals or in client discussions. Although many planning issues apply to all evaluations, those conducted at a distance introduce additional issues. This discussion addresses distance evaluations at the earliest stage of planning, when-Ľ-Ľ determining if it makes sense to propose a distance evaluation. The discussion is organized around a series of questions that can be used to determine the feasibility of successfully conducting a distance evaluation on time and within budget. Questions address the client's attitude toward or experience with prior local and distance evaluations, their views on collaborative approaches to evaluation-Ľ-Ľ-especially as it relates to the capacity and willingness of program staff or clients to participate in data collection, the viability of Internet-based data collection methods such as online surveys or focus groups, and client expectations concerning frequency and type of communication with the evaluator.

Booklets, Telephone Focus Groups, and Training Videos: Getting the Data Collected From a Distance

Cheryl Kessler, Blue Scarf Consulting, cheryl@bluescarfconsulting.com

Three different clients, three separate projects, three different approaches to getting data collected from a distance. A six-week parenting workshop at a children's museum required data collection that was easily integrated into sessions to accommodate the limited time of each session and concerns with participants' literacy skills. A project with limited travel budget and eight communities that were distant from each other as well as from the evaluator required strong client-evaluator-participant communication and consideration of participants' relatively low technical abilities. A large, multi-site project required easily accessible training for staff with varying levels of experience collecting data and understanding of evaluation. Solving the specific data collection issues of each project required collaboration with clients, follow-up with participants, and vigilant oversight of the data collection to insure protocols were followed, explain why half the questions were unanswered, and know when the data promised the prior week would actually arrive.

Helping Build Commitment and Participation Among Reluctant Long-distance Clients

Denise Roseland, MetroPoint Research & Evaluation, metropointeval@gmail.com

Sometimes, the client doesn't choose you as an evaluator and may struggle with the unique challenges of the long-distance relationship. This is especially true in many multi-site evaluations, where the primary "client" is a funder but the work of the independent evaluator is with grantees or local project sites spread across a state or the country. This presentation will share a number of lessons learned (and a few promising practices from the literature) about using a variety of participatory approaches to evaluation to build client trust across the miles and to build capacity of project staff to effectively collaborate using technological tools.

Evaluation from a Distance: Conquering Communication and Data Collection Challenges

Kathleen Dowell, EvalSolutions, kathy@eval-solutions.com

Long distance evaluations - where the evaluator and the program being evaluated are separated geographically - pose unique problems for evaluators. Two of the major issues revolve around communication and data collection. This presentation will focus on how one independent evaluator worked in partnership with a client to plan and manage a long-distance, multi-site evaluation that presented multiple challenges to both the client and the evaluator. Challenges encountered included developing a mutually beneficial partnership with the client organization; maintaining adequate communication; and remotely planning, monitoring, and supporting data collection efforts. While technology exists to mitigate some of the challenges that arise during long-distance evaluations, problems still occur that require flexibility and creativity on the part of the evaluator. Strategies to overcome data collection and communication challenges will be discussed and lessons learned in successfully working on a long-distance evaluation will be presented.

Session Title: An Embedded Theory of Change: How One Organization Embraced Their Theory of Change as the Core Framework for Evaluation, Strategy and Decision-making

Panel Session 402 to be held in Avalon B on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Organizational Learning and Evaluation Capacity Building

Chair(s):

Jara Dean-Coffey, jdcPartnerships, jara@jdcpartnerships.com

Abstract: This session highlights the approach and experience of mid size national nonprofit, CompassPoint NonProfit Services as it integrates a theory of change framework in to its organizational culture. Instead of developing a multi-year strategic plan, leadership chose to engage board and staff in an organizational wide theory of change development process. This reflected a transition from a focus on strategy to outcomes. The theory of change is complemented with an annual plan that includes performance metrics to assess progress towards clearly identified targets in evaluation as well as other business dimensions. During this session, participants will learn about the choices, strategies, lessons learned, changes and challenges resulting from this transition to an outcome frame and how this is balanced with issues of organizational sustainability and relevance.

Why a Theory of Change?

Jeanne Bell, CompassPoint NonProfit Services, jeanneb@compasspoint.org

Adriana Rocha, CompassPoint NonProfit Services, adrianar@compasspoint.org

CompassPoint has served the nonprofit community for more than 30 years and is recognized as one of the premier capacity building entities in the nation. It works across all sectors designing and delivering an array of leadership and management services along a continuum from foundation or corporate sponsored customized initiatives to leading national research on the issues and trends of an increasingly diverse social sector workforce. Like most nonprofits, it could have stayed the course and engaged in the traditional 3-5 year cycle of strategic planning. But it chose a different path, a theory of change. CompassPoints' Executive Director and Practice Director will share thoughts about why they made this decision and their experience in developing and applying this framework in an increasingly complex environment with multiple stakeholders balanced with business sustainability.

Developing an Organizational Theory of Change

Jara Dean-Coffey, jdcPartnerships, jara@jdcpartnerships.com

In 2009, CompassPoint engaged jdcPartnerships to evaluate two initiatives, Fundraising Academy for Communities of Color and Next Generation Leaders of Color. As the work evolved, discussions began about the value of creating an overarching framework, a Theory of Change, to which individual efforts would align and ideally leverage to create a collective impact at the organizational level. In addition, it was recognized that the evaluative capacity of the organization needed to be strengthened. With two initiative evaluations underway and a commitment to evaluation capacity building, jdcPartnerships embarked on a journey with CompassPoint board and staff in a year-long process to develop a Theory of Change. This presenter will provide an overview of the process, the consideration in design and the challenges and success in doing this work in partnership.

Building Evaluation Capacity in a Non-profit Capacity Builder

Jill Casey, jdcPartnerships, jill@jdcpartnerships.com

Evaluation capacity building is not easy. It depends on so many things, from organizational financial resources to staff readiness and willingness to actively engage with a new practice and set of tools and methods. In much of the work with CompassPoint, jdcPartnerships has purposely created or made transparent evaluation design and administration considerations as well as inquiry tools and analysis frames that can subsequently be used by the client. It has also convened an Evaluation Work Group (EWG) that serves as the internal think tank and champion of evaluation within CompassPoint. This presenter will discuss jdcPartnerships' approach to this work, what has worked and remains a challenge and the ways in which it has strengthened jdcPartnerships evaluation skills and practices.

Embedding the Theory of Change

Marla Cornelius, CompassPoint NonProfit Services, marlac@compasspoint.org

As the Director of Next Generation Leaders of Color and a member of the Evaluation Work Group, this presenter has had different exposures to and experiences with jdcPartnerships in its role as evaluator and evaluation capacity building with CompassPoint. Insights on the changes that have occurred and challenges that remain from this unique vantage point will be informative to those contemplating similar endeavors.

Session Title: Valuing Systems Thinking in Designing an Evaluation of a Complex System Intervention

Panel Session 403 to be held in California A on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Systems in Evaluation TIG

Chair(s):

Beverly Parsons, InSites, bparsons@insites.org

Discussant(s):

Melissa Brodowski, Administration for Children and Families, melissa.brodowski@acf.hhs.gov

Abstract: This session focuses on two tools and the overall framework used in the cross site evaluation of a complex multi-site national research project. Each site has a unique intervention designed to build protective factors among caregivers to prevent child abuse and neglect of children 0-5 year old. Each site focus on multiple levels of systems (e.g., direct intervention with caregivers/children plus community or state policy levels). The evaluation was designed to generate new tools and methodologies for evaluating complex systems as well as to generate results about how differences in the valuing and expression of protective factors affect the outcomes for caregivers and children. The presentation includes reflections on (a) how our values regarding complex systems and methodology shape our evaluation and (b) how our evaluation approaches shape our capacity to understand how differences in values within a complex system influence the changes that occur in that system.

The Challenge of Valuing Both Evidence-based Practice and Practice-based Evidence

Charlyn Harper Browne, Center for the Study of Social Policy, charlyn.harperbrowne@cssp.org

The Quality Improvement Center for Early Childhood (QIC-EC) is unique in its efforts to use a complex systems orientation to the cross-site evaluation of the four 40-month research projects it funded in 2010. Each project uses an experimental or quasi-experimental design. The QIC-EC, with the encouragement and support of its federal program officer, selected research sites which each had designed an intervention that focuses on at least two levels of the social ecology (that is, operates at the caregiver/child level and also at community, program services, social norms, and/or policy levels). This presentation addresses the QIC-EC's overall research question and how a common set of outcome measures of three desired results- optimal child development, increased family strengths, and decreased likelihood of child maltreatment-were developed/selected. The presentation addresses the design challenges of valuing both evidence-based practice and practice-based evidence in complex systems.

Cross-Site Evaluation Design for Interventions Addressing Protective Factors to Prevent Child Maltreatment

Beverly Parsons, InSites, bparsons@insites.org

This presentation describes the initial theory of change for the cross-site evaluation design; the analytical framework for the quantitative data gathered about the intervention and the outcomes for caregivers and children; and the strategy for bringing together the quantitative and qualitative data gathered through data collection tools described in later presentations. The change theory addresses the patterns of change over time across multiple levels of the social ecology as people shift their attention from risk to protective factors as their primary focus. It involves reaching a tipping point and moving to a new system dynamic of a sustainable adaptive balance across levels of the social ecology. A key feature of the change theory and thus the cross site evaluation is the role of partnerships in changing complex systems. Understanding the value of partnerships in complex systems change is emphasized.

Instruments to Assess Support for Building Protective Factors to Prevent Child Maltreatment

Patricia Jessup, InSites, pjessup@insites.org

This presentation describes the development and use of two instruments to assess the relationship of partnerships' interventions to support the building of protective factors at multiple system levels to prevent child maltreatment. Results of the instruments' initial use in sites visits will be available for discussion at the presentation. Each of these instruments provides data that are synthesized and used in multiple ways-to develop a quantitative measure that can be used in the quantitative analysis; to gather qualitative data to identify examples of how the partnerships support the building of protective factors; and to contribute to a systems-oriented analysis in which we look at patterns of action and system structures involved in building protective factors. Our valuing of mixed methods is seen in the development of tools that integrate quantitative and qualitative methods, provide data for both types of analyses, and break down the artificial distinction between these methods.

The Cross-site Evaluation Site Visit: A Moment in Time to Understand the Unfolding of Change Over Time

Marah Moore, i2i Institute, marah@i2i-institute.com

Conducting site visits for a cross-site evaluation is fairly straight-forward-or is it? This presentation will explore the design and implementation of site visits when viewed through a complexity lens. Issues addressed will include: using site visits to surface patterns of change across multiple levels of the social ecology; acknowledging and leveraging the site visit as a point of influence in the system; and, following up on the previous presentation, the role of the site visit in a mixed-methods approach. The first of three yearly site visits will occur prior to this AEA presentation. Of particular interest will be how methodological values shape the site visit approach, and how differences in values across stakeholders are given evaluative "voice" through the site visit.

Session Title: Valuing Performance Measurement and Program Evaluation: Current Practice in the Federal Government

Panel Session 405 to be held in California C on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Government Evaluation TIG

Chair(s):

Kathryn Newcomer, George Washington University, kathryn.newcomer@gmail.com

Valuing Performance Measurement and Program Evaluation: Current Practice in the Federal Government - Analysis and Use of Performance Information

Kathryn Newcomer, George Washington University, kathryn.newcomer@gmail.com

The view that the federal government should operate in the most efficient and effective manner is shared by members of the government's executive and legislative branches, civil servants, the general public and industry alike. This view has resulted in a myriad of performance management and improvement initiatives designed to promote the use of performance measurement and program evaluation across the federal government. However, one crucial question remains: how has the use of these tools played out across the federal government? Kathryn Newcomer will present preliminary findings regarding: the procedures federal agencies have in place to analyze and use performance information; and the use of performance information by senior managers to inform decisions. These findings provide a glimpse into the current practices of a limited number of federal agencies and help illuminate the successes and challenges of implementing government-wide performance management and performance improvement initiatives.

Valuing Performance Measurement and Program Evaluation: Current Practice in the Federal Government - Examining the Effects of the Budget and the President's Performance Improvement Initiatives

Rick Kowalewski, United States Department of Transportation, rick.kowalewski@dot.gov

The current Administration, like others before it have adopted performance improvement initiatives designed to improve the efficient and effective operation of the federal government. While the calls for a lean, transparent and accountable government are not new, concerns about an increasing deficit and the elimination of duplicate government programs have raised questions regarding the extent these newly introduced initiatives can be used to better manage results government-wide. In support of the topic of current performance measurement and program evaluation practices in the federal government, Rick Kowalewski will present findings regarding the role and progress of the Performance Improvement Officer in moving forward the President's performance improvement initiative and performance improvement efforts. He'll also discuss the effects of current budget pressures and the current Administration's efforts to facilitate federal agencies' use of performance information through the development of High Priority Performance Goals and the Government Performance and Results Modernization Act.

Valuing Performance Measurement and Program Evaluation: Current Practice in the Federal Government - The Priority and Use of Program Evaluation

Yvonne Watson, United States Environmental Protection Agency, watson.yvonne@epa.gov

Program evaluation is an important component of a complete system of performance management, complementing and using performance measurement information to shed light on how well programs or strategies are working and why. Information obtained through better performance measurement and quality program evaluations provide insights that enable us to understand and replicate successes, and continuously improve programs. As part of the session that highlights current performance measurement and program evaluation practices in the federal government, Yvonne Watson will present findings regarding the current priority given to program evaluation in a select number of federal agencies. In addition, she will discuss the extent to which federal agencies value performance measurement and program evaluation as well as the extent to which value is synonymous with use.

Session Title: ACT I: Exploring the Complexities of Evaluating Infrastructure Development in Cluster, Multi-site, or Multi-level Initiatives

Panel Session 406 to be held in California D on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Cluster, Multi-site and Multi-level Evaluation TIG

Chair(s):

Rene Lavinghouze, Centers for Disease Control and Prevention, shl3@cdc.gov

Discussant(s):

Rene Lavinghouze, Centers for Disease Control and Prevention, shl3@cdc.gov

Abstract: ACT I: As evaluators and researchers of public health systems and policies we know that systems, environmental, and policy interventions are dynamic, complex, and unpredictable. One task is, through evaluation, to understand the stages of growth and development of a given initiative. Then and only then can optimal alignment of focus, people and resources for achievement of program goals be actualized. The task is complex and daunting because the line from infrastructure elements to distal outcomes is circuitous at best and does not lend itself to simple evaluation solutions. The complexity of the evaluation is further increased when programs involve multiple-sites and/or multiple-levels of implementation. A mixture of quantitative, qualitative, and out-of-the-box methods are required. This panel will discuss the complexities of evaluating infrastructure and its link to outcomes across sites and levels of program implementation as well as the relevance of evaluating infrastructure to the overall logic model.

The Difficulties and Complexities of Evaluating "Inputs:" The Forgotten Box of the Logic Model

Rene Lavinghouze, Centers for Disease Control and Prevention, shl3@cdc.gov

Systems, environmental, and policy interventions are dynamic, complex, and unpredictable. Many initiatives in multi-site or multi-level evaluations have similar outcome measures but often dissimilar or unpredictable inputs. As the foundation to achieve outcomes, the "inputs" or infrastructure of an initiative are often overlooked in the evaluation plan or logic model. The challenge is to consider how to evaluate the effect of typical inputs such as partnerships, leveraging of resources, or leadership roles as direct or indirect links to the achievement of outcomes. Omitting these aspects in the evaluation plan, initiatives can fail to realize the importance of core infrastructure elements for intervention development and sustainability. This is true for all types of multi-site, multi-level initiatives. Mixed methods and "out-of-the-box" methods are often required to evaluate the complex nature of infrastructure elements. This presentation will focus on the complexities of evaluating infrastructure elements and making necessary links between inputs and outcomes.

Evaluating the Infrastructure of Success

Martha Engstrom, Centers for Disease Control and Prevention, mengstrom@cdc.gov

Rene Lavinghouze, Centers for Disease Control and Prevention, shl3@cdc.gov

Ray Maietta, ResearchTalk, ray@researchtalk.com

Public health programs often have well defined outcome measures such as the Key Outcome Indicators for tobacco control and prevention programs (www.cdc.gov/tobacco). However, these multi-site initiatives often lack the ability to delineate the exact nature of the inputs required to achieve those outcomes and program sustainability. This is particularly complex when considering multi-site/multi-level programs where contexts for infrastructure and implementation differ. Using the success case method, we examined successful tobacco control programs to identify the core infrastructure elements that were required to achieve program outcomes and sustainability. This presentation will focus on the evaluation methods employed and the need to carefully evaluate the inputs section of the logic model for program success and sustainability.

Moving Quality to the Community Level: The Challenges and Possibilities of Taking Improvement Efforts Outside the Walls of Individual Health Care Organizations

Dennis Scanlon, Pennsylvania State University, dpscanlon@psu.edu

Jeff Alexander, University of Michigan, jalexand@umich.edu

Jeff Beich, Pennsylvania State University, jjb235@psu.edu

Laura Bodenschatz, Pennsylvania State University, ljb191@psu.edu

Jillian Harvey, Pennsylvania State University, jbh221@psu.edu

Many health care leaders are promoting improving health care at the community level (AHRQ; Berwick; Lavizzo-Mourey; IOM). The motivation for this approach is the belief that combining the efforts and resources of health plans, purchasers, providers, public and private agencies, and consumers into a unified initiative will result in better and more sustainable outcomes than the independent activities of the participants (Beich/Scanlon). Using data from the Aligning Forces for Quality (AF4Q) initiative, this presentation will focus on using qualitative methods to evaluate how communities are approaching infrastructure development and to ultimately develop a framework for community quality improvement infrastructure. The presentation will also discuss the process and challenges associated with using large amounts of qualitative data (interviews and secondary documents) collected across multiple sites, stakeholders, and time periods.

Infrastructure Development for the Continued Provision of Evaluation Technical Assistance Through the Establishment of a National Network

Karen Debrot, Centers for Disease Control and Prevention, kdebrot@cdc.gov

Infrastructure development is important for organizations that want to survive, especially given potential future funding limitations. Having a strong infrastructure is akin to a tree having a healthy root system, allowing it to withstand droughts and heavy storms alike. Identifying areas for infrastructure development and devoting resources to this development will likely pay long-term dividends for organizations that can plan ahead. The Office on Smoking and Health (OSH), Centers for Disease Control and Prevention (CDC), has begun to explore expansion of its evaluation technical assistance infrastructure through the development of a national level evaluation network. This strategy will be reviewed and discussed in the context of ensuring that CDC's current level of evaluation technical assistance can continue to be provided to states and local tobacco control programs in the future.

Session Title: Program Theory and Theory-driven Evaluation TIG Business Meeting and Think Tank: Theories of Value and the Value of Theories: A Theory-based Evaluation Perspective

Business Meeting Session 407 to be held in Pacific A on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Program Theory and Theory-driven Evaluation TIG and the Presidential Strand

TIG Leader(s):

John Gargani, Gargani + Company Inc, john@gcoinc.com

Katrina Bledsoe, Education Development Center Inc, katrina.bledsoe@gmail.com

Chair(s):

Katrina Bledsoe, Education Development Center Inc, katrina.bledsoe@gmail.com

Presenter(s):

John Gargani, Gargani + Company Inc, john@gcoinc.com

Discussant(s):

John Gargani, Gargani + Company Inc, john@gcoinc.com

Stewart Donaldson, Claremont Graduate University, stewart.donaldson@cgu.edu

David Fetterman, Fetterman & Associates, fettermanassociates@gmail.com

Abstract: In this think tank presentation, we will debate competing answers to fundamental questions underlying theory-based/theory-driven evaluation. We start from the premise that program theories (frequently represented as logic models) reflect the values of those who construct them. Given this, are there theories of values that can inform evaluation practitioners as they work with stakeholders to construct program theories? Help stakeholders decide whose values to include? Guide us as we navigate conflicting values? Theory-based evaluators look to theories when wrestling with difficult questions such as these. That, of course, assumes that theories add value to our uncertain pursuits. Do they? Or do they merely reflect the values of social scientists who have been indoctrinated to accept an outdated, overly academic world view? Through brief presentations and lively discussion, we will hash out these and other questions of how considering values can improve evaluation practice.

Roundtable: Valuing Voices: Conversations With Senior Evaluators

Roundtable Presentation 408 to be held in Pacific B on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Multiethnic Issues in Evaluation TIG

Presenter(s):

Tamara Bertrand Jones, Florida State University, tbertrand@fsu.edu

Ceasar Jackson, National Science Foundation, crjackso@nsf.gov

Rodney Hopson, Duquesne University, hopson@duq.edu

Michelle Jay, University of South Carolina, mjay@sc.edu

Katherine Tibbetts, Kamehameha Schools, katibbet@ksbe.edu

Pamela Frazier-Anderson, Frazier-Anderson Research & Evaluation, pfa@frazier-anderson.com

Henry Frierson, University of Florida, hfrierson@ufl.edu

Kevin E Favor, Lincoln University, kfavor@lincoln.edu

Tamara Bertrand Jones, Florida State University, tbertrand@fsu.edu

Elmima Johnson, National Science Foundation, ejohnson@nsf.gov

Stafford Hood, University of Illinois at Urbana-Champaign, slhood@illinois.edu

Joan LaFrance, Mekinak Consultin, lafrancejl@gmail.com

Abstract: In many cultures community elders are respected and revered for their wisdom and years of personal and/or professional experiences, as well as the guidance provided to younger generations. As the field of evaluation continues to grow and evaluation training programs become available to increasingly diverse populations, the relationships with and experiences of senior evaluators become a valuable resource. It is rare that senior evaluators, considered experts in their respective fields, are readily accessible to those who would like to learn from them. Participants in Bertrand (2006) identified participation in professional association meetings/conferences as a major influence on the development of professional relationships. This roundtable session brings national and/or international senior evaluation leaders, novice, and mid-career evaluators to engage in evaluation discourse. The expectation is that these conversations will assist participants with improving the quality of their own evaluation activities, as well as lay the foundation for lasting professional relationships.

Session Title: Master Teacher Series: Using Equivalence Tests to Prove That Groups Don't Differ: How to Generate "Evidence of Absence" Rather Than "Absence of Evidence"

Demonstration Session 409 to be held in Pacific C on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Quantitative Methods: Theory and Design TIG

Presenter(s):

Steven Pierce, Michigan State University, pierces1@msu.edu

Abstract: This intermediate session will demonstrate the use of equivalence tests, which are statistical methods designed to use the evidence in the data to explicitly prove that two groups are actually equal on some outcome measure. This departs from the goal of classical statistical methods (e.g., t-tests, chi-square tests, etc.), which aim to use the data to prove that two groups actually differ from one another. The session will explain what kinds of evaluation questions equivalence tests can answer, how these tests work, and why they provide more credible evidence of equivalence than offered by simply finding a non-significant effect with a classical method. The session will offer practical advice on how and when to use equivalence tests in your evaluation work. Because equivalence tests are rarely covered in basic graduate level statistics courses, this session aims to provide the audience with a user-friendly introduction to new statistical methods.

Session Title: 25 Years of Values: American Evaluation Association (AEA) Values: Then, Now, and Tomorrow

Panel Session 410 to be held in Pacific D on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the AEA Conference Committee

Chair(s):

Molly Engle, Oregon State University, molly.engle@oregonstate.edu

Discussant(s):

Melvin Mark, Pennsylvania State University, m5m@psu.edu

Abstract: In 1985, there was no AEA; no Guiding Principles; an early edition of Program Evaluation Standards existed (1980). AEA's guiding principles were adopted in 1992 and the AEA's value statement was not adopted until 2005. Documenting AEA's values development, the expectation arising from those nascent values and their subsequent evolution is important. The values statement is the lens through which we view the work of AEA; the Guiding Principles provides the same for evaluators and the Standards for individual evaluations. As evaluators, the AEA's values statement, the Guiding Principles and the Program Evaluation Standards are used as evaluations are planned, implemented, and reported. Now that the breadth of evaluation encompasses voices seldom heard in 1986, we must take steps to assure that these values continue to evolve to fit the times, places, and stakeholders. Making core values explicit and providing clear guidance ensures AEA continuing success and enduring survival.

AEA's Contributions to Development and Application of the Program Evaluation Standards

Daniel Stufflebeam, Western Michigan University, dlstfbm@aol.com

Dan Stufflebeam was the founding chair of the Joint Committee on Standards for Educational Evaluation in 1975 and principal author of both the 1981 edition of the Program Evaluation Standards and the 1988 Personnel Evaluation Standards. For eight years he was AEA's representative on the U.S. Comptroller General's Advisory Council on Government Auditing Standards. He contributed critical comments on the draft 2011 edition of the Joint Committee's Program Evaluation Standards, especially regarding the need for a fifth category labeled EVALUATION ACCOUNTABILITY. Recently he developed a new metaevaluation checklist keyed to the 2011 Program Evaluation Standards. He will summarize the early development of the Joint Committee's standards and stress the importance of their applications, especially in metaevaluations.

AEA and the 2011 Edition of the Program Evaluation Standards

Donald Yarbrough, University of Iowa, d-yarbrough@uiowa.edu

Don Yarbrough represented AEA in the revision process for the 3rd edition of the Program Evaluation Standards (Sage 2011). Beginning in 2004, members of the Joint Committee on Standards representing 16 North American professional organizations, a special task force of four writers, and more than 400 volunteers revised the Standards. Under the influence of new reviewer cohorts, hearing participants, field testers, and Joint Committee members, the 3rd edition infused culture and cultural competence considerations throughout the Standards and example cases. This edition of the Standards explicitly addresses the Guiding Principles and the International Declaration of Human Rights as key documents informing the topics of values and valuing. Because of the highly inclusive process used to guide reviews and revisions of the standards, the future will bring new focuses and emphases in response to the contributions of new cohorts of stakeholders and the scholarship and practices that inform them.

Developing the AEA Guiding Principles

Maryann Scheirer, Scheirer Consulting, maryann@scheirerconsulting.com

Mary Ann Scheirer was on the Task Force that developed the Guiding Principles, growing from her work on AEA's Board of Directors at the time. A major focus of this effort was to contribute to the professionalization of evaluation, by developing a statement of ethical principles that all members could endorse. She will discuss the reasons and processes behind the initial development of the Guiding Principles, which embody the key ethical values underlying AEA.

The AEA Guiding Principles Today and Tomorrow

Rebecca Woodland, University of Massachusetts, Amherst, rebecca.woodland@educ.umass.edu

Rebecca Woodland is the co-chair of the Guiding Principles review group for 2011. In that capacity, she will address the Guiding Principles as a foundational piece for AEA and the need to keep them current and relevant. She will also discuss possible directions for the Guiding Principles for the future.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Identifying Key Indicators of Nonprofit Advocacy Capacity: Can we Agree?

Roundtable Presentation 411 to be held in Conference Room 1 on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Advocacy and Policy Change TIG

Presenter(s):

Susan Hoechstetter, Alliance for Justice, sue@afj.org

Jared Raynor, TCC Group, jraynor@tccgrp.co

Abstract: In the dynamic field of advocacy evaluation, many practitioners, evaluators, and funders agree that assessing an organization's advocacy capacity is an important component of advocacy evaluations. As more nonprofits and funders seek to aggregate evaluative information and learn from each others' experiences, developing a basic set of key indicators for use across the sector becomes a worthy evaluative objective. The presenters both have advocacy capacity frameworks they have widely used in the last few years and look forward to discussing with AEA colleagues what they have learned, what others have learned and see if there is an emerging consensus on key advocacy capacities. Questions to be discussed include: What are the most basic capacities necessary for successful advocacy work? Can they be adapted to varying types of nonprofit organizations? How can aggregated capacity information be presented in a way that best helps organizations plan and evaluate capacity progress?

Roundtable Rotation II: Evaluating Movement Building and Social Change Work: When Funder and Grantee Outcomes Differ

Roundtable Presentation 411 to be held in Conference Room 1 on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Advocacy and Policy Change TIG

Presenter(s):

Catherine Borgman-Arboleda, Indendent Evaluation Consultant, cborgman.arboleda@gmail.com

Rachel Kulick, Action Evaluation Collaborative, rakulick@gmail.com

Julie Poncelet, Action Evaluation Collaborative, julie_poncelet@yahoo.com

Abstract: Grantmakers seeking to develop strategies to support broader systemic change face the challenge of how to articulate program outcomes that reflect the needs and aims of social change work happening on the ground. Tensions often arise in practice as funders look primarily to policy or other product-oriented outcomes that differ from what systemic efforts seek to achieve. Evaluators are in a unique position to advocate for a more balanced perspective to these dilemmas; however, currently there are few accepted measures for evaluations of community-driven outcomes especially when they deviate from the expectations of grantmakers. Drawing from previous evaluations conducted with foundations, we will facilitate a discussion to explore challenges and strategies associated with how to develop and evaluate outcome indicators that legitimate, integrate and reflect the values, interests and aims of both funders and social change efforts in ways that better support the longer-term goals of social change work.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Fibs, Funders, Power and Politics: The Dilemma of Unflattering Evaluation Results

Roundtable Presentation 412 to be held in Conference Room 12 on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Non-profit and Foundations Evaluation TIG

Presenter(s):

Kelly Firesheets, The Health Foundation of Greater Cincinnati, kfiresheets@healthfoundation.org

Deborah Cohen, Community Mental Health Center Inc, debbie.cohen@cmhcinc.org

Abstract: When it comes to program evaluation, the data don't always go your way. In this roundtable discussion, two Directors of Evaluation - one in a community mental health center and one in a regional foundation - will share their experiences with reporting and using "unflattering" evaluation results. Together, the two will make suggestions about how to discuss evaluation results with funders, clients, Board Members, and other stakeholders when those results don't meet their expectations. Roundtable participants will then discuss their own experiences with unflattering data, the dilemmas that arise in using and reporting unexpected evaluation results, and possible solutions to those dilemmas. Participants will leave the session with new ideas about how to use unexpected outcomes to strengthen their evaluation work.

Roundtable Rotation II: Evaluating Program Quality in Nonprofits and Foundations: Systemic Interactions Between Program Practices, Organizational Values and Behaviors, and Social Impact

Roundtable Presentation 412 to be held in Conference Room 12 on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Non-profit and Foundations Evaluation TIG

Presenter(s):

Ritu Shroff, Independent Consultant, ritushroff2003@yahoo.co.uk

Martin Reynolds, Open University, m.d.reynolds@open.ac.uk

Abstract: Based on experience with nonprofit organizations in Africa and Asia, we are developing a systemic framework for evaluating program quality that is comprised of three prongsâ€”program practices, organizational behavior and ultimate impact. The first prong describes those program practices (interactions between program staff and partners and communities with whom they work) that are seen in quality programs. This framework recognizes that such practices are nested within the environment where implementing staff operate. Thus, we also describe agency-wide behaviors, values, and attitudes necessary to create incentives under which quality programming becomes the norm. Finally, an essential reason to think about program quality is based on the assumption of correlation between program quality and impact. If staff that understand program quality exhibit certain program practices, do those practices result in greater or deeper social impact? This framework describes likely effects of these practices that are conducive to positive impact in social change/development interventions.

Session Title: Examining Our Own Cross-Cultural Competences & Perspectives in International Contexts

Think Tank Session 413 to be held in Conference Room 13 on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the International and Cross-cultural Evaluation TIG

Presenter(s):

Mary Crave, National 4-H Council, mcrave@fourhcouncil.edu

Discussant(s):

Kerry Zaleski, University of Wisconsin, Extension, kerry.zaleski@ces.uwex.edu

Hubert Paulmer, Harry Cummings and Associates Inc, hubertpaulmer@gmail.com

Liudmila Mikhailova, CRDF Global, mikhailova@msn.com

Abstract: Evaluators working in contexts different from their own cultures are continually challenged to explore not only their practices, but also their beliefs and assumptions about evaluation. This session is not about how to understand different cultures but to start a discourse on how evaluators' own beliefs about evaluation and the programs they are evaluating influence evaluation design, execution, and communication in international contexts. The speakers will illustrate examples from their work in several continents and suggest critical ways in which the cultural perspectives of both the evaluator and of the respondent influenced the evaluation. Small group discussions will lead participants to explore beliefs, assumptions, and practices from one of three stakeholder perspectives: the program sponsor; the evaluator; and, evaluation respondents. Participants will discuss together some specific challenges and cultural beliefs in international evaluation such as methodology, indicators of success, evaluation purpose and use, to improve their own international evaluation practice.

Session Title: Determining Value with Evaluation Stakeholders

Skill-Building Workshop 414 to be held in Conference Room 14 on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG

Presenter(s):

Pauline Dickinson, Massey University, p.m.dickinson@massey.ac.nz

Jeffery Adams, ,

Abstract: This interactive skills based workshop will enable participants to engage in a participatory process of determining evaluation criteria and standards with key stakeholders - in other words - determining value. Criteria are the aspects of an evaluation that define whether it is good or bad and whether or not it is valuable or not valuable. It is important to be clear when planning an evaluation about the evaluation criteria to be used as it helps to ensure that relevant data is collected, that stakeholders know how quality and success will be determined and provides clarity about what you need to get right. Participants will practice developing evaluation criteria for determining the value of an object and will set standards for what makes the object 'good'. We will then use a model programme logic from which participants will develop evaluation criteria and standards for an intervention and an outcome. Participants will also be asked to consider the key sources of data and data collection methods that could be used to ensure data relevant to the criteria is collected. Participants will be provided with a resource and templates to use in their future evaluation planning.

Session Title: Mirror, Mirror on the Wall: Strategies to Enhance Evaluation Use and Influence Across Stakeholders

Multipaper Session 415 to be held in Avila A on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Evaluation Use TIG and the Government Evaluation TIG

Chair(s):

Susan Tucker, Evaluation & Development Associates LLC, sutucker1@mac.com

Discussant(s):

Susan Tucker, Evaluation & Development Associates LLC, sutucker1@mac.com

Applying Social Media and On-Line Engagement for Greater Evaluation Use and Influence at The World Bank's Independent Evaluation Group

Presenter(s):

Bahar Salimova, The World Bank, bsalimova@worldbank.org

Nik Harvey, The World Bank, nharvey@worldbank.org

Alex McKenzie, The World Bank, amckenzie@worldbank.org

Abstract: This paper will takes stock of the systematic application of social media and on-line engagement approaches during the past year by the World Bank's Independent Evaluation Group (IEG), for enhancing communication and knowledge-sharing practices that seek greater use of findings, recommendations, and lessons from its evaluations. The past twelve months have been a period of learning and experimentation. Not only are social media channels and engagement approaches fairly new, they also not often seen applied in support of the evaluation discipline, or in a more specialized business context (as opposed to consumer markets). The paper will reflect on what worked well, and what didn't work as expected. The analysis will be based on data collected during recent engagements.

Differences in Stakeholder and Evaluators Views of Evaluation Use and Influence

Presenter(s):

Andy Thompson, Carleton University, arocznik@connect.carleton.ca

Shevaun Nadin, Carleton University, snadin@connect.carleton.ca

Bernadette Campbell, Carleton University, bernadette_campbell@carleton.ca

Abstract: Evaluators want to the see the results of their work influence decisions about programs and policies; however, too often the knowledge generated by evaluations has little influence on the day-to-day operations of the program. Although efforts to understand reasons for lack of use dominate the evaluation literature, these reports are mainly based on evaluator impressions of stakeholder behaviour. Missing from this literature is research that documents stakeholders' own accounts of the reasons for use and non-use of evaluations. To address this gap in the literature interviews were conducted with both evaluators and program officers of National Science Foundation (NSF) sponsored programs. Comparison of evaluator and stakeholder responses revealed similarities as well as differences in their assessments of the reasons evaluation findings were used or not used. Implications for practice and future research are also discussed.

Using Evaluation During Program Implementation: The Data-to-Action Framework

Presenter(s):

Ronda Zakocs, Independent Consultant, rzakocs@bu.edu

Jessica Hill, Centers for Disease Control and Prevention, hss9@cdc.gov

Pamela Brown, Centers for Disease Control and Prevention, pbrown8@cdc.gov

Abstract: Although utility is one standard of high quality evaluations, the challenge remains how evaluators can generate actionable data useful to stakeholders. The Data-to-Action Framework aims to produce relevant information during program implementation that enables decision-makers to: gain insights into the implementation process; make program improvements; and communicate progress with stakeholders. The Framework was developed and pilot-tested with DELTA PREP, a three-year national initiative seeking to build the capacity of 19 statewide coalitions to prevent domestic violence. The Framework outlines a step-by-step process collaboratively implemented among evaluators and program decision-makers to identify implementation questions and data needs, efficiently collect data, quickly generate data synthesis memos, discuss implications, make usage decisions, and take action. The presentation will describe how DELTA PREP implemented the Framework; demonstrate various ways decision-makers used data from 18 synthesis memos generated over a 2 year period; and share the benefits and challenges of using the Framework.

Dueling Federal Funder Preferences and Program Evaluation Needs: Challenges with Conflicting Interests and Evaluator Roles

Presenter(s):

Holly Downs, University of North Carolina, Greensboro, hadowns@uncg.edu

Abstract: A growing tension exists between federal funder preferences for an 'external' evaluator and the evaluation needs of a program. This paper will explore this dilemma encountered when a federally-funded undergraduate science and mathematics program at a large university shifted from an external to an internal evaluation team (i.e., a team at the university but outside the program departments). While the funder encouraged an external evaluator in the RFP, a shift by the program coordinators, who wanted better onsite data collection to improve use and Institutional Review Board coordination, trumped this funder preference post-award. These dueling dynamics can cause a chasm between what is valued and encouraged by funding agencies and the needs of the actual program. The goal of this paper is twofold: to discuss the impact of federal funders' preferences of external evaluators on this program and to consider the implications of these issues on the field of evaluation.

Session Title: Using Visual Data to Engage Stakeholders in a Community Assessment of a Placed-based Initiative

Multipaper Session 416 to be held in Avila B on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Data Visualization and Reporting TIG

Chair(s):

Joelle Greene, Harder+Company Community Research, jgreene@harderco.com

Discussant(s):

Lee Werbel, First 5 Los Angeles, lwerbel@first5la.org

Abstract: Evaluations of comprehensive place-based initiatives require evaluators to engage multiple stakeholders in the evaluative process in meaningful ways. The methods of engagement utilized in a particular project reflect values in two important ways: First, the way in which evaluators value stakeholders as informants and data sources and second, how evaluators value the very process of stakeholder engagement itself. This presentation focuses on the use of data visualization techniques to engage stakeholders in meaningful ways in a community assessment process. Each presenter will highlight a method of visual data presentation/collection (participatory photography, video, story maps, GIS mapping and Social Network Analysis) and will illustrate how it was utilized to engage particular stakeholder groups. Presentations include illustrations of the work produced, an analysis of how the visual method influenced the nature of the data garnered and lessons learned from implementing these approaches in a community-based setting.

Participatory Photography and Video as a Tool for Engaging Community Members in Community Assessment

Loraine Park, Harder+Company Community Research, lpark@harderco.com

Participatory photography is an approach that emphasizes the use of digital media to provide community members with the opportunity to represent themselves directly (in contrast to researchers or evaluators representing community views indirectly) and to create tools for advocacy and communication as a catalyst for social change. This presentation will highlight how participatory photography was used to involve stakeholders in a community assessment process. This will include examples of products produced and lessons learned about implementing participatory photography in the context of a community assessment. The unique data provided by the approach will be contrasted to other data sources to illustrate the utility of the approach to providing alternate community perspectives to the evaluative context.

Story Maps as a Tool for Community Engagement and Public Involvement

Carolyn Verheyen, MIG Inc, cverheyen@migcom.com

Story Maps represent a relatively new way to report community assessment findings in an interactive visual format. This presentation will highlight the use of Story Maps as part of the web-based dissemination of community assessment findings. Story maps utilize a combination of participatory photography and narrative and/or video embedded in an interactive map as part of a website. The presentation will highlight the process of helping communities design the Story Map and discuss the impact of the approach on the community engagement process on the dissemination of findings.

Using Geographical Information System (GIS) Maps to Engage Focus Groups in Asset Mapping

Eric Wat, Special Services for Groups, ewat@ssgmain.org

Asset Mapping is a common activity in most community assessment processes. GIS mapping technology provides evaluators with an opportunity to create geographic asset maps that community members can interact with during traditional asset mapping activities. This presentation will highlight the use of GIS maps of existing community boundaries and resources to conduct asset mapping activities as part of a larger community assessment effort. Community members and organizations were asked to respond to maps of resources created by evaluators based on publically available information as well as maps created by other community members in earlier asset mapping sessions. The discussion will include samples of products produced by both evaluators and community members, the utility of the approach and lessons learned.

Social Network Analysis as a Tool for Engaging Communities in the Assessment of Inter-organizational Collaboration

Joelle Greene, Harder+Company Community Research, jgreene@harderco.com

The success of place-based initiatives is largely contingent upon the ability of communities, leaders and organizations to collaborate. From the assessment of needs, to resource allocations and coordination of services, organizations are required to work towards mutual goals and outcomes. In this community assessment an organizational survey was conducted, in part, to help determine current levels of collaboration among organizations and to help the community partnership envision and set future goals related to the nature and degree of collaboration in their community. In the presentation we will provide examples of Social Network Analysis (SNA) maps used to interact with the community partnership, highlight how the data was used in the planning process, and how SNA will be used longitudinally to map changes in levels of collaboration over time in a place-based comprehensive community initiative.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Using Self-report to Measure Implementation Fidelity

Roundtable Presentation 417 to be held in Balboa A on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Presenter(s):

Kelli Millwood, Pearson, kelli.millwood@pearson.com

Kate Clavijo, Pearson, kateclavijo@gmail.com

Abstract: Self-report measures have been heavily used in many program evaluations to assess fidelity of implementation. In this rountable session we will discuss a range of strategies and questions from online surveys designed to measure implementation fidelity of a professional development initiative. These data are an essential step to ensure the professional development is being used efficiently and effectively. But the big question, is how valid is the self-report data? We are interested in facilitating discussion about how to best measure implementation fidelity of teacher professional development, without directly observing classroom practice. We will discuss lessons learned from our experiences and soliciting suggestions from other evaluators. As more and more evaluation data is being collected on-line, what techniques have others found of value to accurately measure practice from participant self-reports?

Roundtable Rotation II: How Does Our Measurement Measure Up?

Roundtable Presentation 417 to be held in Balboa A on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Presenter(s):

Elizabeth Whipple, Research Works Inc, ewhipple@researchworks.org

Jeffrey Wasbes, Research Works Inc, jwasbes@researchworks.org

Abstract: A popular professional development model in education has school personnel attend off-site professional development consisting of a combination of subject matter content and pedagogy. For instance, federally funded Math Science Partnership (MSP), Teaching American History (TAH), and Enhancing Education Through Technology (EETT) programs follow this model. After receiving this professional development, teachers are adapting what they have learned to their existing practice and context. Thereby, teachers themselves are constructing new innovations in their classrooms leading to great variability in which professional development components are implemented and to what extent they are used. Measurement systems that have been developed over the past several decades that address the variability in use and implementation are the focus of this roundtable discussion. How well are existing measurement systems aligned to the professional development model described above? Are there commonalities among methods used by evaluators adapting these systems to address the variability dilemma?

Session Title: Lessons Learned From Creating Successful Systems Designed to Promote Evaluation Use in non-profit Organizations

Panel Session 418 to be held in Balboa C on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Non-profit and Foundations Evaluation TIG and the Internal Evaluation TIG

Chair(s):

Pranav Kothari, Mission Measurement, pranav@missionmeasurement.com

Discussant(s):

Pranav Kothari, Mission Measurement, pranav@missionmeasurement.com

Abstract: Nonprofit organizations have become more sophisticated in evaluating and reporting on their effectiveness and impact. However, internal systems often need to be built and maintained to ensure that evaluation information is used to address stakeholder reporting needs and to promote organizational learning. The panelists are each responsible for implementing such systems. The panelists will present the lessons they have learned from engaging in this work. The goal of the panel is to provide a range of strategies, tools, processes and political savvy that evaluators can use to build and maintain systems designed to promote internal and external evaluation use.

Lessons Learned From Creating a Performance Culture in a School-Reform Non-profit

Eric Barela, Partners in School Innovation, ebarela@partnersinschools.org

Eric Barela is the Chief Organizational Performance Officer for Partners in School Innovation, a school-reform nonprofit working with high-poverty, high-minority schools in the San Francisco Bay Area. Dr. Barela is responsible for maintaining the organization's performance management system and for coordinating internal evaluation efforts. His presentation will concentrate on his efforts to marry performance management and internal evaluation to generate useful data on both programmatic and organizational effectiveness. This has resulted in a shift of organizational culture with respect to accountability and data use. He will share successful, and not-so-successful, strategies and the political acumen needed to incorporate an internal system into the fabric of a nonprofit.

Knowledge Management Systems That Promote Organizational Learning: Lessons From New Leaders for New Schools

Gina Ikemoto, New Leaders for New Schools, gikemoto@nlns.org

Gina Ikemoto is the Executive Director of Research and Policy Development for New Leaders for New Schools, an organization that works with urban school districts to improve school leadership. New Leaders also serves as an action tank that builds knowledge through an iterative process of implementing, testing, collecting feedback and refining. Dr. Ikemoto oversees a team and range of projects, including organizational impact studies, program implementation studies, and other leadership research studies. Her team is responsible for synthesizing findings from organizational learning and external research to inform programmatic decisions and the broader field. She will focus her presentation on the knowledge management and knowledge sharing systems that New Leaders has developed to facilitate the integration and use of various knowledge sources. She will share lessons learned about factors that have enabled and hindered these efforts.

Successes and Challenges in Creating a New Impact Initiative at a National Non-profit

Srik Gopalakrishnan, New Teacher Center, srik@newteachercenter.org

Srik Gopalakrishnan is the Senior Director of Impact at the New Teacher Center, a national nonprofit that works to accelerate the effectiveness of new teachers and school leaders in order to improve student learning. He leads the organization's efforts to define and measure impact, as well as set up systems and processes to promote organizational learning and improvement based on evidence. Successful strategies used thus far have been the creation of an "impact spectrum", use of a cross-functional impact team, and implementation of an annual impact summit. His presentation will focus on the successes and challenges in setting up a new organizational initiative around impact. The lessons learned touch on critical attributes that someone in this role needs to have, such as creative problem solving, thoughtful systems building and savvy change management skills.

Real Time Results for Real Time Impact in Turnaround Schools

Christina Krasov, Academy for Urban School Leadership, ckrasov@cps.k12.il.us

Melissa Peskin, University of Texas, Houston, melissa.f.peskin@uth.tmc.edu

Christina Krasov is the Director of Performance Management at the Academy for Urban School Leadership, a nonprofit organization that turns around Chicago's most in-need schools. Dr. Krasov jointly oversees the Performance Management and IT teams who have built the organization's first data warehouse and real-time reporting systems to help drive AUSL's bottom line-student impact. Her presentation will discuss the design and implementation of three tools that measure success for 19 schools in various stages of turnaround: a school operational excellence rubric, data-driven instruction instruments, and an ongoing research and development engine. She will share her insights into the harmony of numbers, tools and conversations necessary to make a real difference in education.

Session Title: The Value of Training Clients and Program Personnel in Logic Models and Evaluation Techniques (or Evaluators are From Mars, Stakeholders are From Venus, and We All Want to Work on Earth)

Think Tank Session 419 to be held in Capistrano A on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Organizational Learning and Evaluation Capacity Building

Presenter(s):

Laran Despain, University of Wyoming, ldespain@uwyo.edu

Discussant(s):

Laran Despain, University of Wyoming, ldespain@uwyo.edu

Laura Feldman, University of Wyoming, lfeldman@uwyo.edu

Mariah Storey, Wyoming Department of Health, mariah.storey@health.wyo.gov

Abstract: As evaluators, we are trained in using logic models and other specific evaluation techniques (e.g., measurement) to provide clients with information about programs. Stakeholders generally are not trained in the same techniques. This disparity can create difficulties in developing evaluation plans, data collection, and communication of evaluation findings. During this think tank, the presenters will discuss how they have collaborated with programming personnel at the state and rural county levels to teach techniques of evaluation to improve collaboration with stakeholders and evaluation. Collaboration occurred through face-to-face meetings, document sharing, and web-based meetings. The presenters will discuss the benefits, obstacles, and lessons they encountered in the process of training stakeholders in basic evaluation skills. The liaison between the evaluator and the state-level client will be present to provide a client perspective. Presenters will also encourage attendees to share their experiences working with non-evaluator stakeholders to improve collaboration on evaluation and programming.

Session Title: Internal Evaluation TIG Business Meeting and Paper Presentations: Internal Evaluation - Issues, Challenges, and Opportunities

Business Meeting and Multipaper Session 420 to be held in Capistrano B on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Internal Evaluation TIG

TIG Leader(s):

Boris Volkov, University of North Dakota, boris.volkov@med.und.edu

Kimberly Hall, Los Angeles Universal Preschool, kimberly.m.hall@gmail.com

Wendi Kannenberg, Southcentral Foundation, wkannenberg@scf.cc

Chair(s):

Imelda Castaneda-Emenaker, University of Cincinnati, castania@ucmail.uc.edu

Discussant(s):

Stanley Capela, HeartShare Human Services, stan.capela@heartshare.org

Examining Critical Issues in Internal Evaluation Practice

Presenter(s):

Boris Volkov, University of North Dakota, boris.volkov@med.und.edu

Abstract: This presentation will illuminate critical issues for developing and practicing internal evaluation in organizations. It will provide an overview of organizational, societal, and professional demands on internal evaluation function. Illuminated in a context of modern organizations influenced by various traditions and movements, critical issues for internal evaluation include potential ethical dilemmas, internal evaluator role clarification, and developing evaluation capacity. It has become more evident that the internal evaluation is not just about using appropriate evaluation methodology or building complex M&E systems but increasingly about dealing with intra-organizational obstacles to quality evaluation. Modern internal evaluation practice certainly requires reconfigured, unorthodox methods and styles of work to effectively meet the needs of the emerging learning organizations and other stakeholders. The importance of building evaluation capacity across the entire organization while cultivating independence and credibility of internal evaluation will be highlighted. Implications for future research on internal evaluation will be suggested.

"Rough Road" Evaluation: The Benefits and Consequences of Supporting the Primacy of Stakeholders' Interests

Presenter(s):

Imelda Castaneda-Emenaker, University of Cincinnati, castania@ucmail.uc.edu

Kathie Maynard, University of Cincinnati, sundk@ucmail.uc.edu

Thaddeus Fowler, University of Cincinnati, fowlertw@ucmail.uc.edu

Abstract: Beyond concerns of evaluating a professional development program in science, the presenters share the internal evaluation process that evolved in their practice where the primacy of stakeholders' interests was given foremost importance. The practice centered on day-to-day reflections and feedback about stakeholders' situations and what they needed as the professional development program implementation progressed. This project was part of a state-wide program evaluation, which led to negotiations of tensions involving different expectations of the state external evaluators. Priorities and timelines in the implementation of specific evaluation activities were not in sync with the state-wide intent; thus, the 'rough road' evaluation.

Problems and Possibilities While Evaluating a Field Trip Program on an Ongoing Basis

Presenter(s):

Heather Harkins, Connecticut Science Center, hharkins@gmail.com

Abstract: This presentation discusses the process used to evaluate a school field trip program at a science center in the Northeast United States during its first two years of operation. The process included an internal evaluator and staff from across the institution (educational programs, visitor services, exhibits) in data collection. This evaluation evolved from a process orientation toward an outcome evaluation. Changes during the course of the evaluation were based on institutional needs and capacities, as well as interim findings and discussions. This presentation reports on both the process used to work internally with this diverse group and some of the results relevant to the larger evaluation community. It also raises the discussion of how to manage stakeholder demand for immediate feedback in the form of raw data. The presentation will highlight the lessons learned from this evaluation, and how the process has impacted the science center's overall evaluation program.

Session Title: Prevention and Public Safety Initiatives: Evaluation for Program Development and Improvement

Multipaper Session 421 to be held in Carmel on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Social Work TIG

TIG Leader(s):

Donna Parrish, Clark Atlanta University, donnadparrish@hotmail.com

Tracy Wharton, University of Michigan, trcharisse@gmail.com

Karen Anderson, Independent Consultant, kanderson.sw@gmail.com

Aisha Tucker-Brown, Centers for Disease Control and Prevention, atuckerbrown@cdc.gov

Session Title: Document Analysis: An Inductive Approach to Determine What End-of-Course Evaluations Measure

Demonstration Session 436 to be held in Monterey on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Assessment in Higher Education TIG

Presenter(s):

Jacqueline Singh, Indiana University Purdue University Indianapolis, jhsingh@iupui.edu

Ingrid Ritchie, Indiana University Purdue University Indianapolis, iritchie@iupui.edu

Randi Stocker, Indiana University Purdue University Indianapolis, rstocker@iupui.edu

Abstract: The original purpose of end-of-course evaluation was to provide formative feedback to faculty. However, as the focus shifted to inform decisions about instructors' performance, questions have been raised about the adequacy of end-of-course evaluation instruments for summative purposes. For this session, a seven-step approach to document analysis that analyzes textual information is demonstrated. Document analysis teases out what is missed in course evaluation instruments when we do not holistically account for what is evaluated. It makes explicit the measurement foci of survey items; identifies concepts, constructs, and criteria embedded within these instruments; documents the presence or absence of what is "valued;" and determines whether or not a coherent survey design is evident. Document analysis is useful in all types of evaluation. It is an inductive approach that higher education can leverage to gather facts and think through strategic data sources for summative decision-making.

Session Title: The Role of Human Resources in Sustainability: Studies and Recommendations

Multipaper Session 437 to be held in Oceanside on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Business and Industry TIG

Chair(s):

Jennifer Dewey, James Bell Associates Inc, dewey@jbassoc.com

Discussant(s):

Jennifer Martineau, Center for Creative Leadership, martineauj@leaders.ccl.org

Gaining Innovation Through Empowerment

Presenter(s):

Alexander Manga, Western Michigan University, alexander.manga@wmich.edu

Tala Davidson, Western Michigan University, tala.j.davidson@gmail.com

Stacy French, Western Michigan University, portagefrench@sbcglobal.net

Steve Dibble, Western Michigan University, steve378810@gmail.com

Abstract: Empowerment has been found to be positively related to innovation capability for profit and non-profit organizations. This paper contributes to evaluation of human resource management, and offers several insights for chief executive officers, consultants, and human resource managers. The results of this study may assist managers in decision making to adapt different organizational strategies that may influence sustainable innovative outcomes. In this study, the impact of empowerment on innovation capability; and the peculiarities of these effects are investigated. The study's hypotheses are tested by performing a meta- analysis on both published and unpublished studies and data collected for the purpose of measuring the effects of empowerment on innovation. This study specifies perceptions of empowerment as the independent variable and innovation as the dependant. The results of this meta- analysis will also lead to further research questions that can develop deeper understanding of the topics empowerment and innovation.

Values in Human Resource Development (HRD) Evaluation: What is the Role of Sustainability?

Presenter(s):

Darlene Russ-Eft, Oregon State University, darlene.russeft@oregonstate.edu

Abstract: The purpose of this paper is to explore issues related to the value of sustainability in HRD evaluation. The paper will begin by examining the various meanings of sustainability and the related research. It will then explore the meaning of HRD evaluation and the current approaches to such evaluation. Finally, it will discuss ethics and values within HRD evaluation, arguing that sustainability should be a major concern for HRD evaluators. Two case studies exemplifying the issues will be described: one involving teacher development within the K-12 sector and another involving an employee development effort within a high tech firm. Questions will be posed as to the role of the evaluator in identifying internal and external factors that might affect sustainability.

Session Title: Evidence and Evaluations for Large-scale Arts Education Programs

Panel Session 438 to be held in Palisades on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Evaluating the Arts and Culture TIG

Chair(s):

Don Glass, Boston College, donglass@gmail.com

Discussant(s):

Don Glass, Boston College, donglass@gmail.com

Abstract: The purpose of the session is to share knowledge and practices for evaluating large-scale arts education programs with program staff or evaluators. The presenters represent the three arts-focused USDoE Innovation (i3) grants that were awarded at the Development level that include external multi-year program evaluations of district-level arts education opportunities and their impact on student achievement. Find out more about the evaluation design, use, and sharing for these important i3 grants.

Everyday Arts for Special Education

Robert Horowitz, Columbia University, artsresearch@aol.com

Everyday Arts for Special Education(EASE)is designed to improve student achievement in communication, socialization, academic learning, and arts proficiency through integrated, arts-based approaches. The evaluation is investigating differences among potential effects for students on the autism spectrum, students with intellectual disability, students with emotional disturbance, and students with multiple disabilities.

Arts for Learning

Jan Norman, Young Audiences, jan@ya.org

Sandy Sobolew, WestED, ssobole@wested.org

Young Audiences, Inc., its Oregon affiliate, and partners will implement the Arts for Learning (A4L) Program in the Beaverton School District for 3rd - 5th grade students over the next five years. A4L integrates standards-focused, text-based content and arts strategies to improve students' achievement in literacy, learning, and life skills. WestEd will conduct a three-year multi-method evaluation with formative and summative components. A cluster-randomized trial in 32 elementary schools in grades 3-5 will be used to examine the impact of the A4L curriculum on student achievement, as well as inform the effectiveness of professional development, and the fidelity of implementation. Full proposal narrative: http://www2.ed.gov/programs/innovation/2010/narratives/u396c100900.pdf.

Arts Achieve: Impacting Student Success in the Arts

Susanne Harnett, Metis Associates, sharnett@metisassociates.com

Tom Cahill, Studio in a School Association, tcahill@studioinaschool.org

Studio in a School and its partners will design 5th grade, 8th grade, and high school benchmark arts assessments in visual arts, music, theater, and dance aligned with the New York City Blueprint for the Arts. Metis Associates will conduct the evaluation using a cluster randomized assignment (50% treatment/50% control schools). The evaluation will assess the extent to which the project produces assessments aligned with arts standards, the impact of the initiative on arts teachers' instruction and their integration of technology into the classroom, and the impact of the project on students' arts skills and academic achievement. Full proposal narrative: http://www2.ed.gov/programs/innovation/2010/narratives/u396c100448.pdf

Session Title: Evaluating a Complex Informal Science Education Program Through a Partnership Model

Panel Session 441 to be held in Redondo on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Extension Education Evaluation TIG

Chair(s):

Suzanne Le Menestrel, United States Department of Agriculture, slemenestrel@nifa.usda.gov

Abstract: This panel is focused on the creation of a partnership model for an evaluation of the 4-H Science initiative between an external evaluator and the evaluation's stakeholders. The 4-H youth development organization is a complex public-private partnership between the United States Department of Agricultures' National Institute of Food and Agriculture, the nation's Cooperative Extension system operated by the more than 100 land grant universities and colleges, and a private, non-profit partner, National 4-H Council. The traditional model of evaluation for many Cooperative Extension programs has been to rely on internal evaluation expertise through either a full-time evaluation faculty member or other specialist with evaluation expertise. This panel will include perspectives from national and state-level stakeholders as well as the external evaluator and will discuss the advantages of and limitations of using a partnership model to conduct an evaluation of a complex multi-sector youth development initiative.

A National Perspective on Evaluating a Complex Informal Science Education Program

Suzanne Le Menestrel, United States Department of Agriculture, slemenestrel@nifa.usda.gov

This paper presents a national perspective on the challenges in creating an evaluation design for a complex informal science education program such as the 4-H Science Initiative. The presenter will describe the creation of several national-level science evaluation design teams that were comprised of both internal and external experts and stakeholders, including the funder of the evaluation. The presenter will also discuss the management structure of the national evaluation and the relationship between the client/stakeholders and the external evaluator. The advantages and limitations of this partnership model will be reviewed and recommendations for others who are interested in forming partnerships between organizations and evaluators will be provided.

A State Perspective on Evaluating a Complex Informal Science Education Program

Jill Walahoski, University of Nebraska,Lincoln, jwalahos@unlnotes.unl.edu

Participation in a national evaluation presents both challenges and opportunities for land grant university 4-H Programs that are likely relevant to other multi-site youth development programs. While there are shared outcomes among the institutions and their respective local program sites, there is less consistency in how programs are delivered and how data are collected. There is a need for common measures and data collection strategies. However, there are varying circumstances that impact a location's ability to participate. Integral in overcoming the barriers to participation is the investment of the state and local program sites in the methods and tools utilized. It was crucial in our work to engage program leaders system-wide in the development of the evaluation plan design, methods and tools. They identified the needs and challenges of the program sites and developed strategies that allowed for flexibility and tailoring, especially in the early rounds of data collection.

An External Evaluator's Perspective on Evaluating a Complex Informal Science Education Program

Jenny LaFleur, Policy Studies Associates, jlafleur@policystudies.com

An external evaluation can be a key tool for an organization looking to examine its practices and assess its progress toward stated goals. However, the path to creating a successful partnership often requires that evaluators: (1) navigate steep learning curves in order to become familiar with the contracting organization; (2) regularly (re)calibrate evaluation activities to respond to the realities of a complex, national program; (3) be responsive to all stakeholders, from both the organization and its funders; and (4) present data and findings in a way that allows clients to use the evaluation results to effectively guide their work. This presentation will review lessons learned from Policy Studies Associates' ongoing evaluation of the 4-H Science Initiative from the perspective of the external evaluator.

Session Title: System-wide Assessments in Higher Education

Multipaper Session 442 to be held in Salinas on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Assessment in Higher Education TIG

Chair(s):

William Rickards, Alverno College, william.rickards@alverno.edu

Creating a Culture of Student Learning Assessment at University of Hawaii at Manoa Through Social Capital

Presenter(s):

Marlene P Lowe, University of Hawaii, Manoa, mplowe@hawaii.edu

Abstract: The University of Hawaii at Manoa Assessment Office takes a social capital approach to creating a culture of student learning assessment on its campus, a research university with over 2,000 faculty members. The Assessment Office chose this approach because it believes that by utilizing its own social capital and developing networks of like-minded faculty members, the critical mass necessary for student learning assessment to take hold and become part of the university culture can be achieved. In the past 3 years, the Assessment Office has seen marked increases in faculty member participation and buy-in of student learning assessment. This presentation will share some of the social capital strategies the Assessment Office incorporates into its faculty development activities such as workshops, consultations, events, and general education assessment to facilitate the development of faculty networks and a culture of student learning assessment.

A Seat at the Strategy Table: Integrating Strategy and Evaluation

Presenter(s):

Zita Unger, Independent Consultant, zitau@bigpond.com

Abstract: The emerging field of Strategic Evaluation provides a valuable framework for tracking strategies and asking questions about strategic intent, implementation and what is realized, well beyond single programs or initiatives. Evaluation can be a key support for strategy development and management, bringing timely data for reflection and use into strategic decision-making. However, is it one step too far to suggest that evaluation should have a seat at the strategy table? This presentation explores, from a governance and management perspective, the influence of evaluation in the strategic decision-making process and the extent to which evaluation can be positioned as a partner.

Creating a University-wide Culture of Assessment, Evaluation, and Ongoing Reflective Learning: Facilitating Student, Faculty, and Organizational Learning

Presenter(s):

Joan Slepian, Fairleigh Dickinson University, jslepian@fdu.edu

Marlene Rosenbaum, Fairleigh Dickinson University, marlene_rosenbaum@fdu.edu

Abstract: Mandatory assessment and quality-improvement initiatives provide extraordinary opportunities for focused, data-driven reflection on learning, on curriculum and instructional practice, and on transformative organizational learning and renewal. Not only do they drive us to focus on 1) student learning, but also on 2) faculty learning and curriculum development, and 3) developing systems for ongoing organizational learning as well. As adult learners, educators, evaluators, and students of organizational change, this has provided us with a rare opportunity to put our theories and methods into practice as organizational participants. In this paper, we draw from our professional research and practice traditions to outline a set of conceptually-grounded, systemic approaches and interventions that we have used to work with each of the three intervention targets. In each section we outline several of the conceptual foundations, methodologies, tools and 'guiding principles' that we have used to attempt to facilitate complex, systemic change in our university.

Higher Education Evaluation Practice and Knowledge Production

Presenter(s):

William Rickards, Alverno College, william.rickards@alverno.edu

Abstract: Program evaluation is often interpreted as a procedure for (a) demonstrating the effects of program concepts and practices or (b) accountability for implementation, with these serving a lesser function in the activities of knowledge construction. However, there are reasons to examine the roles for evaluation and its special potentialâ€”particularly in the form of theory-based evaluation--in knowledge construction in higher education research and practice. From a brief critique of some common practices in program evaluation, this paper examines three different approaches that position evaluation as a critical element in knowledge construction, specifically: modeling the complex context of practice (Selman, 2006); examining theoretical concepts as fundamental to program activities (Weiss, 1997); and using evaluation processes to contribute to knowledge production and practice improvement (National Science Foundation, 2006). Related examples link these perspectives to emerging concerns in higher education.

Session Title: Scaling Up the Replication Ladder: Considerations in Evaluating Scale-up Efforts

Think Tank Session 443 to be held in San Clemente on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the AEA Conference Committee

Presenter(s):

Ginger Fitzhugh, Evaluation & Research Associates, gfitzhugh@eraeval.org

Discussant(s):

Julie Elworth, Evaluation & Research Associates, jelworth@eraeval.org

Carolyn Cohen, Cohen Research & Evaluation LLC, cohenevaluation@seanet.com

Davis Patterson, Independent Consultant, davispatterson@comcast.net

David Reider, Education Design, david@educationdesign.biz

Abstract: Participants in this session will explore challenges and promising practices associated with evaluating programs that are either considering or undertaking scale-up into new geographic regions or contexts. When are projects ready to scale up? What is the evaluators' role in working with projects that are scaling up? The facilitators will share a theoretical framework they have used to evaluate programs that have scaled up. Small groups will discuss: (1) What are evaluation considerations before programs scale up? (2) How must the evaluation itself be scaled up and what are effective strategies for a smooth transition? (3) What are useful organizing frameworks for evaluating projects as they scale up? (4) What implications do stakeholders' different values and motivations for scaling up in new contexts have for evaluation design and communication of findings? The whole group will reconvene to share insights and identify questions that merit further attention.

Session Title: Getting Your Findings Used: Making Evaluation Results Relevant, Accessible, and Clear

Think Tank Session 444 to be held in San Simeon A on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Collaborative, Participatory & Empowerment Evaluation TIG

Presenter(s):

Paul St Roseman, DataUse Consulting Group, paul@mydatause.com

Discussant(s):

Trena Anastasia, University of Wyoming, tanastas@uwyo.edu

Javan Ridge, Colorado Springs School District 11, ridgejb@d11.org

Abstract: This interactive session builds upon round table dialogues (occurring during the 2009 and 2010 AEA Conferences) that compile approaches and strategies supporting client capacity to use evaluation reports. Participant feedback from the first two sessions has been segmented into four topic areas: 1) Approaches used to frame the way evaluation reports may be re-purposed; 2) Strategies that leverage online resources that make data analysis more transparent and inclusive; 3) Steps taken to increase the timeliness and use of evaluation findings; and 4) Methods used to include evaluation stakeholders in the development of evaluation reports. For each topic area participants will list activities, indicate items to consider when administering an activity, and communicate the evaluation value modeled by the activity. The information compiled from this session will continue to extend the knowledge-base regarding the use or evaluation reports, as well as access the varied and deep expertise of AEA membership.

Session Title: Values and Valuing in After-School Program Evaluation: Perspectives From Local, District, and Statewide Evaluators

Panel Session 445 to be held in San Simeon B on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Chair(s):

Krista Collins, Claremont Graduate University, krista.collins@cgu.edu

Discussant(s):

James Sass, Rio Hondo College, jimsass@earthlink.net

Abstract: After-school programs present evaluators with a maelstrom of conflicting values. In addition to the typical concerns of compliance, effectiveness, and improvement, after-school program evaluators must address the tension between child/youth development outcomes and academic achievement, respond to the conflicting needs of funders and program providers, promote evaluation use in a funding environment that prioritizes mandatory reporting, and maintain methodological rigor while being flexible in response to ever-changing demands. The presenters address these issues from different perspectives: two community-based organizations (one long-estabished and one rapidly growing) serving schools in multiple districts, an urban school district managing programs provided by 30 community-based organizations in 400 schools, and the nation's largest statewide after-school initiative. Turning principles into practice, they discuss actual decisions and techniques for providing high-quality evaluations in the context of conflicting values. Specifically, the presenters address issues of evaluation design, data collection, analysis and interpretation, utilization and consultation, and budget management.

Incorporating Client Values in After-school Program Evaluations: Results From Woodcraft Rangers' Nvision After-School Program Quality Assessment and Student Outcomes Evaluation

Kristen Donovan, EVALCORP Research & Consulting, kdonovan@evalcorp.com

Lisa Garbrecht, EVALCORP Research & Consulting, lgarbrecht@evalcorp.com

Our evaluation approach involves working with clients to assess and quantify what is most valuable to them, then providing consultation on where implementing such information would be of highest use and value. Over the past several years, EVALCORP has collaborated with Woodcraft Rangers' after-school program, which is provided at more than 60 schools across Los Angeles County, to measure and establish benchmarks for valuing the quality of its Nvision program model and link it to student outcomes. Findings from the assessment were used by Woodcraft Rangers to enhance its program, staffing, and other decisions. Using the Woodcraft Rangers' Assessment of Program Quality and Youth Outcomes study as an example, this presentation will discuss how valuing is a key component throughout the evaluation process - guiding which evaluation design is implemented, the types of data collection methods employed, and how the data are ultimately put to use

Valuing the Role of Children and Youth in Local After-School Evaluations

Tiffany Berry, Claremont Graduate University, tiffany.berry@cgu.edu

Ana Campos, After School All-Stars, anacampos@la-allstars.org

Evaluations of after-school programs often exclude a "whole child" perspective from the evaluation process given that many frequently (1) preference academic achievement (in the form of test scores) over the social/emotional development of children; (2) ignore the contextual variables surrounding the child (their perceptions of neighborhood safety, how children spend their time after school when not in the program, etc.); and (3) disregard the notion that children actively choose their environments (reason they join). During this presentation, the Executive Director of After School All-Stars, Los Angeles, as well as their external evaluator from Claremont Graduate University will explain the development, rationale, and theory behind our "whole child" evaluation approach, discuss measurement issues using this approach, and illustrate how our approach has facilitated program improvement throughout the organization.

Responding to Stakeholders' Values and Priorities in After-School Program Evaluation: Tools and Strategies for Presenting Data to Managers, Policy-Makers, and Funders

Harry Talbot, Los Angeles Unified School District, harry.talbot@lausd.net

Steven Frankel, Research Support Services, stevefrankel@ca.rr.com

Program managers and research directors must play multiple roles if they are to be truly effective. For example, program managers must manage the activities of after-school programs serving thousands of students as well as provide data and policy recommendations supporting the multimillion dollar budgets of these programs. Program managers need unique reporting methods, rapid turnaround times, and priorities that can be shifted on a dime. Research directors must provide an ongoing stream of findings and recommendations that will improve the programs and repay funding for evaluations. They must also develop evaluation strategies that identify promising findings and trends regardless of whether these lines of investigation were included in the original research design. Using examples from Beyond the Bell within the Los Angeles Unified School District, this presentation will describe methods of meeting these multiple demands without compromising objectivity, creativity, and fixed-price budgets.

The Role of Values and Stakeholder Perspectives in Conducting and Reporting Statewide Evaluations of After-School Programs

Denise Huang, University of California, Los Angeles, dhuang@cse.ucla.edu

Since the provisions of California Proposition 49 became effective, annual funding for the After School Education and Safety Program (ASES) has increased from $120 million to $550 million. This funding requires the California Department of Education to contract for an independent statewide evaluation on the effectiveness of programs receiving funding. The National Center for Research on Evaluation, Standards, and Student Testing (CRESST) has taken on the responsibility of this task, and is conducting two statewide evaluations of after-school programs: one for programs serving elementary and middle school students; and the second for programs serving high school students. Using these studies as examples, CRESST will discuss the purpose of evaluations from the State's perspective and from the practitioners' points of views, and values and valuing in evaluation, as well as how evaluators can be forthright and transparent about the values that are being promoted during and in reporting the study.

In a 90 minute Roundtable session, the first rotation uses the first 45 minutes and the second rotation uses the last 45 minutes.

Roundtable Rotation I: Reflections on the Influence of Spiritual and Religious Values on Our Own Particular Evaluation Practice

Roundtable Presentation 446 to be held in Santa Barbara on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Graduate Student and New Evaluator TIG

Presenter(s):

Paula Bilinsky, University of Wageningen, pbilinsky@hotmail.com

Irene Guijt, Learning by Design, iguijt@learningbydesign.org

Abstract: The roundtable is intended to allow an examination and sharing of participants evaluation practice as an expression of values inspired by spiritual/religious traditions. How do the spiritual/religious traditions in which we have been or are steeped bear on evaluation choices and practices? Is it in what we choose to do or not, what 'mastery' we bring to method use, how we construct analysis processes, , what values are present in questions? Or none of these? Are belief systems present or entirely tacit? . We expect an inspiring discussion with people who are influenced by a diversity of religious backgrounds and spiritual schools of thought. The discussion will start off with a description of Jewish and Buddhist values, as only two streams of thought, and how they are reflected in the evaluation practices of the roundtable convener and discussant.

Roundtable Rotation II: Understanding Values and Valuing: A Practical Approach

Roundtable Presentation 446 to be held in Santa Barbara on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Graduate Student and New Evaluator TIG

Presenter(s):

Jason Burkhardt, Western Michigan University, jason.t.burkhardt@wmich.edu

Abstract: At its heart, Evaluation is the science and art of values and value applications. This session seeks to use a practical exercise to support discussion and collaborative understanding of the way in which we identify, communicate, and utilize values to support evaluative conclusions. The roundtable will mirror the process of engaging multiple stakeholders in the evaluation process, and will help the participants to improve their understandings of the group process. The session will begin with a group exercise that focuses on applying values, and will explore the group processes in resolving the value task. Participants will be asked to employ their understandings of and experiences with the evaluation process, and will also be asked to self-reflect on their experiences. Participants will also be given handouts that can facilitate future discussion and reflection.

Session Title: Structural Equation Modeling and Longitudinal Perspectives in Evaluation Research: Promises and Pitfalls

Panel Session 447 to be held in Santa Monica on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Quantitative Methods: Theory and Design TIG

Chair(s):

Georg Matt, San Diego State University, gmatt@sciences.sdsu.edu

Abstract: Path coefficients from longitudinal designs, variance accounted for as r-square, validity coefficients, effect sizes among many others are quantitative indicators and measures many evaluative judgments and decisions are based on. Can we trust these indicators are there any alternatives? Structural equation modeling computes these coefficients on the assumption of analyzing measurement error free latent variables. Analyzing our program evaluation studies with these tools hold the promises of coming close to the truth of the problems? The first presentation focuses on various alternative in analyzing longitudinal data, the second challenges whether path coefficients and effect sizes for structural equation modeling are an overestimate or not. The third one describes the evaluative conclusions to be drawn by applying multiple act criteria analyzed as reflective or formative constructs and compares them to strategies used in meta-analysis. Examples used are based on longitudinal ambulatory psychotherapy outcome research all the presenters had been involved.

Time Matters!: The Analysis of Dynamic Models in Continuous Time - An Example From Evaluating Outpatient Psychotherapy

Manuel Voelkle, Max Planck Institute for Human Development, voelkle@mpib-berlin.mpg.de

This presentation has three goals: First, we will discuss problems with several popular approaches to analyze change over time, such as autoregressive models, cross-lagged models, or dynamic factor models for single or multiple subjects from a structural equation perspective. We will argue that most real-world processes happen in continuous time, and that ignoring the time interval between measurement occasions - as done in these models - may lead to a false representation of the world and thus wrong conclusions. Second, we will propose a better alternative to these approaches by using stochastic differential equations and demonstrate how to formulate these models within an SEM framework. Finally, we will use data from a recently completed evaluation study on outpatient psychotherapy to illustrate our arguments. Special emphasis will be put on the theoretical basis of continuous time modeling, but we will also discuss software issues and practical implications.

Are There any Biases in Using Effect Sizes From Latent Variables in Structural Equation Modeling?

Werner Wittmann, University of Mannheim, wittmann@tnt.psychologie.uni-mannheim.de

Effect sizes and correlation coefficients between latent constructs derived with structural equation modeling (SEM) are often much larger than what we are familiar with using classical statistical tools. This is no wonder because these coefficients are based on measurement error free latent constructs. We discuss the differences between the effects of corrections for attenuation and other artifacts from classical psychometric theory and these communality based true score effects in SEM. Selected examples from our program evaluation studies will be used to demonstrate the consequences for evaluation and whether the promises of SEM will stand up under the scrutiny of a Feynman test, namely the problem of cargo cult science, i.e. will the air crafts land or not?

Should we Analyze Outcome Criteria Either as Formative or Reflective Latent Constructs or use Something Else? The Case of Single and Multiple Act Criteria in Evaluating Outcomes

Andres Steffanowski, University of Mannheim, andres@steffanowski.de

Werner Wittmann, University of Mannheim, wittmann@tnt.psychologie.uni-mannheim.de

Fishbein and Aizen proposed the use of multiple act criteria in many applied research settings. In our longitudinal psychotherapy outcome studies we continuously capitalize on their ideas. From a methodological stance using a structural equation modeling perspective these criteria can be either analyzed as formative or as reflective ones. We report what conclusions can be drawn from both variants and demonstrate implications for using effect sizes from formative or reflective multiple act outcome criteria. We also illuminate the relationship to strategies used in meta-analysis.

Session Title: Innovation Models for Evaluation of Family-Strengthed Interventions

Panel Session 448 to be held in Sunset on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Human Services Evaluation TIG

Chair(s):

Rose Ann M Renteria, Child Trends, roseannrenteria@email.com

Abstract: The session focuses on the how evaluation work occurs, assessing family-strengthed interventions, including programs which operate with key wraparound principles as well as family and youth leadership ideals. The sessions goals are to share information on how family strengthening evaluation occurs with community and key stakeholders, discuss relevant outcome data, and understand how families and their children are fairing in their effort to remain resilient and intact.

Emerging Trends in Family-Strengthed Interventions: The Brevard CARES (Coordination, Advocacy, Resources, Education and Support) Model and Case Study

Patricia Nellius-Guthrie, Brevard Family Partnership, patricia.nellius@brevardfp.org

Valerie Holmes, Brevard Family Partnership, valerie.holmes@brevardcars.org

How are local and community-based organizations developing and evaluating family-strength interventions? Brevard CARES provides front-end child abuse prevention services to Brevard County (FL) children and families with its innovative Wraparound Family Team Conferencing model. Refining evaluation data collection systems and working towards promising practices in program design and evaluation will be discussed. Presenters will discuss how local and community-based organizations build out and implement innovative evaluation strategies.

National Evaluation Trends in Strength-based, Family-Centered, Youth-Driven Interventions

Andrew Munoz, University of Pittsburgh, amunoz@pitt.edu

What are emerging trends in evaluation research work, specific to strength-based, family-centered, youth-driven interventions? Presenter will discuss trends and implications, resulting from the shift to study the family as the unit of analysis in evaluation research work specific to strength-based, family-centered, and youth-driven interventions. Innovative evaluation approaches will be highlighted to understand how evaluators engage the field of positive family and youth voice and development.

Understanding the Importance of Community Impact in Strenth-based, Family-Centered, Youth-Driven Interventions

Rose Ann M Renteria, Child Trends, roseannrenteria@aed.org

What is the community impact of these interventions? The presenter will discuss how stakeholders are interested is learning about how their communities and families are changing? Evaluation research work linked to community contextual profiles and theory of change will be discussed, specific to strength-based, family-centered, youth-driven interventions. The presenter will discuss shifts and implications to study community change versus the quality of services in this particular field of inquiry.

Session Title: Performance Management in Action: A National Measurement System for Early Childhood Outcomes

Panel Session 449 to be held in Ventura on Thursday, Nov 3, 2:50 PM to 4:20 PM

Sponsored by the Pre-K - 12 Educational Evaluation TIG

Chair(s):

Kathleen Hebbeler, SRI International, kathleen.hebbeler@sri.com

Abstract: In response to the federal push for results-based performance indicators, the U.S. Department of Education undertook the development of a national outcomes measurement system for early childhood programs serving children with disabilities. This system, built around state-reported data on child progress for three outcomes, has been well received by key stakeholders. The panel will describe the background for this system, implementation choices and issues faced by state agencies, and how states are planning to use these data to improve programs in addition to submitting them for federal accountability. The panel will share the national findings and lead an audience discussion about their validity and significance. The panel also will discuss lessons about the supports and barriers to building a large scale measurement system such as the need for appropriate assessment approaches for accountability measurement with young children and the importance of procedures for improving data quality.

Steps in Building the National System: Background and State Choices

Lauren Barton, SRI International, lauren.barton@sri.com

Donna Spiker, SRI International, donna.spiker@sri.com

This paper will provide the background to the national outcomes measurement system for programs serving young children with disabilities, including the impetus at the federal level that led to the development of the system, the parameters set by the federal government, the stakeholder process used to inform the requirements, and the measurement choices left to states. The 1993 Government Performance and Reporting Act, the OMB PART review of 2002, and the emphasis in IDEA 2004 on results over process were part of the impetus for national data on child outcomes for early intervention and early childhood special education programs. The federal requirements will be described along with variations in how states are responding to the requirement. Included will be information on the type and frequency of the various methodologies states are using to collect data on child outcomes.

Highs and Lows on the Road to High Quality Data

Lynne Kahn, University of North Carolina, Chapel Hill, lynne_kahn@unc.edu

In February 2011, states agencies submitted data on the progress of children in early intervention and early childhood special education programs for the fourth time. This paper will present the findings from the most recent child outcomes data and results from analyses that have been conducted to examine the data quality. The presentation will describe how the data have improved since the first data collection, and some of the challenges still to be addressed towards achieving trustworthy data from all states. Several approaches to analyzing the data to obtain the most accurate national picture will be reviewed, for example, using data from all states versus used weighting the data from only those states considered to have quality data. Session attendees will be encouraged to contribute to the discussion of whether the national data are of sufficient quality to be used for accountability and program improvement.

Lessons Learned about How to Support Outcomes Measurement

Donna Spiker, SRI International, donna.spiker@sri.com

Kathleen Hebbeler, SRI International, kathleen.hebbeler@sri.com

Building a national outcomes measurement system is an incredibly complex undertaking. The U.S. went from having no agreement on outcomes for the programs serving more than one million young children with disabilities to reporting national data in 5 years. The development of the system is far from complete as states continue to address ongoing challenges around data quality, but the amount of progress in such a short time is incredible. This paper will describe the kinds of support that states received, reflect on what has been learned about the key contributing factors to developing measurement systems at both the national and state levels, and highlight the barriers working against effective systems. For example, some states have very strong and savvy leaders who understand the value of outcomes data, but others do not. State resources are also critical. Implications for obtaining high quality national data will be discussed.

Return to Evaluation 2011 Search Results for All Sessions